Edited by: Daniel Vujcich, WA Health, Australia
Reviewed by: Jason Richter, Army-Baylor University, USA; Elizabeth Querna Cliff, University of Michigan, USA
Specialty section: This article was submitted to Public Health Policy, a section of the journal Frontiers in Public Health
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
There is little evidence to direct health systems toward providing efficient interventions to address medical errors, defined as an unintended act of omission or commission or one not executed as intended that may or may not cause harm to the patient but does not achieve its intended outcome. We believe that lack of guidance on what is the most efficient way to reduce medical errors and improve the quality of health-care limits the scale-up of health system improvement interventions. Challenges to economic evaluation of these interventions include defining and implementing improvement interventions in different settings with high fidelity, capturing all of the positive and negative effects of the intervention, using process measures of effectiveness rather than health outcomes, and determining the full cost of the intervention and all economic consequences of its effects. However, health system improvement interventions should be treated similarly to individual medical interventions and undergo rigorous economic evaluation to provide actionable evidence to guide policy-makers in decisions of resource allocation for improvement activities among other competing demands for health-care resources.
It is clear that medical errors carry a very high burden of disease (
There are many reports of evaluations of interventions to improve health service delivery to decrease the incidence of errors that lead to adverse health events (
Because HSIIs are often complex social interventions, seeking to influence both provider and patient behaviors as well as health-care organization, economic evaluations of these interventions – including cost-effectiveness, cost–utility, and cost–benefit analyses – can be more difficult to conduct and more complicated to interpret their results. In this paper, we argue that complexity is no excuse not to do such evaluations. Evidence-based HSIIs are urgently needed to best serve those who entrust their welfare to health service providers and the system in which they work. Cost-effectiveness analyses, cost-utility analyses, and other economic evaluations inform decision-makers on how efficient and therefore how worthwhile such interventions are compared to other uses of scarce resources.
We outline challenges to conducting economic analysis on HSIIs to reduce health care-associated errors and increase adherence to evidence-based standards and discuss ways these can be overcome or accommodated (Table
Measurement challenge | Potential solutions |
---|---|
Defining intervention | Empirically define intervention; use modeling, Monte Carlo Simulation, and sensitivity analysis to account for low fidelity to intervention |
Defining effectiveness | Measure all positive and potentially negative effects (balancing measures) as feasible; report all effects clearly |
Process measures versus outcomes | Use epidemiological modeling (e.g., LiST Tool) to convert processes to outcomes; use DALYs or QALYs for all outcomes |
Costing the intervention | Collect cost data prospectively; secure agreement with improvement implementers to report all resources used |
Economic consequences of effects | Collect primary data if possible; report assumptions made and account for them in modeling |
Health system improvement interventions to reduce errors and improve health service quality often have a degree of complexity and adaptation to local context that individual medical interventions usually do not. There may be multiple components in HSIIs, such as training of health-care providers, providing job aids and reminders to reinforce good practice, restoring essential medication supply chains, improving patient record-keeping, supervision or coaching, and enhancing facility infrastructure (
A solution suggested for this problem is to define explicitly all the components of the HSII being implemented and the context in which they were introduced. Then, if it is useful to consider the same intervention in different settings or under different circumstances in the same settings, adjustments can be made in the expected results through the use of different scenarios in decision trees and well-established statistical techniques, such as Monte Carlo simulations and sensitivity analyses. Such modeling could reduce precision in the results but would provide a more objective basis for estimating likely benefits under different scenarios.
Assuming success, HSIIs often have more than one positive effect on health outcomes of those receiving care from the heath provider unit involved. In the Niger example, the primary effect was reduced maternal morbidity and mortality. However, improvements in obstetric care were also accompanied by improvements in newborn care and therefore, a likely reduction in newborn mortality and morbidity, though this was not captured in the study.
A solution for this problem is to consider as many of the positive effects as can be feasibly measured in the analysis, perform separate analyses on each of them, and then report all outcomes as a composite score. In the example, if the intervention decreased the risk of newborn asphyxia and sepsis as well as maternal PPH, then the cost-effectiveness result could be reported as “for each $100 spent, there was a decrease of X1 cases of PPH and X2 cases of neonatal asphyxia and X3 cases of newborn sepsis. Alternatively, it could be reported that for a total of $Y2 in a facility that served Z1 people, there were X4 fewer cases of PPH, X5 fewer cases of neonatal asphyxia, and X6 fewer cases of newborn sepsis.”
Conversely, with implementation of a HSII there may be unintended negative consequences of the intervention. In our example, increased health worker attention toward improving delivery of essential maternal and newborn care may lead health workers to neglect other parts of their duties and consequently a deterioration in care quality for patients not receiving service for the focus condition.
The way to deal with this is to measure all the unintended negative consequences that are feasible to identify and include them in the analysis. This is standard practice in individual medical intervention economic evaluations – to measure the severity and frequency of adverse events and to include these in the analyses.
Also, when considering multiple different health outcomes, both positive and negative, from an HSII, it is useful to denominate the outcomes in burden of disease measures, such as disability-adjusted life years (DALYs) or quality-adjusted life years (QALYs). Converting different outcomes to a single unit also has an advantage of permitting comparisons among HSIIs to identify where the health system as a whole will reap the most benefit from investment in HSIIs.
Often in HSIIs, it is more feasible and accurate to measure the effect of the intervention in process measures rather than patient health outcomes. In the Niger example, measurement of compliance to evidence-based standards of care for active management of the third stage of labor and essential newborn care was easier than measuring some maternal and neonatal outcomes, especially when considering relatively rare outcomes such as maternal mortality from PPH. This can also be the case with individual medical interventions. One possible solution is to measure the process indicators as is feasible but then use epidemiological modeling to relate those process measures to tangible health outcomes using the best available evidence of the link between the process and ultimate outcomes. Models, such as the Lives Saved Tool (LiST) (
Given the lack of fidelity to the prescribed intervention that can occur with HSIIs, determining the intervention’s cost can be difficult. The most accurate method is to collect data on costs prospectively, being sure to record all of the resources consumed, including changes to health providers’ time, compared to the scenario in which the HSII is not implemented. However, retrospective reviews of accounting records can produce reasonable cost estimates. Assumptions often need to be made in calculating accurately the opportunity cost of applying HSIIs, and it is important that these assumptions be stated explicitly in the report of the evaluation. Organizations implementing HSIIs may also be protective of cost data to protect their commercial interests. Therefore, it can be useful for those commissioning the intervention to make such reporting a requirement of conducting the work. The ongoing costs of maintaining the improved level of service often need to be estimated with imperfect information, and any assumptions made here should also be stated transparently in the evaluation.
Given that HSIIs for medical error reduction often involve multiple different health outcomes, determining the economic impact can add significantly to the complexity of data collection and modeling. In the Niger example, the cost from the health system perspective of adverse events such as mild, moderate, or severe PPH, mismanaged obstructed labor, retained placenta and other adverse outcomes were calculated based on daily hospital costs and average lengths of stay. Calculating the economic impact from the societal perspective would have involved including the cost to mothers and their families of additional time in hospital and other related expenses. In such low-income settings, secondary economic data may be scarce, and primary data expensive and difficult to collect. Assumptions on economic impact may be needed, and this is acceptable if reported transparently in the evaluation write-up.
Individual medical interventions often involve a single medication (or class of medications) or medical device. There is often a commercial interest in providing evidence of their effectiveness or cost-effectiveness in treating a specific medical condition. Many health systems, such as the UK’s National Health Service and Australia’s Medicare, mandate that individual medical interventions must undergo rigorous cost-effectiveness analyses before there is financial coverage for them to be used to treat the specific medical conditions. This is not the case for HSIIs, where the intervention is generally not considered a marketable product and there are no mandates to prove their effectiveness or efficiency before they are employed. Consequently, the resources available to thoroughly evaluate their value in improving health are much smaller. Yet, HSIIs do consume resources – resources that could be invested in other interventions to benefit patients or providers. We believe that the donors and national governments should insist on greater transparency in the resources devoted to reducing medical errors and substandard care.
The solution to this is for health system policy-makers to mandate that economic analyses of HSIIs be conducted to produce evidence that an HSII is acceptably cost-effective or cost-saving before resources are expended to implement it, particularly at large scale. It is clear that there are multiple competing demands on health-care resources in all settings, and by definition this is especially the case in low- and middle-income settings. It could be viewed as ethically dubious in such settings to implement an HSII without examination of its economic consequences because, without knowing if the intervention is affordable, sustainable, or acceptably efficient, information on its effectiveness alone is not useful. As with individual medical interventions, studies of HSII effectiveness are necessary but not sufficient to recommend implementing the intervention in a given setting.
This is not to ignore the additional difficulties of performing economic analyses on complex social system interventions. It adds to the cost and time required to evaluate the program and the expertise required to carry it out. However, these factors cannot be excuses to omit such an important aspect of determining the applicability of the HSII. There is no place where health-care resources are inexhaustible and where difficult resource allocation decisions do not need to be made. Economic analysis of HSIIs is a vital part of the science of improvement that helps health systems advance.
EB conceived of the idea and drafted the initial version of the manuscript. LM contributed substantively to the writing and editing the manuscript and responding to reviewer comments.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This work was developed with funding from the United States Agency for International Development (USAID) through the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project, implemented by University Research Co., LLC under Cooperative Agreement Number AID-OAA-A-12-00101.