Challenges in Risk Adjustment for Hospital and Provider Outcomes Assessment
How are we to evaluate outcomes for the care that we provide, and how are we to benchmark ourselves compared with colleagues at other institutions? It is critical that we do this to improve our own performance, and it is also being demanded of us by other stakeholders in society. Indeed, regular public reporting of outcomes is now effectively mandated of providers in many medical disciplines, and standards for statistical adjustment have been developed.1 As a result, performance reports based on data from both administrative and clinical databases are commonplace, with a few elite databases serving as de facto sources of truth for medical performance in the United States. However, important questions must be asked about the registries being used to set performance standards and inform medical regulatory policies.
Within cardiology, one of the most commonly reported metrics is mortality after revascularization, either in-hospital or some time thereafter, often 30 days. Such reporting may be at the level of the individual operator, hospital, or health care system. Other metrics, including nonfatal events and cost, are also reported commonly.2 Initial public reports several decades ago offering raw outcome statistics on these end points resulted in a firestorm of criticism that the published data did not appropriately account for patient differences. In recent years, efforts have been made to account for variation in severity of illness, acuteness of presentation, and comorbidities using statistical methods that risk adjust to create fairer comparisons.1 Concerns now focus on metrics such as risk-adjusted mortality or the ratio of observed to expected event rates.
Risk-adjusted data are certainly more informative than raw data, but are adjusted data sufficiently reliable for the purpose of measuring quality of care? A major determinant of mortality in the setting of acute myocardial infarction is the presence of shock. Providers, being aware of this, may respond appropriately by ensuring that shock is carefully documented or less appropriately by avoiding care of these sicker patients where the expected adverse event rate is high. Evidence exists that such avoidance behavior has, in fact, occurred in parts of the country with public reporting and may lead to deterioration, not improvement, in patient outcomes.3
Other end points besides mortality have some degree of subjectivity. For example, the definition of myocardial infarction after a procedure is fraught with challenges, and troponin is not routinely measured in all hospitals after revascularization procedures and when measured can be difficult to interpret. Covariates used for risk adjustment may be subjective too, even for common cardiovascular conditions. For example, a clinical diagnosis of heart failure captured by a data abstractor from an electronic health record may appear important in a registry but will not reliably identify patients with or without meaningful left ventricular dysfunction. Covariates may not be consistently collected, and data that are missing cannot be assumed to be missing at random. As with mortality, important variables known to influence other outcomes are almost certain to go uncollected. In 2011, McNulty et al4 found that a series of important clinical factors, such as frailty measures, which are not routinely collected, do influence both decisions about elective surgical versus catheter-based revascularization and outcomes of treatment in patients with left main coronary artery disease. Thus, problems with covariates limit the ability to risk adjust accurately and may result in error throughout the spectrum of risk.
Clinicians are generally most concerned about the issues noted previously, but the statistical issues are just as critical. If a hospital does 400 percutaneous coronary intervention procedures annually and the expected mortality is ≈1.5% (6 deaths), then how can we evaluate whether the hospital is truly an outlier if it reports, say, 9 deaths? This situation becomes much worse for the individual: if an operator performs 80 percutaneous coronary interventions annually and has that same expected annual mortality rate of 1.5%, then she is expected to lose 1.2 patients each year of practice. Does she become an outlier if she experiences 2 deaths (66% over target)? Three deaths (150% over target)? Is she truly providing superior care if she experiences no deaths? This problem is exacerbated by having a large number of hospitals and thousands of operators, wherein some hospitals will be seen as being better or worse than their peers simply by the play of chance. This problem has been dealt with statistically by using Bayesian hierarchical modeling.5 This statistical approach will pull hospitals toward the average (ie, an observed to expected rate of 1) and shrink the confidence interval. This approach reduces the issue of multiplicity but tends to make all providers look the same and makes it quite difficult to identify a true outlier among low-volume providers. This notion may be important if low provider and hospital volumes really do contribute to increased risk. Conversely, larger providers can be penalized or rewarded in this scenario because their numbers permit them to stand out even after shrinking the confidence interval.
Another problem arises with this type of risk adjustment. The idea is to find differences between providers. This goal is dependent on risk adjustment accounting for nonquality of care variables. However, for acutely ill patients, mortality and morbidity are largely predicted by patient rather than provider variables. Thus, the smaller influence of the provider must be seen through thethicket of patient variables, which can actually form a type of competing risk (ie, deaths due to patient variables vs provider variables). On the other hand, for nonacute patients, the mortality risk may be so small (a fraction of a percent) that finding evidence of increased risk confidently attributable to a provider or hospital may be impossible and perhaps not a clinically meaningful measure.
The best way to make constructive use of benchmarked provider-level outcome data from large national databases is for performance improvement. In the world of public reporting, there is a much greater challenge. Professional societies have been well aware of the difficulties and have urged those who favor unfettered public reporting to proceed cautiously. Other stakeholders may not be responsive to the challenges, and the public may perceive calls to constrain public reporting as a sign that doctors and healthcare systems are hiding something. Indeed, there may be a perception that providers are using statistical arguments to avoid scrutiny. There is no perfect answer to this, and providers will have to accept that their outcomes will become increasingly available to the public. Professional societies must be responsive to the public and sanction, in some manner, practitioners or hospitals with convincing evidence of consistently poor outcomes while also defending practitioners who find themselves inappropriately in the public eye. Furthermore, agencies advocating for full disclosure and transparency also have a responsibility to inform the public of the limitations of information we have to share with them and of the dangers of misinterpreting numbers taken out of context. It is important to note that formal collaboration between patient advocacy groups and medical societies is sorely needed to develop education plans that can help patients and their families understand the information contained in public performance reports, lest patients make poor judgments about physician quality or physicians conclude that taking care of the sickest patients is simply too risky.
Sources of Funding
Funded in part by an Institutional Development Award from the National Institute of General Medical Sciences of the National Institutes of Health under Grant U54-GM104941 (PI: Binder-Macleod).
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
Circulation is available at http://circ.ahajournals.org.
- © 2017 American Heart Association, Inc.
- Krumholz HM,
- Brindis RG,
- Brush JE,
- Cohen DJ,
- Epstein AJ,
- Furie K,
- Howard G,
- Peterson ED,
- Rathore SS,
- Smith SC Jr.,
- Spertus JA,
- Wang Y,
- Normand SL
- McNulty EJ,
- Ng W,
- Spertus JA,
- Zaroff JG,
- Yeh RW,
- Ren XM,
- Lundstrom RJ.