doi: 10.1161/CIRCULATIONAHA.107.745075
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Circulation. 2007;116:2897-2899.)
© 2007 American Heart Association, Inc.
Editorial |
Cardiac Report Cards
How Can They Be Made Better?
From the Institute for Clinical Evaluative Sciences (J.V.T., P.C.A.), Toronto, Ontario, Canada; Division of Cardiology (J.V.T.), Shulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada; and University of Toronto (J.V.T., P.C.A.), Toronto, Ontario, Canada.
Correspondence to Dr Jack V. Tu, Institute for Clinical Evaluative Sciences, G106-2075 Bayview Ave, Toronto, Ontario, M4N 3M5 Canada. E-mail tu{at}ices.on.ca
Key Words: Editorials • cardiovascular diseases • cardiopulmonary bypass • myocardial infarction
When hospital- and surgeon-specific coronary artery bypass graft surgery death rates were first publicly released in 1991 in New York State, many clinicians reacted with anger and outright hostility.1 Over time, as more jurisdictions adopted the concept of public reporting of outcomes in the form of report cards, the view has shifted to gradual recognition and acceptance that cardiac report cards, however imperfect, are here to stay.
Articles pp 2960 and 2969
Professional associations such as the American College of Cardiology and American Heart Association have recognized the importance of measuring the processes and outcomes of cardiovascular care as a critical step in improving the quality of care delivered to patients.2 The field of cardiovascular medicine has been at the forefront of the "accountability" movement in health care for a number of reasons. Cardiovascular disease affects large sectors of the population, because conditions such as acute myocardial infarction and procedures such as bypass surgery are quite common, and tracking of hard objective outcomes such as death are readily available. Furthermore, much clinical research has focused on cardiovascular disease and generated a large base of scientific evidence on which care can be assessed and improved. Most recently, policy makers have embraced the concept of "pay for performance," which is based on the premise that economic rewards may stimulate better quality of care delivery.3
The debate about report cards has progressed from fundamental disagreements about their purpose and usefulness to discussions that focus on their content and the statistical methodology to be utilized. Poor performance on a public report card can have a potentially devastating impact on an individual’s career or on a hospital’s reputation. As such, the stakes for cardiovascular practitioners such as cardiac surgeons, cardiologists, and their hospitals are particularly high. At this time, however, many clinicians lack the training in epidemiology and statistics necessary to understand the methodology by which the quality of care and their performance are being assessed. For these reasons, it is imperative that cardiovascular practitioners learn about the research being conducted on report cards and their methodologies. This issue of Circulation contains important new contributions from Stukenborg and colleagues4 from the University of Virginia on the topic of improved administrative database comorbidity coding for acute myocardial infarction report cards, whereas O’Brien and colleagues5 from Duke University address the topic of composite measures for coronary artery bypass graft surgery report cards. In this editorial, we discuss how these 2 new studies on cardiac report card methodology advance the field.
Detailed clinical data on patient risk factors collected from chart reviews has historically been regarded as the "gold standard" for report cards; however, the substantial cost and time involved in chart review make this an impractical option in many situations. Several comparative studies have demonstrated a reasonably high correlation between risk-adjusted outcomes determined with clinical data from clinical registries versus administrative datasets available on the same patients.6–9 Consequently, administrative databases will continue to be an attractive data source for policy makers commissioning the development of report cards.
Although administrative databases have important limitations, they have several advantages. These include their relatively low cost, the fact that they are often population based, the relative accuracy of the demographic information provided, and the information provided about multiple patient comorbidities. Until recently, a major limitation of some administrative databases, such as the United States Medicare hospital claims data, has been that they do not distinguish between a comorbidity present on hospital admission (ie, a true patient risk factor) and one that developed during a patient’s stay (ie, a complication that developed during the course of the stay).4 The development of report cards with administrative data that do not distinguish between these 2 types of comorbidities is problematic and potentially misleading. For example, when admission comorbidities are not identified explicitly, it may appear that a hospital’s patient population has a high illness burden when in fact the patients’ illness burden became high because of complications caused by the care provided by that institution. Hospitals that provide better care with no or few complications, in this instance, would actually be penalized under this scenario, because their patient comorbidity profiles (which consist predominantly of admission comorbidities) or burden of illness appears to be lower.
In their article, Stukenborg and colleagues4 describe a study of alternative methods of adjusting for differences in patient case mix in the development of report cards that examine death after an acute myocardial infarction. They demonstrate that statistical models for risk adjustment developed with California hospital discharge administrative databases, which distinguish comorbidities present at admission from those developed after admission, perform significantly better statistically than previously developed models that do not distinguish between these 2 types of comorbidities. Furthermore, hospital rankings changed significantly in several cases in which the improved risk-adjustment models were used. The improvement in performance with the comorbidities-present-at-admission model is probably an overestimate of the true impact of this type of comorbidity coding change, because it is also based on many more comorbidities than the original model developed for the California Hospital Outcomes project.10 Nevertheless, few would disagree with the notion that future report cards developed with administrative databases should use this type of comorbidity coding enhancement whenever possible.
Another methodological challenge related to report cards involves composite performance measures. Composite measures are attractive because a single number or ranking serves to summarize multiple aspects of an institution’s or practitioner’s performance, and such measures can incorporate both process-of-care elements (how the care was delivered) and outcomes-of-care elements (the net result of the care delivered). Although patients are particularly interested in the outcomes of care, it is increasingly recognized that good evidence-based processes of care are necessary to achieve good outcomes. Report cards focused solely on either processes or outcomes are incomplete and do not fully reflect the different dimensions of care quality. However, there is no consensus as yet on how best to derive composite performance measures.
The article by O’Brien and colleagues5 studies how a new composite measure used in the Center for Medicare and Medicaid Services’ Hospital Quality Incentive Demonstration "pay for performance" project affects hospital report cards for isolated coronary artery bypass graft surgery with data from the Society of Thoracic Surgeons’ National Cardiac database.5 The Hospital Quality Incentive Demonstration composite performance measure includes 5 nationally endorsed process measures and the outcome measure of death, with each measure weighed equally. However, when applied to the Society of Thoracic Surgeons’ database, the mortality rate component explained only 4% of the total variance of the composite scores, because the standard deviation (degree of spread) of mortality rates was much smaller than that of the process measures included in the composite indicator. Under this composite measure, a hospital could have excellent outcomes with a very low mortality rate and yet receive a low overall composite score and potentially be penalized financially. In this instance, the financial reward for good mortality rates is very low compared with the financial reward for good performance on the process-of-care measures.
O’Brien et al5 suggest that by conducting standardization (ie, measuring each element on a similar scale or range), a more balanced composite measure can be created that weighs process and outcomes performance more equally. In our view, it does seem reasonable to standardize each element of a composite measure, and it is not technically difficult to do so. The remaining challenge involves determining how much weight to assign to process measures versus outcomes measures, and this is probably best determined by understanding the values and perspectives of the intended audience (eg, patients and payers) for a particular composite measure.
A common finding from both of these studies is that hospital rankings (and the corresponding financial rewards or penalties) can change dramatically depending on which data elements and statistical methods are used to assess performance. These findings are cold comfort for hospital administrators and clinicians whose reputations and financial status lie in the wake. Although it is unlikely we will ever achieve complete agreement among statisticians about the optimal statistical method for assessing quality of care, improved transparency with regard to the statistical methods used and their associated strengths and weaknesses is required so that the playing field is as "level" as possible. As demonstrated in these 2 studies, it is important for independent academic investigators to be able to replicate the methods being used and to test potential enhancements so that the methodology used in report cards can be both explained and further enhanced. Report card developers need to continually work to improve the rigor of their methods to ensure that publicly released report cards are fair and accurate and serve their intended purpose of improving quality of care.
 |
Acknowledgments |
|---|
Sources of Funding
Dr Tu is supported by a Canada Research Chair in Health Services Research and by a Career Investigator award from the Heart and Stroke Foundation of Ontario. Dr Austin is supported by a New Investigator award from the Canadian Institutes of Health Research. This editorial is supported in part by a Canadian Institutes of Health Research (CIHR) team grant in cardiovascular outcomes research (www.ccort.ca).
Disclosures
None.
|
Footnotes |
|---|
The opinions expressed in this editorial are not necessarily those of the editors or of the American Heart Association.
|
References |
|---|
 Top References |
|---|
1. Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards of reporting medical outcomes publicly. N Engl J Med. 1996; 334: 394–398.
2. Krumholz HM, Anderson JL, Brooks NH, Fesmire FM, Lambrew CT, Landrum MB, Weaver WD, Whyte J, Bonow RO, Bennett SJ, Burke G, Eagle KA, Linderbaum J, Masoudi FA, Normand SL, Pina IL, Radford MJ, Rumsfeld JS, Ritchie JL, Spertus JA. ACC/AHA clinical performance measures for adults with ST-elevation and non–ST-elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Performance Measures on ST-Elevation and Non–ST-Elevation Myocardial Infarction). J Am Coll Cardiol. 2006; 47: 236–265.
3. Fisher ES. Paying for performance: risks and recommendations. N Engl J Med. 2006; 355: 1845–1847.
4. Stukenborg GJ, Wagner DP, Harrell FE Jr, Oliver MN, Heim SW, Price AL, Han CK, Wolf AMD, Connors AF Jr. Which hospitals have significantly better or worse than expected mortality rates for acute myocardial infarction patients? Improved risk adjustment with present-at-admission diagnoses. Circulation. 2007; 116: 2960–2968.
5. O’Brien SM, DeLong ER, Dokholyan RS, Edwards FH, Peterson ED. Exploring the behavior of hospital composite performance measures: an example from coronary artery bypass surgery. Circulation. 2007; 116: 2969–2975.
6. Krumholz HM, Wang Y, Mattera JA, Wang Y, Han LF, Ingber MJ, Roman S, Normand SL. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation. 2006; 113: 1693–1701.
7. Krumholz HM, Wang Y, Mattera JA, Wang Y, Han LF, Ingber MJ, Roman S, Normand SL. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction. Circulation. 2006; 113: 1683–1692.
8. Austin PC, Tu JV. Comparing clinical data with administrative data for producing AMI report cards. J R Stat Soc Ser A Stat Soc). 2006; 169: 115–126.[CrossRef]
9. Ghali WA, Rothwell DM, Quan H, Brant R, Tu JV. A Canadian comparison of data sources for coronary artery bypass surgery outcome "report cards." Am Heart J. 2000; 140: 402–408.[CrossRef][Medline] [Order article via Infotrieve]
10. Stukenborg GJ, Wagner DP, Harrell FE Jr, Oliver MN, Heim SW, Price AL, Han CK, Wolf AM, Connors AF Jr. Present-at-admission diagnoses improved mortality risk adjustment among acute myocardial infarction patients. J Clin Epidemiol. 2007; 60: 142–154.[CrossRef][Medline] [Order article via Infotrieve]
This article has been cited by other articles:
 |
|
 |
|
  A. L. W. Shroyer, G. O. McDonald, B. D. Wagner, R. Johnson, L. M. Schade, M. R. Bell, and F. L. Grover Improving Quality of Care in Cardiac Surgery: Evaluating Risk Factors, Processes of Care, Structures of Care, and Outcomes Seminars in Cardiothoracic and Vascular Anesthesia, September 1, 2008; 12(3): 140 - 152. [Abstract] [PDF]
|
|
|
|
 |
|
 E. E. Drye and J. Chen Evaluating Quality in Small-Volume Hospitals Arch Intern Med, June 23, 2008; 168(12): 1249 - 1251. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||