doi: 10.1161/CIRCULATIONAHA.106.686725
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Circulation. 2007;115:1508-1510.)
© 2007 American Heart Association, Inc.
Editorial |
To Understand Cardiac Surgical Report Cards
Look Both Ways
From the Section of Cardiology, Boston University School of Medicine, Boston, Mass.
Correspondence to Dr Thomas J. Ryan, Section of Cardiology, Boston University Medical Center, C822, 88 E Newton St, Boston, MA 02118. E-mail thomas.ryan{at}bmc.org
Key Words: Editorials • bypass • surgery
The aphorism attributed to Sir Winston Churchill, "the longer back you look, the further forward you can see," would seem quite apt to place in perspective the importance of the article by Shahian and colleagues that is published in this week’s issue of Circulation.1 These investigators provide a comprehensive side-by-side comparison of risk-adjusted coronary artery bypass graft (CABG) mortality rates ascertained from validated core clinical variables with CABG mortality rates risk-adjusted from administrative data compiled primarily for billing purposes. The importance of this comparison of in-hospital and 30-day mortality for "isolated CABG" springs from its intended use as a report card to grade the quality of individual hospitals in the performance of cardiac surgery. Their report finds the same limitations and inaccuracies in estimates of mortality rates that use administrative data as had been reported in large and careful studies done 15 years earlier by Hannan,2 Jollis,3 and others. Unquestionably, this article should serve as a wake-up call for the entire cardiology community to realize just how important an issue lies on the doorstep.
Article p 1518
First, these investigators were responding to a mandate of the Massachusetts legislature that data on all cardiac surgical procedures be collected in the Society of Thoracic Surgeons National Adult Cardiac Database, which commenced in 2002. It was further directed that these data be submitted to the Massachusetts Data Analysis Center (Mass-DAC) based in the Department of Healthcare Policy at Harvard Medical School. There, under the direction of Dr Sharon-Lise T. Normand, the data would be validated and subjected to internal and external audit. The subsequent risk-adjusted mortality rates for "isolated CABG" performed at the 14 Massachusetts hospitals authorized to perform adult cardiac surgery would then be used to evaluate each hospital’s performance. Although this was done largely to provide oversight of 3 programs newly added to this list, it reflects an enlightened move on the part of a government body to provide accountability for delivery of a specified portion of health care within its borders.
The initial report of hospital profiling issued under this mandate was based on calendar year 2002 data and published 22 months later in October 2004. Given the desire to bring such information to the public with less expense and more quickly than was possible with audited clinical data, the Massachusetts Executive Office of Health and Human Services published its own fiscal year 2004 "isolated CABG" results derived from administrative data. These were based on hospital discharge billing data collected by the Massachusetts Division of Healthcare Finance and Policy according to the national model of the Agency for Healthcare Research and Quality. It was at this point that the authors seized a golden moment in the time-course of outcome research to compare their calendar year 2003 Mass-DAC–validated clinical model with the Agency for Healthcare Research and Quality administrative model.4 Their results demonstrate just how problematic performance measures (ie, report cards) based on administrative data analyzed by nationally-sanctioned models can be, primarily as a result of case misclassification and nonstandardized end points.
The notable findings were a 27.4% disparity between the Mass-DAC cohort and the contemporaneous Agency for Healthcare Research and Quality cohort in the identification of patients classified as "isolated CABG" (4440 versus 5657 patients). Clearly the administrative database included a higher percentage of complex CABG cases (CABG plus valve, aneurysm repair, or "other") than the audited and validated Mass-DAC cohort, with a resultant difference in mortality of 0.83% (Mass-DAC, 2.05%, versus Agency for Healthcare Research and Quality, 2.88%). Depending on whether in-hospital or 30-day mortality was used as the end point, there was a 9% relative difference in average mortality, which underscores the need to use standardized end points when these data are used for hospital profiling.
To better understand the origin of hospital report cards, we can look back 25 years to 1982 when the American Heart Association and the American College of Cardiology published their first Joint Practice Guidelines.5 Initially these guidelines were directed at the establishment of proper indications and contraindications for cardiovascular procedures such as guidelines for implantation of pacemakers, exercise testing, percutaneous transluminal coronary angioplasty, and other procedures. With the introduction of fibrinolytic therapy for the management of patients with ST-elevation acute myocardial infarction in the 1980s and the accumulation of a vast amount of pertinent new data, it was deemed timely to address the management of this clinical condition with the first-of-a-kind disease-entity guideline in 1990.6 The trademark of all guidelines is the extensive review and critical analyses of the existing data that is culled from randomized trials, observational studies, and consensus opinion of appropriate experts. Normative action is classified according to the now familiar class I = definitely indicated; class IIa = probably indicated; class IIb = probably not indicated; class III = definitely not indicated or even harmful.
Thus, for more than a quarter century we have seen evidence-based medicine lead to the promulgation of practice guidelines that are assumed to represent a national standard of care, which has led to the development of clinical pathways that have begotten performance measures that have morphed into quality measures that are now used to profile hospital performance. The use of this latter metric to rate a provider’s proficiency, efficiency, and skill in the achievement of its stated goals is both laudable and sensible. This is readily understood when given to the provider in private, with the objective to stimulate self-evaluation for the purposes of improvement and any voluntary comparisons that it may wish to obtain from within its peer group. The Northern New England Consortium is emblematic of the success such an approach can achieve.7 When this metric is to be reported publicly to inform the community at large, there arises an understandable tension among the involved providers, but few question the right and probity for the consumer to have this information. The authors of the article by Shahian et al fashioned their approach after the internationally acclaimed New York State Reporting System, which is perhaps the most valid of all large cardiac databases because it captures data by fiat on every cardiovascular procedure performed in the state.8 The main finding of the Massachusetts experience was that, in the case of isolated CABG surgery, it is imperative that the creation of the report card be based on data of the highest quality, derived from prospectively gathered, validated, and audited clinical sources and not from administrative data. Of critical importance, in my view, is the commitment made in Massachusetts to obtain the denominators (all cases) for all numerators (end points). Their report also shows that administrative data are prone to case misclassification, are hampered by nonstandardized end points, and lack sufficient detail to discriminate complications from comorbidities. Additionally, when such sources are used, it is not possible to detect "system gaming" by exclusion of a poor-outcome "isolated CABG" by deceptively coding it as "CABG + other" without audits by experts. In addition to this imperative are recent suggestions that these report cards constitute a good measure by which to reimburse medical providers under the dictum of pay for quality of care rather than quantity of care.
Recent attempts to validate the use of administrative versus clinical data for CABG surgery report cards have fallen prey to these difficulties9,10 or have failed to meet the "Standards for Statistical Models Used for Public Reporting of Health Outcomes" recently issued in a Scientific Statement of the American Heart Association11.
Viewed in the present tense and like all contemporary research, the report by Shahian and colleagues1 represents one small piece in the larger mosaic that itself can only be viewed as a work in progress. Their study design, however, does contribute the most recent and compelling data to deliver the message that we must expend the energy, time, and treasure required to obtain these valid measures of hospital performance rather than accept the available, less costly but imprecise, administrative data. This is a very crucial point for several reasons: (1) CABG is the most frequently performed major surgical procedure in the country and has substantial public health and cost implications; (2) percutaneous coronary interventions, an even more widely utilized counterpart of CABG, is undergoing the same profiling; and (3) there is a need to go beyond short-term mortality rates to truly assess quality. Although short-term mortality can be expected to capture such errors of commission as poor surgery, it is a crude and almost primitive measure of quality of surgery and completely fails to identify most errors of omission. The energy, time, and expense should be spent to obtain robust and sensitive quality measures such as longer-term symptom-free survival, objective estimates of completeness and adequacy of revascularization, as well as suitability for gradations of age, severity of illness, and comorbidities. It is obvious this cannot be done "on the cheap," nor can it be done by others. It will always require dedicated surgeons, interventionalists, clinicians, biostatisticians, and others who are committed to the principle of accountability. It will require a sea change in our thinking to provide the accountability necessary to restore the trust in medicine that has been clouded by the competing interests of market share and the informed consumer.
If the above is a reflection of the clear and present data, what will be the future? Looking back as far as one can to the earliest description of isolated CABG by Dr Rene Favoloro 40 years ago,12 we can reasonably assume that the CABG mortality rate is unlikely to decline much lower than its current rate of 2% to 3%.13 Quite surprisingly, in the early 1970s CABG mortality rates were in the 3% to 4% range, well before all the advances in myocardial protection techniques, pharmacological support, and circulatory assist devices we have today.14 This is largely attributable to the fact that patients in those days were, on average, 20 years younger, had nowhere near the extent of comorbidities, nor the number of risk factors seen in today’s surgical population.
The Society of Thoracic Surgeons Quality Measurement Task Force envisions that a new composite end point will emerge in the very near future that has greater sensitivity and specificity than the current mortality rate. We should also witness some added mathematical certainty to our observations as greater numbers of validated databases with compulsory reporting of "all cases" come on-line. As for 40 years hence, one can only speculate, but these critical questions of outcome and hospital profiling may have long disappeared. Conceivably, with the introduction of a unique individual identifier, coupled with an electronic marvel such as an implantable "medical chip" that could be programmed and interrogated on demand, we will have replaced the paper report card with a continuously updated, multifunctional, text-messaging scoreboard.
 |
Acknowledgments |
|---|
Disclosures
None.
|
Footnotes |
|---|
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
|
References |
|---|
 Top References |
|---|
1. Shahian DM, Silverstein T, Lovett A, Wolf RE, Normand S-LT. Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards. Circulation. 2007; 115: 1518–1527.
2. Hannan EL, Kilburn H Jr, Lindsey ML, Lewis R. Clinical versus administrative databases for CABG surgery: does it matter? Med Care. 1992; 30: 892–907.[Medline] [Order article via Infotrieve]
3. Jollis JG, Ancukiewics M, Delong ER, Pryor DB, Muhbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Ann Intern Med. 1993; 119: 844–850.
4. Shahian DM, Torchiana DF, Normand S-LT. Implementation of a cardiac surgery report card: lessons from the Massachusetts experience. Ann Thorac Surg. 2005; 80: 1146–1150.
5. Frye RL, Collins JJ, DeSanctis RW, Dodge HT, Dreifus LS, Fisch C, Gettes LS, Gillette PC, Parsonnet V, Reeves J, Weinberg SL. Guidelines for permanent cardiac pacemaker implantation, May 1984: a report of the Joint American College of Cardiology/American Heart Association Task Force on Assessment of Cardiovascular Procedures (Subcommittee on Pacemaker Implantation). Circulation. 1984; 70: 331A–339A.[Medline] [Order article via Infotrieve]
6. Gunnar RM, Bourdillon PDV, Dixon DW, Fuster V, Karp RB, Kennedy JW, Klocke FJ, Passamani ER, Pitt B, Rapaport E, Reeves TJ, Russell RO Jr, Sobel BE, Winters WL, Fisch C, Beller GA, DeSanctis RW, Dodge HT, Weinberg SL. Guidelines for the early management of patients with acute myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Assessment of Diagostic and Therapeutic Cardiovascular Procedures (Subcommittee to Develop Guidelines for the Early Management of Patients with Acute Myocardial Infarction). J Am Coll Cardiol. 1990; 16: 249–292.[Medline] [Order article via Infotrieve]
7. O’Connor GT, Plume SK, Olmstead EM, Morton JR, Maloney CT, Nugent WC, Hernandez F Jr, Clough R, Leavitt BJ, Coffin LH, Marrin CA, Wennberg D, Birkmeyer JD, Charlesworth DC, Malenka DJ, Quinton HB, Kasper JF. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery: the Northern New England Cardiovascular Disease Study Group. JAMA. 1996; 275: 841–846.
8. Hannan EL, Kilburn H Jr, O’Donnell JF, Lucacik G, Shields EP. Adult open heart surgery in New York state: an analysis of risk factors and hospital mortality rates. JAMA. 1990; 264: 2768–2774.
9. Geraci JM, Johnson ML, Gordon HS, Petersen NJ, Shroyer AL, Grover FL, Wray NP. Mortality after cardiac bypass surgery: prediction from administrative versus clinical data. Med Care. 2005; 43: 149–158.[CrossRef][Medline] [Order article via Infotrieve]
10. Parker JP, Li Z, Damberg CL, Danielsen B, Carlisle DM. Administrative versus clinical data for coronary artery bypass graft surgery report cards: the view from California. Med Care. 2006; 44: 687–695.[CrossRef][Medline] [Order article via Infotrieve]
11. Krumholz HM, Brindis RG, Brush JE, Cohen DJ, Epstein AJ, Furie K, Howard G, Peterson ED, Rathore SS, Smith SC Jr, Spertus JA, Wang Y, Normand SL; American Heart Association; Quality of Care and Outcomes Research Interdisciplinary Writing Group; Council on Epidemiology and Prevention; Stroke Council; American College of Cardiology Foundation. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council: endorsed by the American College of Cardiology Foundation. Circulation. 2006; 113: 456–462.
12. Favaloro RG. Saphenous vein graft in the surgical treatment of coronary artery disease: operative technique. J Thorac Cardiovasc Surg. 1969; 58: 178–185.[Medline] [Order article via Infotrieve]
13. Jones RH. Does government regulation enhance quality of cardiovascular procedures? Circulation. 2006; 114: 2090–2091.
14. Favaloro RG. Critical analysis of coronary artery bypass grafting: a 30-year journey. J Am Coll Cardiol. 1998; 31: 1B–63B.[CrossRef][Medline] [Order article via Infotrieve]
This article has been cited by other articles:
 |
|
 |
|
  S. Westaby, N. Archer, N. Manning, S. Adwani, C. Grebenik, O. Ormerod, R. Pillai, and N. Wilson Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality BMJ, October 13, 2007; 335(7623): 759 - 759. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||