Published online before print March 12, 2007, doi: 10.1161/CIRCULATIONAHA.106.633008
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Circulation. 2007;115:1518-1527.)
© 2007 American Heart Association, Inc.
Cardiovascular Surgery |
Comparison of Clinical and Administrative Data Sources for Hospital Coronary Artery Bypass Graft Surgery Report Cards
From Tufts University School of Medicine (D.M.S.), Harvard Medical School (T.S., A.F.L., R.E.W., S.-L.T.N.), and Harvard School of Public Health (S.T.N.), Boston, Mass.
Correspondence to Sharon-Lise Normand, Department of Health Care Policy, Harvard Medical School, 180 Longwood Ave, Boston, MA 02115. E-mail Sharon{at}hcp.med.harvard.edu
Received April 10, 2006; accepted December 22, 2006.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Background— Regardless of statistical methodology, public performance report cards must use the highest-quality validated data, preferably from a prospectively maintained clinical database. Using logistic regression and hierarchical models, we compared hospital cardiac surgery profiling results based on clinical data with those derived from contemporaneous administrative data.
Methods and Results— Fiscal year 2003 isolated coronary artery bypass grafting surgery results based on an audited and validated Massachusetts clinical registry were compared with those derived from a contemporaneous state administrative database, the latter using the inclusion/exclusion criteria and risk model of the Agency for Healthcare Research and Quality. There was a 27.4% disparity in isolated coronary artery bypass grafting surgery volume (4440 clinical, 5657 administrative), a 0.83% difference in observed in-hospital mortality (2.05% versus 2.88%), corresponding differences in risk-adjusted mortality calculated by various statistical methodologies, and 1 hospital classified as an outlier only with the administrative data–based approach. The discrepancies in volumes and risk-adjusted mortality were most notable for higher-volume programs that presumably perform a higher proportion of combined procedures that were misclassified as isolated coronary artery bypass grafting surgery in the administrative cohort. Subsequent analyses of a patient cohort common to both databases revealed the smoothing effect of hierarchical models, a 9% relative difference in mortality (2.21% versus 2.03%) resulting from nonstandardized mortality end points, and 1 hospital classified as an outlier using logistic regression but not using hierarchical regression.
Conclusions— Cardiac surgery report cards using administrative data are problematic compared with those derived from audited and validated clinical data, primarily because of case misclassification and nonstandardized end points.
Key Words: coronary artery bypass surgery • databases • health policy • outcome assessment (health care) • statistics
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Cardiac surgeons have consistently led the effort to collect, analyze, and apply outcomes data for the benefit of patients. Cardiac surgery also is the paradigm for public performance reporting.1–3 Its predominant procedure, coronary artery bypass grafting (CABG), is the most frequently performed of all complex surgical operations, has well-defined and measurable end points, and has substantial public health and cost implications.
Editorial p 1508
Clinical Perspective p 1527
Determining the optimal statistical methodology for risk adjustment and provider profiling has been controversial.1,3 No statistical technique, however, regardless of its sophistication, can compensate for flawed data, and it is this concern that motivates the present study. Healthcare data may be derived from administrative or clinical sources,4,5 the latter including both retrospective chart abstraction and prospectively maintained databases such as the Society of Thoracic Surgeons National Adult Cardiac Database. Administrative data, typically derived from discharge billing forms, are the most inexpensive and readily available source of information regarding acute care hospitalizations. Although not originally intended for this purpose, such data also have been used to assess healthcare provider performance and are currently the basis for some public "report cards."
Using administrative data to evaluate the quality of provider performance is plausible when they are the sole available source of information and when it has been demonstrated that their use results in conclusions similar to those derived from clinical data.6 However, are public reports based on hospital discharge data an acceptable alternative when more accurate but also more costly and resource-intensive options are available, as in the case of CABG surgery?
The present study, prompted by recent experience in Massachusetts, investigates the usefulness of a national algorithm based on administrative data, compared with results derived from prospectively maintained clinical data, in assessing hospital CABG outcomes. Beginning in 2002, Massachusetts law mandated that data on all cardiac surgical procedures be collected using the Society of Thoracic Surgeons National Adult Cardiac Database. These data are submitted to the Massachusetts Data Analysis Center (Mass-DAC)7 based at the Department of Health Care Policy at Harvard Medical School, where they are subjected to internal and external audit and validation.8,9 Analyses are then performed using hierarchical logistic regression in which random intercepts are included for each hospital.1,9 The first CABG report card was published in October 2004 and was based on calendar year 2002 data.
Because of its desire to bring information to the public more quickly than was possible with audited clinical data, the Massachusetts Executive Office of Health and Human Services published its own fiscal year 2004 CABG results on their Internet site in 2006. They were based on hospital discharge billing data collected by the Massachusetts Division of Health Care Finance and Policy using a national model proposed by the Agency for Healthcare Research and Quality (AHRQ).10 The Massachusetts Executive Office of Health and Human Services also published the calendar year 2003 Mass-DAC isolated CABG results. This afforded a side-by-side, although not strictly contemporaneous, comparison.
Notably, the CABG cohort based on AHRQ CABG criteria as applied to Massachusetts administrative data consisted of 5315 procedures for fiscal year 2004, whereas Mass-DAC recorded 4393 confirmed isolated CABG cases for calendar year 2003. This 21% discrepancy in volume (as high as 40% at 1 hospital) cast doubt on the accuracy of the AHRQ algorithm, particularly because the volume of isolated CABG surgery declined rather than increased in Massachusetts between 2003 and 2004.
Because we had access to both audited clinical data and administrative data for fiscal year 2003, we conducted a comprehensive, contemporaneous comparison of these 2 algorithms and the hospital quality profiles derived from them.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Creation of Cohorts
Two cohorts were created: 1 based on clinical data (Mass-DAC) and 1 that used hospital discharge billing data (AHRQ). The AHRQ CABG model includes patients on the basis of discharge dates; the Mass-DAC model includes patients on the basis of surgery dates. To obtain strictly contemporaneous groups, the Mass-DAC cohort for the present study was selected on the basis of discharge date, including patients who were discharged from a nongovernmental Massachusetts acute care hospital with a diagnosis of CABG surgery between October 1, 2002, and September 30, 2003.
Clinical Cohort (Mass-DAC)
Hospitals submit clinical information to Mass-DAC for all adult patients (age 
18 years) having cardiac surgery in a Massachusetts non–US governmental hospital. Isolated CABG cases are identified by excluding cases that combine CABG with other significant procedures such as valve replacement or carotid endarterectomy. For certain procedures with problematic classification, additional exclusion criteria were used on the basis of clinical data submitted by the hospital.7 Cases coded by hospitals as "CABG+other" are reviewed by an adjudication committee of senior cardiac surgeons, and a determination is made as to whether the case is legitimately in the "other" category. This review seeks to mitigate gaming of the system in which cases with unfavorable outcomes are shifted into an unreported category.1
Administrative Cohort (AHRQ)
Patients 40 years of age discharged with International Classification of Diseases, 9th revision, clinical modification (ICD-9-CM) codes 36.10 to 36.19 are selected. Thus, the total number of discharges invariably includes cases that are not isolated CABG such as CABG plus cardiac valve replacement. Discharges with missing discharge disposition or those involving transfer to a short-term general hospital are excluded.
The gold standard for this comparison is the audited database and the analytical methodology used by Mass-DAC.7–9
Definition of Outcomes
Mass-DAC uses 30-day all-cause mortality as the primary patient end point. Hospital-reported 30-day status is verified by linking all cardiac surgery cases reported to the Massachusetts Registry of Vital Records and Statistics. The AHRQ model uses all-cause in-hospital mortality reported in hospital billing data.
The primary hospital end point is standardized mortality incidence rate (SMIR), interpreted as the mortality at each hospital standardized to the overall mortality experience in the population of hospitals under study. Confidence intervals are constructed when estimating logistic regression models, and probability intervals are constructed when estimating hierarchical logistic regression models. For ease of exposition, we use the terms interval estimates or confidence intervals interchangeably.
Risk Adjustment
Mass-DAC uses the Society of Thoracic Surgeons National Adult Cardiac Database instrument and definitions, as well as a hierarchical logistic regression model, to determine standardized mortality rates. Risk factors for the model were selected by expert consensus from the best existing clinical models. A hospital-specific random intercept was included for each hospital in the sample.
The AHRQ model uses standard logistic regression, and its risk factors include demographic information and factors based on All Patient Refined Diagnosis-Related Group codes.10
Statistical Analyses
Comparison of Cardiac Surgery Volumes With CABG Volumes and With Observed CABG Mortality Rates
CABG volumes and observed mortality rates were examined overall and by hospital using graphical methods. We plotted ratios of CABG volumes in the original AHRQ and Mass-DAC cohorts as a function of total cardiac surgery procedures to determine whether the disparities between cohorts varied by hospital program size. Total cardiac surgery volume was obtained from the Mass-DAC database. A similar plot was constructed to examine the relationship of the difference between observed in-hospital mortality in the original AHRQ and Mass-DAC cohorts as a function of total cardiac program volume. Scatterplot smoothers11 were constructed to illustrate these relationships.
Comparison of Discharges Included in the AHRQ and Mass-DAC Models
We identified the number of discharges meeting inclusion criteria of both approaches, the number included in the AHRQ but not the Mass-DAC model, and the number included in the Mass-DAC but not in the AHRQ model. The administrative data and Mass-DAC data were merged using a combination of medical record number, discharge date, admission data, and date of birth. When possible, we determined the specific reason that discharges were included in 1 model but not the other. Because we had in-hospital mortality for all patients, we also calculated the observed inpatient mortality rate by exclusion category.
Comparison of Hospital Standardized Mortality and Outlier Determination
In addition to misclassification errors, we hypothesized other potential reasons for differences in the AHRQ and Mass-DAC approaches, including risk models, mortality end points, and statistical methodology. To isolate differences not attributable to classification errors, additional comparative analyses were performed using a common cohort of patients who met both AHRQ and Mass-DAC inclusion criteria.
For this common cohort, we estimated the predicted risk of in-hospital mortality for each patient using simple logistic regression models and the model-specific risk predictors and determined the areas under the receiver-operating characteristics curves. Second, we computed point and 95% CI estimates of in-hospital SMIRs using AHRQ and Mass-DAC predictors, estimated by both standard and hierarchical logistic regressions. The standardized rate is obtained by risk adjusting for the case mix of the hospital, indirectly standardizing its risk to an expected risk, and then multiplying by the state average. If the interval estimate for the hospital was below the state average, we concluded that mortality was lower than expected. If the interval estimate included the state average, then the mortality was as expected. If the interval estimate was above the state average, then mortality was higher than expected. Finally, we determined 30-day SMIRs using Mass-DAC predictors and both standard and hierarchical logistic regression. Models were fitted using the WinBUGS (Medical Research Council Biostatistics Unit, Cambridge, United Kingdom), SAS (SAS Institute, Cary, NC), and S-Plus (Insightful Corp, Seattle, Wash) software systems.
The authors had full access to and take full responsibility for the integrity of the data. All authors have read and agree to the manuscript as written.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
CABG Volumes (Original Cohorts)
The fiscal year 2003 Mass-DAC cohort consisted of 4440 isolated CABG patients who were discharged between October 1, 2002, and September 30, 2003 (inclusive). In contrast, the contemporaneous AHRQ cohort consisted of 5657 patients identified as CABG patients using the criteria noted above, which represents a 27.4% disparity. Figure 1 (left) depicts the ratio of AHRQ CABG volume to Mass-DAC CABG volume on the y axis and hospital total cardiac surgery volume (according to the Mass-DAC sample) on the x axis, along with a scatterplot smoother.11 The ratio increases directly with total cardiac surgery volume (notwithstanding the problematic results from the very lowest-volume startup program). Higher-volume hospitals presumably perform a higher percentage of complex CABG cases (eg, CABG and valve, CABG and aneurysm repair) that are excluded from the Mass-DAC cohort but inappropriately included in the AHRQ cohort.
|
Observed CABG Mortality (Original Cohorts)
Overall observed in-hospital and 30-day mortality rates in the Mass-DAC cohort were 2.05% (91 of 4440) and 2.27% (101 of 4440). The in-hospital mortality observed in the AHRQ cohort was 2.88%, exceeding the corresponding Mass-DAC value by 0.83% (range, 0% to 1.63%). In-hospital mortality from the AHRQ-defined cohort is always higher than that from the Mass-DAC cohort, and the absolute difference increases directly with hospital total cardiac surgery volumes (Figure 1, right).
Risk-Adjusted and Standardized Mortality (Original Cohorts and Methodologies)
Figure 2 depicts hospital-specific 30-day SMIRs (95% probability intervals) using hierarchical logistic regression for the original Mass-DAC cohort (n=4440) and hospital-specific SMIR point estimates (95% CIs) using logistic regression for the original administrative cohort (n=5657). The AHRQ cohort has generally higher mortality rates (overall, 2.88% versus 2.27%). The smoothing effect of the Mass-DAC hierarchical model reduces interhospital variability, as demonstrated by the narrower 95% CIs compared with the AHRQ model. Hospital 14 had no in-hospital mortalities, and the AHRQ model–estimated SMIR is 0% (95% CI undefined), whereas the hierarchical model estimate is 2.30% (95% CI, 1.24 to 3.71). No hospital is identified as a statistical outlier based on the Mass-DAC algorithm, whereas hospital 4 is identified as having statistically higher-than-expected 30-day mortality using the AHRQ algorithm.
|
Comparison of AHRQ and Mass-DAC Cohorts
Figure 3 depicts 3 groups of patients: (1) 1264 patients found only in the AHRQ cohort having an in-hospital observed mortality of 5.93%, (2) 47 patients found only in the Mass-DAC cohort having an observed mortality of 4.25%; and (3) 4393 patients found in both cohorts, with 2.03% mortality in the Mass-DAC cohort (confirmed as accurate) and 2.00% in the AHRQ cohort. Most of the patients (42 of 47) classified in group 2 met the age criteria for Mass-DAC but not for AHRQ, had missing data, or had been transferred. Among the 1264 patients in group 1, about half (n=596) had CABG combined with valve procedures. This misclassification could be corrected with a more refined administrative algorithm. However, an additional 663 cases, 
10% of the original AHRQ cohort, were coded as CABG plus other. Even with a clinical database and well-defined exclusion criteria, adjudication of such cases by an expert panel typically is required. It would be difficult, if not impossible, to correctly categorize such procedures (isolated versus nonisolated CABG) from administrative data using only discharge codes. This subset of CABG plus other cases was associated with high mortality (7.39%) and clearly had substantial impact on the overall AHRQ mortality estimate.
|
Of the 4393 discharges, in-hospital mortality differed by 1 subject between the 2 data sources.
Risk Model Differences (Common Cohort, N=4393)
Tables 1 and 2
list risk factors, frequencies, and associations with in-hospital or 30-day mortality for 4393 patients meeting criteria for both the AHRQ and Mass-DAC models. For example, 2.0% of patients in the Mass-DAC cohort presented with cardiogenic shock, of which 23% died in-hospital and 19.5% died within 30 days. In-hospital mortality exceeds 30-day mortality in this and several other instances in Tables 1, 2 and 3 because the Society of Thoracic Surgeons definition of in-hospital mortality includes all in-hospital deaths, regardless of timing.
|
|
|
Comparisons of Hospital Risk-Standardized Mortalities (Common Cohort, N=4393)
Table 3 lists the individual hospital volumes and observed mortality rates for the 4393 patients common to both the AHRQ and Mass-DAC cohorts.
Figure 4 depicts hospital-specific SMIRs (with 95% posterior intervals) for the common cohort estimated by hierarchical logistic regression using Mass-DAC and AHRQ predictors. The 95% Mass-DAC intervals are generally larger compared with those based on AHRQ predictors. Figure 5 depicts a similar comparison using logistic regression. There were 3 hospitals with no in-hospital mortalities for which the logistic model produces SMIRs of 0%.
|
|
Figure 6 depicts the results for 30-day mortality using Mass-DAC predictors estimated using both hierarchical and standard logistic regression models. Overall, there was an absolute 0.18% (2.21% versus 2.03%) difference in average mortality, depending on whether in-hospital (Figure 5) or 30-day (Figure 6) mortality was used as the end point. This 9% relative difference in mortality suggests that choice of an appropriate, standardized end point is not a trivial concern. The smoothing effect of the Mass-DAC hierarchical model shrinks the estimates toward the state mean (especially for low-volume providers), reduces interhospital variability, and narrows the 95% intervals.
|
Overall, these results demonstrate substantially both higher average mortality at 30 days compared with in-hospital mortality and the shrinkage effect of the hierarchical logistic regression model on point and 95% interval estimates. Notably, using in-hospital mortality, hospital 5 was clearly an outlier using the AHRQ or Mass-DAC predictors, estimated via logistic regression. It was not an outlier using hierarchical regression and either set of predictors.
Comparative Risk Model Discrimination (Common Cohort, N=4393)
The Mass-DAC model using in-hospital mortality as an outcome had an area under the receiver-operating characteristics curve of 0.84, and the observed mortalities in the highest and lowest risk deciles were 11.7% and 0.2%. The AHRQ model had an area under the curve of 0.89 (not significantly different from the Mass-DAC model; P=0.05) with observed mortalities in the highest and lowest risk deciles of 15.6% and 0.4%. Finally, the Mass-DAC model assigned 0.2% of the sample to a risk of dying >0.50; the AHRQ assigned no one to this high-risk group.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
These analyses demonstrate the substantial variability in CABG profiling results that occurs as a result of differences in data source, algorithms to identify isolated CABG, nonstandardized end points, and different risk models and statistical methodologies. In the present study, it appears that the predominant factor is misclassification of isolated CABG cases using an algorithm based on administrative data. Many of these classification errors could not have been corrected even with more sophisticated software algorithms and would require case-by-case review. Because of these classification errors, many of these programs have their true isolated CABG volumes and mortality rates inflated by the inclusion of combined procedures that are inherently higher risk, and this may disproportionately affect higher-volume tertiary centers. Our analyses also demonstrate the smoothing effect of the hierarchical model, which compensates for small sample sizes and clustering. This is manifested graphically as a shift in adjusted mortality rates toward the state average for low-volume programs for which there is relatively little information.
When used as the basis for public report cards,1,3 outcomes data must be of the highest possible reliability and validity, a requirement emphasized by the Institute of Medicine12,13 and the American Heart Association.6 Criteria have been proposed for profiling initiatives that evaluate risk-adjusted outcomes in cardiac surgery.5 Clinical data registries generally incorporate these desirable features, whereas administrative databases do not.
Because clinical data are unavailable for many specialties and because of the emphasis currently being placed on outcomes transparency and public accountability, there has been renewed interest in using administrative data for these purposes. Administrative data originally were intended primarily for reimbursement, although there has been a resurgence of interest in their use for healthcare research (despite having been abandoned by the federal government in 1993 as the basis for publicly reporting Medicare hospital mortality4,14,15). Recent reports suggest that experienced researchers can, with careful design and sophisticated statistical methodology, develop models based on administrative data for certain conditions that may be appropriate for profiling,16,17 but experts have generally cautioned against this practice.4,12,18–29 Administrative data should be used to profile hospitals only after rigorous verification of their validity for such purposes, as was recently demonstrated for mortality after AMI16 and heart failure.17
Concerns regarding administrative data sources for profiling include the following:
- Incorrect number of cases for analysis. This results from errors in diagnosis or procedure coding and from the use of software algorithms that cannot reliably identify isolated CABG cases.30
- Restricted study populations. Performance results based on only 1 group (eg, Medicare patients), more common in studies based on administrative databases, are not necessarily representative of overall program quality and may bias the results of profiling.30,31
- Nonstandardized mortality end points. Most administrative data registries record only in-hospital mortality, whereas clinical databases often record 30-day mortality. This may disadvantage providers having fewer options for early transfer to extended care facilities.32 The additional statistical and clinical implications of nonstandardized time intervals and the increasing ability to prolong time to death of critically ill postoperative CABG patients have recently been reviewed.33
- Misalignment of data sources with intended use. Administrative databases were originally designed for claims submission and benefits coordination. Coding practices and software algorithms were developed for such reimbursement issues, not clinical outcomes profiling.23,34
- Coding accuracy and completeness. Administrative data coding accuracy may be problematic because of imprecise or ambiguous definitions4,12,23,25,31,35 or because the coding is not supported by the clinical record.36 In complex and lengthy hospitalizations, particularly if the patient ultimately dies, there is also a tendency to undercode chronic or asymptomatic comorbidities when using administrative data,28,34,35,37,38 with the result that these may paradoxically appear to be protective.26,35
- Absence of critical clinical variables. Much of the predictive power in surgical risk models is derived from a limited number of critical clinical variables that typically are not available in administrative databases.39,40 Addition of a few such clinical variables to administrative models may substantially improve their performance.25,35,41
- Complications and comorbidities. With administrative data, it is difficult to accurately map ICD-9-CM codes to clinically relevant comorbidities or complications, to consistently capture all complications and comorbidities, and to distinguish comorbidities from complications. In general, undercoding of comorbidities in administrative databases seems to be a much more common problem than overcoding, although either type of misclassification may occur despite the use of sophisticated coding algorithms.4,42–51
As a consequence of failing to differentiate complications from comorbidities, the performance of risk models based on administrative data may be exaggerated. This results from including predictors in the risk model that are actually late-hospitalization, preterminal events and thus highly predictive of subsequent mortality.4,14,19,29
Using an indicator for condition present at admission or date stamping of secondary diagnoses49 may improve differentiation of complications from comorbidities in administrative databases. This is currently the practice in at least 2 states, New York and California.4,52
- Outlier determination. Several studies have shown differences in the outliers determined using administrative versus clinical data, even when the administrative databases were relatively sophisticated.35,41,52–54
Study Limitations
In the present study, we used the actual AHRQ algorithm used by the Massachusetts Executive Office of Health and Human Services to generate CABG profiles for their Web site in 2006. Better administrative algorithms could certainly be developed, especially with regard to the correct identification of isolated CABG cases. The results from administrative-based profiling also could be enhanced by adding a few critical clinical variables and using more appropriate statistical techniques such as hierarchical models.
Conclusions
Government agencies and payers will be increasingly tempted to use administrative data for provider profiling because it is inexpensive and available within a short time frame. Given the inaccuracies demonstrated in our Massachusetts CABG analysis, this practice should be discouraged. Efforts should be made, as they are currently in Massachusetts, to reduce the lag time for producing more accurate reports based on clinical data.
|
Acknowledgments |
|---|
Source of Funding
This work was supported by a contract from the Massachusetts Department of Public Health (No. 620022A4PRE).
Disclosures
Dr Shahian receives an honorarium for providing clinical advice to Mass-DAC. T. Silverstein, A. Lovett, R. Wolf, and Dr Normand receive salary support from a Massachusetts Department of Public Health contract.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
1. Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg. 2001; 72: 2155–2168.
2. Shahian DM. Improving cardiac surgery quality: volume, outcome, process? JAMA. 2004; 291: 246–248.
3. Shahian DM, Blackstone EH, Edwards FH, Grover FL, Grunkemeier GL, Naftel DC, Nashef SA, Nugent WC, Peterson ED. Cardiac surgery risk models: a position article. Ann Thorac Surg. 2004; 78: 1868–1877.
4. Iezzoni LI. Risk Adjustment for Measuring Health Care Outcomes. 3rd ed. Chicago, Ill: Health Administration Press; 2003.
5. Daley J. Criteria by which to evaluate risk-adjusted outcomes programs in cardiac surgery. Ann Thorac Surg. 1994; 58: 1827–1835.[Abstract]
6. Krumholz HM, Brindis RG, Brush JE, Cohen DJ, Epstein AJ, Furie K, Howard G, Peterson ED, Rathore SS, Smith SC Jr, Spertus JA, Wang Y, Normand SL. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council: endorsed by the American College of Cardiology Foundation. Circulation. 2006; 113: 456–462.
7. Massachusetts Data Analysis Center. Available at: http://www.massdac.org Accessed March 1, 2006.
8. Shahian DM, Torchiana DF, Normand SL. Implementation of a cardiac surgery report card: lessons from the Massachusetts experience. Ann Thorac Surg. 2005; 80: 1146–1150.
9. Shahian DM, Torchiana DF, Shemin RJ, Rawn JD, Normand SL. Massachusetts cardiac surgery report card: implications of statistical methodology. Ann Thorac Surg. 2005; 80: 2106–2113.
10. US Department of Health and Human Services, Agency for Healthcare Research and Quality. AHRQ Quality Indicators: Inpatient Quality Indicators: Technical Specifications. June 2002. Available at: http://www.qualityindicators.ahrq.gov/downloads/iqi/iqi_technical_specs _v30.pdf. Accessed March 1, 2006.
11. Cleveland WS. Robust locally weighted regression and smoothing scatterplots. J Am Stat Assoc. 1979; 74: 829–836.[CrossRef]
12. Iezzoni LI. Assessing quality using administrative data. Ann Intern Med. 1997; 127: 666–674.[Medline] [Order article via Infotrieve]
13. Institute of Medicine. Performance Measurement: Accelerating Improvement. Washington, DC: National Academies Press; 2006.
14. Iezzoni LI. The risks of risk adjustment. JAMA. 1997; 278: 1600–1607.
15. Krumholz HM, Rathore SS, Chen J, Wang Y, Radford MJ. Evaluation of a consumer-oriented internet health care report card: the risk of quality ratings based on mortality data. JAMA. 2002; 287: 1277–1287.
16. Krumholz HM, Wang Y, Mattera JA, Wang Y, Han LF, Ingber MJ, Roman S, Normand SL. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction. Circulation. 2006; 113: 1683–1692.
17. Krumholz HM, Wang Y, Mattera JA, Wang Y, Han LF, Ingber MJ, Roman S, Normand SL. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation. 2006; 113: 1693–1701.
18. Iezzoni LI. How much are we willing to pay for information about quality of care? Ann Intern Med. 1997; 126: 391–393.
19. Iezzoni LI, Ash AS, Shwartz M, Landon BE, Mackiernan YD. Predicting in-hospital deaths from coronary artery bypass graft surgery: do different severity measures give different predictions? Med Care. 1998; 36: 28–39.[CrossRef][Medline] [Order article via Infotrieve]
20. Dans PE. Looking for answers in all the wrong places. Ann Intern Med. 1993; 119: 855–857.
21. Torchiana DF, Meyer GS. Use of administrative data for clinical quality measurement. J Thorac Cardiovasc Surg. 2005; 129: 1223–1225.
22. Romano PS. Asking too much of administrative data? J Am Coll Surg. 2003; 196: 337–338.[CrossRef][Medline] [Order article via Infotrieve]
23. Romano PS, Chan BK, Schembri ME, Rainwater JA. Can administrative data be used to compare postoperative complication rates across hospitals? Med Care. 2002; 40: 856–867.[CrossRef][Medline] [Order article via Infotrieve]
24. Romano PS, Mark DH. Bias in the coding of hospital discharge data and its implications for quality assessment. Med Care. 1994; 32: 81–90.[Medline] [Order article via Infotrieve]
25. Best WR, Khuri SF, Phelan M, Hur K, Henderson WG, Demakis JG, Daley J. Identifying patient preoperative risk factors and postoperative adverse events in administrative databases: results from the Department of Veterans Affairs National Surgical Quality Improvement Program. J Am Coll Surg. 2002; 194: 257–266.[CrossRef][Medline] [Order article via Infotrieve]
26. Finlayson EV, Birkmeyer JD, Stukel TA, Siewers AE, Lucas FL, Wennberg DE. Adjusting surgical mortality rates for patient comorbidities: more harm than good? Surgery. 2002; 132: 787–794.[CrossRef][Medline] [Order article via Infotrieve]
27. Ghali WA, Rothwell DM, Quan H, Brant R, Tu JV. A Canadian comparison of data sources for coronary artery bypass surgery outcome "report cards." Am Heart J. 2000; 140: 402–408.[CrossRef][Medline] [Order article via Infotrieve]
28. Hartz AJ, Kuhn EM. Comparing hospitals that perform coronary artery bypass surgery: the effect of outcome measures and data sources. Am J Public Health. 1994; 84: 1609–1614.
29. Landon B, Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD. Judging hospitals by severity-adjusted mortality rates: the case of CABG surgery. Inquiry. 1996; 33: 155–166.[Medline] [Order article via Infotrieve]
30. Mack MJ, Herbert M, Prince S, Dewey TM, Magee MJ, Edgerton JR. Does reporting of coronary artery bypass grafting from administrative databases accurately reflect actual clinical outcomes? J Thorac Cardiovasc Surg. 2005; 129: 1309–1317.
31. Hannan EL, Racz MJ, Jollis JG, Peterson ED. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough? Health Serv Res. 1997; 31: 659–678.[Medline] [Order article via Infotrieve]
32. Green J, Wintfeld N. Report cards on cardiac surgeons: assessing New York State’s approach. N Engl J Med. 1995; 332: 1229–1232.
33. Osswald BR, Blackstone EH, Tochtermann U, Thomas G, Vahl CF, Hagl S. The meaning of early mortality after CABG. Eur J Cardiothorac Surg. 1999; 15: 401–407.
34. Jencks SF. Accuracy in recorded diagnoses. JAMA. 1992; 267: 2238–2239.
35. Hannan EL, Kilburn H Jr, Lindsey ML, Lewis R. Clinical versus administrative data bases for CABG surgery: does it matter? Med Care. 1992; 30: 892–907.[Medline] [Order article via Infotrieve]
36. McCarthy EP, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K, Phillips RS, Davies DT, Jr. Does clinical evidence support ICD-9-CM diagnosis coding of complications? Med Care. 2000; 38: 868–876.[CrossRef][Medline] [Order article via Infotrieve]
37. Romano PS, Roos LL, Luft HS, Jollis JG, Doliszny K. A comparison of administrative versus clinical data: coronary artery bypass surgery as an example: Ischemic Heart Disease Patient Outcomes Research Team. J Clin Epidemiol. 1994; 47: 249–260.[CrossRef][Medline] [Order article via Infotrieve]
38. Iezzoni LI, Foley SM, Daley J, Hughes J, Fisher ES, Heeren T. Comorbidities, complications, and coding bias: does the number of diagnosis codes matter in predicting in-hospital mortality? JAMA. 1992; 267: 2197–2203.
39. Tu JV, Sykora K, Naylor CD. Assessing the outcomes of coronary artery bypass graft surgery: how many risk factors are enough? Steering Committee of the Cardiac Care Network of Ontario. J Am Coll Cardiol. 1997; 30: 1317–1323.[Abstract]
40. Jones RH, Hannan EL, Hammermeister KE, Delong ER, O’Connor GT, Luepker RV, Parsonnet V, Pryor DB. Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery: the Working Group Panel on the Cooperative CABG Database Project. J Am Coll Cardiol. 1996; 28: 1478–1487.[Abstract]
41. Geraci JM, Johnson ML, Gordon HS, Petersen NJ, Shroyer AL, Grover FL, Wray NP. Mortality after cardiac bypass surgery: prediction from administrative versus clinical data. Med Care. 2005; 43: 149–158.[CrossRef][Medline] [Order article via Infotrieve]
42. Jollis JG, Ancukiewicz M, Delong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Ann Intern Med. 1993; 119: 844–850.
43. Hawker GA, Coyte PC, Wright JG, Paul JE, Bombardier C. Accuracy of administrative data for assessing outcomes after knee replacement surgery. J Clin Epidemiol. 1997; 50: 265–273.[CrossRef][Medline] [Order article via Infotrieve]
44. Powell H, Lim LL, Heller RF. Accuracy of administrative data to assess comorbidity in patients with heart disease: an Australian perspective. J Clin Epidemiol. 2001; 54: 687–693.[CrossRef][Medline] [Order article via Infotrieve]
45. Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher ES, Duncan C, Hughes JS, Coffman GA. Identifying complications of care using administrative data. Med Care. 1994; 32: 700–715.[Medline] [Order article via Infotrieve]
46. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992; 45: 613–619.[CrossRef][Medline] [Order article via Infotrieve]
47. Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol. 1993; 46: 1075–1079.[CrossRef][Medline] [Order article via Infotrieve]
48. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998; 36: 8–27.[CrossRef][Medline] [Order article via Infotrieve]
49. Glance LG, Dick AW, Osler TM, Mukamel DB. Does date stamping ICD-9-CM codes increase the value of clinical information in administrative data? Health Serv Res. 2006; 41: 231–251.[CrossRef][Medline] [Order article via Infotrieve]
50. Lawthers AG, McCarthy EP, Davis RB, Peterson LE, Palmer RH, Iezzoni LI. Identification of in-hospital complications from claims data: is it valid? Med Care. 2000; 38: 785–795.[CrossRef][Medline] [Order article via Infotrieve]
51. Quan H, Parsons GA, Ghali WA. Validity of information on comorbidity derived from ICD-9-CCM administrative data. Med Care. 2002; 40: 675–685.[CrossRef][Medline] [Order article via Infotrieve]
52. Parker JP, Li Z, Damberg CL, Danielsen B, Carlisle DM. Administrative versus clinical data for coronary artery bypass graft surgery report cards: the view from California. Med Care. 2006; 44: 687–695.[CrossRef][Medline] [Order article via Infotrieve]
53. Gordon HS, Johnson ML, Wray NP, Petersen NJ, Henderson WG, Khuri SF, Geraci JM. Mortality after noncardiac surgery: prediction from administrative versus clinical data. Med Care. 2005; 43: 159–167.[CrossRef][Medline] [Order article via Infotrieve]
54. Jollis JG, Romano PS. Pennsylvania’s Focus on Heart Attack: grading the scorecard. N Engl J Med. 1998; 338: 983–987.
Related Article:
This article has been cited by other articles:
CLINICAL PERSPECTIVE
Demand is increasing for public accountability in health care through the use of "report cards." Numerous states have implemented coronary artery bypass grafting surgery (CABG) report cards. Hospitals are concerned that only the most accurate data and statistical methodologies are used for such purposes because their reported outcomes may substantially affect referral practices. Ideally, public report cards should be based on prospectively collected, audited, and validated clinical data, using the most appropriate risk-adjustment models and statistical approaches. The present study was motivated by the observation that results from a Massachusetts public CABG report card based on these principles differed substantially from those derived from a Massachusetts administrative database. Administrative data are widely accessible at low cost, and analyses based on them are generally available more quickly than from audited clinical registries. We determined that much of the difference between the 2 Massachusetts CABG report cards was attributable to case misclassification, typically with some combined procedures categorized as isolated CABG. Because combined operations have higher risk, their inclusion inflates the apparent mortality rate of a hospital. High-volume tertiary centers may be particularly disadvantaged by procedure classification errors because of their larger proportion of combined procedures. About half of the misclassified cases could not have been correctly classified through the use of more sophisticated coding algorithms and would require case-by-case adjudication. Administrative data are valuable for population-based studies, for quality improvement activities, and for profiling some diseases. When possible, however, audited data registries are preferable for public profiling of CABG surgery outcomes.
Circulation 2007 115: 1503.
 |
|
 |
|
  D. L. Davenport, C. W. Holsapple, and J. Conigliaro Assessing Surgical Quality Using Administrative and Clinical Data Sets: A Direct Comparison of the University HealthSystem Consortium Clinical Database and the National Surgical Quality Improvement Program Data Set American Journal of Medical Quality, September 1, 2009; 24(5): 395 - 402. [Abstract] [PDF]
|
|
|
|
 |
|
 J. G. Modrall, E. B. Rosero, S. T. Smith, F. R. Arko, R. J. Valentine, G. P. Clagett, and C. H. Timaran Effect of Hospital Volume on In-Hospital Mortality for Renal Artery Bypass Vascular and Endovascular Surgery, August 1, 2009; 43(4): 339 - 345. [Abstract] [PDF]
|
|
|
|
 |
|
 J. P. Jacobs, R. J. Cerfolio, and R. M. Sade The Ethics of Transparency: Publication of Cardiothoracic Surgical Outcomes in the Lay Press. Ann. Thorac. Surg., March 1, 2009; 87(3): 679 - 686. [Full Text] [PDF]
|
|
|
|
|
|
K. F. Welke, B. S. Diggs, T. Karamlou, and R. M. Ungerleider Comparison of Pediatric Cardiac Surgical Mortality Rates From National Administrative Data to Contemporary Clinical Standards Ann. Thorac. Surg., January 1, 2009; 87(1): 216 - 223. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Garg, S.-L. T. Normand, T. S. Silbaugh, R. E. Wolf, K. Zelevinsky, A. Lovett, M. R. Varma, Z. Zhou, and L. Mauri Drug-Eluting or Bare-Metal Stenting in Patients With Diabetes Mellitus: Results From the Massachusetts Data Analysis Center Registry Circulation, November 25, 2008; 118(22): 2277 - 2285. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. F. Hernandez and S. M. O'Brien Sex Differences in Hospital Risk-Adjusted Mortality Rates for Medicare Beneficiaries Undergoing CABG Surgery--Invited Commentary Arch Intern Med, November 24, 2008; 168(21): 2323 - 2325. [Full Text] [PDF]
|
|
|
|
|
|
L. Mauri, T. S. Silbaugh, R. E. Wolf, K. Zelevinsky, A. Lovett, Z. Zhou, F. S. Resnic, and S.-L. T. Normand Long-Term Clinical Outcomes After Drug-Eluting and Bare-Metal Stenting in Massachusetts Circulation, October 28, 2008; 118(18): 1817 - 1827. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. C. Hammill and J. Curtis Publicly Reporting Implantable Cardioverter Defibrillator Outcomes: Grading the Report Card Circ Arrhythm Electrophysiol, October 1, 2008; 1(4): 235 - 237. [Full Text] [PDF]
|
|
|
|
|
|
T.R. Vogel, R.G. Symons, and D.R. Flum Longitudinal Outcomes After Endovascular Repair of Abdominal Aortic Aneurysms Vascular and Endovascular Surgery, October 1, 2008; 42(5): 412 - 419. [Abstract] [PDF]
|
|
|
|
 |
|
 L. Mauri, T. S. Silbaugh, P. Garg, R. E. Wolf, K. Zelevinsky, A. Lovett, M. R. Varma, Z. Zhou, and S.-L. T. Normand Drug-Eluting or Bare-Metal Stents for Acute Myocardial Infarction N. Engl. J. Med., September 25, 2008; 359(13): 1330 - 1342. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. Westaby, N. Archer, N. Manning, S. Adwani, C. Grebenik, O. Ormerod, R. Pillai, and N. Wilson Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality BMJ, October 13, 2007; 335(7623): 759 - 759. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. J. Ryan To Understand Cardiac Surgical Report Cards: Look Both Ways Circulation, March 27, 2007; 115(12): 1508 - 1510. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||