(Circulation. 1996;93:403-406.)
© 1996 American Heart Association, Inc.
Articles |
Operator-Specific Outcomes
A Call to Professional Responsibility
From the Department of Medicine, Duke University Medical Center, Durham, NC.
Correspondence to Robert M. Califf, MD, Professor of Medicine, Duke University Medical Center, 2024 W Main St, Durham, NC 27705.
Key Words: Editorials • clinical outcome • quality control
 |
Introduction |
|---|
 Top Introduction Model Accuracy Problems With Low-Volume...Gaming the Results Consistency of Results Patient Selection and Long-term...References |
|---|
One of the fundamental changes in the transformation of medical practice involves the ability of large purchasers of health care to select among competing providers for health care services. Lacking sufficient information about quality and outcomes, decisions concerning provider selection are driven primarily by price considerations. In the progression to a managed care–dominated system, prices may be driven down to the point where the quality of care is at risk. Sufficient information about healthcare quality will become critical in maintaining an effective healthcare system. The article by Ellis and colleagues1 in this issue of Circulation raises the important question of what outcomes should be measured to assess the quality of percutaneous revascularization and, more importantly, what the potential limitations are of these measures.
The recent release of operator-specific outcomes data has ushered in a new terminology known as "scorecard medicine."2 3 The use of the term "scorecard," although having some negative connotations, provides symbolic insight into the issues that must be addressed by the profession. For decades in this country, thousands of fans have carefully followed the statistics of their favorite baseball players (without the benefits of a postgraduate mathematics degree). Intuitive familiarity through years of observation has led the public to widespread sophistication about interpreting these statistics. Small differences in batting averages early in the year are not regarded as important. Hitters with few "at bats" are not considered to have stable estimates of their hitting capabilities; a hitter with 3 hits in his first 10 at bats is not considered to be a ".300 hitter" but rather a batter who needs more exposure to determine his capabilities. Likewise, the fan understands that the runs-batted-in statistic is dependent on the ability of preceding hitters to get on base so that the hitter has an opportunity to drive in runs. Batting averages are often stratified as a function of batting against right-handed and left-handed pitching because of the recognition that expected outcomes differ depending on the pitching. In many ways, the medical scorecard has similar characteristics; the problem is that the important modifying factors are complex and poorly understood and the methods of adjusting for differences are even less well understood. As this field of quantitative outcomes evolves from its current early stage of development, we believe that constant exposure to medical outcome data will eventually lead to intuitive understanding similar to that of the interested baseball fan.
Just as managers in other industries depend on information systems to provide support for policy decisions, managers of organized medical systems (managed care organizations) will do the same. Assessment of the value of strategies of medical care—including the value provided by individual practitioners—has become a reality; clinicians and researchers must accept this fact and move forward quickly to guide the development of efficient, fair, and effective methods of measurement.
Outcomes can be defined in terms of cost, patient satisfaction with the delivery of medical care, freedom from adverse medical events, and quality of life in its many dimensions. The ease of measuring and understanding outcomes is directly related to the order in which they are listed above. Financial systems now allow measurement of cost at a detailed level, and the methods of analyzing cost data are common to all businesses. Similarly, patient satisfaction surveys are fundamentally similar to customer satisfaction surveys in other industries. Adverse outcomes can range from highly objective (mortality end points) to highly subjective (infarction rates, procedural success), as pointed out by Ellis and colleagues.1 Finally, the measurement of quality of life is extremely complicated because of its multidimensional nature and difficulty with subjective interpretation.
Clinical outcomes, previously the sole interest of the medical profession as an intellectual research endeavor, have become an area critical to the understanding of the value and quality of medical care. A central tenet of medical practice is that cost or satisfaction cannot be properly interpreted unless the corresponding medical outcomes are taken into consideration. The complexity of outcome measurement and interpretation must be demonstrated to consumers and managers of health care without self-serving attempts to derail the process, and methods to make outcome measures practical must be developed.
Because of its substantial expense and critical impact on life and death, procedure-based cardiovascular medicine has become a pivotal testing ground for medical outcome measurement. Ellis and colleagues provided important insight into many of the possibilities and pitfalls of using risk-adjusted outcome measures to assess the performance of individual practitioners. By developing a regression analysis to understand the major factors responsible for outcomes in percutaneous intervention and applying the analysis to individual operator outcomes, they provided data that can be used to evaluate the methods and applications of "cardiovascular scorecards."
The basis for using statistical models to assess practitioner performance is straightforward. The observed results of procedures are dependent on a number of factors, most of which are unrelated to the skill of the individual practitioner. Most importantly, the severity of illness of the patient far outweighs the effect of treatment in almost all quantitative studies. A practitioner performing procedures on higher-risk patients will obviously have worse outcomes than practitioners selecting "easy" cases. Risk adjustment with regression models is a statistical method of "leveling the playing field" to account for baseline differences so that the influence of care can be more precisely quantified.
The simplified risk-adjustment model developed by Ellis and colleagues provides an important first step, although considerably more work needs to be done to provide the best statistical model to level the field for interventional cardiology. For example, given the low event rates in their database, considerably more than 5000 patients would be required to accurately identify more than four or five major independent predictors of outcome without risking "overfitting" the model4 and to ensure that risk factors increasing the odds of adverse outcomes by 1.5-fold will be detected.5 Continuous characteristics such as ejection fraction and age can be modeled much more effectively as continuous functions; for example, although patients with an ejection fraction of 15% are obviously quite different in terms of risk than are patients with an ejection fraction of 39%, both would be assigned the same weight in the model developed by Ellis and colleagues. As large clinical databases are accumulated, without doubt the common risk factors for poor outcomes with medical and surgical treatment of coronary disease will emerge as prognostic indicators for percutaneous intervention. Rather than three-vessel disease or proximal left anterior descending coronary artery disease being critical, a continuum of risk almost certainly exists with more complex and diffuse lesions.6 7 Comorbid illnesses such as diabetes and peripheral vascular disease will also likely be identified as important predictors of outcomes after coronary intervention.8 9
Beyond developing regression models, Ellis and colleagues point out many of the potential pitfalls facing scorecarding; these problems include considerable uncertainty remaining after the factors in the models are considered, difficulty in making inferences about individual low-volume operators, subjectivity of outcome measures and baseline characteristics with corresponding ability to "game" the system, and lack of reliability in identifying outliers. To this list we add the lack of ability to define the success of an operator not only by the acute outcomes observed but also based on an understanding of the overall effect on the long-term outcomes of the population served.10
|
Model Accuracy |
|---|
|
TopIntroductionModel Accuracy Problems With Low-Volume...Gaming the Results Consistency of Results Patient Selection and Long-term...References |
|---|
The models developed by Ellis and colleagues, for the reasons enumerated above, may be improved with more outcome events, more high-quality data, and more sophisticated statistical modeling. Nevertheless, the current mortality model and combined end point (death, Q-wave infarction, or bypass surgery) were used to accurately discriminate those who are likely to have an event from those who are not in 85% and 77% of cases, respectively. Although reported statistical measures of variance explained by the model were low, the reader must be cautioned in interpreting these results as they may be affected by the frequency of events and may not reflect a model's predictive accuracy when binary end points are used.11 12 The most sophisticated predictive models in other disease areas provide receiver operator characteristic areas of >0.90 and measures of variance similar to those described by Ellis and colleagues. Even under ideal circumstances, statistical models cannot be expected to explain all uncertainty in clinical outcome, in part because of the play of chance and in part because many biological and procedure-related factors affecting outcome have not been identified. Despite the imperfections of statistical models, such models have been shown to be equal to or better than expert clinicians in predicting patient outcomes.13 14
|
Problems With Low-Volume Operators |
|---|
|
TopIntroductionModel Accuracy Problems With Low-Volume...Gaming the Results Consistency of Results Patient Selection and Long-term...References |
|---|
The ability of outcomes measures to reliably determine the quality of low-volume compared with high-volume operators is an important demonstration of a phenomenon equivalent to the "type II" statistical error.15 Absence of proof of a difference (outlier status) is not the same as proof of no difference (equivalence) in outcomes. The policy decisions made about the statistical interpretation of outcome data from low-volume operators are far reaching. On the one hand, an argument could be made that the consequences of failing to detect a few inadequate low-volume operators affects far fewer patients' lives than failing to detect a single inadequate high-volume operator. On the other hand, the consequences of failing to understand the "type II" error could be substantial, given increasing evidence that low-volume operators and low-volume institutions on average have inferior outcomes compared with high-volume operators and high-volume institutions16 17 18 and given the knowledge that hundreds of low-volume operators continue to practice in the United States. Approaches to this problem could include following the results of low-volume operators over time, thus accumulating sufficient sample size, requiring minimal procedure volumes based on empirical observations, and following process of care measures with interventional procedures such as the appropriateness of cases selected.19
|
Gaming the Results |
|---|
|
TopIntroductionModel Accuracy Problems With Low-Volume...Gaming the Results Consistency of Results Patient Selection and Long-term...References |
|---|
The subjectivity of both the baseline measures used to adjust for severity of illness and the clinical outcomes is a reason for concern. The steady decline in the risk-adjusted mortality for bypass surgery in New York State20 has sparked a vigorous debate about whether the mandatory database and public reports on results were responsible. Supporters of the system argue that the better risk-adjusted outcomes result from improvement in practice related to knowledge of outcomes, whereas detractors point to evidence of "coding creep," leading to overestimation of baseline risk in less ill patients by clinicians who have learned the system.21 The approach advocated by Ellis and colleagues is to rely on objective baseline characteristics and outcomes that cannot be debated. Although this approach has substantial appeal because of its simplicity, it misses the opportunity to provide important and worthwhile feedback about critical outcomes such as non–Q-wave myocardial infarction22 to operators who are interested in the impact of their practices on the well-being of their patients. Data auditing, or independent confirmation of self-reported results, remains another option for improving the quality of scorecarding systems.
|
Consistency of Results |
|---|
|
TopIntroductionModel Accuracy Problems With Low-Volume...Gaming the Results Consistency of Results Patient Selection and Long-term...References |
|---|
The lack of reliability of statistical analyses to identify outliers when different end points are evaluated deserves serious consideration. As defined by Ellis and colleagues, an outlier was an operator with results exceeding the 95% CI for the average outcome observed with all other operators. With this definition, it is not surprising that the
values are low; individuals on the border
of statistical significance with one outcome would understandably fall
within the 95% CI with another measure. These perturbations based on
marginal statistical differences should prompt caution on the part of
those who advise public disclosure of outcome information based solely
on "outlier" status. A more sensible method of displaying the
information would be to provide a rank-ordering of
operator-adjusted outcomes, identifying a range of providers with
results within the bounds of good practice and results deserving
commendation. Public disclosure of true-negative outliers should
require both definitive statistical evidence and a peer review. For
example, previous studies have found that although providers designated
as "outliers" can vary significantly depending on the
risk-adjustment methods used, overall hospital performance
rankings relative to their peers were remarkably
consistent.23 The fixation on "bad apple" identification is a component of scorecarding that is destructive to the practice of continuous quality improvement, as enunciated by Deming.24 This approach provides sensational headlines but drives the profession into a defensive posture rather than a positive engagement in the use of data to improve practice. Although we agree that operators with persistently poor results should undergo further scrutiny, we urge a shift in the fundamental focus of scorecarding. Variation in risk-adjusted outcome should serve as a starting point for which more detailed examination of care processes can be initiated. In this manner, the play of chance can be considered and the focus becomes a genuine search for the practices that lead to the best outcomes. After experiences in the northeast United States, physicians are wary of the potential impact of public release of scorecard data without sufficiently balanced presentation of the data. Because of the uncertainties outlined above, we strongly believe that public dissemination of poor outcome data should follow a peer review of practice. To the extent that such data collection systems can be supported and maintained by healthcare providers rather than government entities, they will not be subject to premature public release.
|
Patient Selection and Long-term Benefits |
|---|
|
TopIntroductionModel Accuracy Problems With Low-Volume...Gaming the Results Consistency of Results Patient Selection and Long-term...References |
|---|
In addition to the elements of proper risk adjustment noted by Ellis and colleagues, other important measures of quality would involve starting the data collection process at the point of referral for revascularization rather than at the point where angioplasty is performed. Such a system would monitor high-risk patients who may be turned down for procedures, the same patients who may derive the greatest survival benefit from revascularization. A second important measure of quality would be to follow long-term outcomes. As emphasized by the results of recent randomized trials,25 the implications of revascularization procedures cannot be appreciated without long-term follow-up. In particular, with percutaneous revascularization, the implications of patient selection and completeness of revascularization can be appreciated only when the likelihood of need for subsequent revascularization is considered. An operator with more appropriate patient selection may have more acute adverse outcomes but better long-term outcomes compared with the expected results with medical therapy. Thus, the concepts of "doing the right things" and "doing things right" must be linked.
Returning to the scorecard analogy, we believe that public and professional interests will be best served if physicians "play ball" rather than "strike" against systems designed to assess quality and outcome. Despite all of the difficulties and uncertainties, the quest for quantitative evidence of quality in medical care will continue to gain momentum. We believe that systematic measurement of baseline risk and clinical outcome can provide a powerful tool for well-meaning operators to develop an understanding of how to practice better medicine. Patients and their families entrust their future with us when they choose to undergo percutaneous revascularization procedures. A voluntary engagement of the cardiovascular community could rapidly lead to a concise, well-defined set of data reflecting severity of illness, procedural methodology, and outcomes. We have a responsibility to improve our practice to produce the best outcomes possible. If cardiovascular practitioners do not heed the call to gather outcome data and improve practice through its use, nonpractitioners will impose policies and select providers without adequately considering the quality of care provided; our patients deserve our involvement.
|
Footnotes |
|---|
The opinions expressed in this editorial are not necessarily those of the editors or of the American Heart Association.
|
References |
|---|
|
TopIntroductionModel Accuracy Problems With Low-Volume...Gaming the Results Consistency of Results Patient Selection and Long-term...References |
|---|
1. Ellis SG, Omoigui N, Bittl JA, Lincoff AM, Wolfe M, Howell G, Topol EJ. Analysis and comparison of operator-specific outcomes in interventional cardiology: from a multicenter database of 4860 quality-controlled procedures. Circulation. 1996;93:431-439.
2. Zimman D. Heart surgeons rated: state reveals patient-mortality records. Newsday. December 18, 1991:3.
3.
Topol EJ, Califf RM. Scorecard
cardiovascular medicine: its impact and future
directions. Ann Intern Med. 1994;120:65-70.
4.
Harrell FE, Lee KL, Pollock BG. Regression
models in clinical studies: determining relationships between
predictors and response. J Natl Cancer
Inst. 1988;80:1198-1202.
5. Tenaglia AN, Fortin DF, Califf RM, Frid DJ, Nelson CL, Gardner L, Miller M, Navetta FI, Smith JE, Tcheng JE, Stack RS. Predicting the risk of abrupt vessel closure after angioplasty in an individual patient. J Am Coll Cardiol. 1994;24:1004-1011. [Abstract]
6. Jones RH. Complex angioplasty: a surgeon's perspective. Am J Cardiol. 1992;69:22F-24F. [Medline] [Order article via Infotrieve]
7.
Mark DB, Nelson CL, Califf RM, Harrell FE Jr, Lee KL,
Jones RH, Fortin DF, Stack RS, Glower DD, Smith LR. Continuing
evolution of therapy for coronary artery disease: initial
results from the era of coronary angioplasty.
Circulation. 1994;89:2015-2025.
8. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis. 1987;40:373-383. [Medline] [Order article via Infotrieve]
9.
O'Connor GT, Plume SK, Olmstead EM, Coffin LH, Morton
JR, Maloney CT, Nowicki ER, Tryzelaar JF, Hernandez F, Adrian L.
A regional prospective study of in-hospital mortality
associated with coronary artery bypass grafting: the Northern
New England Cardiovascular Disease Study Group.
JAMA. 1991;266:803-809.
10. Pryor DB, DeLong ER. Programmed outcome research teams (PORTs) and implications for clinical practice. Am J Cardiol. 1994;73:34B-38B. [Medline] [Order article via Infotrieve]
11. Cox DR, Wermuth N. A comment on the coefficient of determination for binary responses. Am Stat. 1992;46:1-4.
12. Iezzoni LI, ed. Risk Adjustment for Measuring Health Care Outcomes. Ann Arbor, Mich: Health Administration Press; 1994.
13. Pryor DB, Shaw L, Harrell FE Jr, Lee KL, Hlatky MA, Mark DB, Muhlbaier LH, Califf RM. Estimating the likelihood of severe coronary artery disease. Am J Med. 1991;90:553-562. [Medline] [Order article via Infotrieve]
14.
Knaus WA, Harrell FE Jr, Lynn J, Goldman L, Phillips
RS, Connors AF Jr, Dawson NV, Fulkerson WJ Jr, Califf RM, Desbiens N.
The SUPPORT prognostic model: objective estimates of survival
for seriously ill hospitalized adults: study to understand prognoses
and preferences for outcomes and risks of treatments. Ann
Intern Med. 1995;122:191-203.
15. Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The importance of beta, the type II error, and sample size in the design and interpretation of the randomized controlled trial. N Engl J Med. 1978;299:690-694. [Abstract]
16.
Kimmel SE, Berlin JA, Laskey WK. The
relationship between coronary angioplasty procedure volume and
major complications. JAMA. 1995;274:1137-1142.
17.
Jollis JG, Peterson ED, DeLong ER, Mark DB, Collins SR,
Muhlbaier LH, Pryor DB. The relationship between hospital volume
of coronary angioplasty and short term mortality in patients
over age 65 in the United States. N Engl J
Med. 1994;331:1625-1629.
18.
Ritchie JL, Phillips KA, Luft HS.
Coronary angioplasty: statewide experience in
California. Circulation. 1993;88:2735-2743.
19. Ryan TJ, Bauman WB, Kennedy JW, Kereiakes DJ, King SB III, McCallister BD, Smith SC Jr, Ullyot DJ. Guidelines for percutaneous transluminal coronary angioplasty: a report of the American College of Cardiology/American Heart Association Task Force on assessment of diagnostic and therapeutic cardiovascular procedures. J Am Coll Cardiol. 1993;22:2033-2054. [Medline] [Order article via Infotrieve]
20.
Hannan EL, Siu AL, Kumar D, Kilburn H Jr,
Chassin MR. The decline in coronary artery bypass graft
surgery mortality in New York State: the role of surgeon
volume. JAMA. 1995;273:209-213.
21.
Green J, Wintfeld N. Report cards on
cardiac surgeons: assessing New York State's approach.
N Engl J Med. 1995;332:1229-1232.
22. Harrington RA, Lincoff AM, Califf RM, Holmes DR Jr, Berdan LG, O'Hanesian MA, Keeler GP, Garratt KN, Ohman EM, Mark DB. Characteristics and consequences of myocardial infarction after percutaneous coronary intervention: insights from the Coronary Angioplasty Versus Excisional Atherectomy Trial (CAVEAT). J Am Coll Cardiol. 1995;25:1693-1699. [Abstract]
23. Peterson ED, Muhlbaier LH, DeLong ER, Rosen AB, Fortin DF, Ellerbeck EF, Jencks SF, Pryor DB. Are provider profiles affected by risk-adjustment methodology? Results from the Cooperative Cardiovascular Project. J Am Coll Cardiol. 1995;(suppl):98A.
24. Deming WE. Out of the Crisis. Cambridge, Mass: Massachusetts Institute of Technology Press; 1986.
25. Yusuf S, Zucker D, Peduzzi P, Fisher LD, Takaro T, Kennedy JW, Davis K, Killip T, Passamani E, Norris R. Effect of coronary artery bypass graft surgery on survival: overview of 10-year results from randomised trials by the Coronary Artery Bypass Graft Surgery Trialists Collaboration. Lancet. 1994;344:563-570.[Medline] [Order article via Infotrieve]
This article has been cited by other articles:
 |
|
 |
|
  R. M. Califf and E. D. Peterson Public reporting of quality measures what are we trying to accomplish? J. Am. Coll. Cardiol., March 10, 2009; 53(10): 831 - 833. [Full Text] [PDF]
|
|
|
|
 |
|
 D. M. Shahian, S.-L. Normand, D. F. Torchiana, S. M. Lewis, J. O. Pastore, R. E. Kuntz, and P. I. Dreyer Cardiac surgery report cards: comprehensive review and statistical critique Ann. Thorac. Surg., December 1, 2001; 72(6): 2155 - 2168. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. P. Brown, M. J. Mack, A. W. Simon, S. L. Battaglia, L. G. Tarkington, S. D. Culler, and E. R. Becker Comparing clinical outcomes in high-volume and low-volume off-pump coronary bypass operation programs Ann. Thorac. Surg., September 1, 2001; 72(3): S1009 - 1015. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. D. McGrath, D. E. Wennberg, J. D. Dickens Jr, A. E. Siewers, F. L. Lucas, D. J. Malenka, M. A. Kellett Jr, and T. J. Ryan Jr Relation Between Operator and Hospital Volume and Outcomes Following Percutaneous Coronary Interventions in the Era of the Coronary Stent JAMA, December 27, 2000; 284(24): 3139 - 3144. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. D. Peterson, E. R. DeLong, L. H. Muhlbaier, A. B. Rosen, H. E. Buell, C. I. Kiefe, and T. F. Kresowik Challenges in comparing risk-adjusted bypass surgery mortality results: Results from the Cooperative Cardiovascular Project J. Am. Coll. Cardiol., December 1, 2000; 36(7): 2174 - 2184. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. J. Kereiakes, S. C. Smith, A. K. Jacobs, M. J. Kern, and D. P. Faxon Angioplasty for acute myocardial infarction in community hospital without surgical back-up: Response to Wharton and Angelini publications "should guidelines be changed?: Not whether but when" J. Am. Coll. Cardiol., July 1, 2000; 36(1): 299 - 300. [Full Text] [PDF]
|
|
|
|
|
|
S. G. Ellis, D. Miller, T. F. Keys, K. Brown, R. Ellert, G. Howell, A. M. Lincoff, and E. J. Topol Comparing physician-specific two-year patient outcomes after coronary angiography: Methodologic issues and results J. Am. Coll. Cardiol., April 1, 1999; 33(5): 1278 - 1285. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Kastrati, F.-J. Neumann, and A. Schomig Operator volume and outcome of patients undergoing coronary stent placement J. Am. Coll. Cardiol., October 1, 1998; 32(4): 970 - 976. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. G. Jollis, E. D. Peterson, C. L. Nelson, J. A. Stafford, E. R. DeLong, L. H. Muhlbaier, and D. B. Mark Relationship Between Physician and Hospital Coronary Angioplasty Volume and Outcome in Elderly Patients Circulation, June 3, 1997; 95(11): 2485 - 2491. [Abstract] [Full Text]
|
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||