This Article Abstract Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Ellis, S. G. Articles by Topol, E. J. Search for Related Content PubMed PubMed Citation Articles by Ellis, S. G. Articles by Topol, E. J.

(Circulation. 1996;93:431-439.)
© 1996 American Heart Association, Inc.

Articles

Analysis and Comparison of Operator-Specific Outcomes in Interventional Cardiology

From a Multicenter Database of 4860 Quality-Controlled Procedures

Stephen G. Ellis, MD; Nowamagbe Omoigui, MD; John A. Bittl, MD; Michael Lincoff, MD; Mark W. Wolfe, MD; Georgiana Howell; Eric J. Topol, MD

From the Department of Cardiology (S.G.E., N.O., M.L., G.H., E.J.T.), the Cleveland Clinic Foundation, Cleveland, Ohio; and Brigham and Women's Hospital (J.A.B., M.W.W.), Harvard Medical School, Boston, Mass.

Correspondence to Stephen G. Ellis, MD, the Cleveland Clinic Foundation, 9500 Euclid Ave, F-25, Cleveland, OH 44195.

	Abstract

Top Abstract Introduction Methods Results Discussion Appendix References

 
Background Medical consumers are increasingly requesting methods to discriminate among the results of different providers. Standards for appropriate modeling, risk adjustment, and evaluation ("scorecarding") in this setting are not well developed, although such evaluation is being performed by the medical insurance industry and by several states in the United States. Our objectives were to develop and examine clinically meaningful methodology for assessing the operator-specific results for percutaneous coronary revascularization.

Methods and Results From a multicenter database of patients treated since January 1, 1990, we used training and validation samples (n=4860) to develop several models for risk adjustment and applied them to 38 providers performing 25 to 523 procedures in the database. Models were developed using multivariable logistic regression techniques for combinations of the end points of death, myocardial infarction, bypass surgery, and procedural success. Models were evaluated for predictive accuracy by using receiver operating characteristic (ROC) analysis, for the capacity to discriminate between superior and inferior provider outcomes, and for subjectivity and concordance. Major complications occurred in 3.6% of patients. The area under the ROC curve (with perfect discriminatory accuracy, area=1.0; with no apparent accuracy, area=0.5) in the validation sample, and frequency of identification of operators with outcomes outside the 95% CI for the outcome in question for the models were for death, 0.85 and 7.9%; for death, Q-wave infarction, and bypass surgery, 0.77 and 13.2%; for death, all infarction, and bypass surgery, 0.66 and 10.5%; and for procedural success, 0.76 and 23.7%. For the models as a group, identification of outliers was inversely related to provider volume (P=.05). Models evaluating non–Q-wave infarction or requiring measurement of percent diameter stenosis were identified as being most susceptible to provider manipulation.

Conclusions For percutaneous coronary revascularization, modeling to discriminate between provider outcomes is limited by the low incidence of major adverse events, subjectivity or susceptibility to manipulation of more frequently occurring adverse events, the generally modest predictive capacity of the models, and the low volume of individual provider treatments. Modeling will be most useful in the identification of providers with extremely poor outcomes and for discrimination between providers with very large procedural volume. Until improved understanding of the biological and mechanical correlates of major complications allows the development of more predictive models, interpretation of the results of scorecarding, particularly for low-volume providers, should be made with caution.

Key Words: angioplasty • myocardial infarction • statistics

	Introduction

Top Abstract Introduction Methods Results Discussion Appendix References

 
It has been known for some time that the results of some medical procedures, such as coronary bypass surgery, vary from surgeon to surgeon and hospital to hospital.¹ Increasingly, consumers of healthcare have become aware of this phenomenon and are requesting access to the outcome data of individual providers. Operator-specific mortality rates for bypass surgery have been made public in some states since 1991.²

Especially for procedures with serious complications, comparison of appropriately risk-adjusted complication rates between providers may be beneficial for a number of reasons. First, patients can be triaged to providers of high-quality care. Second, providers of lesser quality care may be identified and allowed to acquire further training, modify their technique, or refer high-risk patients elsewhere. Third, there is no disincentive to attempt to treat the most complex and high-risk patients, who often stand the most to gain from intervention.^{3 4} However, currently available data sets may frequently have the responsible physician coded incorrectly,⁵ may not be designed to record variables important in modifying and therefore adjusting risk,⁵ may have inadequate numbers of treatments for individual physicians to be able to discriminate results⁶ (Fig 1), and therefore, when applied to "scorecarding," may discourage physicians from accepting either new and potentially improved treatments or high-risk patients.⁷

View larger version (10K): [in this window] [in a new window]   Figure 1. Relation of statistical certainty of estimated risk to number of procedures analyzed showing 95% CIs for observed risk of 1.0%, 5.0%, and 10.0% based on known procedural volume (see "Appendix"). Providers with the lowest volumes have the widest CIs.

The results of coronary balloon angioplasty also vary by operator⁸ and by hospital.^{9 10} Both government¹¹ and the insurance industry^{12 13} have begun to develop models to risk adjust for the outcomes of many procedures, including angioplasty, but standards for modeling and their appropriate application for comparison of provider results (scorecarding) in this setting have not been developed. Therefore, we sought to assess the strengths and limitations of several models that could be developed to compare the results of percutaneous intervention of different physicians or physician groups.

	Methods

Top Abstract Introduction Methods Results Discussion Appendix References

 
Patients and Studies
Data on 4860 treated patients (consecutively treated [Cleveland Clinic Interventional Database] or enrolled in several prospective trials of percutaneous coronary intervention^{14 15 16 17 18} meeting the criteria listed) were obtained. Criteria for study inclusion were treatment on or after January 1, 1990; treatment for stable or unstable angina, silent ischemia, or acute MI; documentation of baseline data and procedural complications by independent audit (not the physician providers themselves); routine postprocedure measurement of creatine kinase; and measurement of the angiographic result by either computer edge-detection¹⁹ or calipers.²⁰ No patients were excluded due to missing data or for other reasons.

Variables and Definitions
The following preprocedural variables (available from all studies and in 95% of patients, except where noted) were collected in a relational database²¹ —baseline demographic: acute MI (within 24 hours of onset), age, cardiogenic shock, congestive heart failure class (New York Heart Association) (available in 80%), sex, prior bypass surgery, prior MI (available in 82%), prior percutaneous treatment at site to be treated, and unstable angina; and baseline angiographic: LVEF (available in 60%), most complex lesion with attempted treatment (modified American College of Cardiology/American Heart Association lesion classification¹⁶ ), number of diseased (50% diameter stenosis) vessels, and vessel(s) treated. LVEF frequently was not available for patients who died. To avoid excluding patients without known LVEF from the mortality analysis, LVEF was imputed according to the formula: LVEF=65 minus 7.8 (heart failure class) minus 10.1 (prior MI) for that analysis only. This formula was derived from multivariable analysis of the end point LVEF observed in the training sample (adjusted multiple r²=.54, P<.001). Additional variables included treatment (balloon angioplasty, directional atherectomy, excimer laser [xenon chloride], extraction atherectomy, infrared laser [holmium], primary stent implantation, and rotational atherectomy).

Outcome variables collected were coronary bypass surgery (at any time during the hospitalization), death, MI (CK at least twofold to threefold upper limit of normal, depending on study standards), procedural success (final stenosis <50% and no death, MI, or bypass surgery), and Q-wave MI.

Additional variables available only from some studies were also tabulated: baseline heart rate, evaluation as not a candidate for bypass surgery, prior cerebrovascular accident or transient ischemic event, recent MI (within 2 weeks), renal insufficiency (creatinine 2.0 mg/dL), symptomatic peripheral vascular disease, and symptomatic pulmonary disease.

Statistical Analysis
All data are presented as mean±1 SD unless otherwise indicated. Patients were arbitrarily divided into a training sample (from the six studies available when the analysis was begun) and a test sample (from the two studies that became available soon afterward). Nineteen baseline demographic and angiographic variables were analyzed as potential covariates of the clinical end points. Treatment variables (eg, use of balloon angioplasty rather than directional atherectomy) were not included in this analysis because they involved choice by the operator. ² and Student's t tests were used to assess univariate relation of categorical and continuous variables, respectively, with the various end points. Statistical modeling in the training set was done using logistic regression analyses,²² testing candidate variables with univariate P.05, with conversion to risk-adjusted odds ratios and 95% CIs. When >10% of data were missing for a variable with multivariate P.05, the regression analyses were evaluated both with and without that variable included. Candidate end points for the models were death; death, Q-wave MI, or bypass surgery; death, all MI, or bypass surgery; and procedural success. Simplification of the models for use in this study to allow easy application by individual physicians was done by dividing the odds ratio for each significant covariate by that of the smallest significant odds ratio of more than 1 and rounding off to the nearest integer so as to derive a "point score" for each variable. For odds ratios <1, their inverse was handled in the same manner and the point score assigned was negative.

The primary measure of the models' predictive accuracy was ROC analysis (see "Appendix"), but McFadden's ² statistic was also calculated.²³ Possible covariates and end points for models were evaluated for subjectivity and possible susceptibility to manipulation (eg, not measuring CK levels after a successfully treated abrupt closure so as not to diagnose a "small" non–Q-wave MI) by a panel of seven cardiologists on a 1-to-5 scale (1, not subjective or susceptible to manipulation; 5, subjective and most subjective to manipulation). Each cardiologist had previously coded these variables for 300 procedures and was aware of possible subjectivity in their definition and of pressures from cardiology staff attending physicians to make their results "look good." Models were also evaluated for their capacity to identify provider outliers in relation to operator volume and for concordance with the results of other models using Cohen's analyses. The results of these analyses are graphically depicted (see Fig 7) for all 19 providers with procedural volume >75 and for an equal number of randomly selected operators with volume of 25 to 74 procedures.

View larger version (39K): [in this window] [in a new window]   Figure 7. Point estimates and 95% CIs for absolute and risk-adjusted incidences of the end point in-hospital death, Q-wave MI, or bypass surgery, listed by provider. A, For providers with 75 cases (actual number of cases is listed in parentheses next to the code letter of the provider). B, For providers with 25 to 74 cases.

After the development of the four models, each individual operator was entered into the multivariable analysis as a covariate possibly altering the outcome of the patient he or she treated (see "Appendix"). Providers with "superior" outcome were defined as those with results exceeding the 95% CI for outcome, whereas those with "inferior" outcome were defined as those with results worse than the 95% CI for all other providers.

	Results

Top Abstract Introduction Methods Results Discussion Appendix References

 
Patient Population
Baseline demographic and treatment information for the training and validation samples are given in Table 1. Adverse outcomes were distributed among the training and validation samples as follows: death (34 [0.9%], 22 [2.0%]); death, Q-wave MI, and bypass surgery (150 [4.0%], 33 [3.1%]); death, all MI, and bypass surgery (256 [6.8%], 87 [8.1%]); and procedural failure (499 [13.2%], 117 [10.9%]). Although the two groups were similar in many ways, significant (P<.05) differences exist with respect to the incidences of some variables, reflecting not unexpected differences in practice patterns between centers and studies, as well the large number of patients studied. The 4860 procedures were performed by 66 providers (case volume range, 2 to 523).

View this table: [in this window] [in a new window]   Table 1. Baseline Demographic, Treatment, and Outcome Characteristics

Reliability of Variable Coding
The results of the physician panel assessment of the susceptibility of candidate covariates and end points to manipulation are shown in Fig 2. Some candidate variables, such as death and Q-wave MI, were judged to be objective and reliably determined. Conversely, the variables of unstable angina, non–Q-wave MI, stenosis morphology, symptomatic obstructive pulmonary disease, procedural success, and class IV angina were judged to be especially unreliable (score 3.5) due to either interobserver variability or the potential for manipulation.

View larger version (12K): [in this window] [in a new window]   Figure 2. Point estimates and 95% CIs for physician estimates of subjectivity and susceptibility of selected potential covariates and end points to manipulation (5, most susceptible; 1, least susceptible).

Modeling Candidate End Points
The variables correlated with the four candidate end points, their contribution to potential models, and the predictive accuracy of the models in the validation sample (area under ROC curve and McFadden's ²) are shown in Tables 2 through 5. The incidences of adverse outcome by model result as applied in the training and validation samples are shown in Figs 3 through 6.

View this table: [in this window] [in a new window]   Table 2. Multivariate Correlates of In-Hospital Death1

View this table: [in this window] [in a new window]   Table 3. Multivariate Correlates of Death, Q-Wave MI, or Bypass Surgery1

View this table: [in this window] [in a new window]   Table 4. Multivariate Correlates of Death, MI, or Bypass Surgery1

View this table: [in this window] [in a new window]   Table 5. Multivariate Correlates of Procedural Failure1

View larger version (16K): [in this window] [in a new window]   Figure 3. Actual risk and predictive accuracy of in-hospital death based on point score from model for death (Table 2) as applied in the training and validation samples.

View larger version (16K): [in this window] [in a new window]   Figure 4. Actual risk and predictive accuracy of in-hospital death, Q-wave MI, or bypass surgery based on point score from derived model (Table 3) as applied in the training and validation samples.

View larger version (17K): [in this window] [in a new window]   Figure 5. Actual risk and predictive accuracy of in-hospital death, MI, or bypass surgery based on the point score from derived model (Table 4) as applied in the training and validation samples.

View larger version (23K): [in this window] [in a new window]   Figure 6. Actual risk and predictive accuracy of procedural success based on the point score from derived model (Table 5) as applied in the training and validation samples.

In the training sample, the candidate end point of death was independently correlated with acute MI within 24 hours before the coronary intervention, age, ejection fraction, three-vessel disease, lesion morphology, renal function, vessel site treated, and presence of cardiogenic shock. The end point of death, Q-wave MI, or bypass surgery was correlated with ejection fraction, lesion morphology, presentation with acute MI, and shock and was inversely related to prior bypass surgery and restenosis. Data for the end points death, all MI, or bypass surgery and for procedural success are presented in Tables 4 and 5.

The model for death had the best predictive accuracy (area under ROC curve, 0.85), whereas the other models were less predictive. The estimated variance in the outcome data explained by the models was low, however (McFadden's ²=.05 to .25).

Performance of the Models in Adjusting for Risk and Identifying Outliers
Absolute and risk-adjusted 95% confidence limits for the likelihood of death, Q-wave MI, or bypass surgery for patients treated by 38 of the providers are shown in Fig 7. The variation and uncertainty of estimates of provider outcome, the influence of risk adjustment, and the capacity of the model to identify outliers are illustrated. Providers are classified by the number of procedures in the database (<75 or 75) to illustrate the differences in the models' capacity to discriminate results for both low- and moderately high-volume providers. Similar analyses were done for the other three candidate models (data not shown). After risk adjustment and inclusion of data from all four models, 5 of 152 provider evaluations (3.3%) were "superior" and 16 of 152 (10.5%) were "inferior" (P<.001). Risk adjustment changed the categorization (superior, average, inferior) in 7 of 152 evaluations (4.6%), most frequently for the models of procedural success, and death, Q-wave MI, and bypass surgery (both 7.9%, respectively) and only infrequently for the models of death, and death, MI, and bypass surgery (2.6% and 0.0%, respectively). The models' capacity to identify outliers was inversely related to the number of procedures each operator contributed for analysis (P=.05). For the end point with the lowest incidence (death), only three providers (7.7%) could be identified as having results significantly different than the average, and none of these had a procedural volume of fewer than 167 cases. For the models of more frequently occurring events, 10% to 24% of providers could be identified as having significantly different outcomes. Only one provider (2.8%) with a volume of fewer than 57 cases was identified as an "outlier" by any of the four models.

Concordance Among Models in the Identification of Outlier Providers
The values for concordance between models in identifying outliers are displayed in Table 6. Values are generally low, either in the fair-to-good range (.40 to .79) or, more commonly, in the poor range (<.40), indicating that the choice of outcome evaluated often determines which providers are identified in the superior or inferior outlier groups. To put these values in perspective, for the 14 operators who were identified as being an outlier in any of the four models, no operator was identified by all models, 1 operator was identified by three models, 4 operators were identified by two models, and 9 operators were identified by one model.

View this table: [in this window] [in a new window]   Table 6. Agreement in Identification of Provider Outliers1

	Discussion

Top Abstract Introduction Methods Results Discussion Appendix References

 
It is estimated that 410 000 percutaneous coronary revascularization procedures were performed in the United States in 1994,²⁴ and the "average" physician performing this type of procedure did only 20 to 25 (Society for Cardiac Angiography and Intervention, unpublished data, 1994). Complications have been shown to be inversely related to both hospital and physician experience with these techniques.^{8 9 10}

This analysis is the first to evaluate the suitability of various models to assess differences in provider outcomes for this commonly performed group of procedures, even though provider scorecards for these techniques are being used and will soon be published (unpublished data). We have made five major observations.

First, many of the possible covariates and end points were believed by physicians who perform these procedures to be either quite subjective, possibly susceptible to manipulation, or both (Fig 2). For example, the identification of the end point "non–Q-wave MI" often relies on determination of enzyme markers of myocardial damage, most often, CK. Elevations in this enzyme may be brief²⁵ and can easily be missed if assays are not performed at routine and frequent intervals. Guidelines for CK measurement in this setting are not standardized, and a physician desiring to "look good" might simply choose not to test for CK elevation. Determination of procedural success typically requires assessment of the posttreatment dimensional result (percent diameter stenosis) of the treated lesion. This is most often done visually, a method of assessment known to be highly subjective.²⁶ The result may also be measured by any of several techniques,^{19 27} but this is laborious and the operator might choose simply not to record the angiographic projection in which the lesion appears worst. The end point "emergency surgery" is also difficult: Should evidence of ischemia be required to meet the definition, and if so, which ones? Should a patient referred urgently for bypass surgery, for whom a balloon pump or stent temporarily alleviates ischemia yet is considered so unstable that the surgeon limits the number or type of grafts placed, be considered to have had an "elective" procedure? Even for the more objective end points, the clinical variables that are required to optimize the predictive capacity of models describing variation in end point occurrence are often susceptible to similar problems. In particular, the assessment of lesion morphology, a parameter strongly correlated with all four of the potential end points tested, is recognized as having considerable subjectivity even when assessed by supposedly unbiased core angiographic laboratories,^{16 28} and it certainly could be manipulated to make treatment of almost any lesion appear to be high risk.

Second, even with this quality-controlled and presumably unmanipulated database, the models explaining the variation and occurrence of the four candidate end points have modest predictive capacity at best. The predictive capacity of these models are similar to the familiar Goldman index²⁹ used to assess cardiac risk of noncardiac surgery (area under ROC curve of 0.63 to 0.81, depending on the population studied³⁰ ), yet they explain only a modest amount of the variance of the outcome. It should be noted, however, that our model for death has predictive capacity very similar to that developed and published by the State of New York Department of Health (area under ROC curve, 0.85 versus 0.88), and our models for death, Q-wave MI, and/or bypass surgery and procedural success appear to be marginally superior to the New York models (0.77 versus 0.67 and 0.76 versus 0.70, respectively).¹¹

Third, given the low incidence of all (especially the most objectively determined) end points, the low yearly procedural volume for most interventionalists in the United States (Society for Cardiac Angiography and Intervention, unpublished data, 1994) serves as a major impediment to the evaluation of most physicians performing these procedures (see Fig 1). This is particularly important given the recognized inverse relation between poor outcome with this technique and procedural volume; there appears to be a "low-volume operator paradox" (the lower the volume, the more difficult it is to be certain of the results [wider confidence limits], yet the more likely it is that the results are poor).

Fourth, of the models tested, we believe that our model for the end point death, Q-wave MI, or bypass surgery serves best in the capacity to simply and reasonably discriminate among physicians' or hospital providers' superior, average, or inferior outcome. Each of its components is inarguably an adverse outcome; all are relatively objectively determined; its predictive capacity is at least modest; and it has demonstrated capacity to discern differences in its outcome measure, even for relatively low-volume providers (see Fig 7).

Fifth, and perhaps in part as a result of these limitations but also because the models measure different clinical outcomes, the agreement is limited between models in the identification of physician outliers. The choice of a clinical end point to use in modeling, therefore, will affect which operators are identified as outliers. As such, evaluating providers using both the end point death and that of death, Q-wave MI, and bypass surgery may be valuable.

Limitations of Analysis
The models described are dependent on the populations from which they are derived. To the extent that they might be unrepresentative of future populations in which they might be used, they may not function well. We believe, however, that due to their multicenter and multistudy origin and because demographic and outcome data are similar to those of several series,^{9 14 31 32 33} they are reasonably representative of percutaneous coronary revascularization as it was practiced in the United States during 1990 through 1993. We recognize that the use of more sophisticated models (perhaps using variables less routinely recorded, larger patient samples, or first- or second-order interaction terms) might improve the ability to predict major outcomes of these procedures and therefore better discriminate among the results of different providers. However, we suspect that the major limitation to our ability to predict outcome is our current relative inability to understand or measure parameters that are strongly correlated with important adverse outcomes. Furthermore, it was our intent in this analysis to generate relatively simple models that might be useful to the practicing physician and to individual hospitals interested in assessing their performance relative to our multicenter standard. In addition, we acknowledge that many statistical analyses were performed, especially for the identification of outlier providers for the various models; therefore, identification of some outliers would be expected to occur by chance (multiple comparisons problem). No statistical correction was made for this. It might be argued that a value of P=.01 would be a better cutoff point for identification of outliers, and this only underscores the limited capacity of all models to function in their intended capacity for providers with analyzable volumes of less than approximately 200 to 500. As in most, if not all, large clinical data sets, some data points are missing. For coronary intervention databases from referral centers, this is particularly common for the data element LVEF, as the diagnostic procedure from which this is derived is often performed elsewhere. We acknowledge this to be a limitation but suspect that it will be common to all modeling attempts that use this kind of clinical information. Finally, the differences between the training and validation samples might be viewed as a limitation. However, there will likely be differences between treatment and physician groups under surveillance, and our method of assessment may result in a more robust assessment than the more traditional random split between training and validation samples.

Conclusions
The models that we describe, statistically powerful yet limited by modest explanatory power as well as the low incidence of major adverse outcomes, may be useful to the medical and patient communities if they are used appropriately; that is, to identify providers with very poor clinical results or to define more modest differences in outcomes between very high-volume institutions. Validation testing and further refinement of the models will be required.

To achieve such goals, common definitions and complete and unbiased recording of demographic and end point data will be required, which will likely require some form of an audit system to achieve credibility.

Given the limitations of these models, it is important to underscore that they cannot be expected to reliably discriminate between subtle differences in provider outcome. As a consequence, it would be inappropriate to divide any reporting scheme into more than three to five levels. It must be recognized that it will be difficult to fairly assess the results of any low-volume provider. In addition, the limited capacity of these or any other published models to predict adverse outcomes, generally most notable at the extremes of risk, carries the concern that physicians will become reluctant to treat the highest-risk patients and that physicians achieving good results with especially high-risk patients will not be "rewarded." Many such patients, for example, those with acute MI and cardiogenic shock, appear to gain the most in terms of survival advantage with successful revascularization.^{3 4} A useful and practical system might exclude such patients from analysis or might "overweight" certain demographic characteristics to encourage treatment of these patients. Finally, the ideal comparative system would not discourage productive research. The uncertainty of outcome is often greatest when a new therapy is being developed. Physicians should not therefore be subjected to excess risk by participating in well organized clinical trials designed to access new forms of therapy. In the end, the insights gained from such research endeavors may well lead to improved understanding of the origin and mechanisms of the complications we analyzed and to improved models describing them.

	Selected Abbreviations and Acronyms

 

CK = creatinine kinase LVEF = left ventricular ejection fraction MI = myocardial infarction ROC = receiver operating characteristic

	Acknowledgments

 
We thank Patti Durnwald for her excellent secretarial support.

	Appendix

Top Abstract Introduction Methods Results Discussion Appendix References

 
Evaluation of Model Predictive Value by ROC Curve Analysis
ROC curve analysis is a validated method of assessing the "diagnostic accuracy" of a decision performance methodology.³⁴ In essence, the true-positive fraction is compared with the false-positive fraction as the decision threshold is varied. For the present analysis, the true state of the potential model end point (eg, death, yes, or no) was compared with the model's estimate of the risk of that end point (the sum of all "points" accrued from the presence or absence of the independent correlates of the end point [see Tables 2 through 5]) as the decision threshold (point score) was varied. In the event that the number of categories of estimate of risk (sum of point scores) exceeded 11, adjacent categories were collapsed to accommodate the analysis package. Normal distribution and nonparametric models were tested.³⁵ For all analyses, results by these two methods were nearly identical, and therefore, results from the normal distribution model are provided in Tables 2 through 5. To further adjust for potential overfitting of the models, 10-fold 80%/20% cross-validation was performed for each model,³⁶ and the adjusted ROC values are provided in "Results."

Evaluation of Individual Operators
Operators were added one at a time to the model of each outcome, and their results were compared with those of all other operators. The estimated logistic regression coefficient (ß) and corresponding standard error SE(ß) were then used to determine an adjusted odds ratio for each operator (exp{ß}) and a corresponding 95% CI {exp[ß-1.96SE(ß)]}, {exp[ß+1.96SE(ß)]}, which was converted back to adjusted risk by applying the adjusted odds ratio to the known risk of the event in the population. Model-based 95% CIs of these types correspond to P<.05 when the CI does not include the mean incidence of the event in the population.

Relation of Uncertainty in Estimates of Risk to Procedural Volume
This is calculated from the following equation: ±Z(1-/2) , where is the proportion of events observed in the study population, Z is the Z statistic, is the desired level of statistical significance, and n is the number of patients studied (see Fig 1).

Received July 18, 1995; revision received September 20, 1995; accepted October 4, 1995.

	References

Top Abstract Introduction Methods Results Discussion Appendix References

 

1. Hannan EL, O'Donnell JF, Kilburn H Jr, Bernard HR, Yazici A. Investigation of the relationship between volume and mortality for surgical procedures performed in New York State hospitals. JAMA. 1989;262:503-510. [Abstract]
   
2. Zimman D. Heart surgeons rated: state reveals patient-mortality records. Newsday, Nassau and Suffolk Ed. December 18, 1991:3.
   
3. Grines CL, Browne KF, Marco J, Rothbaum D, Stone GW, O'Keefe J, Overlie P, Donohue B, Chelliah N, Timmis GC, Vlietstra RE, Strzelecki M, Puchrowicz-Ochocki S, O'Neill WW, for the PAMI Study Group. A comparison of immediate angioplasty with thrombolytic therapy for acute myocardial infarction. N Engl J Med. 1993;328:673-679. [Abstract/Free Full Text]
   
4. Lee L, Bates ER, Pitt B, Walton JA, Laufer N, O'Neill WW. Percutaneous transluminal coronary angioplasty improves survival in acute myocardial infarction complicated by cardiogenic shock. Circulation. 1988;78:1345-1351. [Abstract/Free Full Text]
   
5. McNeil BJ, Pedersen SH, Gatsonis C. Current issues in profiles: potentials and limitations. In: Physician Payment Review Commission Conference on Profiling. Washington, DC: Physician Payment Review Commission; 1992:46-70, No. 92-2.
   
6. Coronary Artery Bypass Graft Surgery: A Technical Report. Harrisburg, Pa: Pennsylvania Health Care Cost Containment Council; November 1992.
   
7. Kassirer JP. The use and abuse of practice profiles. N Engl J Med. 1994;330:634-636. [Free Full Text]
   
8. Hamad N, Pichard AD, Lyle HR, Lindsay J Jr. Results of percutaneous transluminal coronary angioplasty by multiple, relatively low frequency operators: 1986-1987 experience. Am J Cardiol. 1988;61:1229-1231. [Medline] [Order article via Infotrieve]
   
9. Ritchie JL, Phillips KA, Luft HS. Coronary angioplasty: statewide experience in California. Circulation. 1993;88:2735-2743. [Abstract/Free Full Text]
   
10. Jollis JG, Peterson ED, DeLong ER, Mark DB, Collins SR, Muhlbaier LH, Pryor DB. The relation between the volume of coronary angioplasty procedures at hospitals treating Medicare beneficiaries and short-term mortality. N Engl J Med. 1995;331:1625-1629. [Abstract/Free Full Text]
    
11. Hannan EL, Arani DT, Johnson LW, Kemp KH, Lukacik G. Percutaneous transluminal coronary angioplasty in New York State: risk factors and outcomes. JAMA. 1993;268:3092-3097.
    
12. Iezzoni LI, Hotchkin EK, Ash AS, Schwartz M, Mackiernan Y. MedisGroups data bases: the impact of data collection guidelines on predicting in-hospital mortality. Med Care. 1993;31:277-283. [Medline] [Order article via Infotrieve]
    
13. Blumberg MS. Biased estimates of expected acute myocardial infarction mortality using MedisGroups admission severity groups. JAMA. 1991;265:2965-2970. [Abstract]
    
14. Topol EJ, Leya F, Pinkerton CA, Whitlow PL, Hofling B, Simonton CA, Masden RR, Serruys PW, Leon MB, Williams DO, King SB III, Mark DB, Isner JM, Holmes DR Jr, Ellis SG, Lee KL, Keeler GP, Berdan LG, Hinohara T, Califf RM, for the CAVEAT Study group. A comparison of directional atherectomy with coronary angioplasty in patients with coronary artery disease. N Engl J Med. 1993;329;221-227.
    
15. Wolfe MW, Roubin GS, Schweiger M, Isner JM, Ferguson JJ, Cannon AD, Cleman M, Cabin H, Leya F, Bonan R, Strony J, Adelman B, Bittle JA, on behalf of the Heparin Registry Investigators. Length of hospital stay and complications after percutaneous transluminal coronary angioplasty: clinical and procedural predictors. Circulation. 1995;92:311-319. [Abstract/Free Full Text]
    
16. Ellis SG, Vandormael MG, Cowley MJ, DiSciascio G, Deligonul U, Topol EJ, Bulle TM. Coronary morphologic and clinical determinants of procedural outcome with angioplasty for multivessel coronary disease: implications for patient selection. Circulation. 1990;82:1193-1202. [Abstract/Free Full Text]
    
17. Ellis SG, De Cesare NB, Pinkerton CA, Whitlow P, King SBI, Ghazzal ZMB, Kereiakes DJ, Popma JJ, Menke KK, Topol EJ, Holmes DR. Relation of stenosis morphology and clinical presentation to the procedural results of directional coronary atherectomy. Circulation. 1991;84:644-653. [Abstract/Free Full Text]
    
18. Ellis SG, Popma JJ, Buchbinder M, Franco I, Leon MB, Kent KM, Pichard AD, Satler LF, Topol EJ, Whitlow PL. Relation of clinical presentation, stenosis morphology and operator technique to the procedural results of rotational atherectomy. Circulation. 1994;89:882-892. [Abstract/Free Full Text]
    
19. Mancini GBJ, Simon SB, McGillem MJ, LeFree MT, Friedman HZ, Vogel RA. Automated quantitative coronary arteriography: morphologic and physiologic validation in vivo of a rapid digital angiographic method. Circulation. 1987;75:452-460. [Abstract/Free Full Text]
    
20. Uehata A, Matsuguchi T, Bittl JA, Orav J, Meredith IT, Anderson TJ, Selwyn AP, Ganz P, Yeung AC. Accuracy of electronic digital calipers compared with quantitative angiography in measuring coronary arterial diameter. Circulation. 1993;88:1724-1729. [Abstract/Free Full Text]
    
21. Wilkinson L. SYSTAT: The System for Statistics. Evanston, Ill: Systat, Inc; 1990.
    
22. Steinberg D, Colla P. LOGIT: A Supplementary Module for SYSTAT. Evanston, Ill: Systat, Inc; 1991.
    
23. McFadden D. Quantitative methods for analyzing travel behavior of individuals: some recent developments. In: Hensher DA, Stopher PR, eds. Behavior Travel Modelling. London, UK: Croom Helm; 1979.
    
24. USCI, Division of CR Bard, Inc. Industry Assessment, 1994. Internal publication.
    
25. Lee TH, Goldman L. Serum enzyme assays in the diagnosis of acute myocardial infarction. Ann Intern Med. 1986;105:221-233.
    
26. Trask N, Califf RM, Conley MJ, Kong Y, Peter R, Lee KL, Hackel DB, Wagner GS. Accuracy and interobserver variability of coronary cineangiography: a comparison with postmortem evaluation. J Am Coll Cardiol. 1984;3:1145-1154. [Abstract]
    
27. Strauss BH, Juilliere Y, Rensing BJ, Reiber JHC, Serruys PW. Edge detection versus densitometry for assessing coronary stenting quantitatively. Am J Cardiol. 1991;67:484-490. [Medline] [Order article via Infotrieve]
    
28. Kleiman NS, Rodriguez AR, Raizner AE. Interobserver variability in grading of coronary arterial narrowings using the American College of Cardiology/American Heart Association grading criteria. Am J Cardiol. 1992;69:413-415.[Medline] [Order article via Infotrieve]
    
29. Goldman L, Caldera DL, Nussbaum SR, Southwick FS, Krogstad D, Murray B, Burke DS, O'Malley TA, Goroll AH, Caplan CH, Nolan J, Carabello B, Slater EE. Multifactorial index of cardiac risk in noncardiac surgical procedures. N Engl J Med. 1977;297:845-850. [Abstract]
    
30. Goldman L. Multifactorial index of cardiac risk in noncardiac surgery: ten-year status report. J Cardiothorac Anesth. 1987;1:237-244.[Medline] [Order article via Infotrieve]
    
31. Serruys PW, Klein W, Tijssen JPG, Rutsch W, Heyndrickx GR, Emanuelsson H, Ball SG, Decoster O, Schroeder E, Liberman H, Eichhorn E, Willerson JT, Anderson HV, Khaja F, Alexander RW, Baim D, Melkert R, van Oene JC, Van Gool R. Evaluation of ketanserin in the prevention of restenosis after percutaneous transluminal coronary angioplasty: a multicenter randomized double-blind placebo-controlled trial. Circulation. 1993;88:1588-1601. [Abstract/Free Full Text]
    
32. Hannan EL. Arani DT, Johnson LW, Kemp HG Jr, Lukacik G. Percutaneous transluminal coronary angioplasty in New York State: risk factors and outcomes. JAMA. 1992;268:3092-3097. [Abstract]
    
33. The EPIC Investigators. Use of a monoclonal antibody directed against the platelet glycoprotein IIb/IIIa receptor in high-risk coronary angioplasty. N Engl J Med. 1994;330:956-961. [Abstract/Free Full Text]
    
34. Egan JP. Signal Detection Theory and ROC Analysis. New York, NY: Academic Press; 1975.
    
35. Stenson H. SIGNAL: A Supplementary Module for SYSTAT and SYGRAPH. Evanston, Ill: Systat, Inc; 1988.
    
36. Gong G. Cross-validation, the jacknife, and the bootstrap: excess error estimation in forward logistic regression. J Am Stat Assoc. 1986;81:108-113.
    

This article has been cited by other articles:

				S. B. King III, T. Aversano, W. L. Ballard, R. H. Beekman III, M. J. Cowley, S. G. Ellis, D. P. Faxon, E. L. Hannan, J. W. Hirshfeld Jr, A. K. Jacobs, et al. ACCF/AHA/SCAI 2007 Update of the Clinical Competence Statement on Cardiac Interventional Procedures: A Report of the American College of Cardiology Foundation/American Heart Association/American College of Physicians Task Force on Clinical Competence and Training (Writing Committee to Update the 1998 Clinical Competence Statement on Recommendations for the Assessment and Maintenance of Proficiency in Coronary Interventional Procedures) J. Am. Coll. Cardiol., July 3, 2007; 50(1): 82 - 108. [Full Text] [PDF]

				C. R. Narins, A. M. Dozier, F. S. Ling, and W. Zareba The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians Arch Intern Med, January 10, 2005; 165(1): 83 - 87. [Abstract] [Full Text] [PDF]

				M. Singh, C. S. Rihal, F. Selzer, K. E. Kip, K. Detre, and D. R. Holmes Jr Validation of Mayo clinic risk adjustment model for in-hospital complications after percutaneous coronary interventions, using the National Heart, Lung, and Blood Institute dynamic registry J. Am. Coll. Cardiol., November 19, 2003; 42(10): 1722 - 1728. [Abstract] [Full Text] [PDF]

				P. J. de Feyter and E. McFadden Risk score for percutaneous coronary intervention: forewarned is forearmed J. Am. Coll. Cardiol., November 19, 2003; 42(10): 1729 - 1730. [Full Text] [PDF]

				R. E. Shaw, H. V. Anderson, R. G. Brindis, R. J. Krone, L. W. Klein, C. R. McKay, P. C. Block, L. J. Shaw, K. Hewitt, W. S. Weintraub, et al. Development of a risk adjustment mortality model using the American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR) experience: 1998-2000 J. Am. Coll. Cardiol., April 3, 2002; 39(7): 1104 - 1112. [Abstract] [Full Text] [PDF]

				P. A. Cowper, E. D. Peterson, E. R. DeLong, M. B. Wightman, R. P. Wawrzynski, L. H. Muhlbaier, and M. H. Sketch Jr The impact of statistical adjustment on economic profiles of interventional cardiologists J. Am. Coll. Cardiol., November 1, 2001; 38(5): 1416 - 1423. [Abstract] [Full Text] [PDF]

				B. A. Vakili, R. Kaplan, and D. L. Brown Volume-Outcome Relation for Physicians and Hospitals Performing Angioplasty for Acute Myocardial Infarction in New York State Circulation, October 30, 2001; 104(18): 2171 - 2176. [Abstract] [Full Text] [PDF]

				M. Moscucci, E. Kline-Rogers, D. Share, M. O'Donnell, A. Maxwell-Eward, W. L. Meengs, P. Kraft, A. C. DeFranco, J. L. Chambers, K. Patel, et al. Simple Bedside Additive Tool for Prediction of In-Hospital Mortality After Percutaneous Coronary Interventions Circulation, July 17, 2001; 104(3): 263 - 268. [Abstract] [Full Text] [PDF]

				T. M. Bashore, E. R. Bates, P. B. Berger, D. A. Clark, J. T. Cusma, G. J. Dehmer, M. J. Kern, W. K. Laskey, M. P. O'Laughlin, S. Oesterle, et al. American College of Cardiology/Society for Cardiac Angiography and Interventions Clinical Expert Consensus Document on Cardiac Catheterization Laboratory Standards: A report of the American College of Cardiology Task Force on Clinical Expert Consensus Documents endorsed by the American Heart Association and the Diagnostic and Interventional Catheterization Committee of the Council on Clinical Cardiology of the AHA J. Am. Coll. Cardiol., June 15, 2001; 37(8): 2170 - 2214. [Full Text] [PDF]

				D. R. Holmes Jr, P. B. Berger, K. N. Garratt, V. Mathew, M. R. Bell, G. W. Barsness, S. T. Higano, D. E. Grill, L. N. Hammes, and C. S. Rihal Application of the New York State PTCA Mortality Model in Patients Undergoing Stent Implantation Circulation, August 1, 2000; 102(5): 517 - 522. [Abstract] [Full Text] [PDF]

				J. Lindsay Jr., E. E. Pinnow, and A. D. Pichard Frequency of major adverse cardiac events within one month of coronary angioplasty: a useful measure of operator performance J. Am. Coll. Cardiol., December 1, 1999; 34(7): 1916 - 1923. [Abstract] [Full Text] [PDF]

				S. G. Ellis, V. Guetta, D. Miller, P. L. Whitlow, and E. J. Topol Relation Between Lesion Characteristics and Risk With Percutaneous Intervention in the Stent and Glycoprotein IIb/IIIa Era : An Analysis of Results From 10 907 Lesions and Proposal for New Classification Scheme Circulation, November 9, 1999; 100(19): 1971 - 1976. [Abstract] [Full Text] [PDF]

				P. D. McGrath, D. J. Malenka, D. E. Wennberg, S. J. Shubrooks Jr., W. A. Bradley, J. F. Robb, M. A. Kellett Jr., T. J. Ryan Jr., M. J. Hearne, B. Hettleman, et al. Changing outcomes in percutaneous coronary interventions: A study of 34,752 procedures in Northern New England, 1990 to 1997 J. Am. Coll. Cardiol., September 1, 1999; 34(3): 674 - 680. [Abstract] [Full Text] [PDF]

				M. Moscucci, G. T. O'Connor, S. G. Ellis, D. J. Malenka, J. Sievers, E. R. Bates, D. W. M. Muller, S. W. Werns, E. K. Rogers, D. Karavite, et al. Validation of risk adjustment models for in-hospital percutaneous transluminal coronary angioplasty mortality on an independent data set J. Am. Coll. Cardiol., September 1, 1999; 34(3): 692 - 697. [Abstract] [Full Text] [PDF]

S. G. Ellis, D. Miller, T. F. Keys, K. Brown, R. Ellert, G. Howell, A. M. Lincoff, and E. J. Topol Comparing physician-specific two-year patient outcomes after coronary angiography: Methodologic issues and results J. Am. Coll. Cardiol., April 1, 1999; 33(5):