(Circulation. 1996;93:431-439.)
© 1996 American Heart Association, Inc.
Articles |
Analysis and Comparison of Operator-Specific Outcomes in Interventional Cardiology
From a Multicenter Database of 4860 Quality-Controlled Procedures
From the Department of Cardiology (S.G.E., N.O., M.L., G.H., E.J.T.), the Cleveland Clinic Foundation, Cleveland, Ohio; and Brigham and Women's Hospital (J.A.B., M.W.W.), Harvard Medical School, Boston, Mass.
Correspondence to Stephen G. Ellis, MD, the Cleveland Clinic Foundation, 9500 Euclid Ave, F-25, Cleveland, OH 44195.
 |
Abstract |
|---|
 Top Abstract IntroductionMethods Results Discussion Appendix References |
|---|
Background Medical consumers are increasingly requesting methods to discriminate among the results of different providers. Standards for appropriate modeling, risk adjustment, and evaluation ("scorecarding") in this setting are not well developed, although such evaluation is being performed by the medical insurance industry and by several states in the United States. Our objectives were to develop and examine clinically meaningful methodology for assessing the operator-specific results for percutaneous coronary revascularization.
Methods and Results From a multicenter database of patients treated since January 1, 1990, we used training and validation samples (n=4860) to develop several models for risk adjustment and applied them to 38 providers performing 25 to 523 procedures in the database. Models were developed using multivariable logistic regression techniques for combinations of the end points of death, myocardial infarction, bypass surgery, and procedural success. Models were evaluated for predictive accuracy by using receiver operating characteristic (ROC) analysis, for the capacity to discriminate between superior and inferior provider outcomes, and for subjectivity and concordance. Major complications occurred in 3.6% of patients. The area under the ROC curve (with perfect discriminatory accuracy, area=1.0; with no apparent accuracy, area=0.5) in the validation sample, and frequency of identification of operators with outcomes outside the 95% CI for the outcome in question for the models were for death, 0.85 and 7.9%; for death, Q-wave infarction, and bypass surgery, 0.77 and 13.2%; for death, all infarction, and bypass surgery, 0.66 and 10.5%; and for procedural success, 0.76 and 23.7%. For the models as a group, identification of outliers was inversely related to provider volume (P=.05). Models evaluating non–Q-wave infarction or requiring measurement of percent diameter stenosis were identified as being most susceptible to provider manipulation.
Conclusions For percutaneous coronary revascularization, modeling to discriminate between provider outcomes is limited by the low incidence of major adverse events, subjectivity or susceptibility to manipulation of more frequently occurring adverse events, the generally modest predictive capacity of the models, and the low volume of individual provider treatments. Modeling will be most useful in the identification of providers with extremely poor outcomes and for discrimination between providers with very large procedural volume. Until improved understanding of the biological and mechanical correlates of major complications allows the development of more predictive models, interpretation of the results of scorecarding, particularly for low-volume providers, should be made with caution.
Key Words: angioplasty • myocardial infarction • statistics
|
Introduction |
|---|
|
TopAbstractIntroductionMethods Results Discussion Appendix References |
|---|
It has been known for some time that the results of some medical procedures, such as coronary bypass surgery, vary from surgeon to surgeon and hospital to hospital.1 Increasingly, consumers of healthcare have become aware of this phenomenon and are requesting access to the outcome data of individual providers. Operator-specific mortality rates for bypass surgery have been made public in some states since 1991.2
Especially for procedures with serious complications, comparison
of appropriately risk-adjusted complication rates between providers
may be beneficial for a number of reasons. First, patients can be
triaged to providers of high-quality care. Second, providers of lesser quality
care
may be identified and allowed to acquire further training, modify their
technique, or refer high-risk patients elsewhere. Third, there is
no disincentive to attempt to treat the most complex and high-risk
patients, who often stand the most to gain from
intervention.3 4 However, currently available data
sets
may frequently have the responsible physician coded
incorrectly,5 may not be designed to record
variables important in modifying and therefore adjusting
risk,5 may have inadequate numbers of treatments for
individual physicians to be able to discriminate results6
(Fig 1
), and therefore, when applied to
"scorecarding," may discourage physicians from accepting either
new and potentially improved treatments or high-risk
patients.7
|
The results of coronary balloon angioplasty also vary by operator8 and by hospital.9 10 Both government11 and the insurance industry12 13 have begun to develop models to risk adjust for the outcomes of many procedures, including angioplasty, but standards for modeling and their appropriate application for comparison of provider results (scorecarding) in this setting have not been developed. Therefore, we sought to assess the strengths and limitations of several models that could be developed to compare the results of percutaneous intervention of different physicians or physician groups.
|
Methods |
|---|
|
TopAbstractIntroductionMethods Results Discussion Appendix References |
|---|
Patients and Studies
Data on 4860 treated patients (consecutively treated [Cleveland Clinic Interventional Database] or enrolled in several prospective trials of percutaneous coronary intervention14 15 16 17 18 meeting the criteria listed) were obtained. Criteria for study inclusion were treatment on or after January 1, 1990; treatment for stable or unstable angina, silent ischemia, or acute MI; documentation of baseline data and procedural complications by independent audit (not the physician providers themselves); routine postprocedure measurement of creatine kinase; and measurement of the angiographic result by either computer edge-detection19 or calipers.20 No patients were excluded due to missing data or for other reasons.
Variables and Definitions
The following preprocedural
variables (available from all
studies and in 
95% of patients, except where noted) were collected
in a relational database21 —baseline demographic: acute MI
(within 24 hours of onset), age, cardiogenic shock, congestive heart
failure class (New York Heart Association) (available in 80%), sex,
prior bypass surgery, prior MI (available in 82%), prior
percutaneous treatment at site to be treated, and
unstable angina; and baseline angiographic: LVEF (available in 60%),
most complex lesion with attempted treatment (modified American College
of Cardiology/American Heart Association lesion
classification16 ), number of diseased (50% diameter
stenosis) vessels, and vessel(s) treated. LVEF frequently was
not available for patients who died. To avoid excluding patients
without known LVEF from the mortality analysis, LVEF was
imputed according to the formula: LVEF=65 minus 7.8 (heart failure
class) minus 10.1 (prior MI) for that analysis only. This
formula was derived from multivariable analysis of the
end point LVEF observed in the training sample (adjusted multiple
r2=.54, P<.001). Additional
variables included treatment (balloon angioplasty, directional
atherectomy, excimer laser [xenon chloride], extraction atherectomy,
infrared laser [holmium], primary stent implantation, and rotational
atherectomy).
Outcome variables collected were coronary bypass surgery (at any time during the hospitalization), death, MI (CK at least twofold to threefold upper limit of normal, depending on study standards), procedural success (final stenosis <50% and no death, MI, or bypass surgery), and Q-wave MI.
Additional variables available only
from some studies were also
tabulated: baseline heart rate, evaluation as not a candidate for
bypass surgery, prior cerebrovascular accident or transient
ischemic event, recent MI (within 2 weeks), renal insufficiency
(creatinine 2.0 mg/dL), symptomatic
peripheral vascular disease, and symptomatic
pulmonary disease.
Statistical Analysis
All data are presented as mean±1
SD unless otherwise
indicated. Patients were arbitrarily divided into a training sample
(from the six studies available when the analysis was begun)
and a test sample (from the two studies that became available soon
afterward). Nineteen baseline demographic and angiographic
variables were analyzed as potential covariates of the
clinical end points. Treatment variables (eg, use of balloon
angioplasty rather than directional atherectomy) were not included in
this analysis because they involved choice by the operator.
2 and Student's t tests were used to
assess univariate relation of categorical and continuous
variables, respectively, with the various end points. Statistical
modeling in the training set was done using logistic regression
analyses,22 testing candidate variables with
univariate P
.05, with conversion to
risk-adjusted odds ratios and 95% CIs. When >10% of data were
missing for a variable with multivariate
P.05, the regression analyses were evaluated both
with and without that variable included. Candidate end points for
the models were death; death, Q-wave MI, or bypass surgery; death, all
MI, or bypass surgery; and procedural success. Simplification of the
models for use in this study to allow easy application by individual
physicians was done by dividing the odds ratio for each significant
covariate by that of the smallest significant odds ratio of more than 1
and rounding off to the nearest integer so as to derive a "point
score" for each variable. For odds ratios <1, their inverse was
handled in the same manner and the point score assigned was
negative.
The primary measure of the models' predictive accuracy
was ROC
analysis (see "Appendix"), but McFadden's
2 statistic was also calculated.23 Possible
covariates and end points for models were evaluated for subjectivity
and possible susceptibility to manipulation (eg, not measuring CK
levels after a successfully treated abrupt closure so as not to
diagnose a "small" non–Q-wave MI) by a panel of seven
cardiologists on a 1-to-5 scale (1, not subjective or susceptible to
manipulation; 5, subjective and most subjective to manipulation). Each
cardiologist had previously coded these variables for 300
procedures and was aware of possible subjectivity in their definition
and of pressures from cardiology staff attending
physicians to make their results "look good." Models were also
evaluated for their capacity to identify provider outliers in relation
to operator volume and for concordance with the results of other models
using Cohen's 
analyses. The results of these
analyses are graphically depicted (see Fig 7) for all 19
providers with procedural volume >75 and for an equal number of
randomly selected operators with volume of 25 to 74 procedures.
|
After the development of the four models, each individual operator was entered into the multivariable analysis as a covariate possibly altering the outcome of the patient he or she treated (see "Appendix"). Providers with "superior" outcome were defined as those with results exceeding the 95% CI for outcome, whereas those with "inferior" outcome were defined as those with results worse than the 95% CI for all other providers.
|
Results |
|---|
|
TopAbstractIntroductionMethods Results Discussion Appendix References |
|---|
Patient Population
Baseline demographic and treatment information for the training and validation samples are given in Table 1
. Adverse
outcomes were distributed among the training and validation samples as
follows: death (34 [0.9%], 22 [2.0%]); death, Q-wave MI,
and
bypass surgery (150 [4.0%], 33 [3.1%]); death, all MI, and
bypass
surgery (256 [6.8%], 87 [8.1%]); and procedural failure
(499
[13.2%], 117 [10.9%]). Although the two groups were
similar in
many ways, significant (P<.05) differences exist with
respect to the incidences of some variables, reflecting not
unexpected differences in practice patterns between centers and
studies, as well the large number of patients studied. The 4860
procedures were performed by 66 providers (case volume range, 2 to
523).
|
Reliability of Variable Coding
The results of the physician
panel assessment of the
susceptibility of candidate covariates and end points to manipulation
are shown in Fig 2. Some candidate variables, such
as death and Q-wave MI, were judged to be objective and reliably
determined. Conversely, the variables of unstable angina,
non–Q-wave MI, stenosis morphology, symptomatic
obstructive pulmonary disease, procedural success, and class IV
angina were judged to be especially unreliable (score 3.5) due to
either interobserver variability or the potential for manipulation.
|
Modeling Candidate End Points
The variables correlated with
the four candidate end points,
their contribution to potential models, and the predictive accuracy of
the models in the validation sample (area under ROC curve and
McFadden's 2) are shown in Tables 2 through
5. The
incidences of adverse outcome by model result as applied in the
training and validation samples are shown in Figs 3 through
6.
|
|
|
|
|
|
|
|
In the
training sample, the candidate end point of death was
independently correlated with acute MI within 24 hours before the
coronary intervention, age, ejection fraction, three-vessel
disease, lesion morphology, renal function, vessel site treated, and
presence of cardiogenic shock. The end point of death, Q-wave MI, or
bypass surgery was correlated with ejection fraction, lesion
morphology, presentation with acute MI, and shock and was
inversely related to prior bypass surgery and restenosis.
Data for the end points death, all MI, or bypass surgery and for
procedural success are presented in Tables 4
and
5.
The model for death had the best predictive accuracy
(area under ROC
curve, 0.85), whereas the other models were less predictive. The
estimated variance in the outcome data explained by the models was low,
however (McFadden's 2=.05 to .25).
Performance of the Models in Adjusting for Risk and
Identifying Outliers
Absolute and risk-adjusted 95% confidence limits
for the
likelihood of death, Q-wave MI, or bypass surgery for patients treated
by 38 of the providers are shown in Fig 7. The variation
and uncertainty of estimates of provider outcome, the influence of risk
adjustment, and the capacity of the model to identify outliers are
illustrated. Providers are classified by the number of procedures in
the database (<75 or 75) to illustrate the differences in the
models' capacity to discriminate results for both low- and moderately
high-volume providers. Similar analyses were done for the
other three candidate models (data not shown). After risk adjustment
and inclusion of data from all four models, 5 of 152 provider
evaluations (3.3%) were "superior" and 16 of 152 (10.5%) were
"inferior" (P<.001). Risk adjustment
changed the categorization (superior, average, inferior) in
7 of 152 evaluations (4.6%), most frequently for the models of
procedural success, and death, Q-wave MI, and bypass surgery (both
7.9%, respectively) and only infrequently for the models of death, and
death, MI, and bypass surgery (2.6% and 0.0%, respectively). The
models' capacity to identify outliers was inversely related to the
number of procedures each operator contributed for analysis
(P=.05). For the end point with the lowest incidence
(death), only three providers (7.7%) could be identified as having
results significantly different than the average, and none of these had
a procedural volume of fewer than 167 cases. For the models of more
frequently occurring events, 10% to 24% of providers could be
identified as having significantly different outcomes. Only one
provider (2.8%) with a volume of fewer than 57 cases was identified as
an "outlier" by any of the four models.
Concordance Among Models in the Identification of Outlier
Providers
The values for concordance between models in
identifying
outliers are displayed in Table 6. Values are generally
low, either in the fair-to-good range (.40 to .79) or, more
commonly, in the poor range (<.40), indicating that the choice of
outcome evaluated often determines which providers are identified in
the superior or inferior outlier groups. To put these
values in perspective, for the 14 operators who were identified as
being an outlier in any of the four models, no operator was identified
by all models, 1 operator was identified by three models, 4 operators
were identified by two models, and 9 operators were identified by one
model.
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethods Results Discussion Appendix References |
|---|
It is estimated that 410 000 percutaneous coronary revascularization procedures were performed in the United States in 1994,24 and the "average" physician performing this type of procedure did only 20 to 25 (Society for Cardiac Angiography and Intervention, unpublished data, 1994). Complications have been shown to be inversely related to both hospital and physician experience with these techniques.8 9 10
This analysis is the first to evaluate the suitability of various models to assess differences in provider outcomes for this commonly performed group of procedures, even though provider scorecards for these techniques are being used and will soon be published (unpublished data). We have made five major observations.
First, many of the possible covariates and end points were believed by
physicians who perform these procedures to be either quite subjective,
possibly susceptible to manipulation, or both (Fig 2). For
example, the
identification of the end point "non–Q-wave MI" often relies
on
determination of enzyme markers of myocardial damage, most often, CK.
Elevations in this enzyme may be brief25 and can easily be
missed if assays are not performed at routine and frequent intervals.
Guidelines for CK measurement in this setting are not standardized, and
a physician desiring to "look good" might simply choose not to
test for CK elevation. Determination of procedural success typically
requires assessment of the posttreatment dimensional result (percent
diameter stenosis) of the treated lesion. This is most often
done visually, a method of assessment known to be highly
subjective.26 The result may also be measured by any of
several techniques,19 27 but this is laborious and
the
operator might choose simply not to record the angiographic
projection in which the lesion appears worst. The end point
"emergency surgery" is also difficult: Should evidence of
ischemia be required to meet the definition, and if so, which
ones? Should a patient referred urgently for bypass surgery, for whom a
balloon pump or stent temporarily alleviates ischemia yet is
considered so unstable that the surgeon limits the number or type of
grafts placed, be considered to have had an "elective" procedure?
Even for the more objective end points, the clinical variables that
are required to optimize the predictive capacity of models describing
variation in end point occurrence are often susceptible to similar
problems. In particular, the assessment of lesion morphology, a
parameter strongly correlated with all four of the
potential end points tested, is recognized as having considerable
subjectivity even when assessed by supposedly unbiased core
angiographic laboratories,16 28 and it certainly
could be
manipulated to make treatment of almost any lesion appear to be high
risk.
Second, even with this quality-controlled and presumably unmanipulated database, the models explaining the variation and occurrence of the four candidate end points have modest predictive capacity at best. The predictive capacity of these models are similar to the familiar Goldman index29 used to assess cardiac risk of noncardiac surgery (area under ROC curve of 0.63 to 0.81, depending on the population studied30 ), yet they explain only a modest amount of the variance of the outcome. It should be noted, however, that our model for death has predictive capacity very similar to that developed and published by the State of New York Department of Health (area under ROC curve, 0.85 versus 0.88), and our models for death, Q-wave MI, and/or bypass surgery and procedural success appear to be marginally superior to the New York models (0.77 versus 0.67 and 0.76 versus 0.70, respectively).11
Third, given the low incidence of all (especially the most
objectively determined) end points, the low yearly procedural volume
for most interventionalists in the United States (Society for Cardiac
Angiography and Intervention, unpublished data, 1994) serves as a major
impediment to the evaluation of most physicians performing these
procedures (see Fig 1). This is particularly important given
the
recognized inverse relation between poor outcome with this technique
and procedural volume; there appears to be a "low-volume
operator paradox" (the lower the volume, the more difficult it is to
be certain of the results [wider confidence limits], yet the more
likely it is that the results are poor).
Fourth, of the models tested, we believe that our model for the
end point death, Q-wave MI, or bypass surgery serves best in the
capacity to simply and reasonably discriminate among physicians' or
hospital providers' superior, average, or inferior
outcome. Each of its components is inarguably an adverse outcome; all
are relatively objectively determined; its predictive capacity is at
least modest; and it has demonstrated capacity to discern differences
in its outcome measure, even for relatively low-volume providers
(see Fig 7).
Fifth, and perhaps in part as a result of these limitations but also because the models measure different clinical outcomes, the agreement is limited between models in the identification of physician outliers. The choice of a clinical end point to use in modeling, therefore, will affect which operators are identified as outliers. As such, evaluating providers using both the end point death and that of death, Q-wave MI, and bypass surgery may be valuable.
Limitations of Analysis
The models described are dependent on
the populations from which
they are derived. To the extent that they might be
unrepresentative of future populations in which
they might be used, they may not function well. We believe, however,
that due to their multicenter and multistudy origin and because
demographic and outcome data are similar to those of several
series,9 14 31 32 33
they are reasonably
representative of percutaneous
coronary revascularization as it was
practiced in the United States during 1990 through 1993. We recognize
that the use of more sophisticated models (perhaps using variables
less routinely recorded, larger patient samples, or first- or
second-order interaction terms) might improve the ability to
predict major outcomes of these procedures and therefore better
discriminate among the results of different providers. However, we
suspect that the major limitation to our ability to predict outcome is
our current relative inability to understand or measure
parameters that are strongly correlated with important
adverse outcomes. Furthermore, it was our intent in this
analysis to generate relatively simple models that might be
useful to the practicing physician and to individual hospitals
interested in assessing their performance relative to our
multicenter standard. In addition, we acknowledge that many statistical
analyses were performed, especially for the identification of
outlier providers for the various models; therefore, identification of
some outliers would be expected to occur by chance (multiple
comparisons problem). No statistical correction was made for this. It
might be argued that a value of P=.01 would be a better
cutoff point for identification of outliers, and this only
underscores the limited capacity of all models to function in their
intended capacity for providers with analyzable volumes of less than
approximately 200 to 500. As in most, if not all, large clinical data
sets, some data points are missing. For coronary intervention
databases from referral centers, this is particularly common for the
data element LVEF, as the diagnostic procedure from which
this is derived is often performed elsewhere. We acknowledge this to be
a limitation but suspect that it will be common to all modeling
attempts that use this kind of clinical information. Finally, the
differences between the training and validation samples might be viewed
as a limitation. However, there will likely be differences between
treatment and physician groups under surveillance, and our method of
assessment may result in a more robust assessment than the more
traditional random split between training and validation samples.
Conclusions
The models that we describe, statistically
powerful yet limited by
modest explanatory power as well as the low incidence of major adverse
outcomes, may be useful to the medical and patient communities if they
are used appropriately; that is, to identify providers with very poor
clinical results or to define more modest differences in outcomes
between very high-volume institutions. Validation testing and
further refinement of the models will be required.
To achieve such goals, common definitions and complete and unbiased recording of demographic and end point data will be required, which will likely require some form of an audit system to achieve credibility.
Given the limitations of these models, it is important to underscore that they cannot be expected to reliably discriminate between subtle differences in provider outcome. As a consequence, it would be inappropriate to divide any reporting scheme into more than three to five levels. It must be recognized that it will be difficult to fairly assess the results of any low-volume provider. In addition, the limited capacity of these or any other published models to predict adverse outcomes, generally most notable at the extremes of risk, carries the concern that physicians will become reluctant to treat the highest-risk patients and that physicians achieving good results with especially high-risk patients will not be "rewarded." Many such patients, for example, those with acute MI and cardiogenic shock, appear to gain the most in terms of survival advantage with successful revascularization.3 4 A useful and practical system might exclude such patients from analysis or might "overweight" certain demographic characteristics to encourage treatment of these patients. Finally, the ideal comparative system would not discourage productive research. The uncertainty of outcome is often greatest when a new therapy is being developed. Physicians should not therefore be subjected to excess risk by participating in well organized clinical trials designed to access new forms of therapy. In the end, the insights gained from such research endeavors may well lead to improved understanding of the origin and mechanisms of the complications we analyzed and to improved models describing them.
|
Selected Abbreviations and Acronyms |
|---|
|
|
Acknowledgments |
|---|
We thank Patti Durnwald for her excellent secretarial support.
|
Appendix |
|---|
|
TopAbstractIntroductionMethods Results Discussion Appendix References |
|---|
Evaluation of Model Predictive Value by ROC Curve Analysis
ROC curve analysis is a validated method of assessing the "diagnostic accuracy" of a decision performance methodology.34 In essence, the true-positive fraction is compared with the false-positive fraction as the decision threshold is varied. For the present analysis, the true state of the potential model end point (eg, death, yes, or no) was compared with the model's estimate of the risk of that end point (the sum of all "points" accrued from the presence or absence of the independent correlates of the end point [see Tables 2 through 5
]) as the
decision threshold (point score) was
varied. In the event that the number of categories of estimate of risk
(sum of point scores) exceeded 11, adjacent categories were collapsed
to accommodate the analysis package. Normal distribution and
nonparametric models were tested.35 For all
analyses, results by these two methods were nearly identical,
and therefore, results from the normal distribution model are provided
in Tables 2 through
5. To further
adjust for potential overfitting of
the models, 10-fold 80%/20% cross-validation was performed for
each model,36 and the adjusted ROC values are provided in
"Results."
Evaluation of Individual Operators
Operators were added one at a time to the model of each outcome,
and their results were compared with those of all other operators. The
estimated logistic regression coefficient (ß) and corresponding
standard error SE(ß) were then used to determine an adjusted odds
ratio for each operator (exp{ß}) and a corresponding 95% CI
{exp[ß-1.96SE(ß)]},
{exp[ß+1.96SE(ß)]}, which
was converted back to adjusted risk by applying the adjusted odds ratio
to the known risk of the event in the population. Model-based 95%
CIs of these types correspond to P<.05 when the CI does not
include the mean incidence of the event in the population.
Relation of Uncertainty in Estimates of Risk to Procedural
Volume
This is calculated from the following equation:
±Z(1-
/2)
,
where
is the proportion of events
observed in the study
population, Z is the Z statistic, is the
desired level of statistical significance, and n is the
number of patients studied (see Fig 1).
Received July 18, 1995; revision received September 20, 1995; accepted October 4, 1995.
|
References |
|---|
|
TopAbstractIntroductionMethods Results Discussion Appendix References |
|---|
1. Hannan EL, O'Donnell JF, Kilburn H Jr, Bernard HR, Yazici A. Investigation of the relationship between volume and mortality for surgical procedures performed in New York State hospitals. JAMA. 1989;262:503-510.
2. Zimman D. Heart surgeons rated: state reveals patient-mortality records. Newsday, Nassau and Suffolk Ed. December 18, 1991:3.
3.
Grines CL, Browne KF, Marco J, Rothbaum D, Stone GW,
O'Keefe J, Overlie P, Donohue B, Chelliah N, Timmis GC, Vlietstra RE,
Strzelecki M, Puchrowicz-Ochocki S, O'Neill WW, for the PAMI Study
Group. A comparison of immediate angioplasty with
thrombolytic therapy for acute myocardial infarction.
N Engl J Med. 1993;328:673-679.
4.
Lee L, Bates ER, Pitt B, Walton JA, Laufer N, O'Neill
WW. Percutaneous transluminal coronary
angioplasty improves survival in acute myocardial infarction
complicated by cardiogenic shock.
Circulation. 1988;78:1345-1351.
5. McNeil BJ, Pedersen SH, Gatsonis C. Current issues in profiles: potentials and limitations. In: Physician Payment Review Commission Conference on Profiling. Washington, DC: Physician Payment Review Commission; 1992:46-70, No. 92-2.
6. Coronary Artery Bypass Graft Surgery: A Technical Report. Harrisburg, Pa: Pennsylvania Health Care Cost Containment Council; November 1992.
7.
Kassirer JP. The use and abuse of practice
profiles. N Engl J Med. 1994;330:634-636.
8. Hamad N, Pichard AD, Lyle HR, Lindsay J Jr. Results of percutaneous transluminal coronary angioplasty by multiple, relatively low frequency operators: 1986-1987 experience. Am J Cardiol. 1988;61:1229-1231. [Medline] [Order article via Infotrieve]
9.
Ritchie JL, Phillips KA, Luft HS.
Coronary angioplasty: statewide experience in
California. Circulation. 1993;88:2735-2743.
10.
Jollis JG, Peterson ED, DeLong ER, Mark DB, Collins SR,
Muhlbaier LH, Pryor DB. The relation between the volume of
coronary angioplasty procedures at hospitals treating Medicare
beneficiaries and short-term mortality. N
Engl J Med. 1995;331:1625-1629.
11. Hannan EL, Arani DT, Johnson LW, Kemp KH, Lukacik G. Percutaneous transluminal coronary angioplasty in New York State: risk factors and outcomes. JAMA. 1993;268:3092-3097.
12. Iezzoni LI, Hotchkin EK, Ash AS, Schwartz M, Mackiernan Y. MedisGroups data bases: the impact of data collection guidelines on predicting in-hospital mortality. Med Care. 1993;31:277-283. [Medline] [Order article via Infotrieve]
13.
Blumberg MS. Biased estimates of expected acute
myocardial infarction mortality using MedisGroups admission severity
groups. JAMA. 1991;265:2965-2970.
14. Topol EJ, Leya F, Pinkerton CA, Whitlow PL, Hofling B, Simonton CA, Masden RR, Serruys PW, Leon MB, Williams DO, King SB III, Mark DB, Isner JM, Holmes DR Jr, Ellis SG, Lee KL, Keeler GP, Berdan LG, Hinohara T, Califf RM, for the CAVEAT Study group. A comparison of directional atherectomy with coronary angioplasty in patients with coronary artery disease. N Engl J Med. 1993;329;221-227.
15.
Wolfe MW, Roubin GS, Schweiger M, Isner JM, Ferguson JJ,
Cannon AD, Cleman M, Cabin H, Leya F, Bonan R, Strony J, Adelman B,
Bittle JA, on behalf of the Heparin Registry Investigators. Length of
hospital stay and complications after percutaneous transluminal
coronary angioplasty: clinical and procedural predictors.
Circulation. 1995;92:311-319.
16.
Ellis SG, Vandormael MG, Cowley MJ, DiSciascio G,
Deligonul U, Topol EJ, Bulle TM. Coronary morphologic
and clinical determinants of procedural outcome with angioplasty for
multivessel coronary disease: implications for patient
selection. Circulation. 1990;82:1193-1202.
17.
Ellis SG, De Cesare NB, Pinkerton CA, Whitlow P, King
SBI, Ghazzal ZMB, Kereiakes DJ, Popma JJ, Menke KK, Topol EJ, Holmes
DR. Relation of stenosis morphology and clinical
presentation to the procedural results of directional
coronary atherectomy. Circulation. 1991;84:644-653.
18.
Ellis SG, Popma JJ, Buchbinder M, Franco I, Leon MB,
Kent KM, Pichard AD, Satler LF, Topol EJ, Whitlow PL. Relation
of clinical presentation, stenosis morphology and
operator technique to the procedural results of rotational
atherectomy. Circulation. 1994;89:882-892.
19.
Mancini GBJ, Simon SB, McGillem MJ, LeFree MT, Friedman
HZ, Vogel RA. Automated quantitative coronary
arteriography: morphologic and physiologic validation in vivo of a
rapid digital angiographic method.
Circulation. 1987;75:452-460.
20.
Uehata A, Matsuguchi T, Bittl JA, Orav J, Meredith IT,
Anderson TJ, Selwyn AP, Ganz P, Yeung AC. Accuracy of electronic
digital calipers compared with quantitative angiography in measuring
coronary arterial diameter.
Circulation. 1993;88:1724-1729.
21. Wilkinson L. SYSTAT: The System for Statistics. Evanston, Ill: Systat, Inc; 1990.
22. Steinberg D, Colla P. LOGIT: A Supplementary Module for SYSTAT. Evanston, Ill: Systat, Inc; 1991.
23. McFadden D. Quantitative methods for analyzing travel behavior of individuals: some recent developments. In: Hensher DA, Stopher PR, eds. Behavior Travel Modelling. London, UK: Croom Helm; 1979.
24. USCI, Division of CR Bard, Inc. Industry Assessment, 1994. Internal publication.
25. Lee TH, Goldman L. Serum enzyme assays in the diagnosis of acute myocardial infarction. Ann Intern Med. 1986;105:221-233.
26. Trask N, Califf RM, Conley MJ, Kong Y, Peter R, Lee KL, Hackel DB, Wagner GS. Accuracy and interobserver variability of coronary cineangiography: a comparison with postmortem evaluation. J Am Coll Cardiol. 1984;3:1145-1154. [Abstract]
27. Strauss BH, Juilliere Y, Rensing BJ, Reiber JHC, Serruys PW. Edge detection versus densitometry for assessing coronary stenting quantitatively. Am J Cardiol. 1991;67:484-490. [Medline] [Order article via Infotrieve]
28. Kleiman NS, Rodriguez AR, Raizner AE. Interobserver variability in grading of coronary arterial narrowings using the American College of Cardiology/American Heart Association grading criteria. Am J Cardiol. 1992;69:413-415.[Medline] [Order article via Infotrieve]
29. Goldman L, Caldera DL, Nussbaum SR, Southwick FS, Krogstad D, Murray B, Burke DS, O'Malley TA, Goroll AH, Caplan CH, Nolan J, Carabello B, Slater EE. Multifactorial index of cardiac risk in noncardiac surgical procedures. N Engl J Med. 1977;297:845-850. [Abstract]
30. Goldman L. Multifactorial index of cardiac risk in noncardiac surgery: ten-year status report. J Cardiothorac Anesth. 1987;1:237-244.[Medline] [Order article via Infotrieve]
31.
Serruys PW, Klein W, Tijssen JPG, Rutsch W, Heyndrickx
GR, Emanuelsson H, Ball SG, Decoster O, Schroeder E, Liberman H,
Eichhorn E, Willerson JT, Anderson HV, Khaja F, Alexander RW, Baim D,
Melkert R, van Oene JC, Van Gool R. Evaluation of ketanserin in
the prevention of restenosis after
percutaneous transluminal coronary angioplasty:
a multicenter randomized double-blind placebo-controlled
trial. Circulation. 1993;88:1588-1601.
32.
Hannan EL. Arani DT, Johnson LW, Kemp HG Jr,
Lukacik G. Percutaneous transluminal
coronary angioplasty in New York State: risk factors and
outcomes. JAMA. 1992;268:3092-3097.
33.
The EPIC Investigators. Use of a monoclonal antibody
directed against the platelet glycoprotein IIb/IIIa
receptor in high-risk coronary angioplasty. N
Engl J Med. 1994;330:956-961.
34. Egan JP. Signal Detection Theory and ROC Analysis. New York, NY: Academic Press; 1975.
35. Stenson H. SIGNAL: A Supplementary Module for SYSTAT and SYGRAPH. Evanston, Ill: Systat, Inc; 1988.
36. Gong G. Cross-validation, the jacknife, and the bootstrap: excess error estimation in forward logistic regression. J Am Stat Assoc. 1986;81:108-113.
This article has been cited by other articles:
 |
|
 |
|
  A. Ernst, M. Simoff, D. Ost, Y. Goldman, and F. J. F. Herth Prospective Risk-Adjusted Morbidity and Mortality Outcome Analysis After Therapeutic Bronchoscopic Procedures: Results of a Multi-institutional Outcomes Database Chest, September 1, 2008; 134(3): 514 - 519. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. B. King III, T. Aversano, W. L. Ballard, R. H. Beekman III, M. J. Cowley, S. G. Ellis, D. P. Faxon, E. L. Hannan, J. W. Hirshfeld Jr, A. K. Jacobs, et al. ACCF/AHA/SCAI 2007 Update of the Clinical Competence Statement on Cardiac Interventional Procedures: A Report of the American College of Cardiology Foundation/American Heart Association/American College of Physicians Task Force on Clinical Competence and Training (Writing Committee to Update the 1998 Clinical Competence Statement on Recommendations for the Assessment and Maintenance of Proficiency in Coronary Interventional Procedures) J. Am. Coll. Cardiol., July 3, 2007; 50(1): 82 - 108. [Full Text] [PDF]
|
|
|
|
 |
|
 C. R. Narins, A. M. Dozier, F. S. Ling, and W. Zareba The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians Arch Intern Med, January 10, 2005; 165(1): 83 - 87. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Singh, C. S. Rihal, F. Selzer, K. E. Kip, K. Detre, and D. R. Holmes Jr Validation of Mayo clinic risk adjustment model for in-hospital complications after percutaneous coronary interventions, using the National Heart, Lung, and Blood Institute dynamic registry J. Am. Coll. Cardiol., November 19, 2003; 42(10): 1722 - 1728. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. J. de Feyter and E. McFadden Risk score for percutaneous coronary intervention: forewarned is forearmed J. Am. Coll. Cardiol., November 19, 2003; 42(10): 1729 - 1730. [Full Text] [PDF]
|
|
|
|
|
|
R. E. Shaw, H. V. Anderson, R. G. Brindis, R. J. Krone, L. W. Klein, C. R. McKay, P. C. Block, L. J. Shaw, K. Hewitt, W. S. Weintraub, et al. Development of a risk adjustment mortality model using the American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR) experience: 1998-2000 J. Am. Coll. Cardiol., April 3, 2002; 39(7): 1104 - 1112. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. A. Cowper, E. D. Peterson, E. R. DeLong, M. B. Wightman, R. P. Wawrzynski, L. H. Muhlbaier, and M. H. Sketch Jr The impact of statistical adjustment on economic profiles of interventional cardiologists J. Am. Coll. Cardiol., November 1, 2001; 38(5): 1416 - 1423. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. A. Vakili, R. Kaplan, and D. L. Brown Volume-Outcome Relation for Physicians and Hospitals Performing Angioplasty for Acute Myocardial Infarction in New York State Circulation, October 30, 2001; 104(18): 2171 - 2176. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Moscucci, E. Kline-Rogers, D. Share, M. O'Donnell, A. Maxwell-Eward, W. L. Meengs, P. Kraft, A. C. DeFranco, J. L. Chambers, K. Patel, et al. Simple Bedside Additive Tool for Prediction of In-Hospital Mortality After Percutaneous Coronary Interventions Circulation, July 17, 2001; 104(3): 263 - 268. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. M. Bashore, E. R. Bates, P. B. Berger, D. A. Clark, J. T. Cusma, G. J. Dehmer, M. J. Kern, W. K. Laskey, M. P. O'Laughlin, S. Oesterle, et al. American College of Cardiology/Society for Cardiac Angiography and Interventions Clinical Expert Consensus Document on Cardiac Catheterization Laboratory Standards: A report of the American College of Cardiology Task Force on Clinical Expert Consensus Documents endorsed by the American Heart Association and the Diagnostic and Interventional Catheterization Committee of the Council on Clinical Cardiology of the AHA J. Am. Coll. Cardiol., June 15, 2001; 37(8): 2170 - 2214. [Full Text] [PDF]
|
|
|
|
|
|
D. R. Holmes Jr, P. B. Berger, K. N. Garratt, V. Mathew, M. R. Bell, G. W. Barsness, S. T. Higano, D. E. Grill, L. N. Hammes, and C. S. Rihal Application of the New York State PTCA Mortality Model in Patients Undergoing Stent Implantation Circulation, August 1, 2000; 102(5): 517 - 522. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Lindsay Jr., E. E. Pinnow, and A. D. Pichard Frequency of major adverse cardiac events within one month of coronary angioplasty: a useful measure of operator performance J. Am. Coll. Cardiol., December 1, 1999; 34(7): 1916 - 1923. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. G. Ellis, V. Guetta, D. Miller, P. L. Whitlow, and E. J. Topol Relation Between Lesion Characteristics and Risk With Percutaneous Intervention in the Stent and Glycoprotein IIb/IIIa Era : An Analysis of Results From 10 907 Lesions and Proposal for New Classification Scheme Circulation, November 9, 1999; 100(19): 1971 - 1976. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. D. McGrath, D. J. Malenka, D. E. Wennberg, S. J. Shubrooks Jr., W. A. Bradley, J. F. Robb, M. A. Kellett Jr., T. J. Ryan Jr., M. J. Hearne, B. Hettleman, et al. Changing outcomes in percutaneous coronary interventions: A study of 34,752 procedures in Northern New England, 1990 to 1997 J. Am. Coll. Cardiol., September 1, 1999; 34(3): 674 - 680. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Moscucci, G. T. O'Connor, S. G. Ellis, D. J. Malenka, J. Sievers, E. R. Bates, D. W. M. Muller, S. W. Werns, E. K. Rogers, D. Karavite, et al. Validation of risk adjustment models for in-hospital percutaneous transluminal coronary angioplasty mortality on an independent data set J. Am. Coll. Cardiol., September 1, 1999; 34(3): 692 - 697. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. G. Ellis, D. Miller, T. F. Keys, K. Brown, R. Ellert, G. Howell, A. M. Lincoff, and E. J. Topol Comparing physician-specific two-year patient outcomes after coronary angiography: Methodologic issues and results J. Am. Coll. Cardiol., April 1, 1999; 33(5): 1278 - 1285. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Kastrati, F.-J. Neumann, and A. Schomig Operator volume and outcome of patients undergoing coronary stent placement J. Am. Coll. Cardiol., October 1, 1998; 32(4): 970 - 976. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. C. Block, E. C. Peterson, R. Krone, K. Kesler, E. Hannan, G. T. O'Connor, and K. Detre Identification of variables needed to risk adjust outcomes of coronary interventions: evidence-based guidelines for efficient data collection J. Am. Coll. Cardiol., July 1, 1998; 32(1): 275 - 282. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. G. Jollis and P. S. Romano Pennsylvania's Focus on Heart Attack -- Grading the Scorecard N. Engl. J. Med., April 2, 1998; 338(14): 983 - 987. [Full Text]
|
|
|
|
|
|
S. G. Ellis, W. Weintraub, D. Holmes, R. Shaw, P. C. Block, and S. B. King III Relation of Operator Volume and Experience to Procedural Outcome of Percutaneous Coronary Revascularization at Hospitals With High Interventional Volumes Circulation, June 3, 1997; 95(11): 2479 - 2484. [Abstract] [Full Text]
|
|
|
|
|
|
J. G. Jollis, E. D. Peterson, C. L. Nelson, J. A. Stafford, E. R. DeLong, L. H. Muhlbaier, and D. B. Mark Relationship Between Physician and Hospital Coronary Angioplasty Volume and Outcome in Elderly Patients Circulation, June 3, 1997; 95(11): 2485 - 2491. [Abstract] [Full Text]
|
|
|
|
|
|
J. A. Bittl Advances in Coronary Angioplasty N. Engl. J. Med., October 24, 1996; 335(17): 1290 - 1302. [Full Text] [PDF]
|
|
|
|
|
|
R. M. Califf, J. G. Jollis, and E. D. Peterson Operator-Specific Outcomes : A Call to Professional Responsibility Circulation, February 1, 1996; 93(3): 403 - 406. [Full Text]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||