| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Circulation. 2003;107:811.)
© 2003 American Heart Association, Inc.
Clinical Investigation and Reports |
From the Sections of Cardiovascular Medicine (S.S.R., H.M.K.) and General Internal Medicine (C.P.G.), Department of Internal Medicine and Section of Health Policy and Administration, Department of Epidemiology and Public Health (H.M.K.), Yale University School of Medicine, New Haven, Conn; Center for Clinical and Genetic Economics (K.P.W.), Duke Clinical Research Institute, Duke University Medical Center, Durham, NC; Qualidigm (H.M.K.), Middletown, Conn; and the Yale-New Haven Hospital Center for Outcomes Research and Evaluation (H.M.K.), New Haven, Conn.
Correspondence to Dr Krumholz, Yale University School of Medicine, Room I-456 SHM, 333 Cedar St, New Haven, CT. E-mail harlan.krumholz{at}yale.edu
| Abstract |
|---|
|
|
|---|
Methods and Results We evaluated the simple risk index using data from 49 711 patients
65 years of age hospitalized with ST-elevation myocardial infarction. We evaluated the distribution of patients in the 5 simple risk index groups, compared observed and published 30-day mortality rates, and assessed the scores discrimination and calibration. The simple risk index provided poor discrimination (c=0.62) and calibration (goodness of fit P<0.001) for survival at 30 days. Risk score distribution was skewed, because two thirds (66.1%) of all patients were classified in the highest-risk group, whereas fewer than 11.0% were classified in the 3 lowest-risk groups. Thirty-day mortality estimates were lower than those observed in the cohort (risk group 2 to 5: 1.9% to 17.4% versus 5.3% to 27.9%). Risk index discrimination, calibration, score distribution, and mortality estimates were worse among patients who did not receive acute reperfusion therapy than among those who did.
Conclusions The limited performance of the simple risk index highlights the limitations of applying prognostic models derived in RCT populations to the general population of patients 65 years and older. Prognostic scores must be validated in community-based cohorts before integration into clinical practice.
Key Words: myocardial infarction prognosis elderly
| Introduction |
|---|
|
|
|---|
Recently, Morrow et al3 proposed a simple risk index based on a patients age, admission heart rate, and admission systolic blood pressure that "is likely to be useful in refining initial risk assessment ... [and] could guide early clinical decision-making" for patients hospitalized with an ST-elevation myocardial infarction. The accompanying editorial indicated the score could facilitate prehospital triage to different cardiac centers and inform choice of reperfusion therapy.4 Derived from patients enrolled in the Intravenous nPA for Treatment of Infarcting Myocardium Early (InTIME II) trial, a randomized controlled trial comparing fibrinolytic agents,5 the simple risk index was validated in the Thrombolysis and Thrombin Inhibition in Myocardial Infarction (TIMI) 9A and 9B trial populations.6,7 In these selected patient groups, the index appeared to provide a simple approach to stratify patients hospitalized with ST-elevation myocardial infarction.
The value of the simple risk index in a typical patient population is not known. The derivation and validation samples included relatively few elderly patients. Furthermore, the simple risk index was derived and validated among patients who received fibrinolytic therapy, whereas in practice, more than 60% of elderly patients with ST-elevation myocardial infarction do not receive this therapy.8 Thus, it is unclear how well the simple risk index would perform in a community-based cohort of elderly patients who are older, have a higher comorbidity burden, and are treated at a more diverse spectrum of hospitals and with a wider variety of treatment strategies than patients enrolled in the InTIME II and TIMI randomized trials.
We assessed the prognostic value of the simple risk index among a community-based cohort of elderly patients hospitalized with a myocardial infarction. Using data from the Cooperative Cardiovascular Project (CCP), we evaluated both the risk stratification and discrimination provided by the simple risk index among elderly patients hospitalized with ST-elevation myocardial infarction. This study was designed to illuminate how well a risk score derived and validated in randomized trial populations, with claims of generalizability to the general population of patients, predicts risk and supports therapeutic decision making in a community-based population of elderly patients hospitalized with myocardial infarction.
| Methods |
|---|
|
|
|---|
Simple Risk Index
The simple risk index is a composite score derived from a combination of a patients age, admission heart rate, and admission systolic blood pressure, such that higher scores indicate a greater risk of short-term mortality. To derive the index, a patients age (in years) is divided by 10; this figure is then squared, multiplied by the patients admission heart rate (in beats per minute), and divided by the patients systolic blood pressure (in mm Hg).3 Patients with an admission heart rate below 50 bpm or above 150 bpm were excluded because these patients would require specific interventions. Applying this same restriction to the CCP, we excluded 2910 patients (5.3%). We similarly excluded 2654 patients (4.8%) with an admission systolic blood pressure <60 mm Hg or an admission systolic blood pressure >200 mm Hg because severe hypotension or hypertension would necessitate a directed intervention regardless of a patients risk index. Thus, of 55 078 eligible patients, 5367 (9.7%) were excluded because of a heart rate or systolic blood pressure that would necessitate prompt intervention; the remaining 49 711 patients constituted the study cohort.
Morrow et al3 identified 5 approximately equal-sized groups of successively increasing risk based on patients risk scores. Patients with a score
12.5 were assigned to risk group 1 (the lowest-risk group); those with a score of >12.5 to 17.5, >17.5 to 22.5, and >22.5 to 30.0 were assigned to risk groups 2, 3, and 4 (the successively higher intermediate-risk groups); and patients with a score >30.0 were assigned to risk group 5 (the highest-risk group). Because the CCP represents a cohort of patients aged 65 years and older, the lowest possible simple risk index score was 10.6 (age 65, heart rate 50 bpm, systolic blood pressure 200 mm Hg). We assumed a maximum possible score of 250 based on a hypothetical patient 100 years of age with a heart rate of 150 bpm and a systolic blood pressure of 60 mm Hg. We agreed a priori to exclude any patients whose scores fell outside of the range of 10.6 to 250; no patient met this criterion.
Statistical Analysis
Simple risk index groups were evaluated for their association with mortality at 30 days after admission by global
2 and
2 test-of-trend analyses. The discriminatory performance of the simple risk index was determined by deriving its c statistic, which represents the area under the receiver operating characteristic curve,10 for prediction of mortality at 30 days after admission from a logistic regression analysis. Risk-score discrimination was assessed with the risk index modeled both as a continuous variable and with the 5 proposed risk groups. Calibration of the simple risk index was assessed by graphical analysis, by the Hosmer and Lemeshow goodness-of-fit test, and by comparison of 30-day mortality rates with published estimates by use of a Pearson
2 test of fit.
Analyses were also repeated with stratification of patients by receipt of reperfusion therapy to determine whether the simple risk index performed similarly among patients who had received reperfusion therapy and those who had not. Distribution of risk scores by receipt of reperfusion therapy was evaluated by a Mann-Whitney test. Differences in simple risk index performance between patients who received reperfusion therapy (overall and by type of therapy) and those who did not were tested with interaction terms in logistic regression models. Analyses were also repeated excluding 27 392 patients with cerebrovascular disease, systolic blood pressure >180 mm Hg, diastolic blood pressure >110 mm Hg, those arriving >6 hours after symptom onset, or those presenting in shock per the enrollment criteria of InTIME II.5 Analyses were conducted with Stata 6.0 (Stata Corp).
| Results |
|---|
|
|
|---|
|
The simple risk index ranged from 12.1 to 174.7, with a median value of 35.2 (Table 2). Nearly two thirds (66.1%) of all patients were classified in group 5 (highest risk). In contrast, only 9 patients were classified in group 1 (lowest risk), 985 patients (2.0%) in group 2, and 4361 patients (8.8%) in group 3 (Figure 1A). Patients who received reperfusion therapy had lower median risk scores (29.6 versus 39.1, P<0.0001) than patients who did not (Figures 1B and 1C).
|
|
The 30-day mortality rate for the study sample was 21.7%. Because only 9 patients were classified as low risk, risk group 1 was not incorporated in evaluations of mortality. The simple risk index identified a significant 5-fold gradient in 30-day mortality between patients in group 2 and group 5 (range 5.3% to 27.9%, P<0.0001 for trend) in the CCP cohort (Figure 2).
|
The simple risk index, however, provided poor calibration; 30-day mortality rates in CCP were markedly higher for all risk groups than those reported in InTIME II, particularly for patients who did not receive reperfusion therapy (Figure 2; P<0.0001 for Hosmer-Lemeshow goodness of fit). Prognostic discrimination was also limited; the c statistic was 0.62 for 30-day mortality when the simple risk index was modeled with the prespecified risk group cutpoints. As would be expected, prognostic discrimination increased modestly (c=0.71) when the simple risk index was modeled as a continuous variable but remained lower than previously reported from InTIME II (c=0.78 for 30-day mortality).3
The prognostic performance of the simple risk index varied on the basis of the patients receipt of reperfusion therapy. Patients who did not receive reperfusion therapy had a slightly less steep mortality gradient across risk groups than patients who were treated. There was a 5-fold increase in mortality from risk group 2 to risk group 5 among patients who received reperfusion therapy (range 4.4% to 21.6%, P=0.001 for trend) but a 4-fold increase in mortality over the same groups among patients who were not treated (range 6.9% to 30.2%, P=0.001 for trend, P<0.0001 for interaction). Discrimination for 30-day mortality provided by the simple risk index groups was worse among those who did not receive reperfusion therapy (c=0.59) than among patients who were treated (c=0.64). Findings did not vary by type of reperfusion therapy among patients who were treated. Results were similar when analyses were repeated among the cohort of patients who met InTIME II enrollment criteria (results not shown).
| Discussion |
|---|
|
|
|---|
The clinical utility of a risk score is determined in part by its ability to ensure a clinically meaningful and balanced distribution of patients across the scores risk gradient. However, the simple risk index provided a poor relative distribution of patients mortality risks; less than 2.0% of the CCP cohort was classified as low risk (groups 1 and 2), and two thirds of patients were classified as high risk (group 5). This skewed distribution makes the relative differentiation of patients mortality risks by the simple risk index impractical.
In addition to ensuring a balanced relative distribution of mortality risks, a risk score must also effectively stratify a population on the basis of its absolute risk of mortality. Thus, a low-risk group should have both a lower relative risk of mortality compared with other patients in the population and an absolute risk of mortality that is sufficiently low as to influence clinical decision making. If one assumes that a 30-day mortality rate of 5.0% or lower could be considered low risk, only the 9 patients (0.02%) in group 1 and the 638 patients (1.28%) in group 2 who received reperfusion therapy would meet this conservative criterion. Thus, the simple risk index cannot identify a low-risk population among elderly patients of sufficient size to meaningfully inform decision making for these patients.
Shortfalls in prognostic discrimination also have implications for clinical practice. The c statistic is a measure of the frequency by which a risk score can accurately differentiate between a patient dying or surviving after myocardial infarction. Models with a c statistic >0.80 are generally assumed to provide discrimination sufficient for predictive use.11 However, we found the c statistic of the simple risk index groups to be 0.62 when applied to the CCP cohort, which is markedly lower than reported in the derivation (c=0.76) and validation (c=0.77) cohorts.3 Poor prognostic discrimination increases the likelihood that patients mortality risks may be misclassified. This may be problematic if high-risk patients are presumed to be low risk (particularly by application of mortality estimates reported from InTIME II) and directed toward a less aggressive treatment strategy.
Poor prognostic calibration and discrimination underscore the limitations of applying estimates of prognosis derived from clinical trial cohorts to real-world populations. Differences between patients recruited to clinical trials and those treated in daily practice are well established.2,12 The magnitude of this difference is best demonstrated by the >15% absolute difference in 30-day mortality rates between patients in CCP and those enrolled in InTIME II (21.7% versus 6.0%). Although systolic blood pressure, age, and heart rate may be sufficient to risk-stratify patients enrolled in a clinical trial population, white blood cell count, serum creatinine, and congestive heart failure are of similar prognostic importance among a general population of elderly patients.13 The significance of these risk factors highlights the extent to which comorbid conditions, in addition to acute coronary injury, determine prognosis among elderly patients hospitalized with myocardial infarction. Because these conditions and other comorbidities are underrepresented in clinical trial cohorts,14 caution must be used when prognostic scores or mortality estimates generated from clinical trial populations are applied to patients in the general clinical population.
The publication of the simple risk score3 and the proliferation of other risk scores for patients with myocardial infarction1 underscores the considerable professional interest in developing methods for readily stratifying patients according to risk at the time of presentation. Although risk stratification may be beneficial after the first 12 or 24 hours when invasive strategies are being considered, it is unclear how risk stratification would change the initial management of a patient presenting with ST-elevation myocardial infarction. Current guidelines specify the prompt provision of reperfusion therapy, aspirin, and ß-blockers for all patients who have no contraindications to such therapies.15 These recommendations are not conditional on a patients risk of mortality and are equally indicated for patients who would be classified as low risk or high risk by the simple risk index or other risk scores. To be effective, a clinical prediction rule should reduce clinical uncertainty and improve physicians therapeutic decision making.16 However, it is unclear how clinically useful the simple risk index would be given that the initial management of ST-elevation myocardial infarction patients is prespecified and independent of patients mortality risk.
The present study has certain issues to consider in its interpretation. We limited our analysis to patients aged 65 years and older and thus cannot determine the performance of the simple risk index among patients younger than 65 years of age. However, elderly patients represent the majority of patients hospitalized with ST-elevation or left bundle-branch block myocardial infarction and bear a disproportionate burden of the mortality among such patients.17 In addition, age is explicitly factored into the simple risk index, which indicates that an older cohort should not influence the calibration of the risk index. Furthermore, the authors of the simple risk index provide no indication that the score should be calculated differently, that the risk score groupings are not applicable, or that the score should not be used in elderly patients. Although this index may have superior prognostic performance in a younger cohort, it has clear limitations when applied to a community-based cohort of elderly patients, which suggests that its wide-scale adoption is unlikely to be effective. Our findings were unchanged when patients were stratified by InTIME II study criteria (a proxy for reperfusion therapy eligibility), which indicates that the decreased performance of the risk score is not simply due to the evaluation of patients ineligible for reperfusion therapy.
In conclusion, we found the simple risk index performs poorly when evaluated in a nationally representative, community-based cohort of elderly patients hospitalized with ST-elevation myocardial infarction. The simple risk index failed to identify a sufficiently low-risk population that would influence clinical decision making, and the mortality estimates reported from the InTIME II cohort markedly underestimated mortality rates, particularly among patients who did not receive reperfusion therapy. The present data illustrate the inherent limitations in the application of prognostic data from randomized trial cohorts to a general patient population. Although we evaluated the simple risk index, the issues we raise concerning the prognostic performance of randomized trialderived risk indices and the clinical utility of risk stratification at the time of admission are clearly generalizable to other risk score systems. Our findings suggest the need for an evaluation of other risk scores in nonselected patient populations before their wide-scale adoption. Furthermore, we believe a discussion of the specific clinical purpose of risk stratification for patients with ST-segment elevation myocardial infarction at the time of admission is also merited to ensure risk stratification improves clinical decision making.
Received September 18, 2002; accepted November 4, 2002.
| References |
|---|
|
|
|---|
2. Lee PY, Alexander KP, Hammill BG, et al. Representation of elderly persons and women in published randomized trials of acute coronary syndromes. JAMA. 2001; 286: 708713.
3. Morrow DA, Antman EM, Giugliano RP, et al. A simple risk index for rapid initial triage of patients with ST-elevation myocardial infarction: an InTIME II substudy. Lancet. 2001; 358: 15711575.[CrossRef][Medline] [Order article via Infotrieve]
4. Gibler WB. Applications of easy score for identifying high risk in acute myocardial infarction. Lancet. 2001; 358: 1566.Editorial.[CrossRef][Medline] [Order article via Infotrieve]
5. The InTIME II Investigators. Intravenous NPA for the treatment of infarcting myocardium early: InTIME-II, a double-blind comparison of single-bolus lanoteplase vs. accelerated alteplase for the treatment of patients with acute myocardial infarction. Eur Heart J. 2000; 21: 20052113.
6. Antman EM. Hirudin in acute myocardial infarction: safety report from the Thrombolysis and Thrombin Inhibition in Myocardial Infarction (TIMI) 9A Trial. Circulation. 1994; 90: 16241630.
7. Antman EM. Hirudin in acute myocardial infarction: Thrombolysis and Thrombin Inhibition in Myocardial Infarction (TIMI) 9B trial. Circulation. 1996; 94: 911921.
8. Berger AK, Radford MJ, Wang Y, et al. Thrombolytic therapy in older patients. J Am Coll Cardiol. 2000; 36: 366374.
9. Marciniak TA, Ellerbeck EF, Radford MJ, et al. Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project. JAMA. 1998; 279: 13511357.
10. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143: 2936.
11. Harrell FEJ. Regression Modeling Strategies With Applications to Linear Models, Logistic Regression, Survival Analysis. New York, NY: Springer; 2001.
12. Jha P, Deboer D, Sykora K, et al. Characteristics and mortality outcomes of thrombolysis trial participants and nonparticipants: a population-based comparison. J Am Coll Cardiol. 1996; 27: 13351342.[Abstract]
13. Krumholz HM, Chen J, Wang Y, et al. Comparing AMI mortality among hospitals in patients 65 years of age and older: evaluating methods of risk adjustment. Circulation. 1999; 99: 29862992.
14. McNeil BJ. Shattuck Lecture: hidden barriers to improvement in the quality of care. N Engl J Med. 2001; 345: 16121620.
15. Ryan TJ, Antman EM, Brooks NH, et al. Update: ACC/AHA guidelines for the management of patients with acute myocardial infarction: executive summary and recommendations: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Management of Acute Myocardial Infarction). Circulation. 1999; 100: 10161030.
16. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules: a review and suggested modifications of methodological standards. JAMA. 1997; 277: 488494.
17. Miller WL, Sgura FA, Kopecky SL, et al. Characteristics of presenting electrocardiograms of acute myocardial infarction from a community-based population predict short- and long-term mortality. Am J Cardiol. 2001; 87: 10451050.[CrossRef][Medline] [Order article via Infotrieve]
This article has been cited by other articles:
![]() |
C P Gale, S O M Manda, C F Weston, J S Birkhead, P D Batin, and A S Hall Evaluation of risk scores for risk stratification of acute coronary syndromes in the Myocardial Infarction National Audit Project (MINAP) database Heart, February 1, 2009; 95(3): 221 - 227. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. S. Alpert A Plethora of Prognostic Pearls Circulation, September 23, 2008; 118(13): 1312 - 1313. [Full Text] [PDF] |
||||
![]() |
D. N. Reddan, L. A. Szczech, V. Hasselblad, E. G. Lowrie, R. M. Lindsay, J. Himmelfarb, R. D. Toto, J. Stivelman, J. F. Winchester, L. A. Zillman, et al. Intradialytic Blood Volume Monitoring in Ambulatory Hemodialysis Patients: A Randomized Trial J. Am. Soc. Nephrol., July 1, 2005; 16(7): 2162 - 2169. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. D. Wiviott, D. A. Morrow, P. D. Frederick, R. P. Giugliano, C.M. Gibson, C. H. McCabe, C. P. Cannon, E. M. Antman, and E. Braunwald Performance of the thrombolysis in myocardial infarction risk index in the National Registry of Myocardial Infarction-3 and -4: A simple index that predicts mortality in ST-segment elevation myocardial infarction J. Am. Coll. Cardiol., August 18, 2004; 44(4): 783 - 789. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. J. Ryan The thrombolysis in myocardial infarction risk index: A formula with a future J. Am. Coll. Cardiol., August 18, 2004; 44(4): 790 - 792. [Full Text] [PDF] |
||||
![]() |
K. A. Eagle, M. J. Lim, O. H. Dabbous, K. S. Pieper, R. J. Goldberg, F. Van de Werf, S. G. Goodman, C. B. Granger, P. G. Steg, J. M. Gore, et al. A Validated Prediction Model for All Forms of Acute Coronary Syndrome: Estimating the Risk of 6-Month Postdischarge Death in an International Registry JAMA, June 9, 2004; 291(22): 2727 - 2733. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. Das, R. Lawrance, A. Hall, S. S. Rathore, K. P. Weinfurt, C. P. Gross, and H. Krumholtz Validity of a Simple ST-Elevation Acute Myocardial Infarction Risk Index: Are Randomized Trial Prognostic Estimates Generalizable to Elderly Patients? * Response Circulation, July 8, 2003; 108 (1): e9 - e10. [Full Text] [PDF] |
||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Circulation Home | Subscriptions | Archives | Feedback | Authors | Help | AHA Journals Home | Search Copyright © 2003 American Heart Association, Inc. All rights reserved. Unauthorized use prohibited. |