Validity of a Simple ST-Elevation Acute Myocardial Infarction Risk Index
Are Randomized Trial Prognostic Estimates Generalizable to Elderly Patients?
Background— Risk-stratification scores derived from randomized clinical trial (RCT) data should be evaluated in community-based populations. A simple risk-stratification index for patients with ST-segment elevation myocardial infarction derived from an RCT population was recently proposed, but it has not been validated in a community-based cohort.
Methods and Results— We evaluated the simple risk index using data from 49 711 patients ≥65 years of age hospitalized with ST-elevation myocardial infarction. We evaluated the distribution of patients in the 5 simple risk index groups, compared observed and published 30-day mortality rates, and assessed the score’s discrimination and calibration. The simple risk index provided poor discrimination (c=0.62) and calibration (goodness of fit P<0.001) for survival at 30 days. Risk score distribution was skewed, because two thirds (66.1%) of all patients were classified in the highest-risk group, whereas fewer than 11.0% were classified in the 3 lowest-risk groups. Thirty-day mortality estimates were lower than those observed in the cohort (risk group 2 to 5: 1.9% to 17.4% versus 5.3% to 27.9%). Risk index discrimination, calibration, score distribution, and mortality estimates were worse among patients who did not receive acute reperfusion therapy than among those who did.
Conclusions— The limited performance of the simple risk index highlights the limitations of applying prognostic models derived in RCT populations to the general population of patients 65 years and older. Prognostic scores must be validated in community-based cohorts before integration into clinical practice.
Received September 18, 2002; accepted November 4, 2002.
Risk-assessment tools have been proposed to assist medical decision making for patients hospitalized with a myocardial infarction.1 Many risk scores are derived from and validated in randomized controlled trial cohorts or similarly selected patient populations. This approach may overestimate the performance of these risk scores when applied to patients typically seen in clinical practice. Of particular concern is the applicability of such scores to elderly patients, who despite making up the majority of patients who experience a myocardial infarction are often not well represented in randomized trials of myocardial infarction therapies.2
Recently, Morrow et al3 proposed a simple risk index based on a patient’s age, admission heart rate, and admission systolic blood pressure that “is likely to be useful in refining initial risk assessment … [and] could guide early clinical decision-making” for patients hospitalized with an ST-elevation myocardial infarction. The accompanying editorial indicated the score could facilitate prehospital triage to different cardiac centers and inform choice of reperfusion therapy.4 Derived from patients enrolled in the Intravenous nPA for Treatment of Infarcting Myocardium Early (InTIME II) trial, a randomized controlled trial comparing fibrinolytic agents,5 the simple risk index was validated in the Thrombolysis and Thrombin Inhibition in Myocardial Infarction (TIMI) 9A and 9B trial populations.6,7 In these selected patient groups, the index appeared to provide a simple approach to stratify patients hospitalized with ST-elevation myocardial infarction.
The value of the simple risk index in a typical patient population is not known. The derivation and validation samples included relatively few elderly patients. Furthermore, the simple risk index was derived and validated among patients who received fibrinolytic therapy, whereas in practice, more than 60% of elderly patients with ST-elevation myocardial infarction do not receive this therapy.8 Thus, it is unclear how well the simple risk index would perform in a community-based cohort of elderly patients who are older, have a higher comorbidity burden, and are treated at a more diverse spectrum of hospitals and with a wider variety of treatment strategies than patients enrolled in the InTIME II and TIMI randomized trials.
We assessed the prognostic value of the simple risk index among a community-based cohort of elderly patients hospitalized with a myocardial infarction. Using data from the Cooperative Cardiovascular Project (CCP), we evaluated both the risk stratification and discrimination provided by the simple risk index among elderly patients hospitalized with ST-elevation myocardial infarction. This study was designed to illuminate how well a risk score derived and validated in randomized trial populations, with claims of generalizability to the general population of patients, predicts risk and supports therapeutic decision making in a community-based population of elderly patients hospitalized with myocardial infarction.
The CCP consisted of a retrospective medical record review of Medicare fee-for-service beneficiaries discharged from US acute-care hospitals with a primary discharge diagnosis of myocardial infarction (International Classification of Disease, Ninth Revision [ICD-9] code 410) between January 1994 and February 1996, with the exception of myocardial infarction readmissions (ICD-9 code 410.x2).9 We limited our analysis to patients 65 years of age and older hospitalized with an ST-elevation infarction or a myocardial infarction with left bundle-branch block. We excluded patients <65 years of age (n=17 593) because these patients enrolled in Medicare because of disability, renal failure, or other criteria that make them unrepresentative of patients under the age of 65. Patients without a clinically confirmed infarction (n=31 186), those with readmissions for myocardial infarction (n=23 773), and patients without ST-elevation or a left bundle-branch block (n=161 458) were also excluded. Patients who were admitted by means of interhospital transfer (n=42 177) were excluded because their admission characteristics could not be evaluated. In addition, 340 patients were excluded because we could not verify their survival status. In total, 181 931 cases met 1 or more of the above exclusion criteria; the remaining 55 538 patients composed the baseline sample. We restricted analysis to patients with complete data for age, admission heart rate, and admission systolic blood pressure (n=55 078, 99.2%).
Simple Risk Index
The simple risk index is a composite score derived from a combination of a patient’s age, admission heart rate, and admission systolic blood pressure, such that higher scores indicate a greater risk of short-term mortality. To derive the index, a patient’s age (in years) is divided by 10; this figure is then squared, multiplied by the patient’s admission heart rate (in beats per minute), and divided by the patient’s systolic blood pressure (in mm Hg).3 Patients with an admission heart rate below 50 bpm or above 150 bpm were excluded because these patients would require specific interventions. Applying this same restriction to the CCP, we excluded 2910 patients (5.3%). We similarly excluded 2654 patients (4.8%) with an admission systolic blood pressure <60 mm Hg or an admission systolic blood pressure >200 mm Hg because severe hypotension or hypertension would necessitate a directed intervention regardless of a patient’s risk index. Thus, of 55 078 eligible patients, 5367 (9.7%) were excluded because of a heart rate or systolic blood pressure that would necessitate prompt intervention; the remaining 49 711 patients constituted the study cohort.
Morrow et al3 identified 5 approximately equal-sized groups of successively increasing risk based on patients’ risk scores. Patients with a score ≤12.5 were assigned to risk group 1 (the lowest-risk group); those with a score of >12.5 to 17.5, >17.5 to 22.5, and >22.5 to 30.0 were assigned to risk groups 2, 3, and 4 (the successively higher intermediate-risk groups); and patients with a score >30.0 were assigned to risk group 5 (the highest-risk group). Because the CCP represents a cohort of patients aged 65 years and older, the lowest possible simple risk index score was 10.6 (age 65, heart rate 50 bpm, systolic blood pressure 200 mm Hg). We assumed a maximum possible score of 250 based on a hypothetical patient 100 years of age with a heart rate of 150 bpm and a systolic blood pressure of 60 mm Hg. We agreed a priori to exclude any patients whose scores fell outside of the range of 10.6 to 250; no patient met this criterion.
Simple risk index groups were evaluated for their association with mortality at 30 days after admission by global χ2 and χ2 test-of-trend analyses. The discriminatory performance of the simple risk index was determined by deriving its c statistic, which represents the area under the receiver operating characteristic curve,10 for prediction of mortality at 30 days after admission from a logistic regression analysis. Risk-score discrimination was assessed with the risk index modeled both as a continuous variable and with the 5 proposed risk groups. Calibration of the simple risk index was assessed by graphical analysis, by the Hosmer and Lemeshow goodness-of-fit test, and by comparison of 30-day mortality rates with published estimates by use of a Pearson χ2 test of fit.
Analyses were also repeated with stratification of patients by receipt of reperfusion therapy to determine whether the simple risk index performed similarly among patients who had received reperfusion therapy and those who had not. Distribution of risk scores by receipt of reperfusion therapy was evaluated by a Mann-Whitney test. Differences in simple risk index performance between patients who received reperfusion therapy (overall and by type of therapy) and those who did not were tested with interaction terms in logistic regression models. Analyses were also repeated excluding 27 392 patients with cerebrovascular disease, systolic blood pressure >180 mm Hg, diastolic blood pressure >110 mm Hg, those arriving >6 hours after symptom onset, or those presenting in shock per the enrollment criteria of InTIME II.5 Analyses were conducted with Stata 6.0 (Stata Corp).
Compared with patients in InTIME II, patients in CCP were on average older; a greater proportion were nonwhite, female, hypertensive, and diabetic; presented with an anterior infarction or left bundle-branch block; and more often had prior cardiovascular disease. Of the 49 711 patients in the study cohort, 18 089 (36.4%) received reperfusion therapy; 15 395 patients (85.1% of patients receiving reperfusion therapy) received fibrinolytic therapy, and 2694 received a primary PTCA. Patients who received reperfusion therapy were younger, were more likely to be male and white, and had fewer comorbid conditions and less severe presentation at admission than patients who did not receive reperfusion therapy (Table 1).
The simple risk index ranged from 12.1 to 174.7, with a median value of 35.2 (Table 2). Nearly two thirds (66.1%) of all patients were classified in group 5 (highest risk). In contrast, only 9 patients were classified in group 1 (lowest risk), 985 patients (2.0%) in group 2, and 4361 patients (8.8%) in group 3 (Figure 1A). Patients who received reperfusion therapy had lower median risk scores (29.6 versus 39.1, P<0.0001) than patients who did not (Figures 1B and 1C).
The 30-day mortality rate for the study sample was 21.7%. Because only 9 patients were classified as low risk, risk group 1 was not incorporated in evaluations of mortality. The simple risk index identified a significant 5-fold gradient in 30-day mortality between patients in group 2 and group 5 (range 5.3% to 27.9%, P<0.0001 for trend) in the CCP cohort (Figure 2).
The simple risk index, however, provided poor calibration; 30-day mortality rates in CCP were markedly higher for all risk groups than those reported in InTIME II, particularly for patients who did not receive reperfusion therapy (Figure 2; P<0.0001 for Hosmer-Lemeshow goodness of fit). Prognostic discrimination was also limited; the c statistic was 0.62 for 30-day mortality when the simple risk index was modeled with the prespecified risk group cutpoints. As would be expected, prognostic discrimination increased modestly (c=0.71) when the simple risk index was modeled as a continuous variable but remained lower than previously reported from InTIME II (c=0.78 for 30-day mortality).3
The prognostic performance of the simple risk index varied on the basis of the patient’s receipt of reperfusion therapy. Patients who did not receive reperfusion therapy had a slightly less steep mortality gradient across risk groups than patients who were treated. There was a 5-fold increase in mortality from risk group 2 to risk group 5 among patients who received reperfusion therapy (range 4.4% to 21.6%, P=0.001 for trend) but a 4-fold increase in mortality over the same groups among patients who were not treated (range 6.9% to 30.2%, P=0.001 for trend, P<0.0001 for interaction). Discrimination for 30-day mortality provided by the simple risk index groups was worse among those who did not receive reperfusion therapy (c=0.59) than among patients who were treated (c=0.64). Findings did not vary by type of reperfusion therapy among patients who were treated. Results were similar when analyses were repeated among the cohort of patients who met InTIME II enrollment criteria (results not shown).
The simple risk index has limited utility in a community-based cohort of elderly patients. Although higher risk scores were associated with increased 30-day mortality rates, the simple risk index exhibited a skewed distribution because fewer than 11% of patients were classified in the 3 “low-risk” groups. With 30-day mortality rates of 5.3% among the lowest-risk patients, the simple risk index did not identify a sufficiently low-risk population that could influence clinical decision making. Prognostic calibration was also poor because the published 30-day mortality rates underestimated mortality rates in the CCP cohort, particularly among patients who did not receive reperfusion therapy. Thus, the simple risk index, despite incorporating age into its calculation, has limited value in the management of “real-world” elderly patients hospitalized with ST-elevation myocardial infarction.
The clinical utility of a risk score is determined in part by its ability to ensure a clinically meaningful and balanced distribution of patients across the score’s risk gradient. However, the simple risk index provided a poor relative distribution of patients’ mortality risks; less than 2.0% of the CCP cohort was classified as low risk (groups 1 and 2), and two thirds of patients were classified as high risk (group 5). This skewed distribution makes the relative differentiation of patients’ mortality risks by the simple risk index impractical.
In addition to ensuring a balanced relative distribution of mortality risks, a risk score must also effectively stratify a population on the basis of its absolute risk of mortality. Thus, a low-risk group should have both a lower relative risk of mortality compared with other patients in the population and an absolute risk of mortality that is sufficiently low as to influence clinical decision making. If one assumes that a 30-day mortality rate of 5.0% or lower could be considered low risk, only the 9 patients (0.02%) in group 1 and the 638 patients (1.28%) in group 2 who received reperfusion therapy would meet this conservative criterion. Thus, the simple risk index cannot identify a low-risk population among elderly patients of sufficient size to meaningfully inform decision making for these patients.
Shortfalls in prognostic discrimination also have implications for clinical practice. The c statistic is a measure of the frequency by which a risk score can accurately differentiate between a patient dying or surviving after myocardial infarction. Models with a c statistic >0.80 are generally assumed to provide discrimination sufficient for predictive use.11 However, we found the c statistic of the simple risk index groups to be 0.62 when applied to the CCP cohort, which is markedly lower than reported in the derivation (c=0.76) and validation (c=0.77) cohorts.3 Poor prognostic discrimination increases the likelihood that patients’ mortality risks may be misclassified. This may be problematic if high-risk patients are presumed to be low risk (particularly by application of mortality estimates reported from InTIME II) and directed toward a less aggressive treatment strategy.
Poor prognostic calibration and discrimination underscore the limitations of applying estimates of prognosis derived from clinical trial cohorts to real-world populations. Differences between patients recruited to clinical trials and those treated in daily practice are well established.2,12 The magnitude of this difference is best demonstrated by the >15% absolute difference in 30-day mortality rates between patients in CCP and those enrolled in InTIME II (21.7% versus 6.0%). Although systolic blood pressure, age, and heart rate may be sufficient to risk-stratify patients enrolled in a clinical trial population, white blood cell count, serum creatinine, and congestive heart failure are of similar prognostic importance among a general population of elderly patients.13 The significance of these risk factors highlights the extent to which comorbid conditions, in addition to acute coronary injury, determine prognosis among elderly patients hospitalized with myocardial infarction. Because these conditions and other comorbidities are underrepresented in clinical trial cohorts,14 caution must be used when prognostic scores or mortality estimates generated from clinical trial populations are applied to patients in the general clinical population.
The publication of the simple risk score3 and the proliferation of other risk scores for patients with myocardial infarction1 underscores the considerable professional interest in developing methods for readily stratifying patients according to risk at the time of presentation. Although risk stratification may be beneficial after the first 12 or 24 hours when invasive strategies are being considered, it is unclear how risk stratification would change the initial management of a patient presenting with ST-elevation myocardial infarction. Current guidelines specify the prompt provision of reperfusion therapy, aspirin, and β-blockers for all patients who have no contraindications to such therapies.15 These recommendations are not conditional on a patient’s risk of mortality and are equally indicated for patients who would be classified as low risk or high risk by the simple risk index or other risk scores. To be effective, a clinical prediction rule should reduce clinical uncertainty and improve physicians’ therapeutic decision making.16 However, it is unclear how clinically useful the simple risk index would be given that the initial management of ST-elevation myocardial infarction patients is prespecified and independent of patients’ mortality risk.
The present study has certain issues to consider in its interpretation. We limited our analysis to patients aged 65 years and older and thus cannot determine the performance of the simple risk index among patients younger than 65 years of age. However, elderly patients represent the majority of patients hospitalized with ST-elevation or left bundle-branch block myocardial infarction and bear a disproportionate burden of the mortality among such patients.17 In addition, age is explicitly factored into the simple risk index, which indicates that an older cohort should not influence the calibration of the risk index. Furthermore, the authors of the simple risk index provide no indication that the score should be calculated differently, that the risk score groupings are not applicable, or that the score should not be used in elderly patients. Although this index may have superior prognostic performance in a younger cohort, it has clear limitations when applied to a community-based cohort of elderly patients, which suggests that its wide-scale adoption is unlikely to be effective. Our findings were unchanged when patients were stratified by InTIME II study criteria (a proxy for reperfusion therapy eligibility), which indicates that the decreased performance of the risk score is not simply due to the evaluation of patients ineligible for reperfusion therapy.
In conclusion, we found the simple risk index performs poorly when evaluated in a nationally representative, community-based cohort of elderly patients hospitalized with ST-elevation myocardial infarction. The simple risk index failed to identify a sufficiently low-risk population that would influence clinical decision making, and the mortality estimates reported from the InTIME II cohort markedly underestimated mortality rates, particularly among patients who did not receive reperfusion therapy. The present data illustrate the inherent limitations in the application of prognostic data from randomized trial cohorts to a general patient population. Although we evaluated the simple risk index, the issues we raise concerning the prognostic performance of randomized trial–derived risk indices and the clinical utility of risk stratification at the time of admission are clearly generalizable to other risk score systems. Our findings suggest the need for an evaluation of other risk scores in nonselected patient populations before their wide-scale adoption. Furthermore, we believe a discussion of the specific clinical purpose of risk stratification for patients with ST-segment elevation myocardial infarction at the time of admission is also merited to ensure risk stratification improves clinical decision making.
Morrow DA, Antman EM, Charlesworth A, et al. TIMI risk score for ST-elevation myocardial infarction: a convenient, bedside, clinical score for risk assessment at presentation. Circulation. 2000; 102: 2031–2037.
The InTIME II Investigators. Intravenous NPA for the treatment of infarcting myocardium early: InTIME-II, a double-blind comparison of single-bolus lanoteplase vs. accelerated alteplase for the treatment of patients with acute myocardial infarction. Eur Heart J. 2000; 21: 2005–2113.
Antman EM. Hirudin in acute myocardial infarction: safety report from the Thrombolysis and Thrombin Inhibition in Myocardial Infarction (TIMI) 9A Trial. Circulation. 1994; 90: 1624–1630.
Antman EM. Hirudin in acute myocardial infarction: Thrombolysis and Thrombin Inhibition in Myocardial Infarction (TIMI) 9B trial. Circulation. 1996; 94: 911–921.
Harrell FEJ. Regression Modeling Strategies With Applications to Linear Models, Logistic Regression, Survival Analysis. New York, NY: Springer; 2001.
Krumholz HM, Chen J, Wang Y, et al. Comparing AMI mortality among hospitals in patients 65 years of age and older: evaluating methods of risk adjustment. Circulation. 1999; 99: 2986–2992.
Ryan TJ, Antman EM, Brooks NH, et al. Update: ACC/AHA guidelines for the management of patients with acute myocardial infarction: executive summary and recommendations: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Management of Acute Myocardial Infarction). Circulation. 1999; 100: 1016–1030.