Circulation. 2007;115:928-935
doi: 10.1161/CIRCULATIONAHA.106.672402
doi: 10.1161/CIRCULATIONAHA.106.672402
(Circulation. 2007;115:928-935.)
© 2007 American Heart Association, Inc.
Special Report |
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
From the Division of Preventive Medicine, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, Mass, and the Department of Epidemiology, Harvard School of Public Health, Boston, Mass.
Correspondence to Dr Nancy R. Cook, Division of Preventive Medicine, Brigham and Women’s Hospital, 900 Commonwealth Ave East, Boston, MA 02215. E-mail ncook{at}rics.bwh.harvard.edu
The c statistic, or area under the receiver operating characteristic (ROC) curve, achieved popularity in diagnostic testing, in which the test characteristics of sensitivity and specificity are relevant to discriminating diseased versus nondiseased patients. The c statistic, however, may not be optimal in assessing models that predict future risk or stratify individuals into risk categories. In this setting, calibration is as important to the accurate assessment of risk. For example, a biomarker with an odds ratio of 3 may have little effect on the c statistic, yet an increased level could shift estimated 10-year cardiovascular risk for an individual patient from 8% to 24%, which would lead to different treatment recommendations under current Adult Treatment Panel III guidelines. Accepted risk factors such as lipids, hypertension, and smoking have only marginal impact on the c statistic individually yet lead to more accurate reclassification of large proportions of patients into higher-risk or lower-risk categories. Perfectly calibrated models for complex disease can, in fact, only achieve values for the cstatistic well below the theoretical maximum of 1. Use of the c statistic for model selection could thus naively eliminate established risk factors from cardiovascular risk prediction scores. As novel risk factors are discovered, sole reliance on the c statistic to evaluate their utility as risk predictors thus seems ill-advised.
Key Words: cardiovascular diseases • epidemiology • follow-up studies • prevention • risk factors • statistics • risk
This article has been cited by other articles:
 |
|
 |
|
  M. S. Pepe and H. E. Janes Gauging the Performance of SNPs, Biomarkers, and Clinical Factors for Predicting Risk of Breast Cancer J Natl Cancer Inst, July 16, 2008; 100(14): 978 - 979. [Full Text] [PDF]
|
|
|
|
 |
|
 J.-P. Empana, F. Canoui-Poitrine, G. Luc, I. Juhan-Vague, P. Morange, D. Arveiler, J. Ferrieres, P. Amouyel, A. Bingham, M. Montaye, et al. Contribution of novel biomarkers to incident stable angina and acute coronary syndrome: the PRIME Study Eur. Heart J., July 10, 2008; (2008) ehn331v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 Ankle Brachial Index Collaboration Ankle Brachial Index Combined With Framingham Risk Score to Predict Cardiovascular Events and Mortality: A Meta-analysis JAMA, July 9, 2008; 300(2): 197 - 208. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. E. Gerszten, F. Accurso, G. R. Bernard, R. M. Caprioli, E. W. Klee, G. G. Klee, I. Kullo, T. A. Laguna, F. P. Roth, M. Sabatine, et al. Challenges in translating plasma proteomics from bench to bedside: update from the NHLBI Clinical Proteomics Programs Am J Physiol Lung Cell Mol Physiol, July 1, 2008; 295(1): L16 - L22. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. K. Simmons, S. Sharp, S. M. Boekholdt, L. A. Sargeant, K.-T. Khaw, N. J. Wareham, and S. J. Griffin Evaluation of the Framingham Risk Score in the European Prospective Investigation of Cancer-Norfolk Cohort: Does Adding Glycated Hemoglobin Improve the Prediction of Coronary Heart Disease Events? Arch Intern Med, June 9, 2008; 168(11): 1209 - 1216. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. Zethelius, L. Berglund, J. Sundstrom, E. Ingelsson, S. Basu, A. Larsson, P. Venge, and J. Arnlov Use of Multiple Biomarkers to Improve the Prediction of Death from Cardiovascular Causes N. Engl. J. Med., May 15, 2008; 358(20): 2107 - 2116. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. A. de Lemos and D. M. Lloyd-Jones Multiple Biomarker Panels for Cardiovascular Risk Assessment N. Engl. J. Med., May 15, 2008; 358(20): 2172 - 2174. [Full Text] [PDF]
|
|
|
|
|
|
R. D. Brook Potential health risks of air pollution beyond triggering acute cardiopulmonary events. JAMA, May 14, 2008; 299(18): 2194 - 2196. [Full Text] [PDF]
|
|
|
|
|
|
M. H. Gail, L. Bernstein, J. P. Costantino, D. Pee, and G. Ursin Response:Re: Projecting Individualized Absolute Invasive Breast Cancer Risk in African American Women J Natl Cancer Inst, May 7, 2008; 100(9): 684 - 684. [Full Text] [PDF]
|
|
|
|
|
|
S. S. Levinson New cardiovascular risk determinants do exist and are clinically useful Eur. Heart J., May 2, 2008; 29(10): 1335 - 1335. [Full Text] [PDF]
|
|
|
|
 |
|
 T. Leong, D. Zylberstein, I. Graham, L. Lissner, D. Ward, J. Fogarty, C. Bengtsson, C. Bjorkelund, D. Thelle, and for The Swedish-Irish-Norwegian (SIN) Collaboratio Asymmetric Dimethylarginine Independently Predicts Fatal and Nonfatal Myocardial Infarction and Stroke in Women: 24-Year Follow-Up of the Population Study of Women in Gothenburg Arterioscler. Thromb. Vasc. Biol., May 1, 2008; 28(5): 961 - 967. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. Palmas, T. G. Pickering, J. Teresi, J. E. Schwartz, L. Field, R. S. Weinstock, and S. Shea Telemedicine Home Blood Pressure Measurements and Progression of Albuminuria in Elderly People With Diabetes Hypertension, May 1, 2008; 51(5): 1282 - 1288. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. Kurth and G. Logroscino The Metabolic Syndrome: More Than the Sum of Its Components? Stroke, April 1, 2008; 39(4): 1068 - 1069. [Full Text] [PDF]
|
|
|
|
|
|
S. Kathiresan, O. Melander, D. Anevski, C. Guiducci, N. P. Burtt, C. Roos, J. N. Hirschhorn, G. Berglund, B. Hedblad, L. Groop, et al. Polymorphisms Associated with Cholesterol and Risk of Cardiovascular Events N. Engl. J. Med., March 20, 2008; 358(12): 1240 - 1249. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. A. Tice, S. R. Cummings, R. Smith-Bindman, L. Ichikawa, W. E. Barlow, and K. Kerlikowske Using Clinical Factors and Mammographic Breast Density to Estimate Breast Cancer Risk: Development and Validation of a New Predictive Model Ann Intern Med, March 4, 2008; 148(5): 337 - 347. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Emmerich and P. M Ridker Can Fishing for New Genes Catch Patients at Risk of Coronary Artery Disease? Clin. Chem., March 1, 2008; 54(3): 453 - 455. [Full Text] [PDF]
|
|
|
|
|
|
P. J. Talmud, J. A. Cooper, J. Palmen, R. Lovering, F. Drenos, A. D. Hingorani, and S. E. Humphries Chromosome 9p21.3 Coronary Heart Disease Locus Genotype and Prospective Risk of CHD in Healthy Middle-Aged Men Clin. Chem., March 1, 2008; 54(3): 467 - 474. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. C. Fang, A. S. Go, Y. Chang, L. Borowsky, N. K. Pomernacki, D. E. Singer, and for the ATRIA Study Group Comparison of Risk Stratification Schemes to Predict Thromboembolism in People With Nonvalvular Atrial Fibrillation J. Am. Coll. Cardiol., February 26, 2008; 51(8): 810 - 815. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
Y. M. Smulders, A. Thijs, and J. W. Twisk New cardiovascular risk determinants do exist and are clinically useful Eur. Heart J., February 2, 2008; 29(4): 436 - 440. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. E. Dewey, J. R. Kapoor, R. S. Williams, M. J. Lipinski, E. A. Ashley, D. Hadley, J. Myers, and V. F. Froelicher Ventricular Arrhythmias During Clinical Treadmill Testing and Prognosis Arch Intern Med, January 28, 2008; 168(2): 225 - 234. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. R. Cook Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve Clin. Chem., January 1, 2008; 54(1): 17 - 23. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. W. Franks, R. L. Hanson, W. C. Knowler, C. Moffett, G. Enos, A. M. Infante, J. Krakoff, and H. C. Looker Childhood Predictors of Young-Onset Type 2 Diabetes Diabetes, December 1, 2007; 56(12): 2964 - 2972. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Loscalzo Association Studies in an Era of Too Much Information: Clinical Analysis of New Biomarker and Genetic Data Circulation, October 23, 2007; 116(17): 1866 - 1870. [Full Text] [PDF]
|
|
|
|
|
|
T. Kempf, S. von Haehling, T. Peter, T. Allhoff, M. Cicoira, W. Doehner, P. Ponikowski, G. S. Filippatos, P. Rozentryt, H. Drexler, et al. Prognostic Utility of Growth Differentiation Factor-15 in Patients With Chronic Heart Failure J. Am. Coll. Cardiol., September 11, 2007; 50(11): 1054 - 1060. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. Latini, S. Masson, I. S. Anand, E. Missov, M. Carlson, T. Vago, L. Angelici, S. Barlera, G. Parrinello, A. P. Maggioni, et al. Prognostic Value of Very Low Plasma Concentrations of Troponin T in Patients With Stable Chronic Heart Failure Circulation, September 11, 2007; 116(11): 1242 - 1249. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. G. Hackam Predicting Stroke Risk in Patients With Atrial Fibrillation Stroke, September 1, 2007; 38(9): 2409 - 2409. [Full Text] [PDF]
|
|
|
|
|
|
M. S. Pepe, H. Janes, and J. W. Gu Letter by Pepe et al Regarding Article, "Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction" Circulation, August 7, 2007; 116(6): e132 - e132. [Full Text] [PDF]
|
|
|
|
|
|
S.-J. Janket, Y. Shen, and J. H. Meurman Letter by Janket et al Regarding Article, "Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction" Circulation, August 7, 2007; 116(6): e133 - e133. [Full Text] [PDF]
|
|
|
|
|
|
W. Koenig Cardiovascular Biomarkers: Added Value With an Integrated Approach? Circulation, July 3, 2007; 116(1): 3 - 5. [Full Text] [PDF]
|
|
|
|
|
|
J. J. Cao, A. M. Arnold, T. A. Manolio, J. F. Polak, B. M. Psaty, C. H. Hirsch, L. H. Kuller, and M. Cushman Association of Carotid Artery Intima-Media Thickness, Plaques, and C-Reactive Protein With Future Cardiovascular Disease and All-Cause Mortality: The Cardiovascular Health Study Circulation, July 3, 2007; 116(1): 32 - 38. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. M. Ridker C-Reactive Protein and the Prediction of Cardiovascular Events Among Those at Intermediate Risk: Moving an Inflammatory Hypothesis Toward Consensus J. Am. Coll. Cardiol., May 29, 2007; 49(21): 2129 - 2138. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. M Ridker and B. M. Everett Letter by Ridker and Everett Regarding Article, "The Inflammatory Hypothesis: Any Progress in Risk Stratification and Therapeutic Targets?" Circulation, May 22, 2007; 115(20): e475 - e475. [Full Text] [PDF]
|
|
|
|
|
|
M. Berkwits and E. Guallar Risk Factors, Risk Prediction, and the Apolipoprotein B-Apolipoprotein A-I Ratio Ann Intern Med, May 1, 2007; 146(9): 677 - 679. [Full Text] [PDF]
|
|
|
|
|
|
K. Musunuru, R. S. Blumenthal, P. M Ridker, N. R. Cook, D. M. Becker, S. Mora, D. C. Goff Jr., R. H. Fletcher, S. W. Fletcher, G. Mints, et al. Biomarkers for Prediction of Cardiovascular Events N. Engl. J. Med., April 5, 2007; 356(14): 1472 - 1475. [Full Text] [PDF]
|
|
|
|
|
|
D. A. Morrow and J. A. de Lemos Benchmarks for the Assessment of Novel Cardiovascular Biomarkers Circulation, February 27, 2007; 115(8): 949 - 952. [Full Text] [PDF]
|
|