Donate Help Contact The AHA Sign In Home
American Heart Association
Circulation
Search: search_blue_button Advanced Search
Circulation. 2007;115:928-935
doi: 10.1161/CIRCULATIONAHA.106.672402
This Article
Right arrow Full Text
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow Request Permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Cook, N. R.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Cook, N. R.
Related Collections
Right arrow Epidemiology

(Circulation. 2007;115:928-935.)
© 2007 American Heart Association, Inc.


Special Report

Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction

Nancy R. Cook, ScD

From the Division of Preventive Medicine, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, Mass, and the Department of Epidemiology, Harvard School of Public Health, Boston, Mass.

Correspondence to Dr Nancy R. Cook, Division of Preventive Medicine, Brigham and Women’s Hospital, 900 Commonwealth Ave East, Boston, MA 02215. E-mail ncook{at}rics.bwh.harvard.edu

The c statistic, or area under the receiver operating characteristic (ROC) curve, achieved popularity in diagnostic testing, in which the test characteristics of sensitivity and specificity are relevant to discriminating diseased versus nondiseased patients. The c statistic, however, may not be optimal in assessing models that predict future risk or stratify individuals into risk categories. In this setting, calibration is as important to the accurate assessment of risk. For example, a biomarker with an odds ratio of 3 may have little effect on the c statistic, yet an increased level could shift estimated 10-year cardiovascular risk for an individual patient from 8% to 24%, which would lead to different treatment recommendations under current Adult Treatment Panel III guidelines. Accepted risk factors such as lipids, hypertension, and smoking have only marginal impact on the c statistic individually yet lead to more accurate reclassification of large proportions of patients into higher-risk or lower-risk categories. Perfectly calibrated models for complex disease can, in fact, only achieve values for the cstatistic well below the theoretical maximum of 1. Use of the c statistic for model selection could thus naively eliminate established risk factors from cardiovascular risk prediction scores. As novel risk factors are discovered, sole reliance on the c statistic to evaluate their utility as risk predictors thus seems ill-advised.


Key Words: cardiovascular diseases • epidemiology • follow-up studies • prevention • risk factors • statistics • risk




This article has been cited by other articles:


Home page
JNCI J Natl Cancer InstHome page
M. S. Pepe and H. E. Janes
Gauging the Performance of SNPs, Biomarkers, and Clinical Factors for Predicting Risk of Breast Cancer
J Natl Cancer Inst, July 16, 2008; 100(14): 978 - 979.
[Full Text] [PDF]


Home page
Eur Heart JHome page
J.-P. Empana, F. Canoui-Poitrine, G. Luc, I. Juhan-Vague, P. Morange, D. Arveiler, J. Ferrieres, P. Amouyel, A. Bingham, M. Montaye, et al.
Contribution of novel biomarkers to incident stable angina and acute coronary syndrome: the PRIME Study
Eur. Heart J., July 10, 2008; (2008) ehn331v1.
[Abstract] [Full Text] [PDF]


Home page
JAMAHome page
Ankle Brachial Index Collaboration
Ankle Brachial Index Combined With Framingham Risk Score to Predict Cardiovascular Events and Mortality: A Meta-analysis
JAMA, July 9, 2008; 300(2): 197 - 208.
[Abstract] [Full Text] [PDF]


Home page
Am. J. Physiol. Lung Cell. Mol. Physiol.Home page
R. E. Gerszten, F. Accurso, G. R. Bernard, R. M. Caprioli, E. W. Klee, G. G. Klee, I. Kullo, T. A. Laguna, F. P. Roth, M. Sabatine, et al.
Challenges in translating plasma proteomics from bench to bedside: update from the NHLBI Clinical Proteomics Programs
Am J Physiol Lung Cell Mol Physiol, July 1, 2008; 295(1): L16 - L22.
[Abstract] [Full Text] [PDF]


Home page
Arch Intern MedHome page
R. K. Simmons, S. Sharp, S. M. Boekholdt, L. A. Sargeant, K.-T. Khaw, N. J. Wareham, and S. J. Griffin
Evaluation of the Framingham Risk Score in the European Prospective Investigation of Cancer-Norfolk Cohort: Does Adding Glycated Hemoglobin Improve the Prediction of Coronary Heart Disease Events?
Arch Intern Med, June 9, 2008; 168(11): 1209 - 1216.
[Abstract] [Full Text] [PDF]


Home page
NEJMHome page
B. Zethelius, L. Berglund, J. Sundstrom, E. Ingelsson, S. Basu, A. Larsson, P. Venge, and J. Arnlov
Use of Multiple Biomarkers to Improve the Prediction of Death from Cardiovascular Causes
N. Engl. J. Med., May 15, 2008; 358(20): 2107 - 2116.
[Abstract] [Full Text] [PDF]


Home page
NEJMHome page
J. A. de Lemos and D. M. Lloyd-Jones
Multiple Biomarker Panels for Cardiovascular Risk Assessment
N. Engl. J. Med., May 15, 2008; 358(20): 2172 - 2174.
[Full Text] [PDF]


Home page
JAMAHome page
R. D. Brook
Potential health risks of air pollution beyond triggering acute cardiopulmonary events.
JAMA, May 14, 2008; 299(18): 2194 - 2196.
[Full Text] [PDF]


Home page
JNCI J Natl Cancer InstHome page
M. H. Gail, L. Bernstein, J. P. Costantino, D. Pee, and G. Ursin
Response:Re: Projecting Individualized Absolute Invasive Breast Cancer Risk in African American Women
J Natl Cancer Inst, May 7, 2008; 100(9): 684 - 684.
[Full Text] [PDF]


Home page
Eur Heart JHome page
S. S. Levinson
New cardiovascular risk determinants do exist and are clinically useful
Eur. Heart J., May 2, 2008; 29(10): 1335 - 1335.
[Full Text] [PDF]


Home page
Arterioscler. Thromb. Vasc. Bio.Home page
T. Leong, D. Zylberstein, I. Graham, L. Lissner, D. Ward, J. Fogarty, C. Bengtsson, C. Bjorkelund, D. Thelle, and for The Swedish-Irish-Norwegian (SIN) Collaboratio
Asymmetric Dimethylarginine Independently Predicts Fatal and Nonfatal Myocardial Infarction and Stroke in Women: 24-Year Follow-Up of the Population Study of Women in Gothenburg
Arterioscler. Thromb. Vasc. Biol., May 1, 2008; 28(5): 961 - 967.
[Abstract] [Full Text] [PDF]


Home page
HypertensionHome page
W. Palmas, T. G. Pickering, J. Teresi, J. E. Schwartz, L. Field, R. S. Weinstock, and S. Shea
Telemedicine Home Blood Pressure Measurements and Progression of Albuminuria in Elderly People With Diabetes
Hypertension, May 1, 2008; 51(5): 1282 - 1288.
[Abstract] [Full Text] [PDF]


Home page
StrokeHome page
T. Kurth and G. Logroscino
The Metabolic Syndrome: More Than the Sum of Its Components?
Stroke, April 1, 2008; 39(4): 1068 - 1069.
[Full Text] [PDF]


Home page
NEJMHome page
S. Kathiresan, O. Melander, D. Anevski, C. Guiducci, N. P. Burtt, C. Roos, J. N. Hirschhorn, G. Berglund, B. Hedblad, L. Groop, et al.
Polymorphisms Associated with Cholesterol and Risk of Cardiovascular Events
N. Engl. J. Med., March 20, 2008; 358(12): 1240 - 1249.
[Abstract] [Full Text] [PDF]


Home page
ANN INTERN MEDHome page
J. A. Tice, S. R. Cummings, R. Smith-Bindman, L. Ichikawa, W. E. Barlow, and K. Kerlikowske
Using Clinical Factors and Mammographic Breast Density to Estimate Breast Cancer Risk: Development and Validation of a New Predictive Model
Ann Intern Med, March 4, 2008; 148(5): 337 - 347.
[Abstract] [Full Text] [PDF]


Home page
Clin. Chem.Home page
J. Emmerich and P. M Ridker
Can Fishing for New Genes Catch Patients at Risk of Coronary Artery Disease?
Clin. Chem., March 1, 2008; 54(3): 453 - 455.
[Full Text] [PDF]


Home page
Clin. Chem.Home page
P. J. Talmud, J. A. Cooper, J. Palmen, R. Lovering, F. Drenos, A. D. Hingorani, and S. E. Humphries
Chromosome 9p21.3 Coronary Heart Disease Locus Genotype and Prospective Risk of CHD in Healthy Middle-Aged Men
Clin. Chem., March 1, 2008; 54(3): 467 - 474.
[Abstract] [Full Text] [PDF]


Home page
J Am Coll CardiolHome page
M. C. Fang, A. S. Go, Y. Chang, L. Borowsky, N. K. Pomernacki, D. E. Singer, and for the ATRIA Study Group
Comparison of Risk Stratification Schemes to Predict Thromboembolism in People With Nonvalvular Atrial Fibrillation
J. Am. Coll. Cardiol., February 26, 2008; 51(8): 810 - 815.
[Abstract] [Full Text] [PDF]


Home page
Eur Heart JHome page
Y. M. Smulders, A. Thijs, and J. W. Twisk
New cardiovascular risk determinants do exist and are clinically useful
Eur. Heart J., February 2, 2008; 29(4): 436 - 440.
[Abstract] [Full Text] [PDF]


Home page
Arch Intern MedHome page
F. E. Dewey, J. R. Kapoor, R. S. Williams, M. J. Lipinski, E. A. Ashley, D. Hadley, J. Myers, and V. F. Froelicher
Ventricular Arrhythmias During Clinical Treadmill Testing and Prognosis
Arch Intern Med, January 28, 2008; 168(2): 225 - 234.
[Abstract] [Full Text] [PDF]


Home page
Clin. Chem.Home page
N. R. Cook
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Clin. Chem., January 1, 2008; 54(1): 17 - 23.
[Abstract] [Full Text] [PDF]


Home page
DiabetesHome page
P. W. Franks, R. L. Hanson, W. C. Knowler, C. Moffett, G. Enos, A. M. Infante, J. Krakoff, and H. C. Looker
Childhood Predictors of Young-Onset Type 2 Diabetes
Diabetes, December 1, 2007; 56(12): 2964 - 2972.
[Abstract] [Full Text] [PDF]


Home page
CirculationHome page
J. Loscalzo
Association Studies in an Era of Too Much Information: Clinical Analysis of New Biomarker and Genetic Data
Circulation, October 23, 2007; 116(17): 1866 - 1870.
[Full Text] [PDF]


Home page
J Am Coll CardiolHome page
T. Kempf, S. von Haehling, T. Peter, T. Allhoff, M. Cicoira, W. Doehner, P. Ponikowski, G. S. Filippatos, P. Rozentryt, H. Drexler, et al.
Prognostic Utility of Growth Differentiation Factor-15 in Patients With Chronic Heart Failure
J. Am. Coll. Cardiol., September 11, 2007; 50(11): 1054 - 1060.
[Abstract] [Full Text] [PDF]


Home page
CirculationHome page
R. Latini, S. Masson, I. S. Anand, E. Missov, M. Carlson, T. Vago, L. Angelici, S. Barlera, G. Parrinello, A. P. Maggioni, et al.
Prognostic Value of Very Low Plasma Concentrations of Troponin T in Patients With Stable Chronic Heart Failure
Circulation, September 11, 2007; 116(11): 1242 - 1249.
[Abstract] [Full Text] [PDF]


Home page
StrokeHome page
D. G. Hackam
Predicting Stroke Risk in Patients With Atrial Fibrillation
Stroke, September 1, 2007; 38(9): 2409 - 2409.
[Full Text] [PDF]


Home page
CirculationHome page
M. S. Pepe, H. Janes, and J. W. Gu
Letter by Pepe et al Regarding Article, "Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction"
Circulation, August 7, 2007; 116(6): e132 - e132.
[Full Text] [PDF]


Home page
CirculationHome page
S.-J. Janket, Y. Shen, and J. H. Meurman
Letter by Janket et al Regarding Article, "Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction"
Circulation, August 7, 2007; 116(6): e133 - e133.
[Full Text] [PDF]


Home page
CirculationHome page
W. Koenig
Cardiovascular Biomarkers: Added Value With an Integrated Approach?
Circulation, July 3, 2007; 116(1): 3 - 5.
[Full Text] [PDF]


Home page
CirculationHome page
J. J. Cao, A. M. Arnold, T. A. Manolio, J. F. Polak, B. M. Psaty, C. H. Hirsch, L. H. Kuller, and M. Cushman
Association of Carotid Artery Intima-Media Thickness, Plaques, and C-Reactive Protein With Future Cardiovascular Disease and All-Cause Mortality: The Cardiovascular Health Study
Circulation, July 3, 2007; 116(1): 32 - 38.
[Abstract] [Full Text] [PDF]


Home page
J Am Coll CardiolHome page
P. M. Ridker
C-Reactive Protein and the Prediction of Cardiovascular Events Among Those at Intermediate Risk: Moving an Inflammatory Hypothesis Toward Consensus
J. Am. Coll. Cardiol., May 29, 2007; 49(21): 2129 - 2138.
[Abstract] [Full Text] [PDF]


Home page
CirculationHome page
P. M Ridker and B. M. Everett
Letter by Ridker and Everett Regarding Article, "The Inflammatory Hypothesis: Any Progress in Risk Stratification and Therapeutic Targets?"
Circulation, May 22, 2007; 115(20): e475 - e475.
[Full Text] [PDF]


Home page
ANN INTERN MEDHome page
M. Berkwits and E. Guallar
Risk Factors, Risk Prediction, and the Apolipoprotein B-Apolipoprotein A-I Ratio
Ann Intern Med, May 1, 2007; 146(9): 677 - 679.
[Full Text] [PDF]


Home page
NEJMHome page
K. Musunuru, R. S. Blumenthal, P. M Ridker, N. R. Cook, D. M. Becker, S. Mora, D. C. Goff Jr., R. H. Fletcher, S. W. Fletcher, G. Mints, et al.
Biomarkers for Prediction of Cardiovascular Events
N. Engl. J. Med., April 5, 2007; 356(14): 1472 - 1475.
[Full Text] [PDF]


Home page
CirculationHome page
D. A. Morrow and J. A. de Lemos
Benchmarks for the Assessment of Novel Cardiovascular Biomarkers
Circulation, February 27, 2007; 115(8): 949 - 952.
[Full Text] [PDF]