Letter by Janket et al Regarding Article, “Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction”
To the Editor:
We read the article by Dr Nancy Cook with great interest.1 Although we agree with Dr Cook that the likelihood ratio test may be more precise, the C-statistic is the only statistical moiety that assesses potential clinical utility of a regression model via juxtaposition of predicted probability of true positives and false-positives.
Relative risk is calculated by comparison of disease probability between those who have high levels of a biomarker and those who have low levels or none at all.2 This reference (usually the first quartile) is chosen without clinical consideration and may generate a large number of false-positives. If regression results were to be applied to clinical practice directly, this could lead to inappropriate clinical decisions (eg, vaccination or prescription of medications with adverse effects to everyone with a biomarker level above the first quartile). Therefore, prior to clinical application of the research results, it is imperative to consider not only true-positives but also accompanying false-positives. The C-statistic provides simultaneous assessment of true- and false-positives at different levels of risk factors.
“No change in C-statistic,” as Cook stated, indicates that other factors already in the model have explained what this marker contributes, and/or this marker may generate high false-positives and provides no additional information. When we tested these properties with a simpler version of The Women’s Health Study,3 C-reactive protein levels above the first quartile yielded a relative risk of 2.74 and identified 86% of true-positives correctly. However, 69% of those who do not have heart disease were incorrectly categorized as heart disease patients. These results are consistent with the clinical observation that not all persons with high C-reactive protein develop cardiovascular events4 and that C-reactive protein has low specificity.5 Would clinicians prescribe statins to patients if they knew the false-positive rate is 69%?
No matter how robust a biomarker appears to be in a regression analysis, it only provides the façade of a clinical picture. A sophisticated stochastical model remains theoretical, whereas the C-statistic, in spite of its limitations, gives us a glimpse of potential clinical relevance.6
As a side note, the inherent properties of sensitivity/specificity of inanimate objects such as an x-ray machine do not vary. The sensitivity/specificity derived from observational studies, however, should be considered cohort-specific because data collection methods, study design, or cohort characteristics engender variability in these test properties.
Sources of Funding
Dr Janket is supported by the National Scientist Development Grant from The American Heart Association. Dr Meurman is supported by Grant TYH 3245 from the Helsinki University Central Hospital, Finland; the Ulf Nilsonne Foundation (SalusAnsvar Prize), Stockholm, Sweden; and the Paivikki and Sakari Sohlberg Foundation.
Cook N. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007; 115: 928–935.
Wald NJ, Hackshaw AK, Frost CD. When can a risk factor be used as a worthwhile screening test? BMJ. 1999; 319: 1562–1565.
Libby P, Ridker, Maseri A. Inflammation and atherosclerosis. Circulation. 2002; 105: 1135–43.
Kushner I, Rzewnicki D, Samols D. What does minor elevation of C-reactive protein signify? Am J Med. 2006; 119: 166.e17–166.e28.