Letter by Lown et al Regarding Article, “Risk of Assessing Mortality Risk in Elective Cardiac Operations: Age, Creatinine, Ejection Fraction, and the Law of Parsimony”
To the Editor:
We read with interest the article by Ranucci et al describing a 3-factor risk score for the prediction of mortality risk in elective cardiac operations and a comparison of performance against 5 previously established mortality risk scores. Of particular note is the aim for parsimony; the selection of the readily obtainable, objective variables, age and creatinine; and the discriminatory power achieved with a C statistic >0.8 in validation.1
The authors have chosen 3 variables for the risk model on the basis of univariate analysis. An alternative would be to undertake multivariable analysis because this would aid the selection of the most discriminative parameters when accounting for the other variables. Although combining age and ejection fraction provides a linear function, this simplification is likely to compromise accuracy because the weighting of age is linear for any given ejection fraction. Indeed, a more sophisticated nonlinear model may achieve improved accuracy. The decision to dichotomise creatinine values may also compromise performance and seems perhaps an arbitrary choice given that the area under curve value for age, creatinine, and ejection fraction are similar. The calibrated model had an accuracy of only 0.744 (see Table 3 in the article), a value that is stated as inadequate for clinical purposes earlier in the text. Did the other models have a similar performance with these data, and in what areas did the 3-factor model perform less well? Moreover, did the authors consider using alternative methods of evaluating the model’s effectiveness?
More generally, inherent limitations and unknowns with biological systems will limit achievable area under curve values, and thus there is a point beyond which no improvements in performance will be achieved even when the complexity and number of variables in a model are increased.2 Socioeconomic elements, service provider information, and appropriateness of the procedure are not considered. Complex models may be overfitted, and additional limitations may exist, such as the availability of all the required data and their subjectivity.3 Conversely, simple models probably only reinforce physicians’ intuitive knowledge,4 and subjective assessment is accurate for the extremes of risk.1
With the emergence of increasing numbers of risk models in medicine with area under curve values between 0.75 (for clinical usability) and inherent ceilings, perhaps our choice of model should be guided by the clinical setting. Simple models using readily obtainable objective variables may be better suited for acute settings where background information is not obtainable and risk might guide immediate management. This is evident in the acute cardiac setting where parsimonious models have comparable performance to more complex risk models.5 Models with larger numbers of parameters that have benefited from international validation may be preferred in elective settings and for case-mix adjustment as part of the performance assessment.
Ranucci M, Castelvecchio S, Menicanti L, Frigiola A, Pelissero G. Risk of assessing mortality risk in elective cardiac operations: age, creatinine, ejection fraction, and the law of parsimony. Circulation. 2009; 119: 3053–3061.
Dorsch MF, Lawrance RA, Sapsford RJ, Oldham J, Greenwood D, Jackson BM, Morrell C, Ball S, Robinson M, Hall AS. A simple benchmark for evaluating quality of care of patients following acute myocardial infarction. Heart. 2001; 86: 150–154.
Gale CP, Manda SO, Weston CF, Birkhead JS, Batin PD, Hall AS. Evaluation of risk scores for risk stratification of acute coronary syndromes in the Myocardial Infarction National Audit Project (MINAP) database. Heart. 2009; 95: 221–227.