| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Circulation. 1999;99:2098-2104.)
© 1999 American Heart Association, Inc.
Clinical Investigation and Reports |
Ready-Made, Recalibrated, or Remodeled?
Issues in the Use of Risk Indexes for Assessing Mortality After Coronary Artery Bypass Graft Surgery
Presented in part at the 70th Scientific Sessions of the American Heart Association, November 9–12, 1997, Orlando, Fla, and published in abstract form (Circulation. 1997;96[suppl I]:I-506).
From the Division of Cardiovascular Surgery at The Toronto Hospital (J.I.), Toronto, Ontario, Canada; The Institute for Clinical Evaluative Sciences (J.V.T., C.D.N.), North York, Ontario, Canada; the Faculty of Medicine (J.I., J.V.T., C.D.N.), University of Toronto, Toronto, Ontario, Canada; and the Department of Medicine and the Clinical Epidemiology and Health Services Research Program (J.V.T., C.D.N.), Sunnybrook Health Science Centre, North York, Ontario, Canada.
Correspondence to Dr C. David Naylor, Institute for Clinical Evaluative Sciences, G106-2075 Bayview Ave, North York, Ontario, M4N 3 M5, Canada. E-mail cdn{at}ices.on.ca
Background—Risk indexes for operative mortality after cardiac surgery are used for comparative profiling of surgeons or centers. We examined whether clinicians and managers should use an existing index without modification, recalibrate it for their populations, or derive a new model altogether.
Methods and Results—Drawing on 7491 consecutive patients who underwent isolated CABG at 2 Toronto teaching hospitals between 1993 and 1996, we compared 3 strategies: (1) using a ready-made model originally derived and validated in our jurisdiction; (2) recalibrating the ready-made model to better fit the population; and (3) deriving a new model with additional risk factors. We assessed statistical accuracy, ie, area under a receiver-operator characteristic curve (ROC); precision, ie, statistical goodness-of-fit; and actual impact on both risk-adjusted operative mortalities (RAOM) and performance rankings for 14 surgeons. The new model was slightly more accurate than the ready-made model (ROC, 0.78 versus 0.76; P<0.05), albeit not different from the recalibrated model (ROC, 0.77). The ready-made model showed poor fit between the predicted and observed results (P<0.001), leading to significant underestimation of RAOM (1.6±0.2%) compared with the other strategies (2.5±0.2%; P=0.048). Remodeling also changed the performance rankings among half the surgeons with higher RAOM.
Conclusions—Poorly calibrated risk algorithms can bias the calculation of RAOM and alter the results of surgeon-specific profiles. Any existing index used for risk assessment in cardiac surgery should be episodically recalibrated or compared with new models derived from local subjects to ensure that its performance remains optimal.
Key Words: bypass • risk factors • mortality • prognosis
This article has been cited by other articles:
 |
|
 |
|
  V. Guru, J. V. Tu, E. Etchells, G. M. Anderson, C. D. Naylor, R. J. Novick, C. M. Feindel, F. D. Rubens, K. Teoh, A. Mathur, et al. Relationship Between Preventability of Death After Coronary Artery Bypass Graft Surgery and All-Cause Risk-Adjusted Mortality Rates Circulation, June 10, 2008; 117(23): 2969 - 2976. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. D'Errigo, F. Seccareccia, S. Rosato, V. Manno, G. Badoni, D. Fusco, C. A. Perucci, and the Research Group of the Italian CABG Outcome Pro Comparison between an empirically derived model and the EuroSCORE system in the evaluation of hospital performance: the example of the Italian CABG Outcome Project Eur. J. Cardiothorac. Surg., March 1, 2008; 33(3): 325 - 333. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Singh, B. J. Gersh, S. Li, J. S. Rumsfeld, J. A. Spertus, S. M. O'Brien, R. M. Suri, and E. D. Peterson Mayo Clinic Risk Score for Percutaneous Coronary Intervention Predicts In-Hospital Mortality in Patients Undergoing Coronary Artery Bypass Graft Surgery Circulation, January 22, 2008; 117(3): 356 - 362. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. N. Wijeysundera, K. Karkouti, J.-Y. Dupuis, V. Rao, C. T. Chan, J. T. Granton, and W. S. Beattie Derivation and Validation of a Simplified Predictive Index for Renal Replacement Therapy After Cardiac Surgery JAMA, April 25, 2007; 297(16): 1801 - 1809. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. L. Hannan, C. Wu, E. V. Bennett, R. E. Carlson, A. T. Culliford, J. P. Gold, R. S.D. Higgins, C. R. Smith, and R. H. Jones Risk Index for Predicting In-Hospital Mortality for Cardiac Valve Surgery Ann. Thorac. Surg., March 1, 2007; 83(3): 921 - 929. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. Brunelli, N. J. Morgan-Hughes, M. Refai, M. Salati, A. Sabbatini, and G. Rocco Risk-adjusted morbidity and mortality models to compare the performance of two units after major lung resections J. Thorac. Cardiovasc. Surg., January 1, 2007; 133(1): 88 - 96. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. Seccareccia, C. A. Perucci, D. Fusco, and P. D'Errigo Reply to hekmat et Al. Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 857 - 858. [Full Text] [PDF]
|
|
|
|
|
|
C.-H. Yap, C. Reid, M. Yii, M. A. Rowland, M. Mohajeri, P. D. Skillington, S. Seevanayagam, and J. A. Smith Validation of the EuroSCORE model in Australia. Eur. J. Cardiothorac. Surg., April 1, 2006; 29(4): 441 - 446. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. Brunelli, F. Xiume', M. Al Refai, M. Salati, R. Marasco, and A. Sabbatini Risk-adjusted morbidity, mortality and failure-to-rescue models for internal provider profiling after major lung resection Interactive CardioVascular and Thoracic Surgery, April 1, 2006; 5(2): 92 - 96. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Nilsson, L. Algotsson, P. Hoglund, C. Luhrs, and J. Brandt Comparison of 19 pre-operative risk stratification models in open-heart surgery Eur. Heart J., April 1, 2006; 27(7): 867 - 874. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. S. Weintraub Evaluating the Risk of Coronary Surgery and Percutaneous Coronary Intervention J. Am. Coll. Cardiol., February 7, 2006; 47(3): 669 - 671. [Full Text] [PDF]
|
|
|
|
|
|
E. L. Hannan, C. Wu, E. V. Bennett, R. E. Carlson, A. T. Culliford, J. P. Gold, R. S.D. Higgins, O. W. Isom, C. R. Smith, and R. H. Jones Risk Stratification of In-Hospital Mortality for Coronary Artery Bypass Graft Surgery J. Am. Coll. Cardiol., February 7, 2006; 47(3): 661 - 668. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. Seccareccia, C. A. Perucci, P. D'Errigo, M. Arca, D. Fusco, S. Rosato, D. Greco, and on behalf of the research group of the Italian CAB The Italian CABG Outcome Study: short-term outcomes in patients with coronary artery bypass graft surgery Eur. J. Cardiothorac. Surg., January 1, 2006; 29(1): 56 - 62. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J.-Y. Dupuis Clinical Predictions and Decisions to Perform Cardiac Surgery on High-Risk Patients Seminars in Cardiothoracic and Vascular Anesthesia, June 1, 2005; 9(2): 179 - 186. [Abstract] [PDF]
|
|
|
|
 |
|
 I-C. Huang, F. Dominici, C. Frangakis, G. B. Diette, C. L. Damberg, and A. W. Wu Is Risk-Adjustor Selection More Important Than Statistical Approach for Provider Profiling? Asthma as an Example Med Decis Making, January 1, 2005; 25(1): 20 - 34. [Abstract] [PDF]
|
|
|
|
|
|
D. M. Shahian, E. H. Blackstone, F. H. Edwards, F. L. Grover, G. L. Grunkemeier, D. C. Naftel, S. A.M. Nashef, W. C. Nugent, and E. D. Peterson Cardiac Surgery Risk Models: A Position Article Ann. Thorac. Surg., November 1, 2004; 78(5): 1868 - 1877. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. Z. Omar, G. Ambler, P. Royston, J. Eliahoo, and K. M. Taylor Cardiac surgery risk modeling for mortality: a review of current practice and suggestions for improvement Ann. Thorac. Surg., June 1, 2004; 77(6): 2232 - 2237. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
V. Guru, S. E. Fremes, and J. V. Tu Time-related mortality for women after coronary artery bypass graft surgery: A population-based study J. Thorac. Cardiovasc. Surg., April 1, 2004; 127(4): 1158 - 1165. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. F. Ibrahim, D. Paparella, J. Ivanov, M. R. Buchanan, and S. J. Brister Gender-related differences in morbidity and mortality during combined valve and coronary surgery J. Thorac. Cardiovasc. Surg., October 1, 2003; 126(4): 959 - 964. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
V. Rao, M. C. Oz, M. A. Flannery, K. A. Catanese, M. Argenziano, and Y. Naka Revised screening scale to predict survival after insertion of a left ventricular assist device J. Thorac. Cardiovasc. Surg., April 1, 2003; 125(4): 855 - 862. [Abstract] [Full Text]
|
|
|
|
 |
|
 P. W. C. ten Broecke, S. G. De Hert, E. Mertens, and H. F. Adriaensen Effect of preoperative {beta}-blockade on perioperative mortality in coronary surgery Br. J. Anaesth., January 1, 2003; 90(1): 27 - 31. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J R Pepper Risk assessment in coronary artery surgery Heart, January 1, 2003; 89(1): 1 - 2. [Full Text] [PDF]
|
|
|
|
 |
|
 V. A. Ferraris and S. P. Ferraris Risk Stratification and Comorbidity Card. Surg. Adult, January 1, 2003; 2(2003): 187 - 224. [Full Text]
|
|
|
|
|
|
P. Pinna Pintor, S. Colangelo, and M. Bobbio Evolution of case-mix in heart surgery: from mortality risk to complication risk Eur. J. Cardiothorac. Surg., December 1, 2002; 22(6): 927 - 933. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. E. Shaw, H. V. Anderson, R. G. Brindis, R. J. Krone, L. W. Klein, C. R. McKay, P. C. Block, L. J. Shaw, K. Hewitt, W. S. Weintraub, et al. Development of a risk adjustment mortality model using the American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR) experience: 1998-2000 J. Am. Coll. Cardiol., April 3, 2002; 39(7): 1104 - 1112. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. L. Grunkemeier, K. J. Zerr, and R. Jin Cardiac surgery report cards: making the grade Ann. Thorac. Surg., December 1, 2001; 72(6): 1845 - 1848. [Full Text] [PDF]
|
|
|
|
|
|
D. M. Shahian, S.-L. Normand, D. F. Torchiana, S. M. Lewis, J. O. Pastore, R. E. Kuntz, and P. I. Dreyer Cardiac surgery report cards: comprehensive review and statistical critique Ann. Thorac. Surg., December 1, 2001; 72(6): 2155 - 2168. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R Marchioli, F Avanzini, F Barzi, C Chieffo, A Di Castelnuovo, M.G Franzosi, E Geraci, A.P Maggioni, R.M Marfisi, N Mininni, et al. Assessment of absolute risk of death after myocardial infarction by use of multiple-risk-factor assessment equations; GISSI-Prevenzione mortality risk chart Eur. Heart J., November 2, 2001; 22(22): 2085 - 2103. [Abstract] [PDF]
|
|
|
|
|
|
O. Pitkanen, M. Niskanen, S. Rehnberg, M. Hippelainen, and M. Hynynen Intra-institutional prediction of outcome after cardiac surgery: comparison between a locally derived model and the EuroSCORE Eur. J. Cardiothorac. Surg., December 1, 2000; 18(6): 703 - 710. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. W. Steyerberg, J. Ivanov, J. V. Tu, C. D. Naylor, and H. M. Krumholz Ranking of Surgical Performance Response Response Circulation, August 29, 2000; 102 (9): e61 - e62. [Full Text] [PDF]
|
|
|
|
|
|
J. Ivanov, M. A. Borger, T. E. David, G. Cohen, N. Walton, and C. D. Naylor Predictive accuracy study: comparing a statistical model to clinicians' estimates of outcomes after coronary bypass surgery Ann. Thorac. Surg., July 1, 2000; 70(1): 162 - 168. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
H. M. Krumholz Mathematical Models and the Assessment of Performance in Cardiology Circulation, April 27, 1999; 99(16): 2067 - 2069. [Full Text] [PDF]
|
|