Multicenter Validation of a Risk Index for Mortality, Intensive Care Unit Stay, and Overall Hospital Length of Stay After Cardiac Surgery
Background A multicenter population-based study was conducted to develop and validate a risk index for mortality, intensive care unit (ICU) length of stay, and postoperative length of stay after cardiac surgery.
Methods and Results Data were collected from 13 098 patients undergoing cardiac surgery between April 1, 1991, and March 31, 1993, at all nine adult cardiac surgery institutions in Ontario, Canada. A six-variable risk index (age, sex, left ventricular function, type of surgery, urgency of surgery, and repeat operation) was developed using logistic regression analysis to predict in-hospital mortality, ICU stay in days, and postoperative stay in days after cardiac surgery in a derivation set of 6213 patients who had cardiac surgery during fiscal year 1991 (April 1, 1991, to March 31, 1992). The index predicted mortality, prolonged ICU stay (≥6 days), and prolonged postoperative length of stay (≥17 days) after cardiac surgery with areas under the receiver-operating characteristic (ROC) curve of 0.75, 0.66, and 0.69, respectively, in an independent validation set of 6885 patients who had cardiac surgery during fiscal year 1992 (April 1, 1992, to March 31, 1993). Increasing risk scores were associated with greater mortality rates and longer ICU and postoperative stays at all nine institutions.
Conclusions Mortality, ICU length of stay, and postoperative length of stay after cardiac surgery can be predicted using a simple six-variable risk index. The index has potential application as a risk stratification tool for comparing patient outcomes and resource use among different hospitals and surgeons.
In an era of widespread concerns about variations in the quality of care and use of health care resources, methods to assess the risks of cardiac surgery are of increasing importance. Cardiac surgery is resource intensive and has a moderately high in-hospital mortality rate. Thus, it is understandable that a number of models have been developed for assessing the quality of cardiac surgical care, with a particular focus on mortality benchmarking.1 2 3 4 5 6 Although several cardiac surgery risk models exist, most of these models were not designed to predict more than one outcome measure and most apply to coronary artery bypass graft surgery (CABG) alone.1 2 3 4 5 6
If any risk model is to be easily used by clinicians, then there will likely be a tradeoff between the simplicity of a model and its statistical precision. Many of the existing models require extensive risk factor collection and are based on complex mathematical equations.1 2 3 4 5 6 Logistic regression models or bayesian equation models are statistically precise, but they require a calculator or computer for their use. Simpler additive models have been developed, but most reflect the surgical experience of a single institution, contain setting-specific risk factors, and have not necessarily validated well when used in other settings.1 4 5 7 8 Although mortality is an important outcome, other outcome measures such as intensive care unit (ICU) and postoperative (PostOp) length of stay (LOS) are also of increasing interest.9 Models are required that apply to both CABG and valve surgery patients. Accordingly, the purpose of the present study was to develop a simple risk index that could serve as a basis for interinstitutional comparisons of multiple outcomes (eg, mortality and LOS) after cardiac surgery in Ontario, Canada.
In 1991, the Ontario Ministry of Health established the Provincial Adult Cardiac Care Network (PACCN), a provincewide computerized registry for monitoring cardiac surgery waiting lists in Ontario. All adult patients who require heart surgery in Ontario are entered into this database at the time of referral for cardiac surgery so that they can be triaged according to the urgency of their surgery.10 Clinical information was gathered and entered into the PACCN database on all patients having cardiac surgery at one of the nine adult cardiac surgery institutions in Ontario beginning in April 1991. The nine institutions include eight teaching hospitals and one nonteaching hospital, all with moderate to large surgical volumes (ie, 380 to 1823 cases per year). Missing and inconsistent data elements were completed or checked using hospital-specific registries and/or chart reviews by the network coordinators.
Clinical data from the PACCN database were linked using unique patient identifiers to information contained in the Hospital Medical Records Institute (HMRI) database, an administrative database containing outcomes (mortality, ICU LOS, PostOp LOS) and comorbidity information. Noncardiac comorbidity was determined from the HMRI database using methods described by Deyo et al.11 Complete information was available on a total of 13 098 patients, representing 97% of the patients identified in the PACCN database as having had CABG, valve surgery, or both during the study period. Other types of adult cardiac operations (eg, congenital, transplantation, and so on) were excluded from the present study.
The combined PACCN/HMRI database was divided into a derivation set consisting of the 6213 patients who had cardiac surgery in fiscal year 1991 (April 1, 1991, to March 31, 1992) and an independent validation set consisting of the 6885 patients who had cardiac surgery in fiscal year 1992 (April 1, 1992, to March 31, 1993). The derivation set was used to develop the risk index, and the validation set was used to test the model that was developed.
The quality of the data in the PACCN database was maintained through built-in logic and range checks and by random chart audits by the data entry coordinators at each site. Major fields were mandatory to ensure complete data collection. Previous audits of the HMRI database have shown the mortality data to be 100% accurate. Further assessment of the quality of data in the combined database was done by comparing common information found in both the PACCN and HMRI databases, and significant discrepancies were resolved by contacting the referring institution.
A recently developed predictive index for ICU LOS after cardiac surgery served as a template for the development of the current risk index.9 Variables that were studied as potential risk factors for inclusion in the index included patient age, sex, left ventricular function, type of surgery, urgency of surgery, repeat operation, recent myocardial infarction, number of vessels bypassed, left main disease, and the noncardiac comorbid diseases contained in the Charlson comorbidity index (eg, diabetes, chronic obstructive pulmonary disease [COPD], and so on).12
The outcomes of interest in this study were in-hospital mortality, ICU LOS, and PostOp LOS. A very long ICU LOS was defined as a stay ≥6 days, whereas a very long PostOp LOS was defined as a stay ≥17 days. These correspond to the 90th percentiles for these two outcomes in the Ontario population. These cutoff points were chosen because they are very likely to reflect prolonged LOS secondary to the development of morbidity rather than differences in discharge practices. ICU LOS information was not available at one of the hospitals.
A multivariate stepwise regression procedure was used to identify risk factors for in-hospital mortality that were included in a logistic regression mortality model. Risk factors were included in the model if they fulfilled two criteria: the variable was a statistically significant predictor of mortality at the P<.05 level, and the variable incrementally improved the area under the receiver-operating characteristic (ROC) curve by ≥0.01. The area under the ROC curve is a commonly used measure of the predictive power of a statistical model.13 The stata statistical package was used to fit all of the statistical models.14 Logistic regression models were also developed with very long ICU LOS and very long PostOp LOS as the dependent variables. First-order interactions were determined between each of the variables in the mortality model.
Odds ratios were calculated from the coefficients of the variables in the three logistic regression models. The odds ratio is a measure of the odds of an outcome occurring in one risk group relative to the odds of that outcome occurring in a reference group (ie, the group at lowest risk). The goodness of fit of the three logistic regression models was assessed using the Hosmer-Lemeshow statistic.15 The risk index was created by rounding the mean of the three odds ratios for each risk factor in the different logistic models to the nearest integer. Outcome-specific risk scores were also evaluated.
An unknown left ventricular function was considered equal to a grade 1 left ventricle, an approach recommended by Pierpont et al16 and adopted by others in their cardiac surgery risk studies.3 Left ventricular function was missing primarily among isolated valve patients, where it was not a mandatory variable. Sensitivity analyses were done to determine whether treating unknown left ventricular function as a grade 2, 3, or 4 left ventricular equivalent improved the model, but it did not.
The risk factors in the risk index that was developed were age, which was categorized into one of three groups (<65, 65 to 74, and ≥75 years); sex (female); grade of left ventricular function based on ejection fraction as assessed by echocardiography or angiography (grade 1, >50%; grade 2, 35% to 50%; grade 3, 20% to 34%; and grade 4, <20%); type of surgery (CABG, single valve, or complex [multivalve or CABG plus valve]); urgency of surgery (emergency, urgent, or elective); and repeat operation (previous CABG). A consensus panel of cardiovascular practitioners in Ontario developed urgency definitions and an urgency rating score for cardiac surgery patients based on the severity and stability of their angina symptoms, coronary anatomy from angiographic studies, and the results of noninvasive tests for ischemic risk.10 Emergency surgery is defined as any surgery that is required within 24 hours at the time of referral (eg, postcatheterization “crash,” unstable angina with hemodynamic instability); urgent surgery is surgery required within the same hospital admission (eg, unstable angina stabilized with maximal medical therapy); and elective surgery applies for all other procedures.
With the resulting six-variable risk index, risk scores (range, 0 to 16) were assigned to each patient in both the 1991 derivation and 1992 validation sets based on their clinical characteristics. The in-hospital mortality rate, mean ICU LOS, and mean PostOp LOS for each risk score level were determined. Patients with risk scores ≥8 were combined into one category because of the small numbers. To assess the performance of the index at the institutional level, risk scores were combined into three categories with similar outcomes—0 to 3, 4 to 7, and ≥8—and the mortality rate, mean ICU LOS, and mean PostOp LOS were determined at each of the nine adult cardiac surgery institutions in Ontario.
The overall predictive ability of the risk index and the three logistic models was assessed by calculating the area under the ROC curve in both the derivation and validation sets using a nonparametric method.13 The ability of the risk index to predict mortality as measured by the area under the ROC curve was compared with that for very long ICU LOS and very long PostOp LOS using methods described by Hanley and McNeil.17 The ability of the risk index to predict outcomes in the validation set was further assessed by comparing observed outcomes in both the 1991 derivation and 1992 validation sets at each risk score level. The 95% confidence intervals (CIs) were calculated for the observed mortality rate, mean ICU LOS, and mean PostOp LOS in the 1991 derivation set. The 1992 validation set outcomes were then compared to determine whether the observed results lay within the 95% CIs of the outcomes predicted from the 1991 reference derivation set.
Table 1⇓ is a summary of the in-hospital mortality rate, mean ICU LOS, and mean PostOp LOS by each of the risk factors contained in the risk index in the 1991 derivation set. The overall Ontario mortality rate for cardiac surgery was 3.69% in fiscal 1991 (3.01% for isolated CABG only), with a mean ICU LOS of 3.21 days and a mean PostOp LOS of 10.60 days. In fiscal 1992, the overall mortality rate for cardiac surgery was 3.54% (2.88% for isolated CABG only), with a mean ICU LOS of 3.01 days and a mean PostOp LOS of 10.21 days. These mortality results are comparable to the best outcomes reported in other population-based registries and large institutional databases.2 3 6
Logistic Regression Models
Table 2⇓ shows the logistic regression models with the three dependent variables of mortality, very long ICU LOS, and very long PostOp LOS. A significant first-order interaction (P<.05) was observed between age and left ventricular function and between left ventricular function and urgency in the mortality model. Because these interaction terms complicate the interpretation of the logistic regression model and do not significantly affect the relative magnitude of the odds ratios associated with the main effects, they were excluded from the final model. It was interesting to note that for three of the six risk factors, the relative magnitude of the odds ratios for each of the three outcomes was about the same. However, emergency surgery was associated with a much greater odds ratio for mortality (5.70) than for very long ICU and PostOp LOS (2.84 and 2.61). A grade 4 left ventricle was also associated with a higher odds ratio for mortality (3.64) relative to that for very long ICU and PostOp LOS (2.18 and 2.18). Repeat operation was a very significant risk factor for mortality (odds ratio, 3.22; P<.001), whereas it did not quite reach statistical significance for long ICU stays (P=.064) or long postoperative stays (P=.054). These latter results are compatible with the hypothesis that repeat operation sharply increases the odds of early postoperative death but that once such patients survive this period, their postoperative course is only slightly prolonged compared with that of primary procedure patients. All three logistic regression models passed Hosmer-Lemeshow goodness-of-fit tests (P=.45 for mortality, P=.14 for ICU LOS, and P=.34 for PostOp LOS; P≤.05 indicates a poor fit).15
The risk index that was developed is shown in Table 3⇓. Tables 4⇓ and 5⇓ show the observed mortality rate, mean ICU LOS, and mean PostOp LOS at each of the different risk score levels in both the 1991 derivation and 1992 validation sets. Increasing risk scores were associated with greater risks for mortality and prolonged ICU and PostOp LOS in both the derivation and validation data sets. The index validated very well, with similar outcome results at nearly all risk score levels. The mean risk score was 3.11 in 1991 and 3.06 in 1992.
Figs 1 through 3⇓⇓⇓ show graphically the observed mortality rates, mean ICU LOS, and mean PostOp LOS in both the 1991 derivation and 1992 validation sets. The 95% CIs for the observed outcomes are also shown for the 1991 derivation set. The observed outcomes in the 1992 validation set lie within the 95% CIs predicted by the 1991 derivation set with two exceptions. In the highest risk group (≥8), the mortality rate fell significantly from 20.62% in 1991 to 13.22% in 1992. The mean ICU LOS was also significantly lower at 3.68 days in 1992 compared with 4.61 days in 1991 for patients with a risk score of 6. These results show that the overall improvement in outcomes noted between the 2 years was predominantly in those at highest risk.
Table 6⇓⇓ shows the performance of the risk index at each of the nine adult cardiac surgery centers in Ontario for each of three different risk score categories in the 1992 validation set. Risk scores of 0 to 3 could be considered a low-risk group; risk scores of 4 to 7, an intermediate-risk group; and risk scores of ≥8, a high-risk group. Despite considerable differences in case mix among the nine adult cardiac surgery institutions in Ontario, the risk index validated very well, with higher risk scores being associated with greater risks for all three outcomes, regardless of the institution. Similar results were also observed in the 1991 derivation set. The mean risk score at each hospital ranged between a low of 2.64 and a high of 3.49 in the 1992 validation set.
Areas Under the ROC Curve
The areas under the ROC curve for the risk index were 0.75, 0.67, and 0.71 for mortality, very long ICU LOS, and very long PostOp LOS predictions in the 1991 derivation set. For the outcome-specific logistic regression models, the corresponding areas were 0.76, 0.68, and 0.72. The areas using outcome-specific integer risk scores were 0.76, 0.67, and 0.71, respectively. These results demonstrate that very little predictive performance was lost by using the mean of the odds ratios from the three logistic models to determine the risk scores in the risk index.
In the independent 1992 validation set, the areas under the ROC curve were 0.75, 0.66, and 0.69 for mortality, very long ICU LOS, and very long PostOp LOS predictions using the risk index, demonstrating that the index validated very well. The index predicted mortality significantly better than very long PostOp LOS (P<.05) in both the derivation and validation sets and very long PostOp LOS better than very long ICU LOS (P<.05) in the derivation set. These results suggest that differences in patient case mix explain variations in mortality outcomes better than they explain variations in resource use, where other factors, such as different practice styles, play a role.
In the present study, we have developed and validated a simple six-variable risk index for mortality, ICU LOS, and overall PostOp LOS after cardiac surgery using data from all nine adult cardiac surgery institutions in Ontario. This simple index yields predictive performance for mortality that is comparable to other cardiac surgery risk models that have been developed.3 4 Our index also predicts resource use (as measured by ICU and PostOp LOS) after cardiac surgery. Thus, the index has multiple potential applications, including comparing patient outcomes and resource use among different surgeons and hospitals, counseling patients about the risks of cardiac surgery, and use in patient and staff scheduling when resources are limited.9
Our index is being used to conduct comparisons of risk-adjusted cardiac surgical outcomes (mortality, ICU LOS, PostOp LOS) among the nine adult cardiac surgical institutions in Ontario. Hospitals are given their risk-adjusted outcomes so that they can evaluate their relative performance with the goal of continuous quality improvement. Expected hospital-level outcomes calculated using the risk index have been compared with those determined using logistic and linear regression models, and the overall results have been essentially the same (unpublished data). The index is also being used to provide resource-intensity adjustments of funding of cardiac surgical services in Ontario under the government-sponsored healthcare system. The aim is to ensure that hospitals with sicker surgical patients (ie, higher risk scores) receive greater reimbursement for their surgical cases so that they are not financially penalized for accepting more resource-intensive cases.
Use of an additive model allows clinicians to see how well they are performing relative to others at different levels of patient risk (eg, low, medium, or high risk), and it provides a summary measure (ie, the mean risk score) of a hospital’s case mix severity. By using population-based data from multiple institutions to determine significant risk factors and to define a reference standard for expected outcomes, we have avoided potential biases that might exist when risk models are developed and validated at only a single institution. Overall cardiac surgical outcomes have been improving in Ontario over time, and thus expected outcomes at different risk score levels are determined annually to reflect temporal improvements in the quality of surgical care.
Several other risk stratification models have been developed for predicting morbidity and mortality at cardiac surgery, including those shown in Table 7⇓. Many different variables have been found to be associated with increased risk at cardiac surgery, but only a few variables have consistently been found to be major risk factors across multiple and very diverse study settings.1 2 3 4 5 6 These include the six risk factors contained in our index (age, female gender, left ventricular function, type of surgery, urgency of surgery, and repeat operation).
The major difference between our model and others that have been developed is its lack of inclusion of comorbid diseases. Although intuitively one might expect that certain noncardiac diseases would be major predictors of mortality, different investigators have found different comorbid diseases to be significant risk factors and no diseases to be consistent risk factors, with the possible exceptions of renal dysfunction and diabetes, as shown in Table 7⇑.1 2 3 4 5 6 Renal dysfunction has recently been shown to be an important risk factor for surgical mortality in patients who have this condition.1 2 4 5 6 However, the spectrum of what constitutes renal dysfunction is broad, with some models defining it as elevated creatinine levels and others defining it as dialysis dependency.1 2 4 5 The incidence of dialysis dependency in the cardiac surgical population is so low (eg, 0.5% in New York State) that it is very unlikely to change the results of any population-based comparisons.2 The association between diabetes and mortality at cardiac surgery has been inconsistent, with some studies supporting and other studies not supporting an association.1 2 4 5 6 18
We have observed that noncardiac comorbidity as determined using HMRI administrative data does not usually affect overall hospital-level results once we have adjusted for the six risk factors found in our index. Other investigators have also found that adjustment for comorbidity had no effect on CABG hospital rankings once clinical risk factors were adjusted for.19 However, these findings could be a reflection of undercoding of comorbid diseases in administrative databases, a phenomenon noted by researchers elsewhere.20
The level of predictive performance for mortality achieved using our index (area under the ROC curve, 0.75 in both the 1991 derivation and 1992 validation sets) is comparable to that achieved using other models that have more risk factors (eg, ROC area, 0.74 in the Cleveland Clinic model developed by Higgins et al4 ; ROC area, 0.76 in the Northern New England model developed by O’Connor et al3 ). Given that the overall predictive performance is similar, we have elected to use our simpler model as a method for risk-adjusted outcome comparisons and funding determinations. For any risk factor not presently in the index to significantly affect overall results it would have to have several properties: it would have to be a statistically significant predictor of the outcome of interest, it would have to have a moderately high prevalence in the cardiac surgical population, it would have to be distributed unevenly between the various hospitals, and it would have to be uncorrelated with any of the variables currently in the index.
The Society of Thoracic Surgeons (STS) National Cardiac Surgery Database model developed by Edwards et al6 is the most widely used model in the United States and is an unparalleled effort in terms of its size and comprehensiveness. The current bayesian equation that is being used incorporates 23 risk factors. However, it is possible that the STS model includes some nominally significant mortality predictors of uncertain clinical impact, given the enormous size of the available database and the resultant statistical power. Some institutions may not have the resources necessary to undertake the significant data collection effort required to use this model; this is certainly a concern in Canada. We could not directly compare our index with other models that have been developed because we did not have information on all of the same risk factors. However, other investigators have compared different cardiac surgery risk models and have found that their overall performance is similar, independent of the level of complexity.7 8
The present study has certain limitations. The performance of our index outside Ontario remains to be determined, and other models that have been developed may have better predictive performance in other settings since the prevalence and importance of certain risk factors may be different. Other models may also provide more precise stratification of risk at the individual patient level. The major advantage associated with our model is its ability to predict three outcomes (mortality, ICU LOS, PostOp LOS) using only six variables that are already being collected in most existing cardiac surgery databases. Clinicians elsewhere may prefer the simplicity of our model or they may prefer to use other more comprehensive models that have been developed.
In conclusion, we have demonstrated that a simple six-variable risk index can be used to predict the risks of in-hospital mortality, prolonged ICU LOS, and prolonged PostOp LOS at all nine adult cardiac surgery institutions in Ontario. Patient age, sex, left ventricular function, type of surgery, urgency of surgery, and repeat operation have been combined into an easy-to-use six-variable risk-stratifying index, and this has been validated using a large population-based sample in Ontario. These risk factors are found in virtually all cardiac surgery databases, so our index could easily be used by practitioners in other centers to compare their patient outcomes and resource use with those observed in Ontario. Further case mix and outcome comparison studies are being planned using our risk index as a framework.
Steering Committee of the Provincial Adult Cardiac Care Network of Ontario
Steering Committee members are Alnoor Abdulla, MD, FRCPC, Sudbury Memorial Hospital, Sudbury, Ontario; Glenn Bartlett, MD, FRCSC (1991 to 1993), London, Ontario; Donald S. Beanlands, MD, FRCPC, University of Ottawa Heart Institute, Ottawa Civic Hospital, Ottawa, Ontario; Robert Chisholm, MD, FRCPC, St Michael’s Hospital, Toronto, Ontario; Martin Goldbach, MD, FRCSC, Victoria Hospital, London, Ontario; Neil McKenzie, MB, BCh, FRCSC, University Hospital, London, Ontario; Christopher D. Morgan, MD, FRCPC, Sunnybrook Health Science Centre, Toronto, Ontario; John Pym, MB, BCh, FRCSC, Kingston General Hospital, Kingston, Ontario; Hugh Scully, MD, FRCSC, Toronto Hospital, Toronto, Ontario; B. William Shragge, MD, FRCSC, Hamilton Civic Hospitals-General Division, Hamilton, Ontario; and James Swan, MD, FRCPC, Scarborough Centenary Health Centre, Scarborough, Ontario.
Dr Tu is supported by a Health Research Personnel Development Program Fellowship (04544) from the Ontario Ministry of Health. Dr Jaglal was supported by an Ontario Heart and Stroke Foundation Post-doctoral Fellowship. Dr Naylor is supported by a Career Scientist Award (02377) from the Ontario Ministry of Health. The results and conclusions are those of the authors, and no official endorsement by the sponsoring agencies is intended or should be inferred. The authors would like to thank the many cardiologists and cardiac surgeons of the Provincial Adult Cardiac Care Network of Ontario, the institutional and regional coordinators who are responsible for data collection and patient follow-up, and the staff of the PACCN coordinating office at Victoria Hospital, London, Ontario.
Reprint requests to Dr C. David Naylor, Institute for Clinical Evaluative Sciences, Rm G-106, 2075 Bayview Ave, North York, Ontario, Canada M4N 3M5.
↵1 List of Steering Committee members is provided in “Appendix.”
- Received December 16, 1994.
- Accepted December 16, 1994.
- Copyright © 1995 by American Heart Association
Parsonnet V, Dean D, Bernstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation. 1989;79(suppl I):I-3-I-12.
O’Connor GT, Plume SK, Olmstead EM, Coffin LH, Morton JR, Maloney CT, Nowicki ER, Levy DG, Tryzelaar JF, Hernandez F, et al, for the Northern New England Cardiovascular Disease Study Group. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery. Circulation. 1992; 85:2110-2118.
Bryson G, Mundi C, Dupuis JY, Bourke M. Can multifactorial classifications predict outcome after cardiac surgery in patients with preoperative renal dysfunction? Can J Anaesth. 1994;41:A13. Abstract.
Orr RK, Maini BS, Sottile F, Dumas E. Severity adjusted models fail to accurately predict mortality in coronary artery bypass graft surgery. Med Decis Making. 1994;14:438. Abstract.
Tu JV, Mazer CD, Levinton C, Armstrong PW, Naylor CD. A predictive index for length of stay in the intensive care unit following cardiac surgery. Can Med Assoc J. 1994;151:177-185.
Naylor CD, Baigrie RS, Goldman BS, Basinski A. Revascularisation Panel and Consensus Methods Group: assessment of priority for coronary revascularisation procedures. Lancet. 1990;335: 1070-1073.
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143:29-36.
STATA. Santa Monica, California: Computing Resource Center; 1992.
Hosmer DW, Lemeshow S. Applied Logistic Regression. New York, NY: John Wiley & Sons; 1989.