Preoperative Renal Risk Stratification
Background After cardiac surgery, acute renal failure (ARF) requiring dialysis develops in 1% to 5% of patients and is strongly associated with perioperative morbidity and mortality. Prior studies have attempted to identify predictors of ARF but have had insufficient power to perform multivariable analyses or to develop risk stratification algorithms.
Methods and Results We conducted a prospective cohort study of 43 642 patients who underwent coronary artery bypass or valvular heart surgery in 43 Department of Veterans Affairs medical centers between April 1987 and March 1994. Logistic regression analysis was used to identify independent predictors of ARF requiring dialysis. A risk stratification algorithm derived from recursive partitioning was constructed and was validated on an independent sample of 3795 patients operated on between April and December 1994. The overall risk of ARF requiring dialysis was 1.1%. Thirty-day mortality in patients with ARF was 63.7%, compared with 4.3% in patients without ARF. Ten clinical variables related to baseline cardiovascular disease and renal function were independently associated with the risk of ARF. A risk stratification algorithm partitioned patients into low-risk (0.4%), medium-risk (0.9% to 2.8%), and high-risk (≥5.0%) groups on the basis of several of these factors and their interactions.
Conclusions The risk of ARF after cardiac surgery can be accurately quantified on the basis of readily available preoperative data. These findings may be used by physicians and surgeons to provide patients with improved risk estimates and to target high-risk subgroups for interventions aimed at reducing the risk and ameliorating the consequences of this serious complication.
Acute renal failure, defined as a 50% increase in the serum creatinine concentration from baseline, occurs in ≈5% of hospitalized medical and surgical patients.1 Risk factors for the development of ARF include decreased renal perfusion and exposure to nephrotoxic agents, such as aminoglycosides and radiocontrast.1 2 Specifically after cardiac surgery, the risk of ARF ranges from 5% to 31%, depending on the criteria used to define the complication.3 4 5 6 7 8 9 10 One percent to 5% of patients develop severe disease, defined as a serum creatinine concentration >442 μmol/L (5 mg/dL) or the need for dialysis, both of which are accompanied by a marked increase in perioperative mortality. Although several studies have identified preoperative and intraoperative risk factors for the development of severe ARF after cardiac surgery, none have had sufficient power to measure the independent effects of these variables on the risk of this serious outcome or to develop a risk stratification algorithm.
We hypothesized that ARF after cardiac surgery would be strongly associated with two major factors: (1) occult renal ischemia (associated with poor cardiac performance, fixed atherosclerotic disease of the renal arteries, and/or prolonged hypoxemia) and (2) reduced renal functional reserve. It is important to develop methods of identifying patients at high risk for perioperative ARF because renal ischemia is generally silent, unlike ischemia of the coronary, cerebral, and peripheral vascular beds, which are usually overt, manifested by angina pectoris, neurological sequelae (eg, hemiparesis, aphasia), and claudication, respectively. Because signs and symptoms are unlikely to direct attention toward patients at high perioperative renal risk, prognostic stratification using reliable surrogates may help guide clinical decision making.
We analyzed data collected for the VA CICSS. Data collection began in April 1987, with the following primary goals: (1) to prospectively collect preoperative risk and postoperative outcome data on all patients undergoing cardiac surgery in VA medical centers, (2) to develop risk-adjusted models for estimating perioperative mortality and morbidity, and (3) to apply these data to promote quality improvement and improve clinical decision making. The CICSS selected 54 preoperative variables as possible risk factors for postoperative mortality and morbidity based on work from the Coronary Artery Surgery Study,11 previous analyses of determinants of mortality and morbidity in valvular heart surgery,12 clinical judgment, and the likelihood of complete data collection. With the first 3 years' experience, 19 preoperative variables were deleted, leaving 35 preoperative variables, along with 12 outcome variables. Data were collected by members of the cardiology–cardiac surgery team during the first half of the study period (April 1987 to September 1990), who completed data collection on 73% of all cardiac operations within the VA system. During the latter half of the study period, data were collected by funded research nurses, who achieved a completion rate of nearly 100%.
For the purposes of this study, 43 642 subjects operated on between April 1987 and March 1994 were considered. We excluded subjects with a baseline serum creatinine of ≥265.2 μmol/L (3.0 mg/dL) (n=537, 1.2%) and/or with active endocarditis at the time of surgery (n=391, 0.9%), leaving 42 773 subjects eligible for analysis. Later, we prospectively validated our models on an independent sample of 3795 CICSS subjects operated on between April and December of 1994.
The demographic and anthropometric variables studied as potential risk factors were age, sex, height, weight, body mass index (weight divided by height squared), and body surface area.13 The clinical and laboratory variables evaluated were cardiomegaly (generalized cardiac enlargement on chest radiograph within 30 days before surgery), cerebral vascular disease (manifested by previous stroke and/or transient ischemic attack), and/or prior surgical repair (eg, carotid endarterectomy), and/or carotid arterial obstruction of ≥50% of luminal diameter by contrast angiography or duplex ultrasonography, chronic obstructive pulmonary disease (resulting in functional disability and/or hospitalization, and/or requiring chronic bronchodilator therapy, and/or FEV1 <75% predicted), current digoxin or diuretic use (within 2 weeks before surgery), current tobacco use (within 2 weeks before surgery), diabetes mellitus (requiring therapy with oral agent or insulin), functional status (independent, partially dependent, totally dependent), intravenous nitroglycerin use (within 48 hours before surgery), percent stenosis of left main coronary artery, left ventricular ejection fraction (assessed by preoperative contrast ventriculography, radionucleotide ventriculography, or two-dimensional echocardiography), NYHA functional class (angina and/or congestive heart failure), number of major coronary artery stenoses (≥50%), number of coronary artery anastomoses, peripheral vascular disease (manifested by exertional claudication and/or rest pain, and/or prior revascularization procedure to legs, and/or absent or diminished pulses in legs, and/or angiographic evidence of noniatrogenic peripheral arterial obstruction of ≥50% of luminal diameter), preoperative use of an intra-aortic balloon pump (within 2 weeks before surgery), prior heart surgery, prior myocardial infarction, prior percutaneous transluminal coronary artery angioplasty, pulmonary rales (not clearing with cough and not due to pneumonic process within 2 weeks before surgery), resting angina, resting ST-segment depression (>1 mm in any lead on standard ECG, and/or ECG diagnosis of subendocardial ischemia, left ventricular strain, or left ventricular hypertrophy with repolarization abnormality), serum creatinine concentration (mg/dL), systolic blood pressure (mm Hg), and surgical priority (elective, urgent, emergent).
Additional preoperative data were obtained on selected patients (eg, pulmonary function tests, pulmonary artery and left ventricular pressure measurements) but were not analyzed in detail, because routine collection of these data was not uniformly required. Operative mortality was defined as death from any cause within 30 days after surgery or death occurring at any time after surgery directly related to a complication of surgery. ARF was defined as a deterioration in renal function sufficient to require dialysis within 30 days after surgery. The Cockcroft-Gault formula was used to estimate baseline creatinine clearance.14 Serial serum creatinine and urea nitrogen concentrations were not obtained, nor were data provided regarding the specific indications for or the modality, membrane, timing, or intensity of dialysis.
The means of continuous variables were compared by Student's t test. Binary variables were compared by the χ2 test stratified by surgery type (coronary artery bypass or valvular surgery).15 Linear tests for trend were used for ordinal variables. Tests for the homogeneity of ORs across strata were based on the weighted sum of squared deviations of the stratum-specific log-ORs from their weighted mean.16 Factors with two-tailed values of P<.05 on univariate tests were considered candidate variables for multivariable analyses. Multiple logistic regression was conducted using both the forward stepwise and backward elimination methods, with entry and exit criteria set at the P=.05 level.17 Variables not selected by the automated logistic regression methods were reentered individually to evaluate for residual confounding. Multiplicative interaction terms were created when the stratified ORs or tests for trend across strata were significantly different. ORs and 95% CIs were calculated on the basis of the estimated model parameter coefficients and standard errors, respectively. Competing logistic models were compared with the log-likelihood test. Missing data were handled in two ways. First, all individuals with missing data were excluded from the analyses. Second, missing binary data were coded as “not present,” while missing categorical data were coded as “missing” (eg, five categories of systolic blood pressure: <120, 120 to 139, 140 to 159, and ≥160 mm Hg, and missing). The final logistic regression model was validated with a 100-sample bootstrap. The bootstrap randomly selects a predetermined number of subjects from the original data set with replacement and repeatedly reestimates regression parameters and standard errors of the model.18
To construct an algorithm for stratifying perioperative renal failure risk, recursive partitioning was used.19 20 The aim of recursive partitioning is to repeatedly divide patients into subgroups, each of which ideally consists of patients with or without ARF. It thus provides a nonparametric discriminating tree whose construction is based on interactions among clinical factors chosen for discriminating power. In contrast to logistic regression, recursive partitioning does not assume linear relationships among variables of interest and the log odds of developing ARF. Although recursive partitioning can more easily identify interaction among variables, it is not as effective as logistic regression at identifying independent risk factors, nor does it rank these factors by importance (eg, OR) or degree of stability (eg, CI, P value).
Two-tailed values of P<.05 were considered significant. Statistical analyses were done with SAS (The SAS Institute) and S-Plus (Statistical Sciences, Inc).
Of the 42 773 patients analyzed, 34 874 (81.5%) underwent coronary artery bypass surgery and 7899 (18.5%) underwent valvular surgery with or without coronary artery bypass. The overall risk of ARF requiring dialysis was 1.1%: 0.9% among coronary artery bypass patients and 2.0% among valvular surgery patients. The majority of patients who received valvular surgery underwent aortic valve replacement (68.8%) with or without other procedures, such as great vessel repair. A smaller percentage underwent mitral valve replacement (21.8%), tricuspid valve replacement (0.7%), or valve repair alone (8.7%). The unadjusted risk of ARF requiring dialysis was similar among all non–coronary artery bypass subgroups; these were considered together as “valvular surgery” in the primary analyses. The overall risk of ARF ranged from 0.9% to 1.3% over the 7-year study period; there were no significant differences in the risk of ARF over time (P=.68).
Table 1⇓ displays baseline characteristics of study subjects according to surgery type and the presence or absence of ARF requiring dialysis. Among both surgery groups, one can appreciate the substantial degree of comorbidity present in the population and trends toward increasing comorbidity among those who developed ARF.
Risk of ARF
The risk of ARF increased with age across both surgical strata (χ2 for trend, P<.0001). Among coronary artery bypass recipients, the risk of ARF was 0%, 0.3%, 0.5%, 0.9%, 1.5%, and 1.8% in patients <40, 40 to 49, 50 to 59, 60 to 69, 70 to 79, and ≥80 years old, respectively. Among valvular surgery patients, the corresponding risks were 1.4%, 1.1%, 1.8%, 2.0%, 2.5%, and 0.8% (Fig 1⇓). There were relatively few patients (1.9% of total sample) at both extremes of age. It was difficult to assess sex-specific risk in this cohort, because <1% of subjects were women. The incidence of ARF in women was 1.2% in coronary artery bypass patients and 2.3% in those who underwent valvular surgery (OR, 1.29; 95% CI, 0.57 to 2.90). Data on race or ethnicity were not obtained. Body mass index, a measure of obesity, was unrelated to the risk of ARF. In contrast, body surface area, an anthropometric measure estimating overall body size, was inversely correlated with ARF risk (P<.0001).
ARF was strongly associated with baseline renal insufficiency. The risk of ARF was 0.5%, 0.8%, 1.8%, and 4.9% in patients with baseline serum creatinine concentrations <88.4 μmol/L (1 mg/dL), 88.4 to 123.8 μmol/L (1.0 to 1.4 mg/dL), 132.6 to 168.0 μmol/L (1.5 to 1.9 mg/dL), and 176.8 to 256.4 μmol/L (2.0 to 2.9 mg/dL), respectively. With the Cockcroft-Gault equation, a more precise estimate of renal function, there was a linear increase in risk with decreasing estimated creatinine clearance (χ2 for trend, P<.0001, Fig 2⇓). The risk of ARF was significantly increased among patients with cardiomegaly (OR, 1.74; 95% CI, 1.43 to 2.12), cerebral vascular disease (OR, 1.89; 95% CI, 1.48 to 2.41), chronic obstructive pulmonary disease (OR, 1.55; 95% CI, 1.28 to 1.88), current digoxin use (OR, 1.46; 95% CI, 1.17 to 1.83), current diuretic use (OR, 1.94; CI, 1.61 to 2.34), diabetes mellitus (OR, 1.43; 95% CI, 1.08 to 1.89), dependent noncardiac functional status (OR, 1.71; 95% CI, 1.26 to 2.34), intravenous nitroglycerin use (OR, 2.19; 95% CI, 1.77 to 2.70), left main coronary artery stenosis >70% (OR, 2.23; 95% CI, 1.74 to 2.85), left ventricular ejection fraction <35% (OR, 2.03; 95% CI, 1.60 to 2.56), NYHA class IV status (OR, 2.12; 95% CI, 1.78 to 2.54), peripheral vascular disease (OR, 2.17; 95% CI, 1.80 to 2.63), preoperative intra-aortic balloon pump (OR, 4.57; 95% CI, 3.57 to 5.86), prior heart surgery (OR, 2.24; 95% CI, 1.82 to 2.75), pulmonary rales (OR, 2.34; 95% CI, 1.88 to 2.91), recent (<7 days) myocardial infarction (OR, 2.71; 95% CI, 1.88 to 3.91), resting angina (OR, 1.56; 95% CI, 1.22 to 1.99), and resting ST-segment depression (OR, 1.41; 95% CI, 1.11 to 1.80). The risk of ARF was unrelated to current tobacco use, number of major coronary artery stenoses or surgical anastomoses, remote myocardial infarction, or a history of percutaneous coronary artery angioplasty.
The association of systolic blood pressure and ARF risk was dependent on the type of cardiac surgery. Among coronary artery bypass patients, the risk of ARF rose monotonically from 0.6% to 1.6% as preoperative systolic blood pressure rose from <120 to ≥160 mm Hg (χ2 for trend, P<.0001). In contrast, valvular surgery patients with low systolic blood pressure were at the highest risk of ARF (2.6%), and no trend in risk with increasing systolic blood pressure was evident (Fig 3⇓). Surgical priority was strongly related to ARF risk, which increased in an exponential fashion from elective to urgent to emergent (χ2 for trend, P<.0001).
In an effort to evaluate the independent contributions of predictors identified by univariate screening, we estimated the probability of developing ARF requiring dialysis by multiple logistic regression. Models developed with and without missing data were not appreciably different. The final model is presented in Table 2⇓. Creatinine clearance and systolic blood pressure are categorized; ORs and CIs are presented for each category. Inclusion of multiplicative interaction terms for systolic blood pressure and surgery type substantially improved the model fit (P<.001). Several of the model's derived ORs were attenuated compared with the corresponding univariate tests, suggesting positive confounding among selected covariates. Advanced age and diabetes mellitus were among those factors not independently associated with the risk of ARF requiring dialysis in the logistic regression model. The area under the receiver operating characteristic curve was 0.76, indicating good model discrimination.
Model cross validation using a 100-sample, 42 773–observation bootstrap analysis was performed. Among the 100 random samples, the number of “cases” of ARF was 459±22 (range, 410 to 509). None of the covariate parameter or standard error estimates differed by ≥5% compared with the primary logistic regression model.
A clinical risk algorithm was created by use of recursive partitioning. This method separated the sample into 11 separate groups based on interactions between key discriminating variables (Fig 4⇓). The risk of ARF requiring dialysis ranged from 0.4% in patients without a history of prior heart surgery and with normal or near-normal baseline renal function who underwent coronary artery bypass to ≥5% in patients with baseline renal insufficiency and (1) with peripheral vascular disease who underwent valvular surgery, (2) with cardiomegaly and NYHA class IV functional status, and (3) who required preoperative placement of an intra-aortic balloon pump (Fig 4⇓).
The risk algorithm was prospectively validated on an independent sample of 3795 patients. The mean age of the validation set was 63.3±9.2 years, and 38 (1.0%) were female. The risk of ARF requiring dialysis in the validation set was 1.4% (53/3795), compared with 1.1% in the larger derivation set (P<.0001). Nevertheless, the risk algorithm performed well on prospective testing. The rates of ARF were similar among subgroups in the derivation and validation sets, as was the overall ranking of subgroups by risk (Table 3⇓). Compared with the derivation set, only group A (creatinine clearance <60 mL/min [1.0 mL/s] with preoperative intra-aortic balloon pump) was substantially different (3.2% versus 9.5%). Groups H and I appeared to be similar in the validation set (4.9% and 5.9%, respectively) compared with the derivation set (2.3% and 5.0%, respectively), although these results were based on relatively few outcomes.
Outcomes Associated with ARF
The overall risk of operative mortality in patients with ARF requiring dialysis was 63.7%, compared with 4.3% in patients without this complication (P<.0001). Surgical morbidity was also more common. There were significantly increased risks of postoperative myocardial infarction (17.8% versus 3.0%, P<.0001), reoperation for bleeding (18.3% versus 3.5%), mediastinitis (11.1% versus 1.7%, P<.0001), and endocarditis (2.0% versus 0.1%, P<.0001) among patients with ARF requiring dialysis. Among these 460 subjects, low cardiac output requiring inotropic support (OR, 2.46; 95% CI, 1.61 to 3.77), prolonged mechanical ventilation (OR, 2.13; 95% CI, 1.15 to 3.94), cardiac arrest (OR, 2.36; 95% CI, 1.28 to 4.35), and stroke or coma (OR, 2.20; 95% CI, 1.40 to 3.44) were independent predictors of perioperative mortality.
ARF is one of the most serious complications of cardiac surgery. When it is severe enough to require dialysis, morbidity and mortality are markedly increased despite dialysis and supportive intensive care. In past years, several investigators have attempted to identify risk factors for the development of ARF after cardiac surgery,3 4 5 6 7 8 9 10 but none have had sufficient sample size to specifically identify risk factors for severe or dialysis-requiring ARF or to adjust for confounding among clinical variables. Abel and colleagues6 evaluated 500 consecutive cardiac surgery patients, of whom 35 (7%) developed moderate to severe ARF, defined as a serum creatinine concentration >442 μmol/L (5 mg/dL), and 15 (3%) required dialysis (all of whom died). An additional 102 (20%) developed more mild degrees of azotemia. Preoperative correlates of ARF included advanced age and baseline renal dysfunction; intraoperative correlates included duration of surgery, cardiopulmonary bypass, and aortic cross-clamping. Similarly, Corwin et al10 found that preoperative serum creatinine levels, advanced age, and concurrent coronary artery bypass and valvular surgery predicted an increased risk of ARF after cardiac surgery. Only six patients in this cohort required dialysis.
The analyses reported here strongly support our initial hypotheses. ARF requiring dialysis was associated with conditions that cause occult renal ischemia, such as reduced ejection fraction, peripheral vascular disease, and clinical signs of poor cardiac performance (pulmonary rales, NYHA class IV status) as well as with reduced renal functional reserve, as demonstrated by the striking association of ARF with preoperative estimates of creatinine clearance. Recursive partitioning confirmed the importance of these risk factors and suggested that there are important interactions among baseline renal function, prior heart surgery, cardiomegaly, and peripheral vascular disease.
Some investigators have argued that the marked increase in morbidity and mortality associated with ARF among postoperative and other critically ill patients is related solely to comorbid disease. We cannot disprove this contention. Although ARF remains a significant predictor of mortality risk even after other important model covariates are controlled for21 (data not shown), it may itself be a proxy for one or several unobserved covariates. Alternatively, complications of ARF (eg, volume overload, immune dysfunction) or complications of dialysis may contribute directly to the risk of death. Of note, recent works exploring the proinflammatory effects of membrane bioincompatibility suggest that modifications in the dialysis procedure may affect the rate of sepsis, multiple organ failure, and renal (and other organ) recovery.22 23 If ARF or its management contribute directly to mortality and morbidity, efforts at securing its prevention become increasingly important.
Although the major findings were not unexpected, our study was the first in this area to have had sufficient power to determine independent risk factors for ARF, to compare various factors by the strength and stability of their associations, and to explore interactions among key variables, such as systolic blood pressure and surgery type. Although most of the identified factors are irremediable (eg, prior heart surgery, peripheral vascular disease), it is important to recognize that other factors, such as pulmonary rales or the use of an intra-aortic balloon pump, may influence the risk of ARF and that this risk be considered in the timing of surgical intervention. Finally, results from this study allow us to identify high-risk subgroups who might be optimal candidates for strategies in prospective clinical trials aimed at reducing the incidence of ARF after cardiac surgery.
There are several important limitations to this study. The sample was largely restricted to white men. Blacks may be more susceptible to the adverse renal effects of systemic hypertension and other vascular diseases, and women and blacks may have reduced renal functional reserve.24 Ideally, our results should be revalidated in other populations, especially those with a large proportion of women, to ensure their validity. Use of the Cockcroft-Gault equation to estimate creatinine clearance may have led to misclassification of some subjects, particularly those at the extremes of body composition, because adjustment for weight alone may not capture differences in creatinine generation among patients who are obese, muscular, or marasmic. A more precise measure of renal function, such as a timed collection or radioisotope study, would be preferable. Although we examined many preoperative clinical variables, several were not considered, including gastrointestinal and hepatic dysfunction, important correlates of renal failure in other settings.25 Prediction of the outcome could probably be improved with intraoperative and/or postoperative data (eg, duration of aortic cross-clamping, prolonged intraoperative hypotension, cholesterol embolization, exposure to nephrotoxic agents), although these data could not be used preoperatively to stratify renal risk. Indeed, the “valvular surgery” variable may be a surrogate for some of these intraoperative factors.
Selection bias may have masked or attenuated several important risk factors. For example, patients with the most severe forms of diabetes mellitus may not have been offered cardiac surgery, so that the impact of this condition on ARF risk might have been diminished in the sample available for analysis. A similar preoperative selection process might have lessened the effect of advanced age, which, like diabetes mellitus, was not associated with ARF after peripheral vascular disease, prior heart surgery, and other related variables were adjusted for. Finally, the validation set was relatively small. Although most groups had similar rates of ARF, some varied by substantially more than the difference in baseline risk (1.4% versus 1.1%, 23% relative increase). Repeated validation of our model in other populations will be required to more precisely assess the stability of the risk estimates.
In summary, the likelihood of developing ARF after cardiac surgery depends on factors associated with poor cardiac performance and advanced atherosclerotic vascular disease. These factors, in combination with reduced baseline renal function, can be used to stratify patients before surgery and to identify several subgroups of patients at substantially increased risk (≥5%). We do not intend for these data to be used to withhold or advise against required cardiac surgery. Rather, we hope that these data will be used to promote quality enhancement in perioperative care and to target high-risk subgroups for interventions aimed ultimately at reducing the risk and ameliorating the consequences of this devastating complication.
Selected Abbreviations and Acronyms
|ARF||=||acute renal failure|
|CICSS||=||Continuous Improvement in Cardiac Surgery Study|
|VA||=||Department of Veterans Affairs|
Dr Chertow was a recipient of the American Kidney Fund–Amgen Clinical Scientist in Nephrology Award. Dr Daley is a Senior Research Associate in the Career Development Program, Health Services Research and Development Service, Department of Veterans Affairs.
- Received March 28, 1996.
- Revision received September 5, 1996.
- Accepted September 30, 1996.
- Copyright © 1997 by American Heart Association
Doberneck RC, Reiser MP, Lillehei CW. Acute renal failure after open-heart surgery utilizing extracorporeal circulation and total body perfusion. J Thorac Cardiovasc Surg. 1962;43:441-452.
Yeh TJ, Brackney EL, Hall DP, Ellison RG. Renal complications of open-heart surgery: predisposing factors, prevention, and management. J Thorac Cardiovasc Surg. 1964;47:79-95.
Yeboah ED, Petrie A, Pead JL. Acute renal failure and open-heart surgery. Br Med J. 1972;1:412-418.
Bhat JG, Gluck MC, Lowenstein J, Baldwin DS. Renal failure after open heart surgery. Ann Intern Med. 1976;84:677-682.
Corwin HL, Sprague SM, DeLaria GA, Norusis MJ. Acute renal failure associated with cardiac operations. J Thorac Cardiovasc Surg. 1989;98:1101-1112.
Kennedy JW, Kaiser GC, Fisher LD, Maynard C, Fritz JK, Myers W, Mudd JG, Ryan TJ, Coggin J. Multivariate discriminant analysis of the clinical and angiographic predictors of operative mortality from the Collaborative Study in Coronary Artery Surgery (CASS). J Thorac Cardiovasc Surg. 1980;80:876-887.
DuBois D, DuBois EF. A formula to estimate the approximate surface area if height and weight be known. Arch Intern Med. 1916;17:863-871.
Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst. 1959;22:719-748.
Rothman KJ. Modern Epidemiology. Boston, Mass: Little, Brown; 1986.
Kleinbaum DG, Kupper LL, Muller KE. Straight-line regression analysis. In: Applied Regression Analysis and Other Multivariable Measures. Boston, Mass: Duxbury Press; 1988:41-79.
Efron B, Gong G. A leisurely look at the bootstrap, the jackknife, and cross validation. Am Statistician. 1983;37:36-48.
Gordon L, Olshen RA. Consistent nonparametric regression from recursive partitioning schemes. J Multivariate Anal. 1980;10:611-627.
Clark LA, Pregibon D. Tree-based models. In: Chambers JM, Hastie TJ, eds. Statistical Models in S. Pacific Grove, Calif: Wadsworth & Brooks, Cole Advanced Books and Software; 1992:377-419.