Letter by Bollati et al Regarding Article, “Off-Pump Versus On-Pump Coronary Artery Bypass Graft Surgery: Differences in Short-Term Outcomes and in Long-Term Mortality and Need for Subsequent Revascularization”
To the Editor:
We read with great interest the recent report by Hannan et al1 on the short-term benefits of off-pump coronary artery bypass graft surgery compared with the standard on-pump approach on in-hospital/30-day mortality and perioperative complication rates. The 3-year follow-up proved comparable outcomes, excluding a lower rate of freedom from revascularization for the off-pump surgery group. The topic is certainly extremely interesting, especially nowadays, when less invasive approaches, such as off-pump coronary surgery and percutaneous coronary interventions, are acquiring dominant roles in the treatment of ischemic heart disease.2
However, we believe that this work is fraught with a number of major drawbacks concerning the statistical methods that may potentially mislead readers and clinicians. The rising importance of evidence-based medicine has given medical statistics a central role in clinical practice, and therefore the medical community must rely on established criteria; recommendations for multivariable and propensity model development and assessment are in fact available.3 An extremely relevant step when analyzing data from a study is to decide which elements should be included as covariables in the regression models, and a number of methods may support this decision. In the study by Hannan et al,1 a stepwise selection was used. The stepwise selection, applied to identify covariables for inclusion in the regression models, has proved an unstable and potentially biased method.3 In fact, including all measured variables in a nonparsimonious model may bias the logistic regression analysis, determining a lower confidence in the predicted probability.4,5
Furthermore, Hannan et al1 conducted a propensity analysis to adjust for selection bias in patients treated with off-pump or on-pump coronary bypass surgery. The propensity score is intended as the conditional probability of being assigned to a treatment group given a set of pretreatment characteristics. With the use of this expedient, the conditional distribution would be similar for both treatment groups. The discrimination of the model, intended to show how well the predicted probabilities derived from the model classify patients into their actual treatment group, should be measured by evaluating the area under the receiver operator curve or the equivalent c statistic. Unfortunately, none of these indexes are reported, and thus there is no proof of the predictive accuracy of the pivotal propensity score.5 In addition, an important method to evaluate how well the model describes the data is the assessment of the model fit. The goodness of fit tests, which evaluate whether the differences between the observed (on-pump or off-pump) treatment and the predicted outcomes from the model (propensity score) are not determinant and not systematic, are unavailable. Thus, the propensity scores used from a potentially suboptimal fit model do not permit any evaluation of the treatment groups.
In conclusion, given the contradictory findings regarding short- and long-term benefits of off-pump coronary surgery and the aforementioned statistical limitations, the results by Hannan et al1 may be considered hypothesis generating at best.