Response to Letters Regarding “Agreement Among Cardiovascular Disease Risk Calculators”
We thank Drs Payne and Webb and Drs Echouffo-Tcheugui and Kengne for their letters regarding our article “Agreement Among Cardiovascular Disease Risk Calculators.”1
Drs Payne and Webb raised concern we did not explore why differences exist in the risk calculator estimations or examine how risk estimation may influence patient/clinician behavior. Those are both important issues. As mentioned in our article, our objective was to assess the consistency of a broad sample of commonly used calculators over a sample of patients with a range of cardiovascular risk factors. We wanted to describe the disagreement among calculators (if it occurred). The next but separate question is why does it occur. We explored some subgroups that might help explain the differences and found focusing on Framingham 10-year Cardiovascular Disease risk calculators improved agreement (same database, same duration, and similar outcomes). Furthermore, we also discussed some of the potential causes of disagreement in the conclusion. We focused on describing agreement primarily and stated that more research is needed in understanding calculator inconsistency. We plan future research to investigate how different risk calculators weigh risk factors. As far as how patients/clinicians are influenced by risk estimation, that is another area entirely.
Drs Echouffo-Tcheugui and Kengne focus much of their discussion on how the comparisons of our study do not take into account the many nuances that may contribute to the variations in the risk estimations from different calculators. In general, this is true. As mentioned, we did some comparisons in grouping by outcomes, duration, or database used to derive the calculators and discussed some potential causes of disagreement. However, the scope of our study was to identify and describe the level of agreement among the calculators, particularly risk assignment. Regardless of the details of why differences may exist, it is important that users of risk calculators understand that the risk category assigned by one calculator may not be the same as another. Our study serves to provide easy to understand values, percent agreement, to help readers and clinicians know the agreement between individual calculators.
Echouffo-Tcheugui and Kengne suggest we should only have used calculators recommended by guidelines. As explained in our Methods, we selected a number of calculators from guidelines but also wanted ones outside of the guidelines for broad and comprehensive comparison. Fortunately, our methods of reporting allow readers to pick guideline-associated calculators and compare them specifically. They also comment that using UK Prospective Diabetes Study risk calculator for nondiabetics was inappropriate. We debated whether to include it but decided we would and fully explained how we tried to simulate non−diabetes mellitus. We thought it would allow readers to see how a calculator that is not designed for nondiabetics fared versus calculators designed for nondiabetics. Interestingly, it fared better than the average (67% agreement with other calculators versus the 64% overall non−diabetes mellitus average). They also raised concerns that the use of the small sample of hypothetical patients may be inadequate. We discussed this issue in the Limitations section of our paper.
Both groups of authors comment that the results are not surprising. This may be the case for researchers and writers who have done a fair amount of work investigating and explaining risk assessment and estimation. However, there was no large comparison of a broad range of risk calculators in the literature to confirm this suspicion (until now). Furthermore, we believe primary care clinicians and perhaps most clinicians managing cardiovascular disease risk are not familiar with the degree of disagreement among the calculators. Lastly, we felt it important to quantify the level of agreement to help clinicians/researchers recognize that risk estimation from one calculator is not necessarily the same as another (or any other).
Although the writers suggest that the analyses may have been too simplistic, we were attempting to simplify a complex issue. Risk calculators are complicated, and the use of different time frames, outcomes, risk cut-offs, and calibration contributes to this. However, we believe most frontline clinicians do not fully comprehend or perhaps even consider these nuances. We believe calculators are often used to attain a risk category for decision-making. We hope no one has interpreted our results as a justification to dispose of risk estimation. It remains a useful tool in shared decision-making and treatment choices. Our study serves as an important reminder that (1) risk calculators are not interchangeable, (2) a calculator should be suited to their patient population, and (3) the estimations have margins of error.
G. Michael Allan, MD
Faeze Nouri, MD
Christina Korownyk, MD
Michael R Kolber, MD MSc
Department of Family Medicine
University of Alberta
Edmonton, Alberta, Canada
Ben Vandermeer, MSc
Alberta Research Centre for Health Evidence
University of Alberta
Edmonton, Alberta, Canada
James McCormack, PharmD
Faculty of Pharmaceutical Sciences
University of British Columbia
Vancouver, British Columbia, Canada
Dr McCormack is developing a free-access cardiovascular disease risk calculator based on Framingham, designed to promote shared informed decision-making. The other authors report no conflicts.
- © 2013 American Heart Association, Inc.