Electronic Alerts Versus On-Demand Decision Support to Improve Dyslipidemia Treatment
A Cluster Randomized Controlled Trial
Background— Indirect evidence shows that alerting users with clinical decision support systems seems to change behavior more than requiring users to actively initiate the system. However, randomized trials comparing these methods in a clinical setting are lacking. We studied the effect of both alerting and on-demand decision support with respect to screening and treatment of dyslipidemia based on the guidelines of the Dutch College of General Practitioners.
Methods and Results— In a clustered randomized trial design, 38 Dutch general practices (77 physicians) and 87 886 of their patients (39 433 men 18 to 70 years of age and 48 453 women 18 to 75 years of age) who used the ELIAS electronic health record participated. Each practice was assigned to receive alerts, on-demand support, or no intervention. We measured the percentage of patients screened and treated after 12 months of follow-up. In the alerting group, 65% of the patients requiring screening were screened (relative risk versus control=1.76; 95% confidence interval, 1.41 to 2.20) compared with 35% of patients in the on-demand group (relative risk versus control=1.28; 95% confidence interval, 0.98 to 1.68) and 25% of patients in the control group. In the alerting group, 66% of patients requiring treatment were treated (relative risk versus control=1.40; 95% confidence interval, 1.15 to 1.70) compared with 40% of patients (relative risk versus control=1.19; 95% confidence interval, 0.94 to 1.50) in the on-demand group and 36% of patients in the control group.
Conclusion— The alerting version of the clinical decision support systems significantly improved screening and treatment performance for dyslipidemia by general practitioners.
Received March 16, 2007; accepted October 5, 2007.
Computerized clinical decision support systems (CDSS) are information systems that aim to optimize physicians’ clinical decision making.1 Investigators report a beneficial effect on physicians’ performance of introducing CDSS in daily practice.2,3 Several determinants appear to be critical to the successful introduction of CDSS in daily practice: providing automatic support as part of the clinician’s workflow, providing decisions at the point of care, and providing recommendations rather than just assessments.4 In a recent systematic review, Garg et al2 studied the effect of CDSS on practitioner performance and patient outcomes. They concluded that studies in which users were automatically prompted (alerted) to use the system seemed to indicate better performance than studies in which users were required to actively (on-demand) initiate the system. However, this conclusion was based on comparing the results of studies conducted in different settings, with the use of different methods, and involved heterogeneous populations. A randomized controlled trial comparing the automatic alerting method with the on-demand method on physicians’ performance is lacking.
Editorial p 336
Clinical Perspective p 378
Cardiovascular disease (CVD) is the leading cause of mortality in industrialized countries.5 Researchers report low primary and secondary prevention performance of physicians.6–11 To improve both primary and secondary prevention of CVD, we developed the noncommercial CDSS, named CholGate, on the basis of recommendations for lipid management from the guidelines of the Dutch College of General Practitioners (DCGP).12–16 For more than a decade, >90% of Dutch primary care physicians have used electronic health records (EHR) for all primary care record activities.17 Taking advantage of this widespread use of EHR by Dutch general practitioners, the CDSS is integrated within the EHR to provide decision support as part of the clinician’s workflow. We conducted a randomized trial to compare the effect of 2 versions of this CDSS with no CDSS on screening and treatment of dyslipidemia among general practitioners.
We hypothesize that a CDSS will improve physicians’ performance compared with having no CDSS. In addition, we hypothesize that the alerting method will have a better effect in increasing physician performance than a CDSS that requires manual activation.
Our practice-level, cluster randomized controlled trial studied alerting decision support or on-demand decision support versus usual care. We randomly assigned general practices to the 3 groups and followed the patients within the assigned groups. Between May and June 2004, we recruited general practices in the Delft region in the Netherlands.
Between May and June 2004, all of the 56 practices using the ELIAS EHR (iSOFT B.V., Leiden, the Netherlands) in the Delft region in the Netherlands were invited to participate in the study. We chose to integrate our intervention into a single EHR to avoid potential bias due to different EHR systems. Only practices that fully replaced paper-based records with electronic records during patient encounters and who have been working in this manner for ≥1 year were eligible for the study.
A total of 38 practices (80 general practitioners) with 92 054 eligible patients agreed to participate in the study. Table 1 lists the practice characteristics. Because the 1999 DCGP guidelines restricted lipid management recommendations to men aged 18 to 70 years and women aged 18 to 75 years, we included only these patients in our study.12–16 All patient data were anonymous. The need to obtain patient consent was waived by the Ethical Review Committee of the Erasmus Medical Center because, in accordance with Dutch law, the intervention constituted accepted care and physicians were not forced to follow any suggestions. The Figure shows the study flow.
The general practice, with ≥1 practitioners, was chosen as unit of randomization because it was a definable entity and a logical foundation for implementing a primary care–based intervention. Additionally, this prevented knowledge gained from decision support in one patient influencing the action on a possible control patient, thereby underestimating the effect of the intervention.18 Practices were randomly assigned into 1 of 3 arms: a control arm and 2 intervention arms (an on-demand arm and an alerting arm). The practices were first stratified by single-handed practices and group practices (ie, ≥2 general practitioners in the same practice). Each practice was subsequently assigned by simple random allocation to CDSS alerting, CDSS on-demand, or control groups for the complete study period. Randomization was performed with a table of random numbers by a researcher not involved in the study and who was blind to the identity of the practices. The practices were blinded to the other versions of the CDSS available.
The CDSS aids a general practitioner in the primary and secondary prevention of CVD with regard to lipid management. The system is based on the guidelines of the DCGP and is integrated into the EHR. The system uses 3 layers of data available in the EHR to provide decision support: coded data, structured data, and free text data. Coded data rely on widely used coding schemes to record data. In the Netherlands, all practices use the International Classification for Primary Care for coding diagnoses and the Anatomic Therapeutical Classification19 for coding medication. Structured data rely on locally defined coding schemes. For each practice we identified codes on the level of an individual practice used for specific conditions or measurements. Free text data consist of data recorded in free text. For each practitioner, we relied on the recording habits of physicians who, in busy daily practice, often use repetitive free text in their medical narrative.20 After gathering relevant data, the system identifies 2 types of patients: (1) those who need screening for lipid abnormalities and (2) those who need treatment for lipid abnormalities. The Dutch guidelines emphasize the need to screen specific, well-defined populations.21 As a result, the guidelines contain detailed, sometimes complex descriptions of the various conditions that should lead to screening a patient.22 For example, a 50-year–old male smoker without diabetes mellitus or CVD but with hypertension needs screening, whereas as if the patient does not have hypertension, screening is not indicated. If the patient is aged ≥60 years and smokes but has no other risk factors, screening is again indicated. All patients with known CVD needed screening. When a patient is identified as requiring screening, the guidelines recommend specific actions. The 1999 DCGP guidelines require calculation of a risk score for every patient who needs primary prevention.12–16 This risk score is based on the Framingham risk function for CVD.23 In the 1999 paper-based guidelines, the physician had to deduce the score from a series of tables stratified by age, gender, diabetes mellitus, systolic blood pressure, smoking, and cholesterol/high-density lipoprotein cholesterol ratio. For every age group, a threshold Framingham score exists in which treatment is recommended. Researchers have argued that the relevant tables are complex and might be difficult to use in daily practice.24 In secondary prevention, any patient with CVD attributed to atherosclerosis and with a serum cholesterol level >5 mmol/L (193 mg/dL) should be treated with statins. Once a patient is treated with statins, the 1999 guidelines did not recommend further serum lipid testing.
We developed 2 versions of the CDSS: on-demand and alerting. Both versions analyze and interpret the patient data in the EHR, generating patient-specific guideline recommendations for preventative activities. An overview screen, or dashboard, accessible from the patient record screen presents a patient’s current risk profile to the user, as well as suggesting an action on lipid management. A user can call on this overview screen at any point during the interaction with the patient’s record. This overview is interactive: The user can enter a new diagnosis or change measurement values, and the risk profile is immediately updated. All changes made in the overview are registered into the patient’s record, preventing duplication of tasks. In both versions the user has access to the overview screen to see a patient’s risk profile and recommendations. The only difference between the 2 versions is the alerting functionality.
In the on-demand version, a user has to actively initiate the overview screen to access the recommendations. In the alerting version, the recommendations are automatically shown to the user during record interaction; that is, the user does not need to initiate the overview screen to access the recommendations.
Outcomes and Measurements
The primary study outcomes were the percentage of screened and treated patients determined by the system according to the DCGP guidelines’ recommendations on lipid management in relation to the primary and secondary prevention of CVD.12–16 The percentage of screened patients is the number of patients requiring screening and then screened, divided by the total number of patients requiring screening. The percentage of treated patients is the number of patients requiring treatment and then treated, divided by the total number of patients requiring treatment.
When the system was implemented by the authors at the practices, it was configured to the local practice setting, and users were given a tutorial on the use of the system. We installed the on-demand version in the on-demand arm and the alerting version in the alerting arm. In the control arm, the CDSS was installed, but all functionality was disabled for daily use and only monitored the actions taken by the general practitioner. At all practices, an initialization procedure was performed directly after installation. This procedure gathered data on the practices’ baseline characteristics as well as baseline characteristics of the patient population.
After initialization, all data on patient record interactions, patient characteristics, and follow-up data were obtained from the EHR by the CDSS. Eligibility of patients for screening and treatment actions were determined from the computerized patient data (diagnoses, problems, prescriptions) that were available in the EHR. Eligible patients entered the study at the moment the patient record was opened in the practice during the study period. At that moment the CDSS classified the patient as requiring screening or requiring treatment. If a patient required screening, the number of days from entry until screening or follow-up ended was counted. If a patient required treatment, the number of days from entry until treatment or follow-up ended was counted. Patients who were screened after being classified as requiring screening and subsequently classified as requiring treatment were included in both the screening analysis and treatment analysis.
All patients were followed for at least 1 year, or until the general practice switched to another EHR vendor (that is, stopped using the ELIAS system) or until the patient died.
Logistic regression analysis was used to compare practice characteristics at baseline between the 2 treatment arms. Cox regression analyses were used for comparison of screening and treatment rates between intervention arms.
For comparison of screening rates, time of follow-up was considered as time axis in the Cox regression analysis. This analysis was restricted to persons identified as requiring screening by the CDSS. Because the number of visits during follow-up affects the possibility of being screened, we adjusted for this covariate. We adjusted for CVD and diabetes mellitus, and no other covariates had a material impact on the results.
For comparison of treatment rates, time of follow-up (from requiring treatment until censoring) was used as time axis in the Cox regression analysis. This analysis was restricted to persons classified by the CDSS as requiring treatment.
To estimate the effect of the interventions while taking into account the clustered randomization, we conducted a clustered Cox regression analysis by using PROC PHREG. All analyses were performed with SAS release 8.2 (SAS Institute Inc, Cary, NC).
The authors had full access to and take full responsibility for integrity of the data. All authors have read and agree to the manuscript as written.
Thirty-six general practices, with a total of 87 866 eligible patients, of the 38 general practices that started the study completed the study. Eleven practices were randomized to the control arm, 13 to the alerting version, and 14 to the on-demand version (Figure). Two on-demand practices with 4168 patients could not be included in the analysis. During a scheduled upgrade, the hardware vendor inadvertently deleted our study and backup data, leaving us unable to analyze the practices.
Practice characteristics differed slightly between the treatment arms. For example, screening and treatment performance in the year preceding the intervention were lower in the on-demand arm (Table 1). Patient characteristics were similar between the study arms, except for the percentage of smokers, family history of CVD, and systolic blood pressure (Table 2).
For 3210 of the 87 866 patients, the system determined that according to the DCGP guidelines, screening was needed. The percentage was highest in the on-demand arm (Table 3). The majority of patients who were identified as requiring screening were male and aged ≈60 years, ≈40% had CVD (primarily angina or a prior myocardial infarction), and ≈26% had diabetes mellitus (Table 3).
For 2953 of the 87 866 patients, the system determined that treatment was needed. A total of 367 patients who were screened during the study period required treatment. Most patients requiring treatment had a Framingham risk score >20%. Between 40% and 50% of patients required treatment because of prior CVD (Table 4).
Sixty-five percent of patients requiring screening in the alerting arm were screened versus 35% in the on-demand arm and 25% in the control arm (Table 5). The alerting arm had a 76% increased likelihood to be screened than patients in the control arm (adjusted relative risk [RR]=1.76; 95% confidence interval [CI], 1.41 to 2.20), whereas patients in the on-demand arm had a 28% increased likelihood of being screened (adjusted RR=1.28; 95% CI, 0.98 to 1.68). Of the patients requiring treatment, 66% were treated in the alerting arm, 40% in the on-demand arm, and 36% in the control arm. After adjustment for differences between arms, the likelihood of being treated was 40% higher in the alerting arm (adjusted RR=1.40; 95% CI, 1.15 to 1.70) and 19% higher in the on-demand arm in comparison to the control arm (adjusted RR=1.19; 95% CI, 0.94 to 1.50). The patients in the alerting arm had a significantly higher likelihood of being screened than the on-demand arm (adjusted RR=1.40; 95% CI, 1.08 to 1.81) (Table 5). The treatment likelihood was also elevated but not significantly so (RR=1.18; 95% CI, 0.96 to 1.45).
In the first study to directly compare on-demand CDSS and alerting CDSS with usual care, we showed that a CDSS improves screening and treatment performance of dyslipidemia in general practice. In addition, the alerting version improved screening rates statistically significantly more than the on-demand version.
Explanation of Findings
In busy general practice, time is frequently insufficient to calculate the complex risk scores necessary for decisions on primary prevention. The time needed to read and interpret paper-based guidelines might hamper physician performance. As our results show, displaying recommendations from guidelines to making complex decisions in a way that fits into the workflow of the physician increases performance by obviating the break in the clinical workflow.4
That a system using alerts is associated with better performance than an on-demand system can be explained by the fact that alerting creates awareness of preventive actions that should be taken for a specific patient. In the absence of an alerting system, the physician has to recognize the situation that calls for action. In a previous study we showed that physicians’ awareness of the need for secondary prevention is frequently triggered by a CVD event25; in these situations, the CVD event functions as an alert. Alerting constitutes a triggering event that prompts the physician to initiate preventive care.
Findings in Relation to Other Studies
In a recent updated major review of 100 CDSS studies on practitioner performance and patient outcomes, the authors found that improved practitioner performance was associated with CDSS that automatically prompted (alerted) users compared with requiring users to activate the system.2 However, none of the 100 studies under review directly studied this observation. Our work is the first study that confirms this association in a randomized controlled trial. Recently, authors have identified the critical determinants for CDSS to be successful.2,4,26 However, we did not evaluate the effect of these factors in our study but rather kept them similar across the intervention groups. This means that we could measure the difference between the intended interventions (alerting and on-demand activation) without needing to consider the various system effects listed by others.
Differences existed between the DCGP guidelines in force during the study period and other international guidelines on dyslipidemia.27,28 The 1999 DCGP guidelines did not advocate routine low-density lipoprotein testing. In addition, various treatment thresholds differed. However, all guidelines required physicians to act on certain information related to CVD risk factors and perform screening or treatment actions. Therefore, although the guidelines differ on what tests should be performed in screening and exactly when to treat, they require the physicians to act on information related to risk factors. Because the need to perform preventive actions is a result of the information about risk factors and not the exact content of a particular guideline, we believe that our results are generalizable to settings in which different guidelines apply.
The clinical implications of our findings could be profound. It is known that both primary and secondary prevention of CVD is suboptimally performed.6–11 Our study shows that interventions such as a CDSS can improve the primary and secondary prevention of CVD and especially if general practitioners are alerted to the need for action. It can be argued that an increase in adherence to the guidelines, as effected by our system, will lead to a decrease in mortality and morbidity due to CVD. However, further research will have to show that this is indeed the case.
Recently, authors argued that evaluation of CDSS should focus on the effect on patient outcomes.2 Factors other than physicians’ actions influence patient outcomes. Patient adherence, for example, is a well-documented factor related to outcomes but is not measured in CDSS studies.29,30 Our study only assesses the effect of CDSS on changing physicians’ behavior by measuring their actions and does not evaluate patient outcomes. In performing the study, we did not question the validity of the guidelines but studied which approach could best influence physicians’ behavior given the guidelines. Thus, we believe that if the CDSS is based on well-accepted evidence-based guidelines, improvement in physician adherence to guidelines will result in better patient outcomes. Guidelines, however, evolve over time. Although a CDSS may be an effective intervention to change behavior and affect outcome, systems based on those guidelines need to be continuously updated to reflect changing guidelines.
To ensure that complete medical records were available to the system, general practitioners were only eligible to participate in this study if they did not use any paper records at all. The performance of the CDSS, however, may be influenced by the way physicians capture data in their EHR. We did not use an independent mechanism to assess the physicians’ EHR.
Randomization was conducted on a practice level. To deal with potential differences in patient characteristics between treatment arms, we adjusted through the Cox regression for baseline differences that confounded the effect estimate while taking the clustered design into account. Because of logistical reasons, the follow-up time differed between practices. Because time is an important determinant for screening and treatment, we adjusted for this difference.
Apart from screening for all CVD risk factors, we focused on treating only 1 risk factor: lipid abnormalities. We did not perform CDSS treatment interventions on other CVD risk factors. Performing these extra interventions will induce additional complexity to the system.21,31 Although we have been able to introduce a CDSS that effectively supports the longitudinal lipid management in primary and secondary prevention of CVD, further research will have to show whether introducing more workflow patterns in the CDSS will have additional effects on physicians’ performance.
In conclusion, this study shows that an alerting version of a CDSS significantly improves screening and treatment performance for dyslipidemia.
The authors would like to thank Theo Stijnen of the Department of Epidemiology and Biostatistics for assisting in the statistical analysis of this article.
Sources of Funding
The study was funded by a nonspecific grant from the Dutch Heart Foundation (2000.B161). The Dutch Heart Foundation in no way interfered with the design or execution or reporting of the study. The system was free to all participants and is not being marketed as a commercial application.
Musen MA, Shahar Y, Shortliffe EH. Clinical decision-support systems. In: Shortliffe EH, Perreault LE, eds. Medical Informatics: Computer Applications in Health Care and Biomedicine. 2nd ed. New York, NY: Springer Verlag; 2000: 600–639.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005; 330: 765.
Fuster V, Gotto AM, Libby P, Loscalzo J, McGill HC. 27th Bethesda Conference: matching the intensity of risk factor management with the hazard for coronary disease events: Task Force 1: pathogenesis of coronary disease: the biologic role of risk factors. J Am Coll Cardiol. 1996; 27: 964–976.
Brady AJB, Oliver MA, Pittard JB. Secondary prevention in 24 431 patients with coronary heart disease: survey in primary care. BMJ. 2001; 322: 1463.
Rutten FH, Bohnen AM, Hufman P, Bruinsma M, Leerink HJG, Strootman FA, Brussen AJSH, Dijkstra RH, van der Laan JR. NHG-Standaard Angina pectoris. Huisarts en Wetenschap. 1996; 36: 398–406.
Rutten GEHM, Verhoeven S, Heine RJ, De Grauw WJC, Cromme PVM, Reenders K, Van Ballegooie E, Wiersma T. NHG-Standaard Diabetes mellitus type 2. Huisarts en Wetenschap. 1999; 42: 67–84.
Thomas S, Van der Wijden T, van Drenth BB, Haverkort AFM, Hooi JD, Van der Laan JR. NHG-Standaard Cholesterol. Huisarts en Wetenschap. 1999; 42: 406–417.
Van Binsbergen JJ, Gelpke JEH, Van Bentum STB, Van der Meer K, Schuling J, Verhoeven S, Eizenga WH, Wiersma T. NHG-Standaard TIA. Huisarts en Wetenschap. 1995; 38: 15–25.
Walma EP, Grundmeijer HGLM, Thomas S, Prins A, Van den Hoogen JPH, Van der Laan JR. NHG-Standaard Hypertensie. Huisarts en Wetenschap. 1999; 42: 598–617.
ATC and DDC Values. Geneva, Switzerland: World Health Organization; 1996.
van Wyk JT, van Wijk MAM, Moorman PW, van der Lei J. Cross sectional analysis of guidelines on cardiovascular disease risk factors: going to meet the inconsistencies. Med Decis Making. 2006; 26: 57–62.
van der Weijden T, Grol RP, Knottnerus JA. Feasibility of a national cholesterol guideline in daily practice: a randomized controlled trial in 20 general practices. Int J Qual Health Care. 1999; 11: 131–137.
Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C, Khorasani R, Tanasijevic M, Middleton B. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003; 10: 523–530.
De Backer G, Ambrosioni E, Borch-Johnsen K, Brotons C, Cifkova R, Dallongeville J, Ebrahim S, Faergeman O, Graham I, Mancia G, Manger Cats V, Orth-Gomer K, Perk J, Pyorala K, Rodicio JL, Sans S, Sansoy V, Sechtem U, Silber S, Thomsen T, Wood D. European guidelines on cardiovascular disease prevention in clinical practice: Third Joint Task Force of European and Other Societies on Cardiovascular Disease Prevention in Clinical Practice. Eur Heart J. 2003; 24: 1601–1610.
Clinical decision support systems have been demonstrated to improve the implementation of evidence-based guidelines by influencing physicians’ behavior. Some systems require the physician to ask for support (that is, the system only provides support when the physician demands that support). Other systems generate alerts to the physician independent of a request by the physician for those alerts. Indirect evidence suggests that alerting users is more effective in changing physician behavior than on-demand systems. However, randomized trials comparing these methods in a clinical setting were lacking. We constructed a clinical decision support system for screening and treatment decisions in the area of dyslipidemia in primary care. The system was integrated into the electronic health record of general practitioners and could function in either an alerting mode or on demand. The system was based on the 1999 guidelines of the Dutch College of General Practitioners. In a clustered randomized trial design, we studied the effect of both alerting and on-demand decision support with respect to screening and treatment of dyslipidemia. In the 38 Dutch practices, with 87 886 eligible patients, we demonstrated that a clinical decision support system alerting users to dyslipidemia-based screening and treatment actions changed physician behavior more than on-demand decision support. In efforts to improve care by using evidence-based guidelines, decision support systems that alert physicians should be considered as an implementation strategy.
Clinical trial registration information—URL: http://www.clinicaltrials.gov. Unique identifier: NCT00348751.