Clinical decision support (CDS) tools are designed to help healthcare providers make better decisions. Significant and mounting evidence suggests that CDS, when used effectively, can improve health care quality, safety, and effectiveness (2-7). Indeed, a good part of the promise of the electronic health record (EHR) to improve patient care relative to a paper-based healthcare ecosystem rests upon CDS. When CDS works well, healthcare providers can come to depend on it. However, we have identified a number of instances where CDS interventions malfunctioned and either stopped providing correct alerts or began providing incorrect alerts. In many cases, these malfunctions persisted for a long period of time and in some cases, they led to patient harm. We propose a novel anomaly-detection-based method for identifying malfunctions in CDS systems so that they can be corrected. Such approaches have been used in other industries to identify deviations from expected behavior, such as credit card fraud or computer network intrusion detection, but never previously applied to the problem of CDS function and failure. In preliminary work, we have shown that even simple anomaly-detection-based approaches can identify many CDS malfunctions. In the proposed project, we will extend these methods to improve their sensitivity and specificity and validate them at three sites: Brigham and Women's Hospital, The Ohio State University Medical Center and the University of Texas. We will also conduct a qualitative assessment and root cause analyses of CDS malfunctions and develop an open-source modular dashboard and alerting system for tracking them. Our project has three aims: 1) to inventory CDS failures and issues that have occurred in three medical centers and conduct root cause analyses to identify causes, indicators, and potential solutions, 2) to develop and validate generalizable anomaly detection approaches to identifying CDS failures and 3) to create, implement, and test a useful and generalizable CDS dashboard and alert system for real-time monitoring for CDS anomalies.

Public Health Relevance

Clinical decision support systems, such as drug-interaction alerts and preventive care reminders, when used effectively, have been shown to the quality, safety and efficiency of care. However, such systems are complex and sometimes fail - these failures are often not noticed for a long period of time and can lead to patient harm. In the proposed project, we will study the causes of such failures and develop and test anomaly detection systems to detect such failures and alert knowledge engineers about them with the goal of improving the safety and reliability of clinical decision support systems.

National Institute of Health (NIH)
National Library of Medicine (NLM)
Research Project (R01)
Project #
Application #
Study Section
Biomedical Library and Informatics Review Committee (BLR)
Program Officer
Sim, Hua-Chuan
Project Start
Project End
Budget Start
Budget End
Support Year
Fiscal Year
Total Cost
Indirect Cost
Brigham and Women's Hospital
United States
Zip Code
Wright, Adam; Wright, Aileen P; Aaron, Skye et al. (2018) Smashing the strict hierarchy: three cases of clinical decision support malfunctions involving carvedilol. J Am Med Inform Assoc 25:1552-1555
Wright, Adam; Ash, Joan S; Aaron, Skye et al. (2018) Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study. Int J Med Inform 118:78-85
Wright, Adam; Aaron, Skye; Sittig, Dean F (2017) Testing electronic health records in the ""production"" environment: an essential step in the journey to a safe and effective health care system. J Am Med Inform Assoc 24:188-192
Schreiber, Richard; Sittig, Dean F; Ash, Joan et al. (2017) Orders on file but no labs drawn: investigation of machine and human errors caused by an interface idiosyncrasy. J Am Med Inform Assoc 24:958-963
Liu, Siqi; Wright, Adam; Sittig, Dean F et al. (2017) Change-Point Detection for Monitoring Clinical Decision Support Systems with a Multi-Process Dynamic Linear Model. Proceedings (IEEE Int Conf Bioinformatics Biomed) 2017:569-572
Liu, Siqi; Wright, Adam; Hauskrecht, Milos (2017) Online Conditional Outlier Detection in Nonstationary Time Series. Proc Int Fla AI Res Soc Conf 2017:86-91
McEvoy, Dustin S; Sittig, Dean F; Hickman, Thu-Trang et al. (2017) Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 24:331-338
Liu, Siqi; Wright, Adam; Hauskrecht, Milos (2017) Change-Point Detection Method for Clinical Decision Support System Rule Monitoring. Artif Intell Med (2017) 10259:126-135
Wright, Adam; Hickman, Thu-Trang T; McEvoy, Dustin et al. (2016) Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 23:1068-1076