Significance: In this SBIR project, we propose to develop novel software, ?HindSight?, that will improve InSight, a machine-learning based clinical decision support (CDS) system for sepsis prediction and detection. HindSight will identify clinicians? sepsis-related decisions in the records of former patients; it will then use these events to supply InSight with labeled examples of sepsis cases, incorporating clinicians? judgement to demonstrate appropriate and inappropriate alarms. Together with an online training module that accordingly refines InSight?s predictors, this capability will enable InSight to quickly adapt to the idiosyncrasies of a particular clinical deployment, reduce false or irrelevant alarms, and do both without explicit human supervision. Research Question: Can a machine-learning-based, retrospective labeler learn to autonomously label sepsis and sepsis treatments by integrating the total clinical record, thereby providing many high-quality examples and labels for training a sepsis CDS? In concert with an online learning algorithm, can this labeler facilitate online, supervised learning without explicit human intervention? Prior Work: We have developed InSight for application in a number of sepsis prediction settings. Existing InSight classifiers attain an area under the receiver operating characteristic curve (AUROC) of 0.88 for sepsis detection, and 0.74 for 4-hour early sepsis prediction.
Specific Aims : To identify patients who were evaluated for sepsis, treated for sepsis, or who actually had sepsis using the retrospective clinical patient record and label them accordingly (Aim 1); to use these labels with an online learning algorithm to implement autonomous, supervised learning of alert behavior which reflects clinician judgement (Aim 2). Methods: We will identify evaluated, treated, and septic patients using a machine learning labeler trained on retrospective data from patients? electronic health records (EHR) at time of discharge. Using a set of 100 test cases (? 20 septic) hand-annotated by our clinician investigators, we will assess the labeler?s performance. Labeling AUROC ? 0.95 will constitute success in Aim 1. We will develop an online learning algorithm which enables InSight to continuously retrain during deployment.
With Aim 1 ?s labeler, we will simulate a deployment with online learning, producing a learning curve of predictive AUROC on a held-out test set versus number of observed patients.
Aim 2 will be successful if the online training results in superior area under the learning curve versus the initial model and periodic retraining. All experiments will be executed using the MIMIC-III data set. Future Directions: Following the proposed work, the InSight system with an online, HindSight-based retraining module will be deployed at partner hospitals for prospective studies.
Clinical decision support (CDS) systems present critical information to medical professionals by examining patient data and providing alerts. Machine learning is a powerful method for creating CDS tools, but it requires labels which reflect the desired alert behavior. We will develop software that examines discharged patients? electronic health records (EHR), identifies clinicians? sepsis treatment decisions and patient outcomes, and passes these labeled examples to an online algorithm for retraining InSight, our machine-learning-based CDS tool for real-time sepsis prediction.
