Systematic reviews constitute the highest quality of evidence and form the cornerstone of evidence-based medicine (EBM). Such reviews now inform everything from national health policy guidelines to bedside care. However, systematic reviews are extremely laborious to produce; researchers can no longer keep pace with the massive amount of evidence now being published. Semi-automation of systematic review production via machine learning (ML) has demonstrated the potential to substantially reduce reviewer workload while maintaining comprehensiveness. However, it is unlikely that machines will fully supplant human reviewers in the near future. Rather, human experts will probably remain in the loop, assisted by automated methods. Methods that exploit the intersection of human workers and ML models in the context of systematic reviews have not been explored at length. Furthermore, we believe there is substantial untapped potential in harnessing distributed crowd-workers to contribute to systematic reviews, and thus economize expert reviewer efforts. This novel avenue has largely been neglected as a means of increasing the efficiency of review production. We propose addressing this gap by developing and evaluating novel, hybrid approaches to generating systematic reviews that jointly incorporate domain experts (systematic reviewers), layperson workers recruited via crowdworking platforms such as Amazon's Mechanical Turk and volunteer citizen scientists, while simultaneously capitalizing on ML models. This innovative, hybrid approach will be the first in-depth exploration of intelligent ML/human systems that aim to reduce the workload in the production of biomedical systematic reviews. Our strong preliminary work demonstrates the promise of this general strategy.
| Marshall, Iain J; Noel-Storr, Anna; Kuiper, Joël et al. (2018) Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide. Res Synth Methods 9:602-614 |
| Marshall, Iain J; Kuiper, Joël; Banner, Edward et al. (2017) Automating Biomedical Evidence Synthesis: RobotReviewer. Proc Conf Assoc Comput Linguist Meet 2017:7-12 |
| Mortensen, Michael L; Adam, Gaelen P; Trikalinos, Thomas A et al. (2017) An exploration of crowdsourcing citation screening for systematic reviews. Res Synth Methods 8:366-386 |
