Unless the current Obuchowski-Rockette (OR) Dorfman-Berbaum-Metz (DBM) multireader-multicase (MRMC) receiver operating characteristic (ROC) software is rewritten in a modern programming language, it will be impossible to incorporate innovations or even release a version that runs on modern operating systems. The OR-DBM MRMC methodology takes into account variability among both patients and radiologists so that study conclusions generalize to both patient and radiologist populations. The OR-DBM MRMC software is the most widely used ROC software for MRMC diagnostic imaging studies. It was recently shown (Hillis, 2014) that the OR-DBM method can be extended to many other study de- signs; however, it does not make sense to include analyses for new study designs in software that is rapidly becoming obsolete. A critical need exists to rewrite OR-DBM MRMC in Java to maintain and extend it going forward. Without rewriting the software in Java, no future innovation can be incorporated into the OR-DBM MRMC software, nor will it be possible to create versions to run on current and future operating systems. Our long-term goal is to continue to develop MRMC analysis methods and to make these available using user- friendly, freely-available software. The primary objective in this application is to create a cross platform OR- DBM MRMC package by rewriting the majority of the components in the current software in Java. The secondary objective is to enhance the software by providing several new features. To achieve these objectives, the following specific aims are proposed: (1) create a cross-platform version of OR-DBM MRMC that will run on current and future versions of Windows, OS X, and Linux; (2) enhance the software by adding several new features such as new analysis options, the graphical display of fitted and empirical ROC curves, and alternative free-response ROC and region-of-interest analysis. The expected outcome of this project is an extensible, cross-platform codebase to serve as the foundation for OR-DBM MRMC for many years to come. In addition, the methodology will be enhanced with several new features, making it much more of a one-stop shop software package where the user can do both single- and multi-reader analyses, single- and multi-modality analyses, alternative free-response ROC (AFROC) and region-of-interest (ROI) analyses, easily produce graphical displays of the ROC curves, and seamlessly export estimates needed for power and sample size estimation in our sample size software These contributions are significant because they preserve OR-DBM MRMC as an accessible methodology for researchers while simultaneously enhancing its usefulness. The positive impact will be the continued availability of the software for the analysis of technology evaluation and observer performance.
Providing researchers with the means to properly assess the diagnostic accuracy of new imaging technology will lead to a reduction in diagnostic errors in radiology. Because the proposed project will improve software used to assess diagnostic accuracy, it is relevant to the mission of the National Institute of Biomedical Imaging and Bioengineering, namely to improve health by leading the development and accelerating the application of biomedical technologies.
| Hillis, Stephen L (2018) Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski-Rockette model parameters. Stat Med 37:2067-2093 |
| Hillis, Stephen L; Chakraborty, Dev P; Orton, Colin G (2017) ROC or FROC? It depends on the research question. Med Phys 44:1603-1606 |
