This proposal addresses a fundamental methodological problem in diagnostic radiology research: evaluation of diagnostic performance of imaging modalities. Receiver operating characteristic (ROC) analysis is an accepted procedure for measuring how well imaging modalities allow physicians to detect disease. Dorfman, Berbaum and Metz (1992) (DBM) developed a statistical methodology to analyze ROC studies with multiple readers, taking into account both reader-sample and case-sample variation. DBM methodology is the most widely used method to assess diagnostic performance of new imaging systems. Unfortunately, the usefulness and effectiveness of DBM method is limited: DBM only handles a narrow range of experimental designs and cannot cope with missing data. The soundness of DBM conclusions may be in question when there are many more or many less diseased patients than non-diseased patients. DBM does not fit ROC curves using ordinal regression, restraining further development of ROC models and preventing the incorporation of more advanced statistical techniques. DBM cannot use patient information to improve statistical power. DBM only compares diagnostic systems on accuracy of diagnosis, not on the benefits and costs for patients and society. Our goal is to remove these limitations by extending DBM method's range of application and by improving its effectiveness.
The specific aims are: (1) formulate a more general model for DBM MRMC analysis using the estimated generalized least squares method to cope with computer-aided diagnosis experiments, multiple covariate effects, missing data, designs split on readers or cases, and readers reading different cases;(2) develop an option that determines the sensitivity of inferences to the variability of the variance-covariance estimates when the number of cases is small, and when the number of cases is moderate but the proportion of normal to abnormal cases is close to one or zero;(3) provide a validated and easy-to-use ordinal regression option for ROC curve fitting within our major DBM MRMC method that offers familiar and interpretable ROC-style results;(4) extend DBM MRMC to include a disease-specific patient-level covariate to discover how the covariate affects accuracy and thereby gain more statistical power by explaining variation in outcomes;and (5) develop DBM MRMC methodology for cost/benefit analysis to focus the comparison of diagnostic systems on clinically relevant parts of ROC curves and provide a quantitative basis for health policy decisions. Completing these aims will enhance the effectiveness of the primary method used to assess the diagnostic performance of new imaging systems. This will lead to better radiology research, which will raise the quality and lower the cost of health care.

National Institute of Health (NIH)
National Institute of Biomedical Imaging and Bioengineering (NIBIB)
Research Project (R01)
Project #
Application #
Study Section
Biomedical Imaging Technology Study Section (BMIT)
Program Officer
Pai, Vinay Manjunath
Project Start
Project End
Budget Start
Budget End
Support Year
Fiscal Year
Total Cost
Indirect Cost
University of Iowa
Schools of Medicine
Iowa City
United States
Zip Code
Hillis, Stephen L (2016) Equivalence of binormal likelihood-ratio and bi-chi-squared ROC curve models. Stat Med 35:2031-57
Berbaum, Kevin S; Krupinski, Elizabeth A; Schartz, Kevin M et al. (2015) Satisfaction of Search in Chest Radiography 2015. Acad Radiol 22:1457-65
Hillis, Stephen L (2014) A marginal-mean ANOVA approach for analyzing multireader multicase radiological imaging data. Stat Med 33:330-60
Schartz, K M; Berbaum, K S; Madsen, M T et al. (2013) Multiple diagnostic task performance in CT examination of the chest. Br J Radiol 86:18244135
Hillis, Stephen L (2012) Simulation of unequal-variance binormal multireader ROC decision data: an extension of the Roe and Metz simulation model. Acad Radiol 19:1518-28
Obuchowski, Nancy A; Gallas, Brandon D; Hillis, Stephen L (2012) Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol 19:1508-17
Zanca, Federica; Hillis, Stephen L; Claus, Filip et al. (2012) Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates: results from independently conducted FROC?ROC studies in mammography. Med Phys 39:5917-29
Hillis, Stephen L; Metz, Charles E (2012) An analytic expression for the binormal partial area under the ROC curve. Acad Radiol 19:1491-8
Edwards, Darrin C; Metz, Charles E (2012) The three-class ideal observer for univariate normal data: Decision variable and ROC surface properties. J Math Psychol 56:256-273
Berbaum, Kevin S; Schartz, Kevin M; Caldwell, Robert T et al. (2012) Satisfaction of search for subtle skeletal fractures may not be induced by more serious skeletal injury. J Am Coll Radiol 9:344-51

Showing the most recent 10 out of 26 publications