Observer performance studies have become an established methodology for evaluating diagnostic imaging devices, and are often required for substantiating claims made in publications and to regulatory agencies such as the FDA. These studies are typically evaluated using receiver operating characteristic (ROC) analysis, with studies designed to generalize to the population of readers and cases. The typical endpoint of such studies is the area under the ROC curve (AUC). However, AUC neglects the fact that most diagnostic tasks are focused on certain regions of the ROC curve. For example, in screening mammography false positive rates above 20% (which constitutes most of the curve) are simply not realistic in practice. In this proposal we seek to develop an alternative figure of merit for evaluating observer performance studies that is based on the concept of expected utility (EU). EU incorporates diagnostic outcomes into the performance measure, and thus confines analysis to the most relevant region of the ROC curve. Preliminary evidence suggests that EU may have greater statistical power than AUC for detecting modality differences in fully-crossed experiments in which all readers score all images in all imaging modalities. We also seek to extend the EU analysis into localization and free-response ROC paradigms where it has not previously been used other than as a theoretical device. The three specific aims we propose reflect these goals of development and dissemination. We propose (Aim 1) to investigate optimal experimental design for EU in ROC studies, by analyzing parameters that lead to the greatest statistical power for the EU endpoint, to compare EU to AUC in localization and free-response assessment paradigms (Aim 2) to understand if there is any benefit in experimental design in these methods, and to develop distributable software (Aim 3) that allows investigators in the field to use EU on their own data or for their own assessments.

Public Health Relevance

Validation of new diagnostic imaging systems and imaging methods is of vital importance for bringing new scanners, image processing, and other techniques into the clinic to benefit patients. We propose a utility based approach that uses benefits and costs of correct and incorrect decisions as part of the validation process. If successful, we believe that our approach will make it easier for better imaging methods to be used to improve healthcare for patients.

National Institute of Health (NIH)
National Institute of Biomedical Imaging and Bioengineering (NIBIB)
Exploratory/Developmental Grants (R21)
Project #
Application #
Study Section
Biomedical Imaging Technology Study Section (BMIT)
Program Officer
Pai, Vinay Manjunath
Project Start
Project End
Budget Start
Budget End
Support Year
Fiscal Year
Total Cost
Indirect Cost
University of California Santa Barbara
Schools of Arts and Sciences
Santa Barbara
United States
Zip Code
Samuelson, Frank W; Abbey, Craig K (2017) The Reproducibility of Changes in Diagnostic Figures of Merit Across Laboratory and Clinical Imaging Reader Studies. Acad Radiol 24:1436-1446
Wu, Yirong; Abbey, Craig K; Liu, Jie et al. (2016) Discriminatory power of common genetic variants in personalized breast cancer diagnosis. Proc SPIE Int Soc Opt Eng 9787:
Aminololama-Shakeri, Shadi; Abbey, Craig K; Gazi, Peymon et al. (2016) Differentiation of ductal carcinoma in-situ from benign micro-calcifications by dedicated breast computed tomography. Eur J Radiol 85:297-303
Wunderlich, Adam; Goossens, Bart; Abbey, Craig K (2016) Optimal Joint Detection and Estimation That Maximizes ROC-Type Curves. IEEE Trans Med Imaging 35:2164-73
Abbey, Craig K; Wu, Yirong; Burnside, Elizabeth S et al. (2016) A Utility/Cost Analysis of Breast Cancer Risk Prediction Algorithms. Proc SPIE Int Soc Opt Eng 9787:
Wu, Yirong; Abbey, Craig K; Chen, Xianqiao et al. (2015) Developing a utility decision framework to evaluate predictive models in breast cancer risk estimation. J Med Imaging (Bellingham) 2:041005