In recent decades the paradigm of the receiver operating characteristic (ROC), namely, the laboratory mapping of the true-positive rate versus the false-positive rate, has played a central role in the assessment of diagnostic technologies, especially in medical imaging and computer-aided diagnosis. In the most recent decade, the growing awareness of the great variability in performance of clinicians reading images, together with the emergence of random-effects ROC analysis, have made the multiplereader, multiple-case (MRMC) ROC paradigm the dominant one in the technology assessment community. The fully crossed version of this paradigm, namely, where all readers interpret the same cases in all competing modalities, is considered by many as the most statistically powerful for a given number of truth-verified cases. This power is achieved in the face of high reader and case variability because only those components of variability that are uncorrelated across competing modalities mask the ability to determine a difference between the modalities, and one of the goals of the fully crossed design is to minimize these very components. Recent developments in the field give us the ability to estimate all of the sources of variability and the underlying correlations in MRMC-ROC type experiments. This information provides the opportunity to understand why some designs are more successful than others; it also allows one to better design a large pivotal trial from the results of a smaller pilot study. We propose to analyze our large database of 18 large ROC-type experiments in these terms. The expected benefit from this project will be the insight into design methods to achieve more statistically powerful approaches to the assessment and comparison of competing diagnostic imaging and computer-assist modalities, i.e., those that are the less demanding of the expensive resources of gathering patient cases and expanding radiologists reading time.

Agency
National Institute of Health (NIH)
Institute
National Institute of Biomedical Imaging and Bioengineering (NIBIB)
Type
Research Project (R01)
Project #
1R01EB001694-01A1
Application #
6817584
Study Section
Diagnostic Imaging Study Section (DMG)
Program Officer
Peng, Grace
Project Start
2004-09-01
Project End
2007-08-31
Budget Start
2004-09-01
Budget End
2005-08-31
Support Year
1
Fiscal Year
2004
Total Cost
$259,875
Indirect Cost
Name
University of Pittsburgh
Department
Radiation-Diagnostic/Oncology
Type
Schools of Medicine
DUNS #
004514360
City
Pittsburgh
State
PA
Country
United States
Zip Code
15213
Bandos, Andriy I; Rockette, Howard E; Gur, David (2010) Use of likelihood ratios for comparisons of binary diagnostic tests: underlying ROC curves. Med Phys 37:5821-30
Gur, David; Bandos, Andriy I; Rockette, Howard E et al. (2010) Is an ROC-type response truly always better than a binary response in observer performance studies? Acad Radiol 17:639-45
Bandos, Andriy I; Rockette, Howard E; Song, Tao et al. (2009) Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 65:247-56
Gur, David; Bandos, Andriy I; King, Jill L et al. (2008) Binary and multi-category ratings in a laboratory observer performance study: a comparison. Med Phys 35:4404-9
Gur, David; Bandos, Andriy I; Cohen, Cathy S et al. (2008) The ""laboratory"" effect: comparing radiologists'performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 249:47-53
Gur, David; Bandos, Andriy I; Rockette, Howard E (2008) Comparing areas under receiver operating characteristic curves: potential impact of the ""Last"" experimentally measured operating point. Radiology 247:12-5
Rockette, Howard E; Gur, David (2008) Selection of a rating scale in receiver operating characteristic studies: some remaining issues. Acad Radiol 15:245-8
Gur, David; Bandos, Andriy I; Fuhrman, Carl R et al. (2007) The prevalence effect in a laboratory environment: Changing the confidence ratings. Acad Radiol 14:49-53
Gur, David; Rockette, Howard E; Bandos, Andriy I (2007) ""Binary"" and ""non-binary"" detection tasks: are current performance measures optimal? Acad Radiol 14:871-6
Gur, David; Bandos, Andriy I; Klym, Amy H et al. (2006) Reader variance in ROC studies--generalizability to reader population at high and low performance levels. Acad Radiol 13:1004-10

Showing the most recent 10 out of 12 publications