In recent decades the paradigm of the receiver operating characteristic (ROC), namely, the laboratory mapping of the true-positive rate versus the false-positive rate, has played a central role in the assessment of diagnostic technologies, especially in medical imaging and computer-aided diagnosis. In the most recent decade, the growing awareness of the great variability in performance of clinicians reading images, together with the emergence of random-effects ROC analysis, have made the multiplereader, multiple-case (MRMC) ROC paradigm the dominant one in the technology assessment community. The fully crossed version of this paradigm, namely, where all readers interpret the same cases in all competing modalities, is considered by many as the most statistically powerful for a given number of truth-verified cases. This power is achieved in the face of high reader and case variability because only those components of variability that are uncorrelated across competing modalities mask the ability to determine a difference between the modalities, and one of the goals of the fully crossed design is to minimize these very components. Recent developments in the field give us the ability to estimate all of the sources of variability and the underlying correlations in MRMC-ROC type experiments. This information provides the opportunity to understand why some designs are more successful than others; it also allows one to better design a large pivotal trial from the results of a smaller pilot study. We propose to analyze our large database of 18 large ROC-type experiments in these terms. The expected benefit from this project will be the insight into design methods to achieve more statistically powerful approaches to the assessment and comparison of competing diagnostic imaging and computer-assist modalities, i.e., those that are the less demanding of the expensive resources of gathering patient cases and expanding radiologists reading time.
Showing the most recent 10 out of 12 publications
