Since the inception of the paradigm in 1961 the analysis of FROC (free-response receiver operating characteristic) data (i.e., mark-rating pairs) has remained an unsolved problem in imaging science. Since the radiologist must search patient images for localized lesions, intrinsically FROC data is generated in virtually all imaging studies in radiology. Currently there is great interest in evaluating different computer aided detection (CAD) algorithms for lung cancer screening with helical CT scans. This task also involves search and depending on the CAD in use there are approximately 4 to 20 suspicious areas reported per patient While in the past the receiver operating characteristic (ROC) paradigm has been the standard method for the evaluation of diagnostic imaging systems, its limitation to one report per patient is now being increasingly felt. This is a proposal to develop the science and perform the requisite testing of the FROC paradigm by accomplishing the following aims. (1) Develop a statistical model for FROC data that includes the effect of search and other relevant perceptual factors and which reduces to the standard ROC model in non-search tasks. (2) Use the model to validate and evaluate currently proposed methods for analyzing location and ratings data, and in particular determine their statistical power. (3) Develop an algorithm to determine the parameters of the model from clinical FROC data and to make statistical inferences regarding the modalities being tested; verify that the estimated search parameters correlate with values inferred from eye-movement recordings. (4) Provide practical and validated software to the user community that will offer more statistical power and therefore require fewer resources of cases and readers. The significance of the proposed work is that on the scale of previous developments, the potential gain in statistical power by the FROC method is very large. Also, the explicit modeling of search and the ability to estimate search related parameters from observer data should open up new areas of vision research that all researchers can exploit.
Showing the most recent 10 out of 38 publications
