The research involves measuring imaging system performance in tasks such as detecting breast cancer. Receiver operating characteristic (ROC) methodology, the current gold-standard, uses patient-level information that a woman has suspected breast cancer. The location-specific free-response ROC (FROC) method uses additional location-level information in the radiologist's report, e.g., the cancer is in the left breast and is present at a particular location. Progress during the funded period has resulted in a novel perceptually-based FROC model and data simulator and several validated methods for analyzing data which are applicable to human observers and computer aided detection (CAD) algorithms. Papers using the PI's ideas and software are being presented in increasing numbers at conferences and in journals, and his work has generated healthy debate. The overall goal of the competing renewal project is to continue advancing the state-of-the-art in this field by addressing a number of limitations of current methods.
Specific Aim 1 : The figure-of-merit (FOM) is a critical determinant of statistical power and clinical relevance but all current FOMs are lesion-based and cases with more lesions contribute more to the FOM than cases with fewer lesions, and clinically less important lesions contribute equally as more important ones;we will develop novel case-based FOMs that overcome these limitations.
Specific Aim 2 : A realistic simulator yields confidence in methodology validation using that simulator. We will extend the current simulator by incorporating more realistic correlation effects and we will develop methodology to calibrate the simulator to real datasets thereby allowing the methodology developer to tune the simulator to specific applications. The simulator will be used to validate the different methods of analysis developed in Aim 1.
Specific Aim 3 : We will address several practical issues with current FROC methodology: arbitrariness of the proximity criterion, i.e., how close a mark must be to a lesion in order to credit the observer for a true detection;lack of sample-size estimation methodology for planning prospective studies;and lack of methods for analyzing clinically realistic data acquisition scenarios such as multiple views and breasts and multiple lesion types per case.
Specific Aim 4 : We will validate the methodology using independently acquired ROC, FROC and outcome-data in mammography. Outcome is defined as GOOD for normal cases returned to screening or abnormal cases sent to biopsy and BAD otherwise. We will test the hypothesis that FROC better correlates with outcome and yields greater statistical power than ROC. The significance is that the field is increasingly moving towards location-specific analyses, because of its intrinsic appeal and clinical realism, therefore methodology capable of analyzing the complex data, well outside the scope of the current gold-standard, is urgently needed. Patients benefit from better designed and optimized equipment leading to early diagnosis and treatment of cancers. Health care benefits because more efficient and cost-effective studies become possible which could serve as surrogates for expensive clinical trials.

Public Health Relevance

The research involves measuring imaging system performance in tasks such as detecting breast cancer where the current gold-standard uses information that a woman is suspected to have breast cancer, but the proposed new method uses additional information, e.g., the suspected cancer is in the left breast at a particular location, thereby yielding a more precise and clinically relevant measurement of performance. Patients benefit from better designed and optimized equipment leading to early diagnosis and treatment of cancers. Health care benefits because more efficient and cost-effective studies become possible which could serve as surrogates for expensive clinical trials.

Agency
National Institute of Health (NIH)
Institute
National Institute of Biomedical Imaging and Bioengineering (NIBIB)
Type
Research Project (R01)
Project #
5R01EB005243-07
Application #
8212119
Study Section
Biomedical Imaging Technology Study Section (BMIT)
Program Officer
Pai, Vinay Manjunath
Project Start
2005-08-01
Project End
2013-12-31
Budget Start
2012-01-01
Budget End
2012-12-31
Support Year
7
Fiscal Year
2012
Total Cost
$404,138
Indirect Cost
$137,380
Name
University of Pittsburgh
Department
Radiation-Diagnostic/Oncology
Type
Schools of Medicine
DUNS #
004514360
City
Pittsburgh
State
PA
Country
United States
Zip Code
15213
Soh, B P; Lee, W B; McEntee, M F et al. (2014) Mammography test sets: reading location and prior images do not affect group performance. Clin Radiol 69:397-402
Thompson, John D; Hogg, Peter; Manning, David J et al. (2014) A free-response evaluation determining value in the computed tomography attenuation correction image for revealing pulmonary incidental findings: a phantom study. Acad Radiol 21:538-45
Warren, Lucy M; Given-Wilson, Rosalind M; Wallis, Matthew G et al. (2014) The effect of image processing on the detection of cancers in digital mammography. AJR Am J Roentgenol 203:387-93
Haygood, T M; Ryan, J; Brennan, P C et al. (2013) On the choice of acceptance radius in free-response observer performance studies. Br J Radiol 86:42313554
Chakraborty, Dev P (2013) A brief history of free-response receiver operating characteristic paradigm data analysis. Acad Radiol 20:915-9
Chakraborty, Dev P (2012) Measuring agreement between rating interpretations and binary clinical interpretations of images: a simulation study of methods for quantifying the clinical relevance of an observer performance paradigm. Phys Med Biol 57:2873-904
Chakraborty, Dev P (2011) Recent developments in imaging system assessment methodology, FROC analysis and the search model. Nucl Instrum Methods Phys Res A 648 Supplement :S297-S301
Chakraborty, Dev P (2011) New developments in observer performance methodology in medical imaging. Semin Nucl Med 41:401-18
Chakraborty, Dev P (2011) How many readers and cases does one need to conduct an ROC study? Acad Radiol 18:127-8
Toomey, Rachel J; Ryan, John T; McEntee, Mark F et al. (2010) Diagnostic efficacy of handheld devices for emergency radiologic consultation. AJR Am J Roentgenol 194:469-74

Showing the most recent 10 out of 28 publications