Assessment of diagnostic performance of technologies and practices constitute an increasingly important part of the decision making process when imaging systems are evaluated for possible use in the clinical environment as well as during the regulatory approval process. In medical imaging a substantial fraction of these evaluations include task specific ratings that are being analyzed using an ROC response or derivative thereof. The overall objective of this project is to improve our understanding of the practical (methodological) and computational (analyses related) issues that we encounter during systems evaluations and comparisons when the observer, a computerized scheme (e.g. CAD), or a combination of both become an integral part of the diagnostic system. During the current period of funding, we developed and tested new approaches to analyzing scoring data ascertained during the performance of ROC studies and began to address issues related to the incorporation of relevance based weights into the analyses. The underlying approach we are exploring uses all paired observations between negative and positive cases (examinations) to construct summary indices of performance using a non-parametric methodology. Both unconditional and conditional permutation based tests have been developed and tested and were shown to have distinct experimental advantages in several experimental settings and in particular, for small sample size studies. Six primary efforts are proposed using this general approach. We propose to expand the method to a multi-reader, multimodality, multi-disease settings as well as to incorporate a variety of utility functions into the analysis. We will also investigate how this and other methods can be used to identify and possibly treat outliers in ROC type studies. Last, using the same underlying concept we propose to expand the investigations and study the relationship of the ideal bootstrap variances of the non-parametric estimators of the AUC to other existing estimators in the multi-reader environment. We will develop closed-form solutions for the ANOVA procedure conducted on all possible bootstrap values. ? ? ?

Agency
National Institute of Health (NIH)
Institute
National Institute of Biomedical Imaging and Bioengineering (NIBIB)
Type
Research Project (R01)
Project #
2R01EB002106-12A1
Application #
7197715
Study Section
Biomedical Imaging Technology Study Section (BMIT)
Program Officer
Cohen, Zohara
Project Start
2002-05-01
Project End
2010-02-28
Budget Start
2007-03-01
Budget End
2008-02-29
Support Year
12
Fiscal Year
2007
Total Cost
$329,695
Indirect Cost
Name
University of Pittsburgh
Department
Radiation-Diagnostic/Oncology
Type
Schools of Medicine
DUNS #
004514360
City
Pittsburgh
State
PA
Country
United States
Zip Code
15213
Gur, David; Bandos, Andriy I; Rockette, Howard E et al. (2010) Is an ROC-type response truly always better than a binary response in observer performance studies? Acad Radiol 17:639-45
Gur, David; Rockette, Howard E (2009) Performance assessments of diagnostic systems under the FROC paradigm: experimental, analytical, and results interpretation issues. Acad Radiol 16:770-1
Gur, David; Bandos, Andriy I; Klym, Amy H et al. (2008) Agreement of the order of overall performance levels under different reading paradigms. Acad Radiol 15:1567-73
Gur, David; Bandos, Andriy I; King, Jill L et al. (2008) Binary and multi-category ratings in a laboratory observer performance study: a comparison. Med Phys 35:4404-9
Gur, David; Bandos, Andriy I; Rockette, Howard E (2008) Comparing areas under receiver operating characteristic curves: potential impact of the ""Last"" experimentally measured operating point. Radiology 247:12-5
Rockette, Howard E; Gur, David (2008) Selection of a rating scale in receiver operating characteristic studies: some remaining issues. Acad Radiol 15:245-8
Song, Tao; Bandos, Andriy I; Rockette, Howard E et al. (2008) On comparing methods for discriminating between actually negative and actually positive subjects with FROC type data. Med Phys 35:1547-58
Gur, David; Rockette, Howard E (2008) Performance assessments of diagnostic systems under the FROC paradigm: experimental, analytical, and results interpretation issues. Acad Radiol 15:1312-5
Gur, David; Rockette, Howard E; Bandos, Andriy I (2007) ""Binary"" and ""non-binary"" detection tasks: are current performance measures optimal? Acad Radiol 14:871-6
Gur, David; Bandos, Andriy I; Klym, Amy H et al. (2006) Reader variance in ROC studies--generalizability to reader population at high and low performance levels. Acad Radiol 13:1004-10

Showing the most recent 10 out of 15 publications