Assessment of diagnostic performance of technologies and practices constitute an increasingly important part of the decision making process when imaging systems are evaluated for possible use in the clinical environment as well as during the regulatory approval process. In medical imaging a substantial fraction of these evaluations include task specific ratings that are being analyzed using an ROC response or derivative thereof. The overall objective of this project is to improve our understanding of the practical (methodological) and computational (analyses related) issues that we encounter during systems evaluations and comparisons when the observer, a computerized scheme (e.g. CAD), or a combination of both become an integral part of the diagnostic system. During the current period of funding, we developed and tested new approaches to analyzing scoring data ascertained during the performance of ROC studies and began to address issues related to the incorporation of relevance based weights into the analyses. The underlying approach we are exploring uses all paired observations between negative and positive cases (examinations) to construct summary indices of performance using a non-parametric methodology. Both unconditional and conditional permutation based tests have been developed and tested and were shown to have distinct experimental advantages in several experimental settings and in particular, for small sample size studies. Six primary efforts are proposed using this general approach. We propose to expand the method to a multi-reader, multimodality, multi-disease settings as well as to incorporate a variety of utility functions into the analysis. We will also investigate how this and other methods can be used to identify and possibly treat outliers in ROC type studies. Last, using the same underlying concept we propose to expand the investigations and study the relationship of the ideal bootstrap variances of the non-parametric estimators of the AUC to other existing estimators in the multi-reader environment. We will develop closed-form solutions for the ANOVA procedure conducted on all possible bootstrap values.
Showing the most recent 10 out of 15 publications
