The Receiver Operating Characteristic curve (ROC) is the most commonly used approach for assessing and comparing performance levels of diagnostic procedures that may involve the observer as an integral part of the system / practice. One of the more clinically relevant methods of ROC analysis is based on the part of the ROC curve which describes operating characteristics that are expected to be actually observable in clinical practice. A well known example of such an approach is the analysis based on partial area under the ROC curve (partial AUC) over the range of operating points with low false positive fraction (FPF). Although analyses based on a part of the ROC curve are frequently considered to be more relevant, these are not commonly employed due to several important issues. Two of these are the difficulty with a consistent definition of the actual range of interest and the potential loss of efficiency, namely higher likelihood of inconclusive results of a study or the need for a larger sample size for a future study. Our research proposal is primarily related to: 1) Defining the appropriate portion of interest of an ROC curve when estimated for a laboratory experiment performed outside of the actual clinical environment (given that the region of interest is known for the clinical ROC curve). Indeed the same decision threshold in a retrospective laboratory experiment is likely to have different operating characteristics (i.e. FPF and TPF) and therefore the direct application of the range of interest defined in terms of FPF is inappropriate and could lead to inconclusive and less, or totally irrelevant, results;2) Empirical operating points in performance assessment studies typically span only the region of low FPF and therefore consideration of the entire ROC curve rather than the empirically supported part of the curve may not actually increase the statistical efficiency of comparisons of diagnostic modalities. The research we propose herein will help researchers use partial ROC analysis in a more relevant manner leading to more conclusive results, as well as provide guidelines on the potential statistical tradeoffs between partial and full ROC analyses in commonly encountered experimentally ascertained data, thereby addressing concerns about efficiency. Our proposed effort will provide researchers with three difference methods to define the clinically relevant portion on a laboratory-estimated ROC curve. We will also develop an approach for comparison of partial AUCs by combining strengths of both parametric and nonparametric approaches. This will provide a more interpretable and statistically efficient technique for this purpose. Last, we will investigate the relative efficiency of the partial and full AUC in different types of data commonly encountered experimentally where the empirical operating points span only the range of low FPF, and develop an approach that combines practical relevance of the partial AUC with statistical stability of the full AUC. All investigated analyses will be evaluated using extensive simulations, as well as an assessment of the possible effects, if any, of the proposed analyses on conclusions of previously performed and published large ROC type studies.

Public Health Relevance

The design and analysis of performance assessment studies based on partial ROC curves is frequently more relevant than comparisons of complete ROC curves. However, many analytical and practical issues need to be resolved if a partial ROC approach is to be commonly used for this purpose. We propose to investigate a number of the more important issues in this regard and to develop new tools that will enable clinically relevant use of partial ROC analyses for this purpose and at the same time increase the possibility of conclusive results while decreasing sample size requirements for partial ROC type study designs.

Agency
National Institute of Health (NIH)
Institute
National Institute of General Medical Sciences (NIGMS)
Type
Research Project (R01)
Project #
5R01GM098253-02
Application #
8299493
Study Section
Biodata Management and Analysis Study Section (BDMA)
Program Officer
Lyster, Peter
Project Start
2011-07-15
Project End
2014-04-30
Budget Start
2012-05-01
Budget End
2013-04-30
Support Year
2
Fiscal Year
2012
Total Cost
$287,850
Indirect Cost
$97,850
Name
University of Pittsburgh
Department
Biostatistics & Other Math Sci
Type
Schools of Public Health
DUNS #
004514360
City
Pittsburgh
State
PA
Country
United States
Zip Code
15213
Ma, Hua; Bandos, Andriy I; Gur, David (2018) Informativeness of Diagnostic Marker Values and the Impact of Data Grouping. Comput Stat Data Anal 117:76-89
Bandos, Andriy I; Obuchowski, Nancy A (2018) Evaluation of diagnostic accuracy in free-response detection-localization tasks using ROC tools. Stat Methods Med Res :962280218776683
Bandos, Andriy I; Guo, Ben; Gur, David (2017) Estimating the Area Under ROC Curve When the Fitted Binormal Curves Demonstrate Improper Shape. Acad Radiol 24:209-219
Bandos, Andriy I; Guo, Ben; Gur, David (2017) Jackknife variance of the partial area under the empirical receiver operating characteristic curve. Stat Methods Med Res 26:528-541
Ma, Hua; Bandos, Andriy I; Gur, David (2015) On the use of partial area under the ROC curve for comparison of two diagnostic tests. Biom J 57:304-20
Hakim, Christiane M; Catullo, Victor J; Chough, Denise M et al. (2015) Effect of the Availability of Prior Full-Field Digital Mammography and Digital Breast Tomosynthesis Images on the Interpretation of Mammograms. Radiology 276:65-72
Bandos, Andriy I; Rockette, Howard E; Gur, David (2013) Subject-centered free-response ROC (FROC) analysis. Med Phys 40:051706
Ma, Hua; Bandos, Andriy I; Rockette, Howard E et al. (2013) On use of partial area under the ROC curve for evaluation of diagnostic performance. Stat Med 32:3449-58