Risk prediction models play an important role in selecting prevention and treatment strategies for many cancers. While extensive effort in both clinical and methodological research is spent on building accurate and reliable models, the science and practice of validating these models is less well-developed. We have identified a potentially critical pitfall in the standard analytic approach to validating a prediction model. Whle it is common to observe poorer performance in a validation set compared to a training set, this difference is generally attributed to optimistic bias in measuring performance in the training set. However, the difference might be rather due to differences in distribution of the predictors in the validation set, which can strongly affect predictive performance. Currently, no analytic tools exis to address this possibility, which leaves investigators unable to reliably interpret results of validation studies. For example, if a validation study includes a higher proportion of cases for which prognostication is more difficult because of clinical characteristics, a standard validation analysis might erroneously give a low rating to a useful risk prediction model. Ultimately, because validation studies determine which prediction models are adopted for research and clinical use, it is critical that their methods be grounded in rigorous cross-study comparisons. Development of methods to enhance validation of prediction models is thus both timely and important. We propose to develop a practical and effective statistical method that will enable investigators to systematically adjust for differences in distribution of predictors among multiple datasets or multiple target populations in validation studies (Aim 1). This proposed approach and resulting estimates will provide a better understanding of how risk prediction models perform in specific target populations, and will make interpretation of results of cross-study validation with multiple validation sets much more reliable. We will also develop software to implement our methods on three very commonly used statistical platforms (R, Stata, and SAS), making immediate public availability possible (Aim 2). Our proposal is relevant to the mission of the NCI, because the methods we will develop are innovative, and have broad applicability to developers of cancer-related prediction models. Because of the emergence of large amount of observational data from electronic medical records (EMR), there is an increased opportunity to harness this information to develop prediction tools for use at the bedside. Moreover, clinical decision support tools are increasingly being built into EMRs and so implementation of prediction models at the point of care has become more common; it stands to reason that the opportunity to exploit the EMR for this purpose will only be successful if the models that inform i are statistically sound.
Implementation of risk prediction models in oncology-related clinical and research activities is becoming more common. This self-contained project will provide a new statistical analysis tool for developing rigorously-validated prediction models for public use. The proposed method has the potential to greatly improve the health of patients with cancer by assuring that selection of their treatments is based on models developed with this new innovative and precise statistical method.
