Medical decision support tools are increasingly available on the Internet and are being used by lay persons as well as health care professionals. The goal of some of these tools is to provide an """"""""individualized"""""""" prediction of future health care related events such as prognosis in breast cancer given specific information about the individual. These tools are usually based on models synthesized from data with a fine granularity of information. Under the umbrella of """"""""personalized"""""""" medicine, these individualized prognostic assessments are sought as a means to replace general prognostic information given to patients with specific probability estimates that pertain to a small stratum to which the patient belongs, and ultimately specifically to each patient (i.e., a stratum with n=1). Subsequently, these estimates are used to inform decision making and are therefore of critical importance for public health. Responsible utilization of prognostic models for patient counseling and medical decision making requires thorough model validation. Verification that the estimated or predicted event probabilities reflect the underlying true probability for a particular individual (i.e., verifying the calibration of the prognostic model) is a critical but often overlooked step in evaluation, which usually favors the verification of the discriminatory ability of the model. Selection of the best predictive model for a given problem should be based on robust comparison that takes into account errors in individual predictions, calibration, and discrimination indices. A robust test for comparison of calibration across different models does not currently exist.
Our specific aims are to: (1) Characterize the main deficiencies of existing calibration indices in the context of individualized predictions and develop a new model-independent calibration index and comparison test that can be used to assess and compare predictive models based on both statistical regression and machine earning methods;(2) Unify the theories on decomposition of error into discrimination and calibration components stemming from the statistical and machine learning communities to derive a refined measure of alteration that can be calculated from measures of error and discrimination. We will compare the performance of the new methods with existing ones in different predictive models derived from real clinical data related to different medical domains.

Agency
National Institute of Health (NIH)
Institute
National Library of Medicine (NLM)
Type
Research Project (R01)
Project #
3R01LM009520-04S1
Application #
7938216
Study Section
Biomedical Library and Informatics Review Committee (BLR)
Program Officer
Sim, Hua-Chuan
Project Start
2009-09-30
Project End
2010-09-29
Budget Start
2009-09-30
Budget End
2010-09-29
Support Year
4
Fiscal Year
2009
Total Cost
$164,827
Indirect Cost
Name
University of California San Diego
Department
Internal Medicine/Medicine
Type
Schools of Medicine
DUNS #
804355790
City
La Jolla
State
CA
Country
United States
Zip Code
92093
Menon, Aditya Krishna; Jiang, Xiaoqian; Kim, Jihoon et al. (2014) Detecting Inappropriate Access to Electronic Health Records Using Collaborative Filtering. Mach Learn 95:87-101
Mohammed, Noman; Jiang, Xiaoqian; Chen, Rui et al. (2013) Privacy-preserving heterogeneous health data sharing. J Am Med Inform Assoc 20:462-9
Thawait, Shrey K; Kim, Jihoon; Klufas, Roman A et al. (2013) Comparison of four prediction models to discriminate benign from malignant vertebral compression fractures according to MRI feature analysis. AJR Am J Roentgenol 200:493-502
Dai, Wenrui; Xiong, Hongkai; Jiang, Xiaoqian et al. (2013) An Adaptive Difference Distribution-based Coding with Hierarchical Tree Structure for DNA Sequence Compression. Proc Data Compress Conf 2013:371-380
Wang, Shuang; Jiang, Xiaoqian; Wu, Yuan et al. (2013) EXpectation Propagation LOgistic REgRession (EXPLORER): distributed privacy-preserving online model learning. J Biomed Inform 46:480-96
Dreiseitl, Stephan; Osl, Melanie (2012) Testing the calibration of classification models from first principles. AMIA Annu Symp Proc 2012:164-9
Jiang, Xiaoqian; Menon, Aditya; Wang, Shuang et al. (2012) Doubly Optimized Calibrated Support Vector Machine (DOC-SVM): an algorithm for joint optimization of discrimination and calibration. PLoS One 7:e48823
Wu, Yuan; Jiang, Xiaoqian; Kim, Jihoon et al. (2012) Grid Binary LOgistic REgression (GLORE): building shared models without sharing data. J Am Med Inform Assoc 19:758-64
Menon, Aditya Krishna; Jiang, Xiaoqian J; Vembu, Shankar et al. (2012) Predicting accurate probabilities with a ranking loss. Proc Int Conf Mach Learn 2012:703-710
Sarin, Pankaj; Urman, Richard D; Ohno-Machado, Lucila (2012) An improved model for predicting postoperative nausea and vomiting in ambulatory surgery patients using physician-modifiable risk factors. J Am Med Inform Assoc 19:995-1002

Showing the most recent 10 out of 37 publications