Analytics (Objective 1) The SSA adjudication process is dynamic, involving a complex sequence of decisions by several offices within SSA as well as the decisions and resources of the claimants themselves. NIH undertook an overarching project, the Adjudication 1 project, to comprehensively model this process. Project goals are to: 1. Develop analytical tools to analyze various aspects of the adjudication process in terms of accuracy, consistency, and timeliness; 2. Develop tools to predict how the system responds to external shocks; 3. Develop methods to analyze data taking into account the multi-stage application process in which data are collected; 4. Develop tools to assist with the disability determination process; 5. Quantify the extent to which SSA can adjust the system to respond to changes; 6. Derive useful statistics to monitor and adjust program performance, based on important outcomes measures (accuracy, timeliness, consistency). Case Status Change Model and Queuing Theory: This project aims to develop methods to analyze system timeliness, measure processing times, and derive optimal flow characteristics. Our work in the area of system timeliness took on two complementary directions. To study system delays, we built a queuing model for the adjudication system that allows the user to obtain system performance statistics. Addressing the separation between wait times and processing time is our second direction of inquiry. For this purpose, we developed a batch processing model. To date, we have scripted code to obtain the needed transition probabilities, determined the rate at which jobs enter the system, completed the queue code, and estimated the distributions of processing/waiting times by using three techniques. Assessing the Predictive Quality of Evidence: The ROC estimation method for clustered data has been applied to simulated data and two real examples in diagnostic medicine. Our method shows more accurate results than existing methods. This method was mainly developed for evaluating a single classification criterion. For multiple classification criteria from the same set of observations, it is important to account for the correlation among these criteria. Case review nominator: The SSA evaluates adjudicated claims for benefits administered by SSA. As a result of the wide varieties of language that might be used, reviews focused on particular issues are typically limited to samples that included entire populations without any ability to screen out unlikely cases. With assistance from SSA's staff, NIH developed an automated document classification tool to identify cases with specific issues. The tool used a set of labeled legal documents to build predictive models for identifying new cases of interest. We implemented the automated document classification tool in Python, using Python NLTK and Scikit-learn for text processing, normalization, and feature extraction, as well as Python Scikit-Learn or Weka for feature selection and classification. We have expanded the classification algorithms available in the tool, and we implemented parameter selection methodology to optimize the document classification model parameters. We optimized the parameters of the document classification models for the SSA Case Review Nominator by running over 10,000 combinations per model to find the best parameters. We also expanded the performance metrics available for reporting and optimizations to include F-measure, precision, recall, and overall accuracy. Because SSA was interested in optimizing the model precision, we delivered to them a model that had 80% accuracy and 94% precision, and 73% recall. SSA started testing the model in production this year, and we are exploring potential uses in another SSA component- the Office of Disability Adjudication and Review (ODAR). Data Mining Feasibility Study: We are pursuing three areas of inquiry: methods of information retrieval (IR) for measuring relevance of documents, methods in natural language processing (NLP) for extracting medical records, and partial NLP methods for analyzing decision files. We performed optical character recognition (OCR) on over 500,000 decision files and developed an optical mark recognition tool to extract check box information from SSA RFC forms and applied it to the 500,000 downloaded forms with an estimated accuracy of 91%. We are in the process of refining methodology to extract text box fields from the RFC forms. We also investigated the different types of medical evidence available in SSAs electronic claimant folders. Listings nominator: Under current SSA rules, the presence of a condition that meets criteria in the Listings of Impairments (or that is of equal severity) is considered sufficient to establish medical considerations for allowances. We are currently building a tool for SSA to quickly find relevant listings from the Listings of Impairments. We are using natural language processing to extract information from the medical records and to create a more formal representation of the listings. To date, we parsed through most information in the Listings downloaded from SSAs website and are devising information retrieval methods to automatically identify ICD and CPT codes relevant to the listings inside the medical documents. We hired a medical annotator to assist in identification of functional terms. Medical Continuing Disability Reviews: The objective of this project is to find ways to improve the CDR process and reduce the backlog of cases for review. This year, we began learning about the current CDR process and how we may improve its different components. We identified the lack of structured medical and functional evidence as the main limitation of this process. We identified key areas of improvement using data analytics provided the right data could be acquired or extracted and have prepared a final paper of our findings, which will be published and disseminated by the SSDI Solutions Initiative. CAT development (Objective 2) The NIH/RMD previously awarded a contract to the Boston University Health and Disability Research Institute (BU-HDRI) to develop a comprehensive set of tools to characterize the full continuum of individual capabilities (i.e. human function) relevant to work. This method uses Computer Adaptive Testing (CAT) coupled with Item Response Theory (IRT). The development of CAT tools (also known as the Functional Assessment Battery or FAB) is a sequential process; one step must be completed before advancing to the next step. A calibration study was conducted to refine the item pools for the Daily Activities and Learning and Applying Knowledge. Additionally, Westat, a research survey firm, conducted replenishment testing of items for the first two domains, physical function and behavioral health, as well as legacy validation on all item pools. Boston University has embarked on a number of additional post development studies to enhance the functionality, utility, and comprehensiveness of the instruments. NIH plans to monitor claimant outcomes for a period of two years, assessing the predictive capacity of the instruments developed under this protocol. Upon expiration of the period of performance for this contract, BU-HDRI submitted a final report detailing efforts during the collaboration. NIH intends to execute a new contract with BU this FY to continue refinement of the instruments and explore implementation opportunities within the SSA process. Publications generated by this year's research: Qingqi Yue, Ao Yuan, Xuan Che, Minh Huynh, and Chunxiao Zhou. Batch model for batched timestamps data analysis with application to the SSA disability program. 22nd ACM SIGKDD international conference on knowledge discovery and data mining (KDD 2016)
Showing the most recent 10 out of 12 publications
