This project will advance computerized adaptive testing (CAT) as an alternative approach to static-reduced batteries. A common way that social scientists learn about individuals' traits and beliefs is to have people answer many closely related questions about the same topic. These large multi-item batteries are used to measure constructs like personality traits, psychological illnesses, political ideology, personal values, levels of knowledge, and more. Public opinion researchers, however, often choose not to use these widely respected multi-item batteries because of concerns regarding respondent burden or the costs of long surveys. The standard solution is to select a subset of the available items, which then are administered to all respondents. Such static-reduced batteries are well known to increase bias and lower measurement precision, however. CAT ameliorates much of the decreased measurement precision and accuracy associated with static-reduced scales. The project builds on existing work in the fields of educational testing and psychometrics to develop software and an online webservice that adopts and adjusts CAT for the specific needs of public opinion researchers.
This research project will apply CAT algorithms to the domain of survey research and provide practical resources and theoretical guidance specific to the needs of public opinion researchers. CAT algorithms adapt dynamically to measure latent constructs while minimizing the number of questions each respondent must answer. This approach uses information about the qualities of each question item to respond to individuals' prior answers by choosing subsequent questions that will place them on the latent dimension with maximum precision and a minimum number of questions. That is, it chooses questions for each respondent based on their previous responses so that researchers can learn as much as possible about the opinions of a respondent. The project will provide basic software infrastructure necessary for developing CAT batteries and administering them on surveys. This includes the completion and hosting of a cloud-based webservice easily integrated into survey-administration software. The project also will provide solutions to problems likely to confront survey researchers fielding adaptive batteries including appropriately handling high levels of measurement error and diagnosing flawed question-item calibrations. Finally, the project will collect data to build adaptive batteries measuring important concepts in the social sciences, with a special emphasis on personality inventories, using a combination of large convenience samples, a nationally representative sample, and existing datasets. The distribution of these calibrations will allow researchers to use CAT techniques without the added cost of extensive pre-testing.
