Duane Alwin Pennsylvania State University, University Park
This research focuses on the reliability of measurement in social surveys. There has been much written about the sources of measurement errors in surveys and best practices in developing high quality survey instruments, that is, good questionnaires. The overarching goal of the project is to establish a factual basis for conjectures that exist in the survey methods literature concerning what are the qualities of good survey questions. The project will build a publicly accessible data base of information for roughly 900 questions representative of typical questions used in social science surveys. The data base will contain estimates of question-specific reliabilities, along with detailed coding of attributes of the questions (e.g. content, response formats, and question length), which can be used to evaluate the optimal properties of survey questions with respect to levels of measurement error. Through an analysis of the reliability information and the attributes of survey questions from several large-scale panel studies, practical suggestions will be made about the attributes of survey questions that will improve the quality of survey data.
This project builds on prior NSF funding, which relied on six nationally (or regionally) representative panel surveys of the American population, all involving probability samples -three National Election panels, the American's Changing Lives panel study, and the Study of American (Detroit Area) Family panel study -which produced question-specific reliability estimates for more than 450 questions. The present study has three major objectives, as follows: (1) extending an existing data base on the reliability of survey measures, (2) replicating prior findings using question-specific reliability estimates for an additional 400 survey questions or more from two new studies (using new analyses of the Health and Retirement Study and General Social Survey panel studies), and (3) developing a long-range plan for a publicly available archive of information on the reliability of survey measurement.
Broader Impact
This research will significantly increase the social science research infrastructure by providing a publicly available data base on reliability estimates for a representative pool of survey questions. Given that survey measurement is a key ingredient in the majority of social science research, the broader impact of the proposed research lies in its contribution to the uses of virtually all types of survey data, which can be evaluated in terms of the results of this study. The research will add to our current knowledge by making the reliability estimates obtained in the prior NSF-supported research, along with the extension proposed here, available to the public in a manner that allows users to search, filter or query the data base in investigating the question reliability of types of survey questions of interest. Thus, the long-range goal of the proposed project to create a public archive of the levels of reliability for the typical kind information gathering approaches used in surveys can have an impact on the development of survey questions for new surveys, as well as increase our understanding of the quality of existing surveys.
