For many decades, numerous important decisions made by governments and businesses around the United States have been informed by data collected through representative sample surveys of Americans. Billions of US dollars are spent every year so that one can learn the nation's unemployment rate, its inflation rate, and much more about the experiences of Americans that govern policy-making, cost-effectively and quickly through surveys. In recent years, there has also been a sharp increase in the use of survey data by academic researchers to enhance theory development in a wide range of important domains, providing insights into the nature of contemporary social life, the workings of the human mind, and approaches to solving pressing national problems.
At the same time, the nation has experienced a sharp increase in the costs of conducting high-quality surveys, coupled with a decline in the response rates of such surveys. The U.S. federal, state, and local governments, the private sector, and the academy therefore need quick innovation in survey methodology to permit cost-effective collection of accurate data from highly representative samples in the future.
This project will take an unprecedented step to test a new methodology to achieve this goal. If this trial works, the methodology will be available for use by all organizations around the country that are committed to making informed policy-making decisions. The tool to be built will bring a new standard of accuracy to data, provide broad accessibility to social scientists at reduced cost, and blend the analytic power afforded by a state-of-the art computer network laboratory with the generalization power afforded by a representative sample survey of American households. Specifically, the project will purchase computer hardware, install the hardware in a representative sample of 1,000 households across the country, and calibrate the national network of computers via the Internet, testing to assure that it works properly to permit social science survey data collection. The network will enable path-breaking research that could not be achieved by any methodology currently in use and would considerably increase the value of the data collected, analyzed, and reported.
Sampling statisticians will draw a representative sample of American households using an innovative method based upon U.S. postal service mailing address lists. Then, installation specialists will visit the selected households, randomly select an adult household member, conduct a brief interview face-to-face (with an expected response rate of about 80%), and then offer him or her a free laptop computer and free high-speed Internet connection in exchange for using the equipment to provide data once a month via the Internet. The installation specialists will install the computers, walk respondents through the process of using them, and calibrate the equipment on site to work properly. Then, once a month, respondents will provide a new round of data by accessing a secure webpage, and calibration of the computer network will be conducted. The opportunity to collect data on this platform will be made available to government agencies, businesses, and academic scholars. A range of different evaluations of the network will be performed to assess its effectiveness.
If this effort is successful, it will provide scientific and practical justification for the implementation of this approach to experimental and non-experimental survey data collection on a much larger scale, which will enable academic researchers, federal, state, and local government agencies, and private commercial and non-commercial organizations to conduct the highest quality research at a practical price.
Survey research is a valuable tool used by businesses, government agencies, and academic researchers to study populations. By drawing a truly random sample from a population and asking unbiased questions, survey researchers measure a wide range of phenomena with high levels of accuracy. This is why the U.S. government, for example, measures the national unemployment rate via surveys of the American public, to count individuals who were without paid work and were looking for a job. Surveys can be conducted in four modes: (1) face-to-face interviewing, usually in people’s homes, (2) telephone interviewing via landlines and cell phones, (3) paper and pencil questionnaires, and (4) computers, usually via the Internet. People answer questions somewhat differently in the different modes, and past research has shown that face-to-face interviewing and computer self-administration elicit the most accurate reports from respondents. Furthermore, face-to-face recruiting at respondents’ homes yields the highest survey response rates. This Major Research Instrumentation grant funded the building of an innovative survey research platform that coupled the high response rates yielded by face to face recruiting with the high accuracy self-reports that computer self-administration elicits. The project created what is called the Face-to-face Recruited Internet Survey Platform, or the FFRISP for short. A random sample of American households was selected in a multistage clustered area probability sample, and highly trained interviewers from Abt SRBI visited the households to randomly select one adult resident per household and recruit him or her to join an Internet panel and answer 30 minutes of questions per month for 12 months. A laptop computer was installed in every household, and households that lacked high speed internet access were provided that access at no cost for the year. At the end of the year, the households were permitted to keep the laptops. Each month, after the respondent completed his/her questionnaire online, he/she received a small amount of cash as compensation. The questionnaires and data collection procedures were designed to permit calibration of the measurement system via evaluations of the methodology in many regards. The resulting response rate for the panel was the highest that has been observed for a probability sample completing interviews via the Internet to date, and the retention rate was extraordinarily high: more than 90% of respondents who joined the panel were actively participating at the end of the year. Furthermore, the accuracy of the obtained measurements was equivalent to that of the measurements acquired in gold-standard high quality surveys involving face-to-face interviewing, such as the General Social Survey, at a fraction of the cost per respondent-minute of data collection. Furthermore, the FFRISP provided a relatively large amount of data on each respondent over a long period of time, allowing the study of individual-level change and causal influences inducing such change. The data collected were used by many investigators to conduct calibration analyses of methodological phenomena such as survey satisficing (the tendency of respondents to sometimes answer questions superficially without careful thought), as well as to study substantive phenomena such as racism and opinions on global warming and the causes of consumer purchasing decisions, to explore the effectiveness of this data collection platform for conducting such research. Findings were very encouraging and in many instances replicated and extended the findings produced using data collected via face-to-face interviewing of area probability samples. This reinforces confidence in the validity of the FFRISP data. Lessons learned from the project include a long list of insights gained about how to effectively recruit people to join such an internet panel, how to maintain their participation through an extended period, how to manage the network of computer equipment and software programming required to manage the panel, how to maintain the good will and morale of participants, and how to prevent problematic behaviors from occurring. At the end of the year, the FFRISP respondents were invited to join the American Life Panel, an online survey panel run by the RAND Corporation, and most respondents accepted this invitation to continue completing online surveys. Therefore, the investment made by NSF in creating the FFRISP panel was successful in creating a data collection platform of perpetual value.
