The proposal requests support for a one-day workshop to review and suggest improvements in the instrumentation used in the Surveys of Public Attitudes Toward and Understanding of Science and Technology. The workshop will bring together survey methodologists and substantive experts at NSF headquarters in Arlington, VA. The participants will consider the wording of questions, the structure of response scales, the meaning of terms used in the questions, and the range and overlap of the items in light of the concepts that they are designed to measure and the need for comparability and continuity of measurement. Participants at the workshop will include leading experts in questionnaire construction, cognitive approaches to survey design, psychometrics, and item response theory, along with substantive experts in measurement of scientific literacy. Scholars who have published leading work on measurement issues in science literacy will also be invited.
The results from the Surveys of Public Attitudes Toward and Understanding of Science and Technology have usually been included in the NSB's annual Science and Engineering Indicators, a report which goes directly to the White House and informs public debate about science education and public support of science in the United States. It is vital that the reported survey results be based on sound, tested survey questions that ask about the right issues and provide high-quality measurement of the key concepts that reveal public understanding of science--or the lack thereof. The workshop will review and improve key items on the survey and align these instruments more closely with the relevant concepts. These improvements will allow for a more revealing and more robust assessment of public understanding in future editions of the Science Indicators report and serve as a more reliable guide for future science policy in the United States.
Since 1979, NSF has sponsored recurrent surveys of the American public to gauge their understanding and awareness of science, sometimes referred to as scientific literacy. The results have appeared in Science and Engineering Indicators. However, the 2010 edition of Science Indicators omitted the results from two true-false items because the National Science Board judged that they "don’t properly reflect what Americans know about science and, thus, are misleading." NSF arranged with Chris Toumey of the NanoCenter at the University of South Carolina, and Thomas Guterbock, Director of the Center for Survey Research at the University of Virginia to conduct two scientific workshops during Fall 2010 to evaluate the questions that the National Science Board has used over the years as indicators of public knowledge of science. The first workshop (supported by a different grant) focused on the concept of adult science knowledge underlying the survey items, including the items that involve both scientific knowledge and religious conviction. The second workshop, supported by this grant, examined the ‘instrumentation’ of the survey, that is, the measurement adequacy of the items themselves. In consultation with NSF Program Officers and staff, Guterbock brought together for the second workshop a group of survey methodologists and substantive experts, with the purpose of developing a set of specifications to identify the measurement qualities that would be desirable in the public science knowledge questions and outlining a protocol for creating additional questions and testing them. The group met at NSF’s Arlington headquarters on November 12, 2010. The workshop participants represented a wide range of expertise from the disciplines of sociology, communication, psychology, political science, and health policy and included survey researchers and methodologists. Some of the participants had been involved with the Science Engineering Indicators for many years. Others brought a fresh perspective stemming from their research on related topics. The workshop generated concrete ideas for better measurement of public knowledge and attitudes toward science and technology. One insight that was especially salient to the group was developed in the preceding, conceptual workshop. Toumey and his colleagues emphasized that persons in the public have different reasons for acquiring scientific knowledge. Sometimes a person is in the role of acquiring information as a consumer. At other times, a person is acting in a civic role and needs scientific knowledge for decision-making about public matters. Finally, acquiring scientific knowledge is often pursued for sheer interest, enjoyment, or personal fulfillment. These three purposes have a direct impact on how to improve the measures of scientific knowledge. The November workshop generated a report that suggested ways of improving the questions that are asked in NSF"s surveys of public knowledge of science and technology. In particular, the report made suggestions on how questions might be improved or made more informative to researchers, by distinguishing more clearly between knowledge of scientific theories and belief in those theories. The work of this group has contributed to a continuing debate in the National Science Board and in the pages of Science about the most appropriate ways to measure public understanding of science and technology in the United States and in other countries. The proposed "Science in the Service of Citizens and Consumers" Framework provides a new perspective essential to the development and improvement of items to measure public attitudes toward and understanding of science. The full report of the proceedings of this meeting as well as the workshop’s recommendations, suggested for presentation to the National Science Board, can be found at: www.coopercenter.org/csr/publications/measurement-and-operationalization-science-service-citizens-and-consumers-framework.
