Social scientists routinely design questionnaires using rating scales to measure social and psychological constructs. There is no consensus, however, among researchers about the optimal number of points to offer on such scales so as to maximize reliability and validity. This project will compare the reliabilities and validities of rating scales ranging from 2 points long to 11 points long presented visually. This project also will explore the moderating roles of three aspects of the task (biopolar vs. unipolar constructs; verbal labeling on all scale points vs. the end points only; topic familiarity) and two respondent characteristics (cognitive skills and motivation to provide accurate reports, as gauged by need for cognition). Longer scales may be optimal for bipolar constructs, scales that are fully verbally labeled, and highly familiar topics. To address these issues, a three-wave survey will be conducted with a nationally representative sample of adults by Knowledge Networks, a survey research firm that collects data weekly via the Internet from an established panel of respondents who have all been equivalently equipped with Internet access. The questionnaire will manipulate number of points on target rating scales addressing a range of topics and will include criterion measures for assessing concurrent validity of the target ratings. Also manipulated will be polarity of the construct and the extent of verbal labeling of the scale points. Issue familiarity, respondents' cognitive skills, and their motivation to provide accurate reports will be measured. Multiple groups longitudinal structural equation modeling will be conducted to assess the effects of these factors on the reliability and validity of responses, explicitly estimating random and systematic measurement error and measurement validity.
Rating scales are ubiquitous in social science research because they are convenient tools for measuring attitudes, beliefs, behavioral intentions, behavioral experiences, and a range of other phenomenon of great interest to scholars, policy-makers, and business planners. Researchers use a wide variety of different scale lengths in different investigations, however, suggesting that there is no consensus as to the optimal length of a rating scale to maximize the accuracy of the measurements obtained. But it is easy to imagine that rating scales may be too long, making it difficult for respondents to use them, because the meanings of the scale points will be ambiguous. And rating scales may also be too short and therefore suboptimal because they clump people into a few crude categories (e.g., separating the people who support a law from those who oppose it, rather than permitting people to express degrees of support or opposition). This project will help to improve the accuracy of social science measurements by comparing the effectiveness of rating scales of various lengths to identify which are optimal. The study will investigate whether the optimal scale length is greater for more familiar topics, for scales that have all points labeled with words (instead of only the end points being labeled as such), and for scales with a zero-point in the middle (e.g., from "dislike a great deal" through "neither like nor dislike" to "like a great deal"), as compared to those with a zero point at the end (e.g., from "not at all important" to "extremely important"). And the project will test whether people with greater cognitive skills and more motivation to report precisely are able to manage longer rating scales more effectively. The findings will contribute to the psychological literature on how ability and motivation moderate task performance, and the findings will provide practical guidance for questionnaire designers in government, in the commercial world, and in academia (in psychology, sociology, political science, economics, education, communication, anthropology, medicine, and many other sciences that make measurements via questionnaires).
