Many decision processes in science, including hiring, tenure, promotion, and funding give considerable weight to citation-based metrics. It has in fact become commonplace to equate the number of citations received by an article with that article's "impact" or importance. Despite the plausibility of this equation, there are many reasons that scholars cite past work and there is no independent empirical evidence to support this fundamental postulate of bibliometrics. This lack of independent verification calls into question the meaning and purpose of the entire field of research, and is often used as an argument against the validity of reliance on citation numbers in assessment exercises. This project will test the feasibility and develop the methodology necessary to generate the first large-scale set of data on "true" scientific impact as actually perceived by scientists.
Measuring the true scientific impact of all articles will require surveying millions of corresponding authors of scientific publications as part of the biggest survey ever conducted in science. The main purpose of the present project is to establish the feasibility of the requisite survey methods. The research will survey small samples of authors to determine the optimal survey format necessary to achieve the appropriate tradeoff between response rate, information, and reliability. In particular, the research will explore variations in two basic ingredients of the survey format: content and design. We will select papers from different sample pools to quantify biases in participant preferences depending on external factors such as authorship, journal of publication, and number of citations accumulated. We will also consider different survey designs, and determine the most effective one able to increase the response of the potential participants and the amount of time that they will dedicate to the survey, without degrading the quality of the data obtained from their answers.
