Software is the means by which scientists harness the power of computing. Much of the most widely used research software are those created by other practicing scientists. Science has a critical interest in identifying and rewarding the production of useful and impactful computing tools. Traditional metrics for assessing scientific significance and impact, like citation, may not adequately reflect the total utility and benefit resulting from a program's creation, dissemination and use. Basic questions of incentive and reward for computational tool creators, even within the traditional frameworks of citation and funding, remain unclear, and questions of accurately assessing use vs. utility vs. productivity have yet to be rigorously examined. The goal of this research is to improve current measures for impact of scientific software, evaluate alternative metrics, develop methods for analyzing the direct measurement of research software usage, and deploy tools to aid research software users and developers to better cite, gauge and evaluate software use, impact and productivity.
The development of sociotechnical systems and methods for disseminating data gathered through the direct tracking of research software usage within the SBGrid consortium of software users will provide unique insight and information to program authors and developers regarding the use, impact and productivity of their software. This tracking information will also provide users with an automated means of recoding and documenting their computational workflows and protocols, and help relieve the difficulties in compiling comprehensive bibliographic information of computational methodologies and procedures. In addition to improving future progress in assessing the metrics of scientific software use, data mining techniques will be employed to retrospectively examine historical and contemporary databases and data repositories containing software usage data and metadata, publication and citation information, and research output to assess the efficacy and accuracy of traditional measures of scientific impact. The mined data will also be analyzed and used to evaluate new and existing alternative metrics for measuring the scientific impact of research software and computational tools.
