Better Science Through Benchmarking: Theory Validation and Application to Software Engineering PI: Susan Elliott Sim
The research seeks to refine a preliminary theory of benchmarking and use the theory to guide development of benchmarks for research results in software engineering. Benchmarks have been used in elsewhere in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. According to the theory of benchmarking, these effects appear because a benchmark is closely tied to the discipline's scientific paradigm (as described by Thomas Kuhn.) Both benchmarks and paradigms emerge through a consensus on technical results and cultural expectations for conduct. Based on a successful preliminary study applying this theory to reverse engineering, this research develops additional benchmarks in software engineering and collect data to validate the theory. This work will have benefits for the field through the specific benchmarks being developed, empirical evaluation of research results, and an improved understanding of how research is conducted. In terms of broader impact, the benchmarks enable geographically under-represented groups to participate in top tier research, because they can be downloaded from a web site and completed in reasonable time.
