Software testing techniques are fundamental to the development of dependable software. Computer scientists studying testing techniques must be able to evaluate and compare them empirically, and this requires access to a wide range of infrastructure support including programs, program versions, test suites, and fault data. This project provides that support, focusing on test generation and on software domains that are increasingly important: concurrent and distributed systems, software product lines, and web services. The PIs are collecting and constructing systems, specifications, version attributes, test attributes and fault attributes relevant to these domains, and organizing them in formats that allow computer scientists to utilize them efficiently in controlled experiments. Th PIs are embedding these artifacts in the existing Software-artifact Infrastructure Repository (SIR), a repository established under prior NSF support to facilitate experimentation with deterministic C and Java systems. They are also enhancing SIR's support for community contributions, by providing moderated write access to the repository; this will be expanded in the latter years of the project, following the spirit of open-source models for system management. The infrastructure being assembled enables several specific research and educational projects related to software testing, by the project researchers themselves and by the software engineering research community as a whole.
As software grows increasingly ubiquitous in our society, the costs of software failures increase. Software engineers rely on program analysis and testing techniques to reduce the incidence of such failures. Researchers are actively pursuing new approaches to analysis and testing, and approaches that automate processes, but these approaches must be assessed empirically. A key challenge in performing empirical studies is the difficulty of acquiring suitable experimental objects. Experiments with testing and analysis techniques can require software (often in multiple versions), test suites, and fault data. Obtaining such artifacts is not simple: free software seldom possesses the required attributes, while commercial systems are hard to get and when available, seldom provide support for rigorous controlled experimentation. The Software-artifact Infrastructure Repository (SIR) has been established, and under this grant, extended, to help provide support for experimentation with testing and analysis techniques. The SIR was originally established and populated with a number of deterministic, sequential C and Java systems. In this work, we extended the SIR to support research across a much wider domain, including new classes of software and testing methodologies. Artifacts have been added to support experimentation with software product lines, C++ and Java concurrency problems, C# software, and artifacts annotated with assertions. These artifacts have been validated by the PIs and students involved in the grant through numerous empirical studies, including studies of test generation techniques, techniques for augmenting (improving) test suites, and testing techniques for software product lines and highly configurable systems. Facilities for allowing user contributions to the SIR have also been created. The largest contribution of this work, however, has been its impact on the software engineering research community. As of the date on which this grant terminated, 2103 persons had registered for access to the SIR. These persons are associated with 629 different institutions (primarily academic, but also including some research laboratories). Even more significantly, 336 journal articles, conference publications, and doctoral dissertations have cited SIR as a source of support for their research. SIR has thus facilitated or enabled research on software engineering topics, that could not previously been as easily conducted, with potential impacts for anyone in society who depends on the reliability and security of software.
