Published reports on software-development costs and our interactions with many companies reveal that testing and debugging consume too much of the development costs. Furthermore, the ubiquitous nature of software requires that it be high quality. Thus, there is an urgent need for technology that companies can use to improve the process to reduce the cost of these activities and improve the quality of the software. The technology that this team will explore for commercialization consists of rigorous techniques that are highly automated, and have the potential to improve reliability and adaptability of today's large, complex software systems at a reduced cost. These techniques have provided the foundations in testing and debugging on which many additional techniques have been developed. The techniques have been implemented in prototype tools that have been used for experimentation that shows their effectiveness and efficiency in improving the quality of the software and reducing the cost of achieving it. However, no commercial tools or systems currently provide this advanced technology. The team's project has the potential to provide such tools and systems.
The technology the team plans to commercialize will improve the efficiency and effectiveness of software development by addressing the process of testing and debugging. Not only will the technology result in higher-quality software for those companies who use it, but it will provide savings in the costs of the software development. Because the technology can be used for testing and debugging at all levels of development (e.g., individual, team, and system), on systems written in different programming languages and on non-code components (e.g., databases), by companies that develop software for their own use or to sell, and by companies of all sizes, the commercialization of the technology can have a significant impact on industry. If successful in commercializing the technology, the tools and systems being built will ultimately improve software quality and, thus, has the potential to have a significant societal benefit. The project will prepare students and faculty who participate in the important area of entrepreneurship, which will help them transition the technology developed to industry.
This Innovation Corps (I-Corps) project considers the problem of how to test and debug software in real-world commercial settings. Its more specific goal and intellectual merit is to study how feasible it would be to deploy the kinds of testing and debugging tools being developed as research projects into such settings. This analysis opens up both new opportunities for research as well as new ideas for how to harden research prototypes into robust tools that can truly address industrial-scale software. The broader impact is to significantly reduce testing and debugging costs in commercial environments by bringing state-of-the-art research tools into wider use. There are two broad categories of outcomes of this project. The first is a qualitative analysis of the state-of-practice; the second relates to technology transfer, and more specifically, uses the analysis to improve and harden existing research prototypes into tools with the potential for commercial deployment. Background. This study concerns two specific research prototype tools that the Principal Investigator (PI) and her team have been developing over a number of years, using NSF support. The first is a debugging tool called Tarantula. It helps software developers to find automatically the parts of their program most likely to have caused a failure. It works by repeatedly testing the program, gathering data about what parts of the program execute during each test, collecting statistics about when the program passes and fails each test, and then analyzing all of this information to estimate the most "suspicious" program statements. The second tool, DejaVOO, focuses on testing itself. It specifically focuses on how to generate and select tests, among other testing tasks. Outcome 1: Analysis of the state-of-practice. The state-of-practice analysis resulted from in-depth discussions with 84 different commercial software development companies. The following summarizes the key findings: Industrial software is very complex, often employing multiple programming languages and depending intricately on varities of third-party software, scripts, and databases. Thus, there would need to be at least some customization to each company's systems and processes. However, such customization for a given company appeared tractable, that is, customizing would be a relatively minor activity. Thus, this aspect of industrial software was not a major impediment to deployment. Automation in industrial settings is far behind the state-of-research. This finding indicates the value of automated tools but an even more pressing need to invest in ways to translate these tools from research into practice. A critical pain point in commercial practice is a lack of use of quality-assurance and testing processes. Indeed, companies largely confirmed that the time spent on testing activities may account for anywhere from 10-50% of a project's budget, with that fraction growing over time. (This fact is especially acute in industries like medical and aerospace software development, where customer requirements are increasing.) Addressing these problems would require both additional research and development. In particular, regression testing time, test generation, test selection, and impact analysis (anticipating the effect of software changes) were found to be especially critical problems. This finding, among others, suggested that the project focus on enhancing the DejaVOO tool to support these tasks, which the project accomplished. Outcome 2: Technology transfer. The initial effort, prior to the analysis, focused on hardening the Tarantula tool, which focuses on problems of automatic bug finding. This software is freely available for download and use at: http://aristotleresearch.com. Following the preliminary analysis, the PI and her team changed the focus to extending and hardening the DejaVOO tool, which could address the pain points related to testing problems and impact analysis. A number of enhancements were made based on the feedback from several large companies, making the tool more robust and capable of operating in a complex software development environment. As this project closes, the team is exploring initial deployments with several large organizations.
