System benchmarking is a critical technique used by application developers and system administrators to help them decide how best to perform a wide spectrum of system management tasks. These tasks range from design-time aspects such as picking which system to use for an application, to all pre-deployment aspects of testing and tuning the applications and system, and to production aspects such as reliability assurance and problem diagnosis. However, the reality today is that the massive potential of benchmarking is seldom harnessed satisfactorily by application developers and system administrators. Benchmarking is very human-effort-intensive. Users may not be aware of or follow correct benchmarking methodology. Often corners are cut to reduce either time or cost without full awareness of the implications of doing so. Repeatability and reuse of benchmarking results are often unattainable. The results of a benchmarking effort may be misinterpreted, leading to incorrect or suboptimal decisions.
This project combines new research with recent technology trends in order to make the power of benchmarking accessible to a diverse community of users. The project is building a Benchmarking-as-a-Service (BaaS) platform that runs on top of an Infrastructure-as-a-Service or Platform-as-a-Service cloud platform. A BaaS provides four integrated services to simplify the overall benchmarking process, while addressing a number of novel technical challenges: 1. Design: Helps the user create the workloads and data that best meet her specific benchmarking needs, using the concept of benchmark design as a search problem; 2. Execute: Helps the user run the selected workloads on appropriate data, system configurations, and hardware and software resources in a timely and cost-efficient manner; 3. Interpret: Helps the user interpret the results of the benchmark correctly since many users who run benchmarks on systems are far from being experts on these systems (many challenges arise here due to the large volume, variety, and velocity of system monitoring data); and 4. Build: Helps the user build new system management services on top of the BaaS. For example, BaaS enables the automatic tuning of production database instances running on cloud platforms like Amazon Web Services and SQL Azure. BaaS is being prototyped and opportunities for deploying it in real-world settings will be pursued. The prototype will be used to evaluate the transformative capabilities of BaaS towards increasing the productivity of humans and systems, and in avoiding wrong decisions involving mission-critical systems.
