The emergence of large-scale Internet applications and services has driven a surge in research on cloud platforms, virtualization, data center architectures, and green computing. However, realistic performance evaluation of new research prototypes continues to be a major challenge. Home-grown performance evaluation tools that are often used by researchers are no longer able to capture the complexity of today's real systems and applications. To address this drawback, the project seeks to develop BenchLab, an open, flexible community infrastructure comprising applications, workloads and tools to enable realistic performance evaluation and benchmarking by systems researchers. BenchLab is an open framework where source code and workload datasets are freely available for modification and use by researchers for their specific experiments. The framework consists of a suite of server-side benchmark applications and workloads that represent cloud, mobile web, and green computing environments. BenchLab employs a modular, extensible architecture that is designed to support a range of server applications and workloads, with the ability to add support for newer applications and workloads and retire outdated ones. BenchLab is designed to be easy to use for experimental systems research "at scale" in commercial clouds, such as Amazon EC2, or virtualized clusters in laboratory settings. BenchLab will provide open tools and workloads for the research community to enable researchers to run larger and more realistic performance evaluation experiments that better emulate today's real-word systems. BenchLab will be incorporated into hands-on lab assignments to teach students the science and art of experimental performance evaluation.
