The computer architecture research community has created a rich and diverse collection of simulators, emulators, and benchmarks. Each individual research group has its own preferred tools, perhaps built on top of an existing or custom infrastructure. These artifacts are used in experiments to evaluate design trade-offs and to analyze implications to performance, energy, reliability, and other metrics.
Unfortunately, the ways in which these tools are used for evaluation often hinder the application of the scientific method. The barrier to entry for using various artifacts (i.e., simulation tools and benchmarks) is usually high, when these artifacts are made available. Experiments are rarely open, nor easily repeatable by other researchers, leading to inaccurate comparisons. Moreover, it is difficult for a single group to explore all variations due to state-space explosion.
To demonstrate that these problems can be addressed to improve computer architecture evaluation, this pilot project takes the first steps to develop an open-access repository that permits sharing artifacts and experimental results among a broad group of stakeholders in computer architecture. The project builds and engages an active community of users, establishes governance and access policies, and determines the requirements for software services of the repository. This initial prototype provides the community an opportunity to supply feedback and suggestions to evolve the prototype and refine its policies.
The project shows that significant collective effort can be saved by sharing simulators and experiments for accountable and repeatable experimentation as part of the scientific method. It avoids the burden of re-implementing and re-creating tools, experiments, data sets, benchmarks, etc. This savings in effort and more sound and complete evaluation can then be directed on innovating new architectural techniques for better computer systems.
