Modern scientific applications, such as analyzing information from large-scale distributed sensors, climate monitoring, and forecasting environmental impacts, require powerful computing resources and entail managing an ever-growing amount of data. While high-end computer architectures comprising of tens-of-thousands or more processors are becoming a norm in modern High Performance Computing (HPC) systems supporting such applications, this growth in computational power has not been matched by a corresponding improvement in storage and I/O systems. Consequently, there is an increasing gap between storage system performance and computational power of clusters, which poses critical challenges, especially in supporting emerging petascale scientific applications. This research develops a framework for bridging the said performance gap and supporting efficient and reliable data management for HPC. Through innovation, design, development, and deployment of the framework, the investigators improve the I/O performance of modern HPC setups. The target HPC environments present unique research challenges, namely, maintaining I/O performance with increasing storage capacity, low-cost administration of a large number of resources, high-volume long-distance data transfers, and adapting to the varying I/O demands of applications. This research addresses these challenges in storage management by employing a Scalable Hierarchical Framework for HPC data storage. The framework provides high-performance reliable storage within HPC cluster sites via hierarchical organization of storage resources, decentralized interactions between sites to support high-speed, high-volume data exchange and strategic data placement, and system-wide I/O optimizations. The overall goal is a data storage framework attuned to the needs of modern HPC applications, which mitigates the underlying performance gap between compute resources and the I/O system. This research adopts a holistic approach where all system components interact to yield an efficient data management system for HPC.
High performance computing (HPC) systems are faced with a deluge from the vast amounts of data that is being processed by the state-of-the-art and emerging petascale scientific computing applications. The goal of this project is to address the storage and I/O challenges arising from such data-intensive operations. We have developed a framework to bridge the performance gap between storage and compute components and support efficient and reliable data management for HPC. We adopted a two-pronged approach: providing high-performance reliable storage within HPC cluster sites via hierarchical organization of distributed storage resources, and enabling decentralized interactions between sites to support high-speed, high-volume data exchange. A key contribution of the project is the design and development of tools to optimize the large-volume data transfers in HPC workflows. First, we developed a contributory storage based solution, which enabled HPC centers to offload data to user-provided distributed storage sites. We also developed cloud-enabled techniques for seamless data transfer between HPC centers and users, and offloading data-intensive workloads from the HPC centers to the cloud. Our off-loading approaches exploit the orthogonal bandwidth available between the users and the HPC center and relieve the center from handling I/O-intensive tasks, thus allowing the center to focus on compute-intensive components for which it is better provisioned. Evaluation of our approach using both real deployments as well as simulations demonstrates the feasibility of decentralized offloading; an improvement in the data transfer times by as much as 81.1% for typical HPC workloads was observed. Second, we explored the use of solid-state storage devices (SSDs) in designing a novel multi-tiered data staging area that can that then be seamlessly integrated with our offloading system, with the traditional HPC storage stack (e.g., Lustre) as the secondary storage. The novelty of our approach is that we employed SSDs in a limited number of participants that are expected to observe the peak load, thus ensuring economic feasibility. Our evaluation showed that the staging area absorbs application checkpoint data and seamlessly drains the data from various storage tiers to the parallel file system, thereby improving the overall I/O performance. We also extended the work to use adaptive data placement, both across various storage layers of an HPC site and with individual nodes within a site. The evaluation yielded better understanding of using the storage layers, and insights into how to incorporate SSDs into the storage hierarchy. Finally, we explored the use of emerging technologies such as accelerators and low-power micro-servers in supporting the HPC I/O stack operations. Specifically, we explored the use of such components in supporting I/O-intensive workloads both for HPC applications as well as the extant cloud programming model, Hadoop. To this end, we designed low-cost GPUs to achieve a flexible, fault-tolerant, and high-performance RAID-6 solution for a parallel file system. We capitalize the resources provided by the file system, such as striping individual files over multiple disks, with the computational power of a GPU to provide flexible and fast parity computation for encoding and rebuilding of degraded RAID arrays. The results demonstrate that leveraging GPUs for I/O support functions, i.e., RAID parity computation, is a feasible approach and can provide an efficient alternative to specialized-hardware-based solutions. The effect would be to reduce the cost of HPC I/O systems, and improve the overall efficiency of the system. The work on designing a robust and scalable data storage framework attuned to the needs of modern HPC applications is complete and has resulted in improving the efficiency of HPC storage systems (as discussed above). With the help of our collaborators, some of the services are being adopted in production class machines at Oak Ridge National Lab. We have also integrated our techniques with the commonly used tools for HPC data management, which will enable easy adoption of our approaches by other researchers and practitioners. The results from the experiments were disseminated via publications in appropriate venues and presentations at various high-quality peer-reviewed conferences. The project has led to five PhD theses (two by females) and four MS theses (two females). The PI also mentored 12 REU students including female and minority students during the course of the project. The PI also created related introductory lectures for Virginia Tech College of Engineering Freshmen Seminar. The research was leveraged and extended to develop courses on introduction to HPC and cloud computing, as well as in integration of a file and storage systems component in the PI’s undergraduate and graduate operating systems classes, and in the systems and networking Capstone course. The participants (graduate and undergraduate students) were educated in HPC and wide-area data transfer systems and data management, as well as on principles of cloud computing.
