The challenges of Exascale computing suggest that a new model of computation is needed upon which a new framework for mass storage may be built. Such a model currently under development is the ParalleX model. ParalleX addresses Exascale parallelism by removing most (if not all) global synchronization and providing a natural means for programs to manage local dependencies that maintain consistency. In a similar manner, we propose to develop a model of parallel storage that achieves the desired semantics without global synchronization. At the same time, the model will adopt a new programmer interface that reflects this approach. This I/O model is named PXFS and is intended to be integrated with the ParalleX model of computation, both by providing an I/O model for ParalleX and by being defined in terms of ParalleX. The goals of the proposed research are to derive a new model of persistent mass storage that unifies it with active in-memory data and develop PXFS, a proof-of-concept file system to enable effective and scalable Exascale computing. The objectives of the proposed project are to describe the PXFS model in complete terms, develop an initial reference implementation integrated with the HPX implementation of ParalleX, and evaluate the model via the reference implementation.
Science and engineering are increasingly reliant on the execution of realistic simulation models. Simulation is faster and cheaper than physical experimentation, and while it doesn’t eliminate the need for experiments it can dramatically reduce the time and cost needed to get to a working solution. Simulations require significant amounts of computation and can both consume and generate very large amounts of data. Today simulations use a computational technique known as parallel processing to provided the needed computational performance. Parallel processing refers to using hundreds or even thousands or more computer processors to work together to solve a single large problem. Parallel processing can be quite difficult to use properly and many problems lose performance dramatically when the number of processing elements gets very large. In order to allow these problems to scale to next generation computer systems, the ParalleX computational model has been proposed. ParalleX describes a way of organizing computer code so that it can scale effectively to extremely large numbers of processors. Until now ParalleX only defined computations, but modern problems perform as much data I/O as computation. This project has focused on defining an I/O framework for ParalleX. Under this project we developed PXFS, an I/O subsystem for a programming environment based on ParalleX called HPX. HPX is a system based on c++ that manages the creation, execution, and interaction of threads on a large distributed computer system. PXFS is based on the Parallel Virtual File System (PVFS) which is an open source system for performing I/O on a large distributed computer system. The primary goal of the project was to develop requirements for PXFS by developing a prototype. The prototype used PVFS file server processes, and part of the PVFS API but built a new interface on top in order to understand how these two complex software systems might interact. One of the main issues discovered in this project was that HPX assumes complete control over threads execution on a given computer in the system. This became problematic, because PVFS regularly interacts with the computer’s operating system in a way that would cause all of the HPX threads to stop while I/O was being done. To remedy this it was clear that the HPX/PVFS interface had to employ a asynchronous model. The first approach employed used an asynchronous layer on top of the PVFS file system interface. This layer was constructed to look much like the common POSIX API, but allowed each call to execute in the background and call a completion function when it had finished. This was implemented using a distinct thread which solved the conflict discussed, but was not particularly efficient. Experiments run on the interface showed very little overhead over standard interfaces, and even a performance benefit because I/O could be submitted concurrently. It was theorized that poor interaction with HPX was the culprit. Subsequent experimentation involved powering the asynchronous interface with an HPX thread which allowed HPX to manage the I/O. This produced good results. Additional experiments involved moving the I/O processing from an HPX compute node to a PVFS server node and using HPX active messaging to send data for I/O. This produced even better results because the I/O processing was removed from the compute nodes completely. Thus, the results of this project were to define an appropriate architecture for PXFS as the addition of HPX communication capability to storage nodes rather than the addition of storage processing on compute nodes. It also demonstrated that this can be done with a very low overhead and that the inherent support for highly asynchronous I/O can be very beneficial. This has led to a new proposal based on these results where we intend to integrate I/O into the HPX model in such a way that it becomes transparent to the user where storage ends and memory begins. This is to say that rather than a model where a program explicitly reads and writes data, a program would simple create and use objects and those objects would migrate between disk and memory as needed under the control of a runtime system. All based on the architecture we have developed under this project. Society needs the ability to process large amounts of data, whether for science, engineering, security, or the arts. Machines of the future will provide hundreds of thousands of processors and thousands of storage devices to deal with this, but the critical aspect will be the software that we use to make harness these devices. Currently systems of this scale are problematic because of a lack of asynchrony both in computation and I/O. The model provided by ParalleX and PXFS will provide a way to use these machines. This model but be powerful, controllable, but also intuitive to use. This project has been the first step towards this goal.
