Current high-end computing (HEC) applications explicitly manage persistent data, including both application state and application output. This practice not only increases development time and cost, but also requires an application developer to be intimately aware of the underlying platform-dependent storage mechanisms to achieve good application I/O performance. Such vertical development also makes the application software less portable.
The Software Persistent Memory (SoftPM) project builds a lightweight infrastructure for streamlining data management in next generation HEC applications. SoftPM eliminates the duality of data management in HEC applications by allowing applications to allocate persistent memory in much the same way volatile memory is allocated and easily restore, browse, and interact with past versions of persistent memory state. This simplifies the implementation of three broad capabilities required in HEC applications -- recoverability (e.g., checkpoint-restart), record-replay (e.g., data-visualization), and execution branching (e.g., simulation model-space exploration).
The SoftPM project is organized in three modules. The first module builds an evolvable SoftPM API and addresses memory management issues. The second module addresses high-performance I/O and the atomicity of persistence points for local storage and parallel file systems. The final module builds several HEC application case-studies to illustrate the different capabilities supported by SoftPM in HEC environments.
Intellectual Merit: This project developed a new software-based persistent memory (SoftPM) interface and library for block-based storage and in doing so, created new techniques that allow location-independent memory allocation of data structures. Such memory data can be modified without operating system involvement directly by the application and large sets of such modifications can made persistent atomically. The developed interface simplifies programming significantly whereby the only required operations are creating a named container, associating a single "root" data structure to it, and specifying persistence points. The SoftPM library seamless discovers all data reachable from the root via pointers and makes all such reachable data persistent. It also allows restoring the container data with one, simple library operation. Finally, the library supports advanced capabilities for container versioning and branching of container versions. Complementarily, this project designed and implemented parallel file system (PFS) delegation to optimize metadata management for deploying SoftPM on parallel storage systems. Evaluation done on PVFS2 shows encouraging results where PFS delegation based checkpointing substantially outperforms traditional many-to-many checkpointing approach. We further incorporated emerging solid-state devices (SSDs)in high-end computing systems. Broader Impact: Our work provided research opportunities for several students from minority groups at the undergraduate and PhD levels. Undergraduate Hispanic minority student, Leonardo Marmol, was an REU participant as part of this project. He decided to pursue a PhD degree to continue his work on the project at the graduate level. The findings related to the interface components of the Software Persistent Memory project add to the knowledge base for persistent memory interfaces and impacts new interface standards for persistent memory such as the one considered by the T10 standards body.
