Humankind's knowledge of the world and its ability to manipulate it for the betterment of quality of life and understanding through science, technology, engineering, and mathematics (STEM) is increasingly dependent on the ability to store, access, and manage extremely large persistent data sets representing scientific and process measurements, results from science and engineering simulations, and long-term knowledge. Supercomputers conventionally operate in dual or separate modes: one to do the computations in their temporary (ephemeral)-main memory-and the other to supervise the use of large persistent data storage. As supercomputers get larger, perhaps to the scale of an Exaflops by the end of this decade, the comparable scale and ease of use of mass storage is severely challenged. This research will address the problems of efficiency and scalability of data migration through the vertical memory hierarchy and will unify the way both main memory data objects and persistent storage data are named creating a single, easy to use programming. This will revolutionize data intensive supercomputing and establish a new path towards future Exascale system design and programming. This research is in collaboration with Clemson University to provide a proof-of-concept system to evaluate the new concepts.
The semantic and performance barriers between computing in main memory and manipulation of mass storage for persistent data have imposed significant limitations to performance and programmability. Because of uncertainties of access latency times combined with overheads and the need to exploit data access parallelism for high throughput, a new relationship between ephemeral storage and persistent objects is needed to unify their association and manage the asynchrony of operation while achieving high efficiency. This research is deriving an innovative execution model and developing a proof-of-concept experimental system to test and evaluate its underlying concepts for a new generation of persistent mass storage at extreme scale. It will address the challenges and provide the means for the unification of the semantics of ephemeral and mass storage through a single abstraction of data manipulation and the integration of meta-data and synchronization to manage asynchrony and uncertainty of response time as well as logical conflicting accesses while automatically hiding latency. The new model will support dynamic data path management for the asynchronous vertical storage hierarchy, exploiting adaptive runtime event-driven techniques for enhanced efficiency and scalability including management of vertical transport of data, which demands an innovative strategy of dynamic control of the entire data path.
