This research is at the nexus of the data deluge in science and business and two major computing thrusts - clouds and exascale scientific systems which are unified with an interoperable runtime system. The project has the potential to transform the approach to applications that varies from data mining of genomic and proteomic data for science to data analytics for business. Computer science areas at the heart of the research - namely Iterative Map Collective runtime, fault tolerance, data-computing co-location and high level languages - will be advanced. Furthermore, the new applications enabled and new software paradigms will feed back into the architecture of cloud and exascale systems possibly suggesting particular storage and communication choices and new directions for the national infrastructure. The investigator will incorporate this novel research into courses and graduate and undergraduate research experiences at both Indiana University and with national and international collaborators. The work blends scientific research (computer science and applications) with mainstream commercial practice (clouds). Thus, curricula built around this research will motivate and inspire the entry of students into the workforce and so it has potential for supporting needed economic development.
The research is based on initial research on Iterative MapReduce with successful prototypes Twister (on HPC) and Twister4Azure (on clouds). The project will architect and prototype a Discovery Environments for Data-Enabled Science and Engineering with the following components developed: (1) a next generation Iterative MapReduce using a Map-Collective model as the runtime for data analysis (mining) interoperably between clouds and clusters; (2) polymorphic collective operations needed to support parallel linear algebra and other data analysis operations such as those in MapReduce; (3) a software message routing using publish-subscribe to scale to tens of thousands of nodes or above; (4) a storage model that builds on current object stores, data parallel file systems (as in Hadoop), and wide area models like Lustre but respects compute-data co-location; (5) a fault tolerance model implemented as a Collective operation with configurable settings that supports checkpointing between iterations for robustness and individual node failure without compromising performance. Later research objectives include security and a higher-level programming model that compiles to an iterative MapReduce runtime.
