Workflows, especially data-driven workflows and workflow ensembles are becoming a centerpiece of modern computational science. However, scientists lack the tools that integrate the operation of workflow-driven science applications on top of dynamic infrastructures that link campus, institutional and national resources into connected arrangements targeted at solving a specific problem. These tools must (a) orchestrate the infrastructure in response to application demands, (b) manage application lifetime on top of the infrastructure by monitoring various workflow steps and modifying slices in response to application demands, and (c) integrate data movement with the workflows to optimize performance.
Project ADAMANT (Adaptive Data-Aware Multi-domain Application Network Topologies) brings together researchers from RENCI/UNC Chapel Hill, Duke University and USC/ISI and two successful software tools to solve these problems: Pegasus workflow management system and ORCA resource control framework, developed for NSF GENI. The integration of Pegasus and ORCA enables powerful application- and data-driven virtual topology embedding into multiple institutional and national substrates (providers of cyber-resources, like computation, storage and networks). ADAMANT leverages ExoGENI - an NSF-funded GENI testbed, as well as national providers of on-demand bandwidth services (NLR, I2, ESnet) and existing OSG computational resources to create elastic, isolated environments to execute complex distributed tasks. This approach improves the performance of these applications and, by explicitly including data movement planning into the application workflow, enables new unique capabilities for distributed data-driven "Big Science" applications.
The project brought together teams of computer scientists from RENCI/UNC Chapel Hill, USC/ISI and Duke University with the purpose of studying the feasibility of complex scientific computational workflows running on dynamically provisioned cloud-like widely distributed cyber-infrastructure. The distributed cyber-infrastructure included a collection of institutional clouds, which included computational and storage resources linked with on-demand network connections that can all be provisioned together using ORCA control software, developed at Duke and RENCI and deployed in ExoGENI testbed (www.exogeni.net). The workflows were managed by the Pegasus workflow management system (http://pegasus.isi.edu/) designed at ISI. The project team developed software elements to tie these two systems together with a performance feedback mechanism called ShadowQ and also provide an easy-to-use portal for scientists to submit and monitor their computational workflows. The main goal of the project was to prove that such a coupling between workflows and infrastructure was indeed possible, i.e. it was possible for a workflow to provision the necessary infrastructure for itself and manage it through the workflow execution process. The benefits of this approach include improved usability by domain scientists compared to existing institutional and grid resources as well better predictability and performance and security isolation, which speed up the discovery process and encourage more scientists to use existing resources. Additionally, learning how to execute workflows in distributed cyber-infrastructure represented by ExoGENI demonstrated how multiple different science domains can effectively use federated resources contributed by multiple institutions thus reducing the need for creating and maintaining separate cyber infrastructures for each science domain and improving the efficiency of investments in infrastructure build-outs. The project focused on two science domains - astronomy and high-throughput gene sequencing. The two domains were chosen because on the one hand they heavily rely on workflows and workflow management systems for performing their computational tasks and on the other, of their widely differing requirements to the cyber infrastructure that must execute those workflows. The ADAMANT team successfully demonstrated both astronomy and gene sequencing applications at various venues including the premiere SuperComputing conference and generated interest among the scientists in those two domains.
