Future cyber-infrastructures are increasingly expected to revolve around the integration of big data, computation, and high performance networking. The High Performance Computing with Data and Networking Acceleration (HPCDNA) project focuses on these issues in the context of campus science and computational requirements. Many researchers have application specific data and a variety of desired compute environments, including purpose built small lab compute resources, medium scale campus High Performance Computing (HPC), cloud computing, and national scale distributed or centralized resources. The inability to flexibly and seamlessly get data to the most appropriate compute resource is often the limiting factor determining where the computation is run and what computation is performed. The HPCDNA project is developing technologies to greatly improve the ability to utilize common data sets across this diverse set of computational resources.
The HPCDNA architecture and technologies include a Network Embedded Storage (NES) system based on a distributed high performance parallel file system deployed in the core of the Mid-Atlantic Crossroads regional network. This system is enhanced for high performance via tight integration with campus HPC, campus networks, and wide area networks. An NES client interface is being developed, which coordinates storage with dynamic network provisioning, and enables a variety of high performance access methods by workflow processes.
It is expected that the HPCDNA system will facilitate expanded use of HPC and clouds by researchers, which will allow the adoption of more ambitious goals with the knowledge that their computing environments can scale up along with their problem space.
