Cloud computing platforms such as Amazon Web Services, Microsoft Azure, Cornell Red Cloud, and the SDSC Storage Cloud have emerged as important research computing resources. These resources are already popular in research and education due to their instant (on-demand) access, seemingly infinite scalability, ease of configuration for widely varying needs, and ability to offer higher-level, easier-to-use services to less-than-expert users. However, as usage of Cloud resources by the NSF community continues to grow, challenging issues of resource allocation, accounting, reporting and billing emerge. This award develops a framework that can simplify Cloud usage for researchers and educators while also enabling allocation, monitoring, and management by those responsible for strategic and budget planning and reporting. The PI's partner with Amazon, Inc., Eucalyptus Systems, Inc., and MathWorks, Inc., the Globus Online team from the University of Chicago and the Red Cloud team from Cornell University to develop and evaluate a prototype service called Globus Usage that can be leveraged by individuals and their research groups, academic institutions, NSF projects, and XSEDE to integrate a range of cloud services into their research computing environments, with the ability to implement allocation, accounting, reporting, and billing all from a single Web-based dashboard.
Cloud computing platforms such as Amazon Web Services, Microsoft Azure, Cornell Red Cloud, and the SDSC Storage Cloud have emerged as important research computing resources. These resources are already popular in research and education due to their instant (on-demand) access, seemingly infinite scalability, ease of configuration for widely varying needs, and ability to offer higher-level, easier-to-use services to less-than-expert users. However, as usage of Cloud resources by the NSF community continues to grow, challenging issues of resource allocation, accounting, reporting and billing emerge. Individual users struggle to track usage across different providers and by different members of a research group. Being able to allocate, monitor and track usage from a user, project, departmental, college, university and even NSF community level of granularity is essential from a budget, planning and cost-benefit perspective. But different cloud providers have different accounting systems and often enforce no usage limits beyond possession of a valid credit card. A framework is urgently needed that can simplify Cloud usage for researchers and educators—while also enabling allocation, monitoring, and management by those responsible for strategic and budget planning and reporting. Intellectual merit: In partnership with Amazon, Inc., Eucalyptus Systems, Inc., and MathWorks, Inc., the Globus team from the University of Chicago and the Red Cloud team from Cornell University created a prototype service that can be leveraged by individuals and their research groups, academic institutions, NSF projects, and XSEDE to integrate a range of cloud services into their research computing environments, with the ability to implement allocation, accounting, reporting, and billing—all from a single Web-based dashboard. Operated as a Software as a Service (SaaS) system by the University of Chicago, hosted for robustness on Amazon Infrastructure as a Service (IaaS) resources, this service supports an "allocation currency" that translates across cloud providers, and delegated usage, consolidated billing, and allocations to provide for management and reporting within groups. [VV1] An extensible provider interface makes it easy to integrate new cloud providers, and experimental integration with both academic and commercial cloud providers offering storage, computing, and application software services enable evaluation of the feasibility of the approach. The project contributes to knowledge of the ways in which the NSF community interacts with cloud resources and of the suitability of different mechanisms for streamlining such interactions from the perspectives of accounting and allocations. The project team brings together unequalled expertise in the development and operation of high-quality cloud services, via Cornell’s work with the Red Cloud IaaS and SaaS systems, and Chicago’s work with the Globus SaaS system and Amazon IaaS. Broader impacts: This work demonstrates mechanisms for cloud resource allocation and accounting; for usage monitoring; for billing; and for providing and managing usage limits on Cloud resources that may or may not enforce those limits. In so doing, the project makes this new class of resources more useful and accessible to the NSF community. By providing users with uniform mechanisms for allocating and tracking usage of diverse cloud resources, the project facilitates use by researchers and educators. By bridging the current gap between the user-oriented cloud accounting and billing offered by current cloud providers and the need within projects, institutions, and funding agencies for more sophisticated allocation and accounting mechanisms, the project enables more widespread use of cloud computing by the NSF community. And by integrating with authentication and allocation mechanisms used within campus and the NSF XSEDE cyberinfrastructure, the project contributes to the development of a more integrated and full-featured national cyberinfrastructure. This work provides an important foundation for anyone in the research community who is looking to Cloud computing to provide researchers in their organization with a competitive advantage.
