Since the inception of the Internet, the problem of sending files between users that were too large for most servers has plagued users. Often the simple process of getting the sender and recipient to set up accounts on a common service to allow exchange of their respective identities results delays in actually sending a file, as this information needs to be known before a file exchange can be initiated. Internet2 pursues in this work the creation of a general bulk data transfer service that will allow researchers to stage datasets in the 10MB-1TB size range for short periods of time. An initial data store of 100TB will be provided as a short-term repository, allowing robust implementation to meet the needs of the community as none of the space will be allocated long-term and will be available for re-use quickly. This work focuses on modest functional goals (solving only the problem of exchanging files, rather than their long term storage and sharing) while emphasizing a global reach to work across a large population with no advanced user planning or provisioning.
The intellectual merit of this work lies in its unique solution to a problem affecting a broad scope of research communities, particularly those that deal with large file data transfers, and in the value it will offer through a solution that already enjoys international deployment. The existence of a simple but comprehensive solution that is independent of any community infrastructure will be a critical component of the overall R&E cyberinfrastructure. FileSender, the core software to accomplish this, has evolved into an international open-source community that is gaining significant traction. By providing a service based on FileSender in the U.S., this project will capitalize on and extend the potential for this to become a ubiquitous and common solution throughout the international research and education community.
The goal of this project has been the creation of short-term large data transfer with controlled access through Internet2's InCommon federated service and its proof of concept within the Internet2 community. Through this project, researchers have been able to stagegeneral bulk datasets in the 10MB-1TB size range for short period (no more than two weeks). Using InCommon's basic level of assurance credentials, the project has provided a controlled short-term repository for data up to 11TB, including coverage of capital equipment and operational costs. Over 450 researchers have used the service during the project period, with an average of 46 users at any given time. This usage rate occurred without any active promotion, although many campuses have shared information regarding its availability. The project has provided a domain- and campus-independent mecahnism for transferring large data files between researchers and campuses without formal provisioning of specific services, saving researchers time and money. In addition to providing a cost-effective solution to the common challenge of transferring large data, the service has demonstrated the value of federated authentication through InCommon. In the process, it has provided a short-term solution for data exchange challanges, faciliating research collaboration without requiring advanced user planning or system provisioning. This solution has been offered with modest cost, and has complemented other efforts in shared community file system management and high-speed file sharing. By doing so, it has capitalized on the success of FileSender software and by using InCommon, it has extended the potential this throughout the R&E community as a more commonly adopted solution. This has resulted in significant benefits to a wide range of sponsored R&E activities in different scientific disciplines.
