In the foreseen future a mix of changes in technologies, user and application requirements and the business model of delivering computing capacity will continue to pose new challenges to the effectiveness of high throughput computing (HTC) technologies. To address these challenges, this ongoing research and development effort will devise new policy-driven capabilities to increase throughput within a defined budget by effectively managing extremely large workloads of homogenous jobs running on homogenous machines provisioned by cloud services. These capabilities will be augmented with effective schedulers for servers that have multiple cores of execution, many disks, perhaps several GPUs each with different capabilities, and multiple networking interfaces. New communities will be introduced to the power of HTC through customized, easy to deploy and secure software. Novel tools for profiling the requirements and dependencies of scientific applications will expand the reach of distributed computing infrastructures that leverage advanced networks to cross institutional and national boundaries.
The advances in HTC technologies will be delivered through the widely adopted HTCondor software tools. More than 150 domestic universities, a growing number of national and international science communities and a wide spectrum of commercial organizations employ HTCondor to improve the throughput of their compute and data intensive applications. This project will sustain a software engineering process that enables translational work to occur in a transitional manner, building upon the previous generation of software while simultaneously continuing to offer and support dependable software that is suitable to handle the ever growing amounts of experimental and simulated scientific data.
