Irregular applications are increasing in importance to the computing industry. For a long time they have underpinned computations of interest to the national security arm of the federal government. But now, they underpin such questions as ad placement in social networks, and analysis of complex data-sets in medicine and science. For example, developing custom drug therapies involves analysis of how cellular pathways, known drug/patient outcomes, and a particular patient's DNA interact. These complex big-data problems require new computational models in order to execute effectively on commodity hardware. The defining characteristic of these applications is poor locality and massive available parallelism. The key idea is use the available concurrency to tolerate memory latency, instead of relying on locality. Using a highly optimized runtime system, tuned for use on commodity processors and networking hardware it has been shown that scalable performance on these applications can exceed custom supercomputer-class hardware.
The research currently being undertaken is to continue exploring and developing this latency tolerant scale-out runtime system. The proposed research directions include exploring ways to mitigate latency for disk (SSD) access, in order to tackle petabyte-scale problems on small clusters of commodity hardware. In addition the research will examine programmable router hardware in order to scale network aggregation into the thousand-node-plus cluster range. Finally the research effort will focus on enhanced language semantics and compiler techniques that make it easier to implement the types of analyses and graph algorithms of interest to the scientific, business, medical, and government communities.
