We live in the age of big data. In many problem domains such as data-mining, machine learning, scientific computing, and the study of social networks, the data deals with relationships between pairs of entities, and is represented by a data structure called a graph. Graphs of interest today may have hundreds of billions of entities, and trillions of relationships between these entities. Large-scale graph processing is typically done in data-centers which are huge clusters of power hungry computers. The proposed Mongo Graph Machine (MGM) project will explore a different solution known as out-of-core processing. In this system, graphs will be stored in flash memory, which is much cheaper, denser and cooler than DRAMs, and processed using a combination of specialized circuits called FPGAs in tandem with a conventional processor. A programming system will be developed to hide this complexity from the end-user. The resulting system will be small enough to fit under a desk and dramatically more energy-efficient while providing powerful graph processing capability.
The MGM project will address the problem of storing and processing extreme-scale graphs by using in-storage acceleration based on NAND flash chips with an attached FPGA. A single machine can accommodate 1 TB to 16 TBs of flash memory using current NAND technology. This configuration provides the flash storage necessary to store very large graphs and the computational power necessary to saturate the bandwidth of the flash. To address the programming problem for this architecture, the project will develop compiler technology and FPGA accelerators that will permit developers to write applications in the high-level programming model, leaving it to the system to exploit parallelism and optimize memory accesses for the access characteristics of flash storage. The software system will be based on the Galois system, which has been shown to scale to hundreds of processors on large shared-memory machines.
