Society is beginning to witness an explosion in the use of Deep Neural Networks (DNNs), with major impacts on many facets of human life, including health, finances, family life, and entertainment. To train DNNs, practitioners have preferred to use GPUs and, recently, specialized hardware accelerators. Despite constituting the bulk of a data center?s compute resources, general-purpose shared-memory multiprocessors have been regarded as unattractive platforms. In this project, the Principal Investigators (PIs) think that these platforms have high potential. Consequently, this project will develop new techniques to dramatically improve shared-memory multiprocessor performance in training DNNs. Already, shared-memory servers are compelling for several reasons: they can support a high-degree of parallelism, are general-purpose and easy to program, and provide flexible, fine-grain inter-core communication. However, efficiently using shared-memory servers to train DNNs imposes significant challenges. First, fine-grain synchronization is still expensive, and latencies are non-trivial. In addition, when DNN training moves to an environment with multiple users sharing the same physical shared-memory platform in the cloud, privacy and integrity become major concerns.
To overcome these challenges, this project will synergistically address architecture and security issues. On the architecture side, it will augment a highly-parallel shared-memory server with support for synchronization, data movement, data sharing, and DNN sparsity structuring. On the security side, it will investigate how shared-memory servers create novel privacy and integrity threats (for example, leaking the DNN?s sparse structure and forcing incorrect model generation), and how to defend against those threats. The project?s broader impact is to help enable ?neural network training for everyone,? by making a ubiquitous and easy-to-program platform a viable and safe target for running these important, emerging workloads.
