Deep neural networks have elicited breakthrough successes in machine learning by achieving impressive accuracies on diverse tasks such as facial recognition, object identification, anomaly detection and monitoring assistance on a large scale. However, deep neural networks are not theoretically guaranteed to always perform well, and they could, although rarely, fail in the presence of previously unseen data or small/imperceptible (adversarial) changes to the data. For instance, a home security system using a deep neural network facial recognition algorithm could mistake a stranger wearing pixelated sunglasses for the homeowner; a slight change of the environment, such as a rainy day, could cause a computer vision based autonomous driving vehicle to wrongly recognize a "STOP" sign as an outdoor commercial sign. The existence of such failure cases in widely used machine learning systems today could put our daily lives and even national security at risk. One way to make machine learning systems robust against these failure cases is to design algorithms that are guaranteed to provide an optimal solution, generalize to unseen scenarios and be robust to adversarial changes even if the attacker is given full knowledge of the algorithm. The methods developed via this research will provide theoretical bases that explain "black-box" deep neural networks and provide guarantees over their performance when applied to high-stakes problems. The project will be integrated with graduate and undergraduate education, fostering collaboration between researchers from Computer Science, Applied Math, Physics and Business. Software programs developed via this project will be released as an open-source toolkit, allowing widespread dissemination to researchers and practitioners in a range of fields.
This project will advocate theoretically guaranteed training and understanding of neural networks via techniques from learning theory, nonconvex optimization and consistent latent variable model learning using spectral methods. The investigator's goal is to design compressed neural networks that are theoretically guaranteed to generalize well, fit into Internet of Things devices with memory constraints, and are robust to adversarial examples. Concretely, the technical aims of the project are divided into three thrusts. (1) Guaranteed training of deep nets. The investigator proposes to develop a theoretical justification of why deep residual networks are easier to optimize than non-residual ones when each layer provides a better-than-a-weak-baseline oracle in predicting labels. The investigator plans to use two approaches to guarantee existence and implement the better-than-a-weak-baseline oracle: (a) exploiting theoretically guaranteed training of shallow convolutional neural networks, a.k.a. convolutional dictionary learning, using spectral methods and (b) ensuring escaping from local optima using Homotopy transformations to "sharpen" local optima of network's objective landscape as recent advances in escaping from local optima showed that SGD will not get stuck at sharp local optima with small diameters. (2) Analyzing generalization ability of compressed deep neural networks. The investigator will introduce deep neural network compression using tensorized tensor decomposition, and develop tighter bounds for generalization error, which takes the input distribution and the compressibility of the network into account. (3) Reliable deep neural networks robust to the worst attackers. To provide a universal defense mechanism against the worst possible adversarial examples using a minimax formulation, the investigator proposes to analyze the robustness of nonlinear single-layer neural nets using tensor decomposition method and ultimately design universal defense mechanisms for deep neural nets.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
