Machine learning is having a tremendous impact on our society and economy, but it depends critically on the ability to efficiently fit a model to data. The technique of automatic differentiation takes software code written to build such a model and automatically performs the calculus derivations necessary to fit it to data. Automatic differentiation tools have been at the heart of the resurgence of neural networks for tackling problems ranging from drug discovery to self-driving cars. This project revisits core assumptions in the way that automatic differentiation works, and identifies new ways that it can take advantage of randomness to find better machine learning models, faster. This research will lead to new tools that expand the frontier of what machine learning systems are possible.
The project will develop new techniques for automatic differentiation when it will be used as part of a stochastic optimization procedure, as is commonly done in training deep neural networks. Rather than exact Jacobian accumulation on the linearized computational graph, this project proposes techniques for selecting random subgraphs such that the Jacobian is preserved in expectation but much less memory and computation is required. Beyond randomization of the central Jacobian accumulation problem, the project will also explore how randomization can enable new approaches to implicit differentiation as used in PDE-constrained optimization and related problems. Additionally, the project will develop techniques for finding good, approximate near-optimal dynamic programming schedules for the linearized computational graph.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
