Learning complex statistical models from data is intractable for many models of interest. The PIs are studying a new approach to learning from data that formulates learning as a weakly chaotic nonlinear dynamical system. They show that this dynamical system, which they call ?herding?, combines learning and inference into one tractable forward mapping. They study the abstract mathematical properties of this nonlinear mapping, such as the properties of its attractor set and the topological and metric entropy of the mapping. They then relate these to properties of learning systems.
The PIs apply herding systems to a wide range of applications in machine learning. In supervised learning they show that herding suggests a natural extension to the ?voted perceptron algorithm? by including hidden variables. In unsupervised learning, herding is used to train Markov random field models from data. Herding is also extended to Hilbert spaces where it naturally leads to a deterministic sampling algorithm. Due to negative autocorrelations, this ?kernel herding? generates samples that have superior convergence properties than random sampling. They also apply herding to active learning problems.
Herding has the potential to radically transform the way we view learning systems. It connects learning to the vast field of nonlinear dynamical systems and chaos theory. As such the impact on machine learning is significant. Scientific results will be disseminated through journal publications and conference proceedings. The PIs also introduce a new course on learning, chaos and fractals to expose students to the intriguing connections between these fields.
Many processes in our complex world are "chaotic". With this we mean that small changes may have very large influences and detailed features of such a process are very hard to predict. An example is the complex firing pattern of neurons in our brain as we think and learn. Many artificial (engineered) learning systems are designed to behave very different and use randomized (Markov Chain Monte Carlo) algorithms to train complex models from data. These methods are flexible but have an important disadvantage: they converge only slowly because of the noise generated through the randomization. In this project we have studied learning algorithms that make use of deterministic (chaotic) learning dynamics instead of randomized MCMC algorithms. Our main finding is that this is not only possible but also advantageous because the algorithms retain their flexibility but at the same time converge much faster. Our research has brought together two scientific disciplines (mathematics of complex systems and machine learning) and opened new research directions that may lead to crossfertilization. Machine learning has fed mathematics with new problems inspired by learning systems while new mathematics has contributed to a better understanding of these learning algorithms. We have now designed and tested algorithms based on chaotic dynamical systems for classification, forecasting, image analysis, compression, and even playing "GO". We envision that deterministic chaotic dynamics may play an important role in learning complex probabilistic models of "big data".
