We are confronted with richer, more detailed and more forms of data than ever before. Eigen-analysis based techniques constitute a powerful class of algorithms for discovering statistical signatures in mountains of data. Such techniques are used widely for the detection, estimation and classification of weak signals in a variety of applications that span the breadth of science and engineering such as gene microarray analysis, and climate science among others. This research will uncover the fundamental limits of these techniques and develop new techniques that can tease out weaker signals from large, noisy datasets. The results of this research will be disseminated broadly to advance relevant technology and clarify both advantages and limitations of eigen-analysis based dimensionality reduction.
Eigen-analysis based dimensionality reduction techniques are ubiquitous in statistical signal processing and machine learning applications. Their popularity is due to a sound theoretical justification, near-optimal computational complexity and strong performance guarantees in regimes where the techniques are known to work well. This research will provide a deeper understanding of when eigen-analysis based dimensionality reduction techniques fail so that these limitations can be potentially overcome. The project focuses on high-dimensional, noisy settings and uses random matrix theory as an enabling mathematical tool in this endeavor. The informational limits developed will provide a principled way of comparing eigen-analysis based algorithms with other techniques, and allow for an objective quantification of any performance losses due to fast implementations, measured in terms of how weak the signal or statistical signature is that can be identified. The research is interdisciplinary - powerful, new tools from random matrix theory are used to prove theorems and establish sharp, asymptotic performance bounds.
