Supervised learning is used to infer an unknown regression function from a set of training data. Applications include time-series prediction, remote sensing, medical image analysis, computer vision, face detection, text categorization, image scene classification, speech recognition, and bioinformatics. Existing learning algorithms have several drawbacks. For example, the set of basis functions usually involves unknown parameters that need to be determined empirically. The optimization problem defined by the learning task depends on a trade-off parameter that requires careful selection. Analytical performance of the existing learning methods is not well studied and the role of the set of basis functions is not yet clear. In addition, currently there is no effective way to select the set of basis functions.
The investigator develops a new framework of sparsity-enforced learning for regression function learning to overcome the drawbacks of existing approaches. This framework includes numerical algorithms for sparse vector estimation, tight bounds for performance analysis, and optimization procedures for dictionary design. A flexible form for the inferred function is a linear combination of basis functions, which are constructed by discretizing the unknown parameters. Algorithms are designed to learn the weight vector by convex sparsity enforcing, hierarchical Bayesian modeling and normalized maximum likelihood principle. The discretization and the sparsity enforcement enable automatic selection of most relevant basis functions with appropriate parameters. The investigator analyzes the performance of the learning framework by deriving tight bounds on the mean-squares error of the learned weights, in particular, the Cramer-Rao bound and Hammersley-Chapman-Rob bins bound. The performance bounds (and their simplifications) are employed to optimally design the overcomplete dictionary. The investigator uses the developed framework to model time series and extract knowledge from images.
Intellectual Merits: In this project we develop statistical methods of data analysis, including both mathematical results and applications to real-world problems. Specifically, we developed methods to extract signals of interest from noisy measurements. We exploited the fact that many signals in nature or man-made systems are sparse, meaning that they are mostly zero and have a limited number of informative components that are non-zero. The contributions of our research are three-fold. First, we designed techniques for implementing methods of learning sparse signal models that have over-complete dictionary of building blocks (kernels) that describe the signals. The over-complete dictionary comprises a large number of kernels, with sampled kernel parameters. To solve this problem we minimized an error cost-function (called l1-regularized) that fits sparse weight vector of the kernels. This minimization automatically selects the kernels with the most suitable parameters to describe the sparse signals. We showed that by utilizing sparsity, we can learn the structure of the signals more efficiently. We introduced a statistical learning procedure using signal thresholding to enhance sparsity, assuming statistical prior information is known. Second, we computed mathematical results of performance analysis to demonstrate that our methods can recover or learn the signals of interest in a stable way. Our performance analysis shows that the new method we proposed provides improved regression accuracy with a small model complexity as measured by the number of non-zero entries of the weight vector. We also developed computational techniques to analyze the performance of various recovery methods of sparse signals, where the sparsity appears in individual or block of signals of interest. We analyzed the performance of matrix completion methods, which originally were used to predict the rating of movie viewers based on partial information on what the viewers rated movies they had watched earlier. Third, we applied our analytical results to several applications. We used a novel approach to accurately estimate positions, velocities of multiple targets using distributed and collocated MIMO radar systems by employing sparse modeling. We also solved a multiple moving-target estimation problem using a collocated MIMO radar system. We considered iterative electromagnetic imaging of metallic targets hidden behind dielectric walls using sparse regularization. Broader Impacts: The completion of this project advanced the knowledge and created methods for mathematical analysis of data in nature and man-made systems in which the signal with information of interest appears only sparsely and at unknown locations in the data. The high efficiency of the proposed methods and their guaranteed performance will be useful to advance current technologies of information retrieval and extraction from noisy data. The derived performance bounds provide computable performance analysis tools, as opposed to alternatives that require extremely high computation effort. The methods developed in this project can be potentially applied to various applications, including sensor signal processing, medical image processing, and other important applications for the society, as we demonstrated in numerical examples. Our project supported and allowed training of Ph.D. students who conducted cutting-edge multidisciplinary research. Under this project, we also created NSF REU program for undergraduate students, and emphasized recruiting of underrepresented groups, including female and minority students to this research project. We prepared both graduate and undergraduate students for joining the work force in high-tech positions.
