The recent successes in image and video analysis have been largely in the domain of supervised learning. Supervised learning methods assume the availability of extensive amounts of manually annotated/labeled training data, which limits the applicability of existing methods to complex and unseen environments. This has motivated growing interest in developing semi-supervised, and even unsupervised, methods for image and video analysis, i.e., methods that have limited or even no manually annotated data. These methods focus on how to learn visual analysis models with limited labeled data; however, the problem of what to label is far less addressed. If one can identify the optimal subset to label, it is likely that the learning process will be more efficient than randomly choosing representatives that are labeled by a human. This project will focus on mathematically rigorous approaches on how to choose these samples to label.
The project will investigate information theoretic approaches, coupled with an understanding of the inherent structure in visual data, that would help identify samples that require annotation. A well-known idea in information theory is typical sets. The concept of a typical set is based on the intuitive notion that some portions of a dataset carry more information than others. The project will investigate the applicability of this notion of typicality from information theory to select a minimal set of most informative samples, which will be manually labeled, such that visual analysis models (e.g., classifiers) designed on this subset can maximize performance (of the task of interest) on the entire dataset. The inherent structure in visual data will be combined within this information theoretic framework to obtain the optimal set of representatives to label. The approach will be analyzed in the context of various learning paradigms, including active learning, transfer learning, and weakly supervised learning. The project will involve theoretical analysis of the representative selection process, algorithm design, and experimental evaluation on applications in image segmentation, activity recognition, and target re-identification.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
