This project is pursuing a novel strategy for video segmentation based on the decomposition of a video into multiple overlapping segments of pixels, and the subsequent composition of these segments into hypotheses about the existence of objects within the video. Given an input video, this approach produces a set of spatio-temporal pixel regions as its output, where the set of output regions has a high degree of overlap with the objects that are present in the video. The project further develops methods for semantic segmentation, occlusion analysis, and activity recognition which can exploit a segment-based video representation. The basis for the approach is a statistical framework known as composite likelihood, which implicitly models the joint distribution of a random vector through distributions of low-dimensional statistics on overlapping subsets of variables. This statistical model is ideally-suited to describing video objects as a collection of multiple overlapping segments. Using this framework, methods are being developed to track overlapping segments within a video and generate object hypotheses. Additional efforts are aimed at improving the computational efficiency of the approach in order to address applications in on-line video analysis.
The resulting algorithms yield improved performance in video object segmentation and tracking, and provide new approaches to content-based video categorization and retrieval, for unstructured video collections such as those found on YouTube. The project is producing a novel publicly-available dataset containing fine-grained ground truth video object segmentations, in order to facilitate research activities in video analysis. The project is integrated with education and outreaches high school students to research in STEM.
