This project is pursuing a novel strategy for the analysis of temporal structure in video through the exploitation of statistical tests of temporal causality. The motivation is the need for unsupervised video analysis methods which do not require a pre-defined set of video categories or a large corpus of labeled examples. The starting point is the classical formulation of Granger causality, which provides a principled statistical test for directed influence between two time series. Modifying the classical pair-wise Granger test leads to a method which is suitable for video events, which are represented as multiple point processes. Using this representation, methods are being developed for grouping visual words into sets based on their interaction over time. This results in a novel bottom-up segmentation approach which can identify interactions between visual words without supervision. A further goal is the development of an integrated approach to modeling visual events and identifying causal relations. Additional efforts are aimed at developing novel features constructed from causal relations with the goal of improved performance on categorization and retrieval tasks.
In summary, the project is developing new unsupervised methods for representing and segmenting video based on temporal causal analysis. The resulting algorithms yield improved performance in video retrieval and categorization tasks, and provide new approaches to organizing and searching unstructured content such as YouTube videos. Novel datasets for video segmentation and categorization are being developed along with a library of analysis software to facilitate adoption by the research community.
