This project explores methodologies for forecasting long-term human group activity from videos. Forecasting future activities in real-world videos is an emerging computer vision problem with important applications in visual surveillance for security. This project systematically and rigorously formulates long-term group activity forecasting problem as causalities of visual entities, and designs visual intelligence systems using machine learning and data mining methods. The research considers multiple visual identities of different types, models their interactions, and mines the sequential causalities between these visual entities in videos. These causalities are used for forecasting future group and individual activities. The project creates new mathematical models for describing and simplifying understanding statistical properties of human activity videos. The project leads to important and timely technology that can help to design future video analysis systems with optimal performance in understanding and searching activities from videos. The project tightly integrates research and education activities for the purpose of providing young researchers with project-based learning opportunities in an interdisciplinary environment that offers exceptional professional and personal growth opportunities.
This research discovers complex causality patterns between visual entities from noisy visual data, in order to gain rich and useful knowledge for the forecasting of future visual activities. This essentially bridges the gap between human understandable visual semantics and high-dimensional noisy visual data. The research enables to efficiently capture interactions between multiple visual entities and their temporal causalities, and provides rich knowledge for guiding long-term group activity forecasting. The project also explores several innovative ways to leverage rich sequential context and builds progress level-invariant features. This naturally enriches feature representations from temporally partially observed data, and allows building more time efficient activity prediction machines. Moreover, the project develops an effective forecasting model that can elegantly utilize causalities mined from visual data for long-term forecasting. The developed technologies can lead to new intelligent systems.
