This EAGER project explores the use of social cameras to reconstruct and understand social activities in the wild. Social cameras are an emerging phenomenon, producing video captures of social activity from the point of view of members of the social group itself. They are proliferating at an unprecedented rate, as smartphones, camcorders, and recently wearable cameras, become broadly adopted around the world. Users naturally direct social cameras at areas of activity they consider significant, by turning their heads towards them (with wearable cameras) or by pointing their smartphone cameras at them. The core scientific contribution of this work is the joint analysis of both the 3D motion of social cameras (that encodes group attention) and the 3D motion in the scene (that encodes social activity) towards understanding the social interactions in a scene. A number of internal models (such as maximizing rigidity or minimizing effort) for event reconstruction are being investigated to address the ill-posed inverse problems involved.
This research is establishing a new area of visual analysis by providing the requisite framework for social activity understanding in 3D rather than in 2D. The ability to analyze social videos in 3D space and time provides useful tools for almost any activity that involves social groups working together, such as citizen journalism, search-and-rescue team coordination, or collaborative assembly teams. The project is integrated with education through teaching and student training, and outreaches industry through collaborations.
