This project is concerned with the study and development of seeing machines capable of recognizing actions in dynamically changing environments with a minimal level of supervision. The emphasis will be on the recognition of face, hand and arm gestures, human gaits, changes in individual behaviors in a crowd (e.g., a person in the middle of a walking crowd starts running), and changes in crowd behavior (e.g., a group of individuals in a walking crowd suddenly start running) occurring in a dynamically changing environment due to camera motion, dynamic backgrounds (e.g., water, fog, fire, smoke, steam) and multiple moving objects and people. The intellectual merit of the proposed research will be a unifying theoretical framework for the recognition of human and crowd activities that combines geometry, dynamics and clustering. The recognition task will be viewed as the inference of a mixture of dynamical models (e.g., rigid motions, non-rigid motions, linear dynamical models) exhibiting abrupt changes both in space (due to multiple activities occurring in a single frame) and in time (due to multiple activities occurring over a period of time). The development of this framework will require significant advances on dynamic scene reconstruction, spatio-temporal video analysis, clustering on geometric spaces, and kernels on dynamical systems. The broad impact of the proposed research includes applications in computer vision, machine learning, control theory, robotics, and biomedical engineering. Techniques for recognition of human and crowd activities are directly applicable to surveillance and security. Techniques for dynamic scene reconstruction and spatio-temporal modeling are useful in traffic monitoring, sports coverage/broadcast, human-computer interaction, image retrieval and search, video motion capture, and image-based rendering. Techniques for clustering on geometric spaces can be applied to identification of hybrid systems in control theory, reconnaissance and mapping problems in multiple robot systems, and modeling of gene expression data and biological networks in biomedical engineering.
