It is natural that events of interest in observed scenes manifest themselves across multiple sensing modalities - vision, hearing, smell, etc. The remarkable perceptions of audio-video signals by natural systems, such as humans, also points to superiority of inferences drawn across modalities. It, therefore, seems natural to enhance performance of automated systems by using joint, cross-modal statistical inferences. However, the detection, organization, and understanding of cues and events in real-world scenarios are difficult tasks. This project seeks to develop a pattern-theoretic framework for achieving these goals. The main research items are: (1) development of mathematical quantities to represent audio and visual events and their spatiotemporal relations, (2) use domain-specific ontologies to impose semantic structure and to incorporate prior knowledge, and (3) derive algorithms for Bayesian inferences using efficient adaptations of Markov Chain Monte Carlo sampling.
The use of pattern theory allows bridging of gaps between raw signals and high-level, domain-dependent semantics, and helps discovers large groups of audio-visual events likely to represent the same underlying event. This effort combines ideas from perceptual organization in computer vision, computational analysis of auditory signals, pattern theory, and prior developments in ontological structures. The methods developed here are applicable to many scenarios that deploy audio and video sensors, including problems of audio annotations of videos, speaker tracking in teleconferencing, and separation of multiple objects in remote surveillance. Broader impact activities involve the development of teaching modules, innovation and entrepreneurial training of the students, and communication of the findings to the community.
