Normal vision is not static: time is a key dimension of the natural world we see. The eventual understanding of biological vision requires understanding the neural mechanisms used to recognize objects and actions over time. Thus the focus of the proposed research is to study how the primate visual system recognizes objects and actions in time sequences of images. A meta-goal of this project is to exploit the synergies between computational approaches and physiological experiments to lead to a better understanding of brain function and at the same time to develop better computer vision algorithms. Object recognition in time sequences of images presents a significant challenge for recognition systems, because it requires both selectivity to shape and invariance to changes of appearance in time.. This project will extend an existing computational model of the ventral stream by adding temporal dynamics in its model neurons and the ability to process video sequences. It will also expand a working model of the dorsal stream to understand the relative roles that it and the ventral stream play in dynamic visual recognition. At the same time, recordings from single units, and multiple single units, from high level visual areas including IT and regions of the STS will be made in order to characterize the tuning of single neurons to the shape dynamics of specific image sequences. By combining modeling and physiology, this work will search for a computational explanation for how the higher areas of the visual cortex recognize objects and actions over time and how they can learn. This integrative effort, which is focused on processing of dynamic perceptual information, can have a significant and direct impact on current theories of autism, dyslexia, and effects of stroke, in addition to directly guiding modeling and engineering efforts in computer vision. The proposed research is tightly coupled to education and teaching, and resources used in the research, including databases of videos, visual stimuli, the modeling software and the experimental data will be made available to the broad scientific community. Information on the project and its progress will be available at http://cbcl.mit.edu/projects/NSF-CRCNS/index.html
