Understanding how animals perceive and act upon complex natural environments is one of the most pressing challenges in neuroscience, with applications that have potential to revolutionize not only our understanding of the brain, but also machine vision, artificial intelligence, and robotics. Until now, studying the neural basis of active vision - how visual stimuli give rise to eye movements under diverse task conditions - has largely been restricted to simplified laboratory stimuli, presented to overtrained animals performing stereotypical tasks. With funding from the National Science Foundation, the Canadian Institute of Health Research, and the National Geospatial Intelligence Agency, Dr. Douglas Munoz at Queens University in Canada and Dr. Laurent Itti at the University of Southern California will combine neurophysiology and computational modeling to investigate free viewing in natural environments. Using multi-electrode arrays, this project will record in a deep brain structure, called the superior colliculus (SC). The SC is a layered structure comprising several well-understood neural maps, from purely sensory representations in the superficial layers, to sensorimotor representations linked to the control of eye movements in the deeper layers. The project will start by characterizing responses of neurons in the SC under simple stimulus conditions: When the animal is simply looking at a central fixation cross on a display while small isolated patterns are presented at other visual locations; when the animal searches for an oddball item among an array of distracting items; and when the animal inspects natural images and video clips. The project will extend the investigators' salience map theories and models, and develop a new model of the SC. The complete model will predict, from any image or video clip, which visual locations are more salient, task-relevant, and candidate targets for eye movements.
The project leverages a cross-disciplinary collaboration between a neurophysiology lab (co-PI Douglas P. Munoz) and a computational modeling lab (PI Laurent Itti). This will allow, through the combination of experiments and modeling, the interpretation of an otherwise undecipherable mass of data collected during natural viewing. Conversely, the theories will guide further experiments. Coupling multi-unit recording with modeling during free-viewing of natural videos has never been attempted before, and it is expected that it will lead to new understanding of how percepts map into actions under natural conditions. The project will support undergraduate and graduate students, and post-doctoral researchers, who will benefit from exposure to combined physiological and computational techniques, as will the investigators' teaching. In addition to publications, all theory and algorithm source code will be freely distributed, and data will be available through the CRCNS data sharing web site. This research is hence expected to lead to new and broadly accessible fundamental advances in the understanding of how animals use visual information to guide behavior, and how one could build machines which act in similar ways when faced with the complex natural world.
The overarching goal of this project was to investigate the neural basis of active vision in awake monkeys, through combined multi-unit recording, psychophysics and modeling. The project has three main aims: 1) Characterizing single-unit response properties when the animal is fixating and presented with single isolated stimuli, to specify the elementary building blocks of the system, in terms of spatio-temporal receptive fields, feature tuning properties, contrast response, and adaptation to spatial and temporal stimulus statistics. These building blocks are essential to specify the detailed operation of the models. 2) Characterizing unit responses from multiple sites while the animal is fixating and presented with complex stimuli, to decipher and allow modeling of spatial interactions across distant recording locations, and task-dependent sensory-to-motor transformations across SC layers as recorded by the multiple contacts along each multi-electrode. 3) Characterizing unit responses from multiple sites while the animal is freely viewing complex stimuli under various task conditions, to test generalization of our theories with complex and ecologically-relevant situations. Importantly, this will allow us, for the first time, to sample SC salience maps for direct comparison with model maps. We are pleased to report that success has been achieved on all aims, especially the last one. As we stated in the proposal, this last aim is a direct, make-or-break test for saliency map theories. We found that neural activity in the SC can be significantly predicted by our saliency model, thereby validating the theory and isolating for the first time the neural correlate of the saliency map in the primate brain. Our results have been published in 24 journal articles (including high-impact journals like J Neuroscience, TiNS, Current Biology) and 11 papers in proceedings from international conferences with full paper review (e.g., IEEE CVPR).
