To optimally estimate a property of the environment such as object size, location or orientation, one should use all available sensory information and combine it with prior information, i.e., a probability distribution across possible world states, reflecting knowledge of scenes one is likely to encounter. Sensory input typically arises from multiple sensory modalities, and is uncertain due to physical and neural noise. How are these sources of information combined? An ideal observer will combine all sources of information, taking into account the relia- bility of each source. In addition, such an observer needs to consider alternative causes of discrepancies be- tween sources of sensory information. Do two sources disagree so much that one should conclude they derive from different objects, and therefore have separate causes in the environment? Or, does a discrepancy indi- cate that one or both sources of information (e.g., sense modalities) have become uncalibrated? Many studies define ?optimal? cue integration as maximizing the reliability of the combined-cue estimate, which is generally consistent with human behavior. Do observers have access to the resulting reliability estimate to determine one's confidence in this estimate, perhaps to inform subsequent behavior? What computation does the brain use to solve these problems and how are these computations implemented? We propose research aimed to answer these questions. In our first aim we propose to develop biologically realistic models of how such computations are implemented, i.e., testable neural-network models of optimal behavior for sensory estimation, causal inference, recalibration and confidence. Second, we propose a series of experiments in an area that has been little studied in the framework of optimal cue integration: the combina- tion of visual, tactile and proprioceptive inputs for localization. These experiments test whether humans per- form optimal integration and recalibration of multisensory cues and priors under unclear causal structures in scenarios that are more complex than typically studied (i.e., involving dynamics, context effects, etc.) and thus more similar to the real world. These studies are important and innovative on their own. In addition, they will also provide the foundation for Aim 3, in which we will probe the implementation of cue combination, influence of priors, causal inference, recalibration and confidence in the human brain using fMRI. Together, the experi- mental data from Aims 2 & 3 will be used to test the models from Aim 1. These studies will shed light on the way in which multisensory stimuli are encoded to form a coherent percept, the information considered when perceptual decisions are made, and how vision is used to guide us in an ever-changing world. These experi- ments on normal humans will provide a starting point for understanding multisensory perception and perceptu- al adaptation in individuals in which these systems are compromised by conditions that impact sensory input (e.g., amblyopia, AMD, stroke).

Public Health Relevance

The proposed work benefits public health by systematically assessing the behavioral and neural mechanisms for making perceptual decisions using multiple sensory systems and how optimal decisions must take into ac- count prior knowledge, the uncertainty of sensory information, and possible changes in the sensory systems themselves. A variety of medical conditions can impact both the reliability and accuracy of sensory information (e.g., AMD, amblyopia, stroke). The proposed research will improve our understanding of how sensory inputs from multiple modalities are combined so as to optimize a perceptual decision, and thus can serve to help in the design of rehabilitative plans and sensory-substitution devices when sensory input is disrupted (change in bias, gain and/or variability) by disease or other health-related conditions.

Agency
National Institute of Health (NIH)
Institute
National Eye Institute (NEI)
Type
Research Project (R01)
Project #
2R01EY008266-28A1
Application #
9819370
Study Section
Mechanisms of Sensory, Perceptual, and Cognitive Processes Study Section (SPC)
Program Officer
Wiggs, Cheri
Project Start
1989-08-01
Project End
2023-08-31
Budget Start
2019-09-01
Budget End
2020-08-31
Support Year
28
Fiscal Year
2019
Total Cost
Indirect Cost
Name
New York University
Department
Psychology
Type
Schools of Arts and Sciences
DUNS #
041968306
City
New York
State
NY
Country
United States
Zip Code
10012
Aschner, Amir; Solomon, Samuel G; Landy, Michael S et al. (2018) Temporal Contingencies Determine Whether Adaptation Strengthens or Weakens Normalization. J Neurosci 38:10129-10142
Protonotarios, Emmanouil D; Griffin, Lewis D; Johnston, Alan et al. (2018) A spatial frequency spectral peakedness model predicts discrimination performance of regularity in dot patterns. Vision Res 149:102-114
Locke, Shannon M; Landy, Michael S (2017) Temporal causal inference with stochastic audiovisual sequences. PLoS One 12:e0183776
Rizzo, John-Ross; Hosseini, Maryam; Wong, Eric A et al. (2017) The Intersection between Ocular and Manual Motor Control: Eye-Hand Coordination in Acquired Brain Injury. Front Neurol 8:227
Norton, Elyse H; Fleming, Stephen M; Daw, Nathaniel D et al. (2017) Suboptimal Criterion Learning in Static and Dynamic Environments. PLoS Comput Biol 13:e1005304
Rizzo, John-Ross; Fung, James K; Hosseini, Maryam et al. (2017) Eye Control Deficits Coupled to Hand Control Deficits: Eye-Hand Incoordination in Chronic Cerebral Injury. Front Neurol 8:330
Rizzo, John-Ross; Hudson, Todd E; Abdou, Andrew et al. (2017) Disrupted Saccade Control in Chronic Cerebral Injury: Upper Motor Neuron-Like Disinhibition in the Ocular Motor System. Front Neurol 8:12
Sun, Peng; Landy, Michael S (2016) A Two-Stage Process Model of Sensory Discrimination: An Alternative to Drift-Diffusion. J Neurosci 36:11259-11274
Westrick, Zachary M; Heeger, David J; Landy, Michael S (2016) Pattern Adaptation and Normalization Reweighting. J Neurosci 36:9805-16
Hudson, Todd E; Landy, Michael S (2016) Sinusoidal error perturbation reveals multiple coordinate systems for sensorymotor adaptation. Vision Res 119:82-98

Showing the most recent 10 out of 113 publications