Research has revealed much about the mechanisms of the visual system. However, perceptual experience is usually multimodal, with close relationships between visual and auditory modalities. Auditory signals influence neural activation throughout the visual pathways including in the mid brain and primary visual cortex. It is therefore important to extend the rigorous theories of vision to integrate multimodal contexts. Prior research on auditory-visual interactions has primarily focused on perception of space, timing, duration, motion, and speech, whereas recent research has demonstrated auditory-visual interactions in the perception of objects and faces. The goal of the proposed research is to fill the gap in our understanding of auditory-visual interactions at the level of visual feature processing. We will characterize which acoustic patterns uniquely interact with processing of low-level (e.g., spatial frequency), intermediate-level (e.g., material texture and 2D shape), and high-level (e.g., common objects, words, face identity, and facial expressions) visual features. To understand these interactions, we will combine psychophysics and computational modeling (AIM 1) to determine how associated sounds influence basic mechanisms of visual feature processing, including those that control image visibility (front-end signal-to-noise ratio and sampling efficiency), those that control signal competition for visual awareness, and those that control the strength and reliability of neural population coding of visual features in the presence of between- and within-receptive-field signal interactions. The results will provide an integrative understanding of how sounds influence visual signals, sampling, competition, and coding, for the processing of low-, intermediate-, and high-level visual features. The proposed research will also allow development of cross- modal methods for assisting visual perception by enhancing specific spatial scales, materials, shapes, objects, and facial expressions. For example, our preliminary results suggest that sounds can be used to boost and tune the perception of facial expressions, and to direct attention to specific spatial frequencies. In the translational aim (AIM 2), we will systematically investigate how sounds can be used to aid visual perception, for example, to direct attention to an object, material, word, or facial expression in search, facilitate object recognition via directing attention to diagnostic spatial-frequency components, and enrich scene understanding via directing attention to multiple spatial scales. Because feature-specific auditory signals are readily presented over headphones, the proposed research may provide a means to, for example, counter biased perception (e.g., perceiving facial expressions as negative due to social anxiety), and to direct attention to specific objects and spatial scales (e.g., details versus gist) for individuals with visual challenges such as low vision, strokes affecting vision, or with attention disorders. Thus, the proposed research will not only systematically integrate auditory influences into the current models of visual feature processing, but it may also provide a means to aid visual processing by using auditory signals.

Public Health Relevance

Visual signals are often accompanied by related auditory signals and therefore understanding auditory influences on visual processes is important for understanding how the visual system works in realistic contexts. Recent results suggest that auditory-visual interactions involve the perception of objects;for example, playing a characteristic sound of a target object (e.g., meow for a cat) facilitates visual search even when the sound is spatially uninformative. Understanding the nature of these interactions may provide new insights for alleviating vision problems such as age-related visual impairments by using auditory stimulation.

Agency
National Institute of Health (NIH)
Institute
National Eye Institute (NEI)
Type
Research Project (R01)
Project #
5R01EY021184-03
Application #
8526466
Study Section
Cognition and Perception Study Section (CP)
Program Officer
Wiggs, Cheri
Project Start
2011-09-01
Project End
2014-08-31
Budget Start
2013-09-01
Budget End
2014-08-31
Support Year
3
Fiscal Year
2013
Total Cost
$355,841
Indirect Cost
$118,341
Name
Northwestern University at Chicago
Department
Psychology
Type
Schools of Arts and Sciences
DUNS #
160079455
City
Evanston
State
IL
Country
United States
Zip Code
60201
Zweig, L Jacob; Suzuki, Satoru; Grabowecky, Marcia (2015) Learned face-voice pairings facilitate visual search. Psychon Bull Rev 22:429-36
Ortega, Laura; Guzman-Martinez, Emmanuel; Grabowecky, Marcia et al. (2014) Audition dominates vision in duration perception irrespective of salience, attention, and temporal discriminability. Atten Percept Psychophys 76:1485-502
Plass, John; Guzman-Martinez, Emmanuel; Ortega, Laura et al. (2014) Lip reading without awareness. Psychol Sci 25:1835-7
Parrott, Stacey; Guzman-Martinez, Emmanuel; Ortega, Laura et al. (2014) Spatial position influences perception of slope from graphs. Perception 43:647-53
List, Alexandra; Iordanescu, Lucica; Grabowecky, Marcia et al. (2014) Haptic guidance of overt visual attention. Atten Percept Psychophys 76:2221-8
Sweeny, Timothy D; Suzuki, Satoru; Grabowecky, Marcia et al. (2013) Detecting and categorizing fleeting emotions in faces. Emotion 13:76-91
List, Alexandra; Grabowecky, Marcia; Suzuki, Satoru (2013) Local and global level-priming occurs for hierarchical stimuli composed of outlined, but not filled-in, elements. J Vis 13:
Sherman, Aleksandra; Grabowecky, Marcia; Suzuki, Satoru (2013) Auditory rhythms are systemically associated with spatial-frequency and density information in visual scenes. Psychon Bull Rev 20:740-6
Xu, Yangqing; Suzuki, Satoru; Franconeri, Steven L (2013) Shifting selection may control apparent motion. Psychol Sci 24:1368-70
Zabelina, Darya L; Guzman-Martinez, Emmanuel; Ortega, Laura et al. (2013) Suppressed semantic information accelerates analytic problem solving. Psychon Bull Rev 20:581-5

Showing the most recent 10 out of 14 publications