Speech perception is one of the most important cognitive functions of the brain. Humans use both the auditory information available in the heard speech and the visual information available from viewing the speaker's face to understand speech. Functional magnetic resonance imaging (fMRI) is the most popular method for examining human brain function, but suffers from the critical limitation of poor temporal resolution: because it measures hemodynamic changes, it has only ~sec resolution, rather than the ~ms timescale of neuronal activity. This renders fMRI a poor choice for examining dynamic neuronal processing. Instead, we propose to use electrocorticography (eCog), in which electrodes implanted in human subjects for the treatment of epilepsy are used to directly measure neuronal activity in the human brain. The major innovation of this proposal will be to use eCog to measure the dynamic neural processes underlying audiovisual speech perception. The central hypothesis is that rapid communication occurs between the human superior temporal sulcus (STS) and sensory areas through neural oscillations, specifically frequency-specific interactions between the multisensory cortex in the STS and earlier sensory areas in the auditory and visual cortex. As a measure of functional connectivity between areas in the speech network, we will measure trial-by-trial correlations in gamma-band power between brain areas. We assume that high correlations in gamma- band power reflect information transfer and a functional connection between the areas. Neuronal oscillations, particularly in the gamma range (~ 30 - 200 Hz), have been found to reflect neuronal spiking activity and several studies suggest that neuronal oscillations might be an important mechanism of information transfer in the brain that modulates spiking activity. We will test the hypothesis that gamma-band activity in the STS correlates with gamma-band activity in auditory association areas during multisensory speech perception. In ordinary speech, the auditory speech signal is much more informative than the visual speech signal. Therefore, we hypothesize that for clear audiovisual speech, the correlation in gamma oscillations between auditory cortex and STS is stronger than the correlation in gamma oscillations between visual cortex and STS. Conversely, we expect a stronger correlation between gamma-band activity in the visual cortex and the STS when the auditory speech signal is noisy. The proposed research will make use of direct neural recordings from the human brain to investigate whether the information content of the different speech signals affects the communication between early sensory areas and multisensory associative areas. These findings will help us understand the neuronal dynamics involved in speech perception and devise new therapies to help treat veterans with hearing loss and other disabilities.
Many U.S. veterans with acquired hearing loss suffer from a disabling loss in the ability to process and understand speech. Multisensory integration through the use of the visual information in the talker's face can help compensate for these auditory deficits. However, little i known about the neural mechanisms for multisensory speech perception. We propose to remedy this knowledge gap using electrocorticography, which is ideally suited to investigate the dynamics of speech perception at high temporal (millisecond) and spatial (millimeter) resolution. A better understand of the neural mechanisms for multisensory speech perception will allow us to design therapies to promote multisensory integration in patients with sensory dysfunction, especially hearing loss, and other language impairments, such as stroke.
