It is well known that the intelligibility of auditory speech perception is enhanced when a listener can view the talker's face. This fact is important for developments in automatic recognition of speech as well for clinical intervention for individuals with hearing impairments. The proposed research addresses the nature of this visual information and what visual information processing takes place during audiovisual speech perception. The experimental strategy is to manipulate image characteristics systematically in order to uncover the necessary and sufficient visual conditions for speech intelligibility and to investigate how eye movement behavior is influenced by these characteristics. One important tool for these purposes is realistic facial animation and two different systems of facial animation are employed in the proposed research. Three main research aims are addressed: 1. The spatial resolution required to perceive visual speech will be assessed. In a series of studies, digital image processing is used to produce image sequences with restricted ranges of spatial frequencies. Differences in the perception of speech in noise with these image sequences will be used to determine how fine grained visual speech information must be. 2. The temporal characteristics of visual speech information will be assessed through studies using temporal filtering and animated talking heads. With the aid of realistic animation, the precise movement of the facial surface and head can be manipulated to test for perceptual sensitivity to movement parameters. 3. Eye movements will be monitored in studies in which image characteristics and visual speech kinematics are manipulated. The studies will contribute to the understanding of how visual stimulus properties and higher level processes determine gaze patterns and thus influence perception.
| Munhall, K G; ten Hove, M W; Brammer, M et al. (2009) Audiovisual integration of speech in a bistable illusion. Curr Biol 19:735-9 |
| Wilson, Amanda; Wilson, Adam; Ten Hove, Martin W et al. (2008) Loss of Central Vision and Audiovisual Speech Perception. Vis Impair Res 10:23-34 |
| Lucero, Jorge C; Munhall, Kevin G (2008) Analysis of facial motion patterns during speech using a matrix factorization algorithm. J Acoust Soc Am 124:2283-90 |
| Buchan, Julie N; Pare, Martin; Munhall, Kevin G (2008) The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception. Brain Res 1242:162-71 |
| Lucero, Jorge C; Maciel, Susanne T R; Johns, Derek A et al. (2005) Empirical modeling of human face kinematics during speech using motion clustering. J Acoust Soc Am 118:405-9 |
| Munhall, K G; Jones, Jeffery A; Callan, Daniel E et al. (2004) Visual prosody and speech intelligibility: head movement improves auditory speech perception. Psychol Sci 15:133-7 |
| Munhall, K G; Kroos, C; Jozan, G et al. (2004) Spatial frequency requirements for audiovisual speech perception. Percept Psychophys 66:574-83 |
| Munhall, Kevin G; Buchan, Julie N (2004) Something in the way she moves. Trends Cogn Sci 8:51-3 |
| Nicholson, Karen G; Baum, Shari; Kilgour, Andrea et al. (2003) Impaired processing of prosodic and musical patterns after right hemisphere damage. Brain Cogn 52:382-9 |
