The perception of speech is typically considered a solely auditory phenomenon. However, research in this lab and others shows that the perception of speech does not lie solely in the province of audition. Visible speech information (the sight of a talker's mouth movements) strongly influences the perception of speech in adult listeners. Moreover, our work has shown that connections between the audible and visible products of articulation are present very early in life; infants only 18 weeks old can relate the auditory consequences of speech to the visible movements that cause it. During the last funding period our studies have focused on the form of the auditory and visual speech information at the point of cross-modal integration. We show that: (a) at the point of auditory-visual conflux speech information has not been phonetically classified; it is still in a precategorical form, (b) very discrepant auditory and visual speech information can be combined, such as when a male's face is combined with a female's voice, and (c) that attentional mechanisms influence auditory-visual speech perception. We also show that in order to link auditory and visual speech information, infants need, on the auditory side, the whole speech signal. The work has led to the formation of a new hypothesis that generates the programmatic series of studies proposed here. The hypothesis is that auditory-visual speech information is mapped on to language-specific stored representations of phonetic units (""""""""prototypes"""""""") that instantiate both the auditory and visual concomitants of speech sounds. In the next funding period, we focus on three themes: (a) auditory-visual prototypes for speech - we will test the notion that speech prototypes include information about articulation that is visually specified, (b) cross-language studies of speech prototypes - we will investigate the influence of specific language experience on the representation of speech units by studying adults from two countries (America and Japan), and (c) development of speech prototypes in infants and young children - we will explore the formation and nature of speech representations in ontogeny. The proposed research has implications for theories of speech perception, more general theories of cognitive categorization, models of human development, and the clinical concerns of deaf and blind populations.
