The proposed research involves the contribution of visible information in face-to-face communication and how it is combined with auditory information in bimodal speech perception. The experimental research methodology utilizes a strength-inference strategy of hypothesis, independent manipulations of multiple sources of information, and the testing of mathematical models against the results of individual participants. Synthetic speech will allow the auditory and visual signals to be manipulated directly, an experimental feature central to the study of psychophysicals and perception. In addition, expanded factorial designs are used to provide the moist powerful test of quantitative models of perceptual recognition. Expanded factorial designs are used to study how auditory speech and visual speech are processed alone and in combination, and under different degrees of ambiguity. Experiments are proposed to clarify the classic McGurk effect , to assess the contribution of segment frequency in the language and the psychophysical properties of the auditory and visual speech, and to contrast the influence of visible speech with written text in terms of how it is integrated with auditory speech. Experiments are also proposed to test whether previous results and theoretical conclusions based on syllable perception extend to meaningful items, such as words and sentences. Experiments will evaluate the integration of paralinguistic information in bimodal speech perception and the relative influence on dynamic and static sources of visible information in speechreading and bimodal speech perception. To further substantiate the model testing, Bayesian selection as well as RMSD goodness-of-fit criteria will be used in the evaluation of extant models. Many communication environments involve a noisy auditory channel, which degrades speech perception and recognition. Having available speech from the talker's face improves intelligibility in these situations. Visible speech also supplements other (degraded) sources of information for persons with hearing loss. The use of visible speech and its combination with auditory speech is therefore critical for improving universal access to spoken language. It has potential to improve the quality of speech of persons with perception and production deficits, 2) enhance learning and communication, 3) provide remedial training for poor readers, and 4) facilitate human-machine interactions.
Showing the most recent 10 out of 17 publications
