This revised proposal describes a project to systematically investigate the facial components, combinations of components, and interactions of components that constitute facial expressions (nonmanual markers) in the grammar of American Sign Language (ASL). Some of these components have already been shown to differ in significant ways from those used by the general hearing population. They may carry semantic, prosodic, pragmatic, and syntactic information that may not be provided by the manual signing itself. We will compile an inventory of facial articulations, construct a database of video images of these in isolation and in context, and use these data and innovative computational tools to construct a model of facial behavior in ASL. To successfully accomplish this, we propose an innovative integrated linguistic and computational approach to the study of nonmanuals. Our goal in this project is to construct an initial phonological model of ASL nonmanuals. We have targeted a relevant set of facial features and have identified 4 experiments to obtain appropriate information on each of them. A necessary step in preparation for these experiments is to develop computer vision and pattern recognition algorithms that automatically extract these facial features from a large quantity of videos. These algorithms will be capable of processing data more accurately and efficiently than can be done by hand. Finally, by comparing these results with those obtained from native ASL signers in a series of perceptual studies, we can determine what further modifications are still needed. The study of facial expressions in ASL has very practical applications to several areas affecting the lives of Deaf individuals. The absence of clear information on the facial components makes teaching them to individuals trying to learn ASL, such as parents, deaf children, future teachers and interpreters, a pedagogical nightmare. Another important practical application is the development of systems that automatically recognize ASL. Such a system is not feasible without the ability to handle ASL nonmanuals, which carry grammatical information. ? ?

National Institute of Health (NIH)
National Institute on Deafness and Other Communication Disorders (NIDCD)
Research Project (R01)
Project #
Application #
Study Section
Biobehavioral and Behavioral Processes 3 (BBBP)
Program Officer
Cooper, Judith
Project Start
Project End
Budget Start
Budget End
Support Year
Fiscal Year
Total Cost
Indirect Cost
Purdue University
Other Health Professions
Schools of Arts and Sciences
West Lafayette
United States
Zip Code
Malaia, Evie; Wilbur, Ronnie B; Weber-Fox, Christine (2012) Effects of verbal event structure on online thematic role assignment. J Psycholinguist Res 41:323-45
Malaia, Evie; Ranaweera, Ruwan; Wilbur, Ronnie B et al. (2012) Event segmentation in a visual language: neural bases of processing American Sign Language predicates. Neuroimage 59:4094-101
Malaia, Evie; Wilbur, Ronnie B (2012) Kinematic signatures of telic and atelic events in ASL predicates. Lang Speech 55:407-21
You, Di; Hamsici, Onur C; Martinez, Aleix M (2011) Kernel optimization in discriminant analysis. IEEE Trans Pattern Anal Mach Intell 33:631-8
Gotardo, Paulo F U; Martinez, Aleix M (2011) Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion. IEEE Trans Pattern Anal Mach Intell 33:2051-65
Fortuna, Jeff; Martinez, Aleix M (2010) Rigid Structure from Motion from a Blind Source Separation Perspective. Int J Comput Vis 88:404-424
Ding, Liya; Martinez, Aleix M (2010) Features versus context: An approach for precise and detailed detection and delineation of faces and facial features. IEEE Trans Pattern Anal Mach Intell 32:2022-38
Malaia, Evie; Wilbur, Ronnie B (2010) Early acquisition of sign language What neuroimaging data tell us. Sign Lang Linguist 13:183-199
Malaia, Evie; Wilbur, Ronnie (2010) Sign Languages: Contribution to Neurolinguistics from Cross-Modal Research. Lingua 120:2704-2706
Wilbur, Ronnie B (2009) Effects of varying rate of signing on ASL manual signs and nonmanual markers. Lang Speech 52:245-85

Showing the most recent 10 out of 23 publications