American Sign Language (ASL) grammar is specified by the manual sign (the hand) and by the nonmanual components (the face). These facial articulations perform significant semantic, prosodic, pragmatic, and syntactic functions. This proposal will systematically study mouth positions in ASL. Our hypothesis is that ASL mouth positions are more extensive than those used in speech. To study this hypothesis, this project is divided into three aims. In our first aim, we hypothesize that mouth positions are fundamental for the understanding of signs produced in context because they are very distinct from signs seen in isolation. To study this we have recently collected a database of ASL sentences and nonmanuals in over 3600 video clips from 20 Deaf native signers. Our experiments will use this database to identify potential mappings from visual to linguistic features. To successfully do this, our second aim is to design a set of shape analysis and discriminant analysis algorithms that can efficiently analyze the large number of frames in these video clips. The goal is to define a linguistically useful model, i.e., the smallest model that contains the main visual features from which further predictions can be made. Then, in our third aim, we will explore the hypothesis that the linguistically distinct mouth positions are also visually distinct. In particular, we will use the algorithms defined in the second aim to determine if distinct visual features are used to define different linguistic categories. This result will show whether linguistically meaningful mouth positions are not only necessary in ASL (as hypothesized in aim 1), but whether they are defined using non-overlapping visual features (as hypothesized in aim 3).
These aims address a critical need. At present, the study of nonmanuals must be carried out manually, that is, the shape and position of each facial feature in each frame must be recorded by hand. Furthermore, to be able to draw conclusive results for the design of a linguistic model, it is necessary to study many video sequences of related sentences as produced by different signers. It has thus proven nearly impossible to continue this research manually. The algorithms designed in the course of this grant will facilitate this analysis of ASL nonmanuals and lead to better teaching materials.
Deafness limits access to information, with consequent effects on academic achievement, personal integration, and life-long financial situation, and also inhibits valuable contributions by Deaf people to the hearing world. The public benefit of our research includes: (1) the goal of a practical and useful device to enhance communication between Deaf and hearing people in a variety of settings;and (2) the removal of a barrier that prevents Deaf individuals from achieving their full potential. An understanding of the non-manuals will also change how ASL is taught, leading to an improvement in the training of teachers of the Deaf, sign language interpreters and instructors, and crucially parents of deaf children.
| Du, Shichuan; Tao, Yong; Martinez, Aleix M (2014) Compound facial expressions of emotion. Proc Natl Acad Sci U S A 111:E1454-62 |
| Benitez-Quiroz, C Fabian; Gökgöz, Kadir; Wilbur, Ronnie B et al. (2014) Discriminant features and temporal structure of nonmanuals in American Sign Language. PLoS One 9:e86268 |
| Benitez-Quiroz, C Fabian; Rivera, Samuel; Gotardo, Paulo F U et al. (2014) Salient and Non-Salient Fiducial Detection using a Probabilistic Graphical Model. Pattern Recognit 47: |
| You, Di; Benitez-Quiroz, Carlos Fabian; Martinez, Aleix M (2014) Multiobjective optimization for model selection in kernel methods in regression. IEEE Trans Neural Netw Learn Syst 25:1879-93 |
| Du, Shichuan; Martinez, Aleix M (2013) Wait, are you sad or angry? Large exposure time differences required for the categorization of facial expressions of emotion. J Vis 13:13 |
| Martinez, Aleix; Du, Shichuan (2012) A Model of the Perception of Facial Expressions of Emotion by Humans: Research Overview and Perspectives. J Mach Learn Res 13:1589-1608 |
| Rivera, Samuel; Martinez, Aleix (2012) Learning Deformable Shape Manifolds. Pattern Recognit 45:1792-1801 |
| Hamsici, Onur C; Gotardo, Paulo F U; Martinez, Aleix M (2012) Learning Spatially-Smooth Mappings in Non-Rigid Structure from Motion. Comput Vis ECCV 7575:260-273 |
| You, Di; Hamsici, Onur C; Martinez, Aleix M (2011) Kernel optimization in discriminant analysis. IEEE Trans Pattern Anal Mach Intell 33:631-8 |
| Gotardo, Paulo F U; Martinez, Aleix M (2011) Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion. IEEE Trans Pattern Anal Mach Intell 33:2051-65 |
Showing the most recent 10 out of 15 publications
