Sign languages are complex, abstract linguistic systems, with their own grammars, and their "articulation" involve not just the hands, but the face, shoulder, and arms. In this project the PI and his team will push the state of the art in scalable automated American Sign Language (ASL) recognition formalisms. Presently, there are methods to recognize isolated signs and, to some extent, continuous signs in short sentences from a single signer, mostly using special equipment such as data gloves or magnetic markers or from visual input against plain background and special clothing. The PI seeks to achieve substantial advances in five areas:: (i) recognition against varied backgrounds and different clothing, (ii) use of non-manual aspects such as facial expression and head movement, (iii) recognition across signers, (iv) design of robust, scalable formalisms, and (v) development of large ASL data corpus that exercise variates such as viewpoint, background, time, and signer, to help benchmark progress. To these ends, the PI will (i) develop robust manual and non-manual (face) feature sets for ASL, (ii) construct formalisms to learn sign models from example sentences, (iii) investigate if elemental forms of signs (signemes) can be learned, (iv) employ Bayesian network based indexing schemes to limit the combinatorics of recognition, and (v) explore techniques to incorporate grammar and syntax information into the recognition process.
Broader Impacts: With the gradual shift to speech based I/O devices for human computer interaction, there is great danger that people who rely on sign languages for communication will be deprived access to state of the art technology unless there significant advances in automated recognition of sign language are achieved. Such advances will also enhance the quality of life of persons with disabilities, by facilitating interaction with the general populace in public situations, such as airports and grocery stores. The PI will identify at least one deaf graduate student or a student with communication disorder to participate in the project, to ensure the outcome is relevant and appropriate to the intended user community. The large ASL data corpus collected in this project will be distributed aggressively; this effort will continue to be supported beyond the project end date.
