The PI hypothesizes that communication between the deaf and hearing communities can be augmented through the use of a mobile one-way sign language translator. His preliminary prototype of such a translator, dubbed Telesign, uses a body-worn computer, a speaker in the brim of the hat, a head-up, hands-free micro-display for user feedback and editing, and a variety of sensors (e.g., a camera embedded in the user's hat, wireless accelerometers worn in wristbands, and the Acceleglove accelerometer/potentiometer system). His first step in this project will be to compare different methods of communication between a deaf and hearing (non-signing) individual - including writing (current practice), typing on mobile devices, a gesture driven sign-by-sign one-way translator, and a phrase-level one-way translator - both to provide a measure for the performance obtained with existing communication technologies and to establish the baseline for future evaluation of Telesign. These methods will be evaluated by speed (propositions conveyed per second and time to task completion), perceived workload (NASA Task Load Index), and clarity of English generation. The PI will then significantly extend the Telesign prototype, by bringing speech recognition techniques to bear on sign language recognition by adapting his Georgia Tech Gesture Toolkit (GT2K), a publicly available hidden Markov model (HMM) toolkit for gesture recognition, for these tasks. The PI will also research sensing techniques, such as visual tracking of the hands in a mobile environment, in order to create reasonable prototypes for sign translation testing over an extended time. Finally, the PI will define research-grade tasks that can be used to evaluate the performance of computer enhanced communication aids for the deaf (both existing techniques and the new improved Telesign system).
Broader Impacts: Everyday communication with the hearing community poses a major challenge to the millions of people with hearing loss. Most hearing people do not know sign language, and know very little about deafness in general. This project attempts to lower the communication barrier between the deaf and hearing populations, by researching aids for language production. However, everyday use of a successful communication aid may also help a signer learn more about the structure of written and spoken English, in much the same way that everyday interactions help students learn a foreign language. This research couples Georgia Tech students with consultants and subjects from the Atlanta Area School for the Deaf (a K-12 state school), exposing some of the nation's top student engineers and scientists to the deaf community, and vice versa. Deaf academics and native signers from the deaf community will be recruited actively as members or the research team, which will also help disseminate the research as it becomes viable.
