The project creates robust, widely-deployable and cost-effective technologies for supporting cross-lingual spoken interaction between people who do not share a common language. The target application supports communication between healthcare personnel who speak English only and patients with limited-English proficiency. The state-of-the-art technologies that enable such cross-lingual interactions are characterized by a pipelined architecture of speech recognition, machine translation and speech synthesis, that largely ignore the rich information present in spoken language beyond those conveyed by words. They also do not take advantage of the humans in the loop for collaboratively managing the interaction. Overcoming these limitations requires improving robust intelligence at all levels ? signal, system, and human ? and set the research goals for this project.
The project?s intellectual merit comes from the unique combination of theoretical, computational model-ing and empirical elements: The theoretical framework is centered on notions of social co-presence to de-velop new models for translation-mediated communication. The computational modeling focuses on capturing prosody, dialog and user state from spoken language for enriching the technology components. The empirical work relies on a participatory approach to iterative design and evaluation of the system, working directly with the stakeholders.
The broader impact can be seen in the potential for facilitating multilingual efforts ranging from disaster relief and global business operations to servicing diverse immigrant populations notably in health care. The effort brings together engineers, linguists, human communication experts, and medical professionals to tackle a broad range of problems, and offers integrated interdisciplinary research training and mentor-ing.
