Despite significant progress in machine translation, little work has been done to address large-scale translation of natural multi-party human interactions in real-world contexts. Multi-party interactions typically take place in a context where multimodal events such as hand gestures, head gestures, gaze, body movements, etc. are available and utilized by participants to interpret human language and establish common ground. This observation leads to the hypothesis that modeling multimodal environment and interaction discourse can improve automated language interpretation for the purpose of machine translation. Based on this hypothesis, this exploratory research investigates the role of multimodality in machine translation of multi-party conversations using a subset of the AMI meeting corpus. An appropriate level of multimodal representation is identified focusing on user gestures and presentation slides. Correct referents to referring expressions and correct senses to ambiguous words are annotated and used to evaluate whether and how multimodal context improves reference resolution and word sense disambiguation. The enhanced semantic processing utilizing multimodal information is incorporated in statistical machine translation for English-German. The objective is to conduct proof-of-concept experiments exploring the usefulness of multimodal and discourse information for statistical machine translation in real-world contexts. The results from this exploratory study will provide insights on algorithms and systems for automatic extraction of multimodal information, language interpretation using multimodal information, and incorporation of this information into statistical machine translation. The annotated data will be made available to the research community to facilitate the development of translation technology that produces more natural, human-like translations in real-world interactive situations.
