*** SRI is investigating the properties of human interactions with a graphical terrain model as they would occur in a crisis situation, with a particular emphasis on elucidating the process of reference through natural language and gesture. First, the research team is augmenting existing theories of discourse salience to account for how entities are introduced through the visual display in addition to speech, and extending theories of reference resolution by incorporating mutually constraining effects of gesturing and properties of linguistic referring expressions. Second, the team is studying how discourse structure is affected by the structure of the terrain model and of the task being executed, and how it can be determined through a combination of speech and gestural cues. Third, they are studying how spatial terms are used in uniquely identifying referents in a terrain model, particularly with respect to perspective- relative relational terms. The data used are generated from a set of wizard-of-oz sessions with an existing multimodal route-planning system. The results of the research are demonstrated in proof-of-concept extensions to this system, and are measured by a systematic set of evaluation methods. This research carries out the first step toward the flexible integration of speech, gesture, and visual display that will be required by the interactive, multimodal systems of the future.***
