Current computing systems do not support human work effectively. They restrict human-computer interaction to one mode at a time (through a keyboard or a mouse) and are designed with an assumption that use will be by individuals (rather than groups), directing (rather than interacting) with the system. To support the ways in which humans work and interact, a new paradigm for computing is required that is multimodal, rather than unimodal, collaborative, rather than personal, and dialogue-enabled, rather than unidirectional. This research will develop principles for, implement, and assess natural, multimodal, multiuser, dialogue-enabled interfaces to geographic information systems (GIS) that make use of large-screen displays and virtual environment technology. These interfaces will support a two-way iterative dialogue between human and machine in which the human participant can use three modalities, speech, gesture, and gaze, in natural ways. Progress toward multimodal human-computer interfaces will require both an understanding of individual modalities and the fusion of information at various levels. The project is concerned, specifically, with the use of computer vision and speech processing as a means of interpreting and integrating information from spoken words, free hand gestures and gaze. It is also concerned with how to enable a human-computer dialogue with an interactive, multi-layered map in the context of a GIS and with how to enable map-mediated dialogue between human collaborators. The work will be bootstrapped through use of an existing test-bed that integrates gesture and speech for simple map queries. The research will investigate how characteristics of tasks, information, and users influence the nature of human-computer and computer-mediated human-human dialogue. In doing so, the research will determine how these factors influence the strategies required to achieve an effective easy-to-use interface that supports information access and productive work by individuals and groups.
This research will advance the science of natural, multimodal human-computer interaction, while addressing, directly, the challenges of developing new knowledge through which advances in information technology can enable universal participation in geospatial information access and applications. Estimates suggest that as much as 80% of all digital data contain some form of georeferencing (coordinates, addresses, zip codes). As a result, GIS have become an important tool for managing and analyzing data across a range of science, policy analysis, resource management, business planning, and educational applications. In spite of the potential applications for GIS and the growing availability of geospatial data, GIS are underutilized because current systems are hard to use. There is a substantial gap between current GIS applications and the promise of GIS to facilitate public access to the large volume of geospatial information and to support public as well as expert input to important decisions affecting their communities and environments. Achieving the research goals outlined will, thus, enable more effective decision-making about risks to public safety and health and will support input to important public decisions by a wider range of participants.
