Advances in technology are making it feasible to explore novel approaches to human-computer interaction in a wide variety of devices and settings, in an effort to achieve interaction that feels more natural. While valuable from both a user and commercial perspective, use of new technology is not well understood from a more principled system design or cognitive science perspective. Establishing even the basic elements of a set of design principles would both produce better designs and increase our confidence in using the technology in high-risk/high-reward domains, such as first responder planning and control. The PI's long-term goal is to develop a set of principles specifying how to design systems that both enable natural human-computer interaction and are informed by an understanding of human factors. Natural interaction refers to the cognitively transparent, effortless multimodal communication that can happen between people; this work aims to make that possible in human-computer interaction. Designs informed by human factors take into account an understanding of human capabilities (e.g., attention, use of multiple information channels, etc.), so that the final system is a good impedance match to human information processing. This research will involve building systems designed in this spirit and articulating principles for their design that, in turn, will facilitate future designs by making explicit both the task conditions under which one or another modality is appropriate (e.g., when to draw, when to talk), and the ways in which multiple modalities can effectively be used simultaneously in human-computer communication. The project is set in the context of a tabletop-based system that assists with planning and coordination in the command center of an urban search and rescue (USAR) operation. The work will proceed by leveraging and combining the team's experience in building novel interaction technologies and in human factors. They will extend the current version of the PI's tabletop system, which permits basic pen-based interaction, so as to give it the ability to handle the kinds of sketching, freehand gestures and speech used in real-world USAR work, thereby providing a far more natural style of interaction. The additional interaction modalities will make the system more powerful, while the real-world, time-pressured character of the task offers a good platform for studying the human factors aspects. This will allow the team to understand how, when and why various modalities are useful, providing the data from which system design principles can be articulated. To help ensure breadth of applicability of project outcomes, the PI will explore the same issues in a second domain, software design with UML diagrams.
Intellectual Merit: This research will provide insight into a model of multimodal interaction by producing empirical data about modality selection, and by articulating a widely useful set of principles for interface design that make explicit the conditions for both modality selection and cognitively effective modality combination. Such principles offer the possibility of transformative change to multimodal interface design, changing it from the current largely ad hoc practice to a design process guided by testable principles. Pursuing the research in two task domains will help to ensure a useful degree of generality to the principles derived. The work will provide benefit to society to the extent it can improve the effectiveness of first responder teams. The ability to handle non-traditional interaction modalities (e.g., gesture) will ultimately make computer interaction more accessible to physically disadvantaged users. The PI will take care, to the extent possible, to use and build upon open source tools, so that he can make available to the community all of the research software produced during the course of the project, thereby adding to the supply of next-generation research and education platforms.
