The state of practice in human-computer interaction today is the graphical user interface, or direct manipulation, interaction style. This project seeks to identify and unify an emerging next generation of user interaction from a range of seemingly disconnected current research (including virtual reality, augmented reality, ubiquitous, pervasive, and handheld interaction, tangible user interfaces, lightweight, tacit, or passive interaction, perceptual interfaces, affective computing, context-aware interfaces, and speech and multi-modal interfaces). Ubiquitous computing, tangible interfaces, and the spread of computers into a wide range of products and objects are changing interacting with computers, from a specialized activity segregated from daily life, to one that is more and more a part of the real world. At the same time, as computers are becoming more a part of the real world, user interfaces seem to be evolving to behave more and more like the real world (e.g., in virtual reality). The PI believes these can be connected through his concept of "reality-based interaction" by focusing on the ways in which interfaces that are based on reality exploit users' built-in abilities, and that this idea will provide the necessary leverage to tie them together and define a new generation of user interfaces, in the hope that this naive notion can be built into a useful theoretical framework. Starting from the idea of natural or "reality-based" interfaces, which gain their strength because they exploit abilities that their users already possess, the PI will formalize the notion of learned knowledge vs. "reality-based" skills needed to use a system. He will develop this into a theoretical framework and flesh it out more formally, and use it to identify specific open issues for investigation. The PI will then modify or reinvent the initial approach, as needed, and will test the theory, first by applying it against a range of published results, and second by devising experiments specifically for testing, where aspects of a user interface can be selectively manipulated for the experiment. For this phase of the work, the PI will design, implement, and evaluate selected new interaction techniques representing gaps or opportunities suggested by the framework.
Broader Impacts: This project will lead to a unified theoretical framework for the next generation of human-computer interfaces, which should serve as a foundation and guide for research in the field. It will also result in prototype implementations and experimental evaluation of new interaction techniques, and will furthermore help extend research in new interaction styles beyond the relatively large and well-funded institutions that can currently pursue it, in that the PI will develop as part of the project an inexpensive workbench (or infrastructure package) to support this work, which he will then disseminate on the web (hardware descriptions and software tools) to other universities.
