The most natural means of human communication is multimodal, involving a mixture of speech, hand and body movement, facial expression, and eye motion. A multimodal framework is developed where several interaction modes are explored as a means of manipulating a 3D virtual display. The main focus of this project is exploring the use of free hand gestures for manipulating virtual objects in an effort to advance the goal of developing the fundamental theory behind hand gestures as a way of manipulating graphical objects together with speech, gaze direction, scene content, and graphical feedback. Existing multi-user virtual reality setups are used as experimental testbeds for validating the usefulness of methodologies for building useful multimodal human-computer interfaces for virtual reality. Since the development of natural human-computer interaction techniques are perceived to be a major hurdle in a greater impact of the information revolution, the results of this projects have wide-reaching potential applications.
