This project will enable full-body nonverbal communication in virtual worlds, applying state-of-the-art machine learning and computer animation techniques to the synthesis of nonverbal communication, advancing and refining these techniques in the process. Virtual worlds are an emerging computer-mediated communication medium that situates geographically distributed participants in a shared communication space and enables embodied interaction with others in a simulated environment. Participants are represented as animated virtual humans that can convey both speech and body language. Yet no viable technology exists that can animate the virtual human's body during a live conversation without resorting to esoteric hardware or brittle algorithmic techniques.
This research will develop an approach that can convey body language through virtual humans in real time, using a natural control interface: the speech and motion of the participants. The proposal is based on, and sometimes advances, the state of the art in machine learning, computer animation, and the relevant aspects of linguistics and cognitive psychology. Body language animation based purely on visual tracking of the participant's motion is prone to significant defects due to tracking noise and failure. Thus this approach analyzes the speech of the participant together with the motion. This principled integration of live speech and motion input constitutes a fundamentally new approach to the control of nonverbal expression of human self-representations in virtual worlds.
The ability to convey rich nonverbal communication in virtual worlds will advance the capabilities of computer-mediated communication as a whole. This will provide basic infrastructure for distributed collaboration in science and engineering. Powerful forms of situated learning and social-scientific inquiry in education will be enabled, with positive impact on the self-efficacy of students who traditionally underperform in science curricula. Social science will be enriched with a new medium for the study of human interaction.
The primary goal of this project was to develop new approaches for the simulation of human motion and behavior in virtual environments. We have developed two major types of approaches. First, we demonstrated that high-fidelity human motion can be produced from first principles, given only a model of the human body and a compact set of objectives. For example, walking is produced by moving the center of mass forward at an appropriate velocity. Running is produced when the desired velocity is increased, without any other modifications to the model or the objective. We have shown that such de novo optimization can produce motion that both qualitatively and quantitatively matches human reference data. Second, we have developed a number of algorithms for learning motor skills from demonstrations. Using these approaches, virtual characters can be programmed by simply demonstrating the desired motions and behaviors. Specifically, our approaches optimize parameterized policies that specify the actions that the character needs to perform, depending on the character's current state. Optimizing such policies is very challenging, since their dimensionality is extremely high: in principle, a policy specifies how each joint in the character's body must move depending on the body's current state. We have developed a number of approaches that deal with this high dimensionality and successfully produce policies for such motor skills as walking, running, driving, and swimming.
