This Small Business Technology Transfer Phase I project aims to enable novel teleimmersive interfaces for digital gaming via the development of multi-site 3D multimedia integration and synchronization technologies. These interfaces will allow geographically distributed individuals to interact seamlessly with each other and learn in shared 3D virtual gaming environments. The focus of this project is to develop, evaluate, and transfer teleimmersive technologies that synchronize multiple correlated and spatial audio and 3D video streams in distributed multi-site games. These technologies are required to create interfaces that will bring together learners in a novel manner and potentially transform cultural education based on digital gaming. Current multimedia technologies only fully address the synchronization of pairs of audio and 2D video streams. The objectives of the proposed work are to (i) implement site-to-site 3D multi-audio/video streams synchronization solutions through rugged prototyping and experimental validation, (ii) identify and incorporate serious gaming requirements, usability and evaluation factors into the teleimmersive interface, and, (iii) design, develop and evaluate the extension of the synchronization capabilities for multi-site, 3D multi-video/audio streams. The project builds on research conducted at the University of Illinois.
The broader impact/commercial potential of this project lies in that the project will enable the creation of a 4D teleimmersive interface that represents a paradigm shift in the manner participants can be engaged in serious gaming through real-time teleimmersion across distances. These new environments will promote cultural awareness and understanding by providing (i) real-time teleimmersive dialogue between gamers/learners separated by geography, (ii) simulated travel to other cultures, and (iii) the means for structured interactions with individuals and groups from other cultures. The proposed interface will allow participants (and any physical objects they wish to bring with them) to be immersed in a shared virtual space and interact freely with each other, easily bringing into the games the complexity of human behavior. The commercial potential lies in greatly reducing game development time and costs by replacing the programming of 3D spaces and avatar behavior with 3D live capture and recordings of individuals in virtual spaces while retaining maximum flexibility and authenticity. The lower game development costs will enable the deployment of many small individual learning-games that may otherwise simply be too costly to develop using current methods. The teleimmersive interfaces and their underlying synchronization technologies will also enable new social interactions and collaboration capabilities in the educational games that are of particular interest to girls and women.
Intellectual Merits This project makes several technical contributions to our understanding of how to implement synchronization between distributed locations in which individuals are interacting through multimedia streams from a variety and multitude of different input and output devices. In particular, the detailed design and implementation of synchronization for a novel distributed 3D media-rich teleimmersive platform were completed with particular attention to the importance of synchronization in digital games for education. The 3D teleimmersive platform allows individuals to be captured through 3D video cameras and "teleported" or rendered in a shared virtual space where they can interact with each other and their virtual environment. The successful implementation of the synchronization algorithms was demonstrated through experiments connecting three different teleimmersion sites via the Internet. The synchronization solutions that were implemented took advantage of the fact that streams from a given location are not only temporally correlated but spatially correlated as well. The technical merits of the project include six contributions that address problems of synchronization at the sender site (where imprecise timing and varying rates of data across different types of sensors disrupt synchronization), during transmission over the Internet (where delays and jitter can disrupt synchronization), and at the receiver site (where different machine decoding and processing buffers can also disrupt synchronization). Broader Impacts The broader impacts of this project relate to the belief that the transformative interactive 3D video platform that is enabled by this project will be the foundation for a new category of digital educational games for cultural education. In particular, the platform allows for full body and facial language interactions as well as the conveyance of spatial relationships between individuals, all factors important to the true understanding of cultural interaction factors. This greater understanding is possible because the 3D video of participants is captured and "teleported" into virtual gaming environments as opposed to relying on avatars. This also eliminates the need for the development and programming of avatars. The latter is expected to lower game developments costs and provide for the development of a wider array of affordable and accessible educational games. In addition, the interactivity enabled by the platform has the potential to significantly impact distributed training and intervention methods in several areas of importance to society including, for example, health care through remote clinician training and remote physiotherapy and disaster intervention where situations will benefit from the close interaction of remote experts with local individuals enable through interactive 3D video teleimmersion.
