With recent advances in multimedia computing and communication, tele-immersive environments in smart room are emerging as enablers for collaborative interaction. There are many examples of smart room systems that offer exciting infrastructures to enhance teaching, distance education, resource sharing and local group activities. Examples include the iLAND, Roomware, EasyLiving, Interactive Workspaces, Oxygen and Gaia systems for support of smart room infrastructures. These smart environments are characterized by a high concentration of single user devices, ranging from handheld devices to high-performance HDTV plasma displays, connected via heterogeneous networks such as 802.11 wireless Ethernet or Gigabit Ethernet. However, most of these smart room devices include multimedia devices that can assist in creating 2D video conferencing environments, but not 3D teleimmersive environments. Examples include VRVS, a web oriented system for video conferencing and collaborative work over IP, Polycom, Netmeeting, vic and vat to broadcast audio and video over MBone IP multicast. On the other hand, outside of smart room environments, expensive immersive environments emerged, which either do not support easily tele-immersion or are complex to be connected via Internet. Examples include CAVE environment, AccessGrid, an immersive group-to-group collaboration system, TIDE, a tele-immersive virtual environment for collaborative visualization and exploration, Colliseum, a desktop tele-immersive system.
The goal of this research is to place a joint 2D and 3D tele-immersive collaboration into smart rooms, consider multicamera 2D/3D sources in one smart room which transmit the video data over LAN and/or WAN to other smart room(s) where the video streams are either rendered into a 3D tele-immersive video or displayed as 2D video streams in a multi-view fashion on multiple displays. The challenges of this goal are tremendous if we consider a true tele-immersive environment as follows: (1) video resolution of a captured video stream from a 3D camera is 640x480 pixels or higher, each pixel includes RGB and depth information encoded as 5 bytes per pixel resolution, and 10 frames per second. This video characteristic means to process and transmit 1.536Mbytes per frame, 15.36 Mbytes per second, 122.480 MBit per second per stream, (2) we consider ten cameras to achieve a satisfactory teleimmersive perception, i.e., 3D video streams are placed in one room in a 180 degree half-circle, which means 15.36 Mbytes or 1.22480 Gigabits per second to store and transmit in/from/to one smart room, and (3) all 3D streams must arrive in a synchronous manner to achieve a proper rendering into a final 3D tele-immersive video. The current multimedia protocol delivery solutions still consider (a) much smaller sizes of 2D video streams, (b) no joint 2D/3D videos, and (c) if high volume data are to be transmitted, then often dedicated networking infrastructure are in place (e.g., in case of supercomputing applications or Virtual Reality applications). Hence, there is a strong need to understand how to organize such large volumes of time-sensitive video data in smart rooms, what algorithms and protocol functions must be part of the environment to assist in 3D multimedia delivery, management and control.
The principal investigators (PIs) propose a unified application-level broadband protocol framework for multi-camera 3D video delivery between smart rooms to reach the goal of sending 3D video streams in time-sensitive manner. This application-level broadband framework will explore the protocol design space above the augmented TCP or UDP-like multimedia-enhanced transport layer as a strong complement to efforts to change the router and transport protocol functions. The PIs will carefully look at service functions residing in the session layer and application layer using a cross-layering approach. Since the multi-camera 3D video streams are sent from a room environment and also received in a room environment, a LAN/WAN computing and communication infrastructure will be available to host the various service functions in a distributed manner. They will explore content-specific and middleware-specific service functions in the protocol space such as (a) a color-reduction-based compression approach, (b) bandwidth/delay management services, (c) configuration service to tune the number of sending cameras for bandwidth/delay management reasons, (d) user-based customization service to tune
