Providing an environment that offers both immersion and interaction is a tough research challenge. Ensuring a reasonable Quality of Experience (QoE) in using these environments installed in geographically distributed cities is even a tougher challenge. This project considers a collaborative, immersive, and interactive environment that not only supports 3D rendering of the participants? video but also other modalities such as Body Sensor Network (BSN) data that can offer highly precise data about a person?s physical movements (as well as physiological data). While creating this environment, one needs to consider the various bottlenecks that choke the data streams carrying the immersive and interactive information: reconstruction delay, ultra-high throughput needed, packet loss, and rendering delays.
The main aim of this project is to design and develop collaborative, multi-modal immersive environments with higher frame rates and frame quality by carrying out research tasks that can take advantage of information from other modalities and handle these bottlenecks.
In a typical tele-immersive environment, participants can see themselves in the locally rendered 3D view and see participants in the remote environments as well. Since the local rendering delays are much smaller, participants can see themselves earlier and in a more smooth fashion compared to the rendering of remote participants that suffers from communication delays and packet losses. This aspect of varying delays among the immersive participants can potentially cause problems during dynamic interactions and affect their QoE. Answers to questions such as what type of problems can be caused and how the participants handle them depend on the application domain of the immersive environments. To study the QoE and validate (with usability studies) the collaborative, immersive environment, a tele-rehabilitation application will be deployed in multiple cities: Berkeley, California; 2 sites in Dallas, Texas; and Urbana-Champaign, Illinois.
Intellectual Merits of this project are (i) The resource adaptation framework for streaming multi-source, multi-destination, multi-rate, multi-modal data incorporates supervisory hybrid control theory based fine-grained resource management, multi-modal coarse-grained management, and a multi-modal multicasting approach. (ii) Graphics Processing Unit (GPU)-based 3D reconstruction and compression algorithms. These algorithms facilitate reconstruction of 3D data points based on 3D camera array data and compress them at a faster pace than their CPU-based counterparts. (iii) GPU-based rendering algorithm of 3D data on the receiver side. This algorithm will handle potential data loss in 3D camera data streams using skeletal information from BSN data streams. (iv) Identification and measurement of Quality of Experience (QoE) metrics and using those metrics to derive Quality of Service (QoS) parameters. The derived QoS parameters will then help the resource adaptation framework to modify its decisions at run-time. This project aims to have transformative aspects in the new set of algorithms that exploits multi-modality while incorporating a feedback based on Quality of Experience for functions such as streaming, 3D reconstruction, and rendering.
Broader Impacts: This project promises significant impact in the fields of education and pervasive health care by providing augmented abilities to carry out intricate programs such as tele-rehabilitation with increased correctness and flexibility. This can also lead to improved productivity in the society considering the ability of health-care professionals to potentially handle a larger population (in remote places) as well as considering the possibility of the affected persons to become independent and productive faster. The project also ensures the results from the proposed research will be incorporated into the courses being taught. 3 women PhD students and 6 under-graduate students (2 are minority students) already working with the investigators of this project. Serious efforts will be undertaken to continue their involvement in this project. Apart from refereed conference and journal publications, the developed software, collected data, and research results will be shared with other researchers through a dedicated website (after ensuring satisfaction of HIPAA regulations).
