Free-viewpoint video is an emerging technology for visual communication that creates the sensation of 3D immersion in the remote scene by allowing the user to dynamically switch between arbitrary viewpoints. It has the potential to advance society by enabling virtual human transportation and boost the global economy and quality of life. At present, free-viewpoint video is limited to high-end computing environments and studio-type settings, due to its higher bandwidth and complexity expansion over single-view video. Furthermore, the fundamental questions of viewpoint sampling (camera location) and resource allocation across the captured views are largely unanswered, due to the nascency of the technology, and are addressed using suboptimal heuristic approaches, thus penalizing system efficiency. These two characteristics would otherwise make free-viewpoint video impractical and preclude its broader deployment, in particular on mobile devices, due to their constrained bandwidth, battery power, and CPU capabilities. However, the latter have become a primary platform for computing and communication needs, anywhere and anytime, a trend that will only accelerate in the future. Thus, it is anticipated that only by enabling ubiquitous and seamless mobile free-viewpoint video may the full potential of immersive communication be achieved. This project seeks to achieve this goal via concerted advances in signal representation, wireless video communication, and user-action modeling that will be integrated holistically. The advances delivered by this investigation will have broad impact across diverse fields that involve live video communication via multiple viewpoints, including telemedicine, telepresence and telecollaboration, remote monitoring and control, entertainment (3D and free-viewpoint TV), gaming and virtual worlds, people-centric sensing and connected-community applications.
The project will pursue the following technical thrusts: (i) Characterization of the fundamental trade-offs between viewpoint space sampling, rate allocation, and signal fidelity in immersive mobile communication; (ii) Derivation of the optimal sampling policy, at the view and data-unit levels, in uplink communication; (iii) Design of novel view and rate scalable coding for multi-view broadcast and derivation of the optimal view embedding policy, as a function of the broadcast rate, in downlink communication; (iv) Characterization of the optimal scheduling policy for local ad-hoc communication between mobile clients; and (v) Integration of energy conservation and characterization of the trade-offs between battery lifetime and multi-view application performance.
