The focus of this project is on the problems associated with reconstruction and visualization of three-dimensional objects from images or video streams. The PI's ultimate goal is to first develop the fundamental algorithms underlying, and then to implement, an end-to-end light portal that can capture and "re-enact" real-world objects or events in a different time or space. A light portal captures a scene using a number of cameras with different viewpoints. From these input images, intelligent software to be developed by the PI will either reconstruct a coherent parametric scene model that includes not only geometry but also lighting and surface reflectance properties (suitable for storage, editing, and playback over time), or directly interpolate the complete bundle of color rays to be displayed (suitable for live transmission over space). These results will then be visualized on a unique light-field display that utilizes an array of projectors and micro-lens screens to emit a large collection of controllable light rays, so that any number of viewers can simultaneously perceive view-dependent stereoscopic effects as if they were in the real scene.
Realization of the light portal poses many scientific and engineering challenges in vision, graphics, image processing, networking, human-computer interaction, optics, etc., which the PI expects to be able to successfully tackle by building on his prior work and focusing on the core scientific problems in 3D reconstruction and visualization. On the reconstruction front, two paths will be explored. One is aimed at allowing robust 3D reconstruction for a much broader class of scene types by finding invariants from the complex interactions among geometry, lighting and surface reflectance properties. Alternatively, an inverse formulation to estimate directly the final image will be developed; by incorporating new constraints that are difficult to adopt in a geometric setting, such as nature image statistics, this inverse formulation should allow more realistic image synthesis even when explicitly reconstructing the 3D model is ill-conditioned. These two approaches are complimentary to each other, with one optimized for storage and editing, and the other for direct transmission and visualization. On the visualization front, the proposed light-field display will have a unique design that de-couples regular 2D displays and 3D screens, made possible by novel calibration and rendering algorithms. Therefore, in addition to its auto-stereoscopic capability, this display will offer an unprecedented level of flexibility and scalability in terms of resolution, size, field of view, etc.
Broader Impacts: In a manner analogous to the ubiquitous digital cameras and video displays in use today, the PI expects the light portal to provide the ultimate three-dimensional recording and viewing experience that may lead to profound changes in the way people communicate, educate, and entertain. In particular, the 3D capability may enable new paradigms for surgical training and tele-collaboration. This project also has an integral educational component; in addition to the usual course development, student participation, and broad dissemination through various venues (publications, demonstrations, a project web site, etc.), the PI will make efforts to attract minority students by taking an active part in the exchange program between the University of Kentucky and the University of Puerto Rico.
