This research aims to advance the theory and practice of image and video processing with the emerging 3D cameras. The recent huge commercial success and excitement with low-cost consumer-grade color plus depth cameras such as Microsoft Kinect promise the wide deployment of depth sensors to every computer and display device. This rapid development demands fundamental new methods in processing depth input and its combination with regular color images and videos. Furthermore, this emerging depth-sensing technology has the potential to radically extend the possibilities of image and video processing.
Toward these aims, this research involves, (i) calibration and denoising the raw measurements from depth cameras via accurate physical and statistical modeling of the new depth sensors and typical 3D scenes, (ii) improving the quality and resolution of depth images by exploiting the synergy between color and depth cameras, (iii) integration of multiple depth frames as an extension of super-resolution in video from 2D to 3D using computer graphics tools, and, finally, (iv) employing the improved and integrated depth information with color videos in challenging and high-impact applications including color plus depth rendering, relighting, and encoding.
