This is the first year funding of a four year continuing award. The objective of this project is to develop a research and educational program for modeling complex real?world environments from photographs. We are motivated by the following questions: suppose one could photograph a scene from every possible viewing position and orientation. What could be inferred about the structure of the scene and via which algorithms? To answer these questions, we propose a formal study of the scene reconstruction problem from the standpoint of the plenopticfunction, a 5D function that encodes the space of all possible images of a scene. The project will focus on two primary issues: (1) acquiring plenoptic representations of real scenes, and (2) reconstructing scene geo metry and radiance from such representations. To address the first problem, we propose a principled approach to the problem of image acquisition, i.e., along which directions should scene radiance be sampled in order to obtain the best possible scene reconstructions. Based on this analysis, we propose novel plenoptic cameras that are optimized for reconstruction tasks. The second problem is to devise algorithms for computing scene geometry and radiance from plenoptic representations. In contrast to traditional approaches which are based on perspective images, the proposed algorithms will operate directly on the plenoptic function and integrate information from a continuum of viewpoints. This formulation introduces unique challenges due to correspondence, visibility, and scale that will be addressed in the proposed work. A key outcome of the proposed work will be practical techniques for constructing 3D models of complex realworld environments. This capability is central for a variety of robotics tasks such as visual servoing, robot navigation, and motion planning. Environment modeling will also facilitate numerous applications in computer?aided design and computer graphics, including visualization of remote objects and environments over the Internet, virtual studios for television and film, and 3D virtual teleconferencing. An integral part of the project is a long term educational program for scene modeling in computer vision and graphics, emphasizing curriculum development, research opportunity for students, and outreach activities. An important component will be to integrate concepts from vision, graphics, and image processing within the undergraduate curriculum at CMU, and to design new interdisciplinary courses on environment capture and synthesis. The involvement of both undergraduate and graduate students is integral to the proposed research plan, and students will be encouraged to contributed to ongoing research activities. Outreach activities will include organizing tutorials and courses to disseminate research results to the larger community.
