This project is focused on research issues associated with producing extremely detailed and accurate 3D geometry and appearance (BRDF) models at city scale from internet collections of photos from various sources containing millions of photos of enormous diversity such as viewing range and conditions, time of day and weather conditions, to name a few. The properties of urban scenes may include low-texture surfaces, reflective and transparent materials, and repeated structures that challenge existing reconstruction algorithms. The investigators will address these challenges with the aim of reconstructing several large US and foreign cities. Historical photos and virtual models may also be incorporated. There are a number of research topics associated with the project. To register photographs and recover sparse geometry at city-scale, a new, unified, structure-from-motion (SfM) algorithm will be designed to take advantage of large, parallel computing platforms. With registered photographs and sparse 3D scene points recovered by SfM, multi-view stereo (MVS) algorithms can reconstruct detailed geometric models. Novel MVS algorithms will then exploit the structure of architectural scenes and volumetric reconstruction methods will be employed to produce annotated models of exceptional accuracy and usability. Digital models are playing an increasingly important role in social, cultural and economic endeavor and are central to next-generation mapping and visualization applications. All models and datasets will be made freely available to researchers and the general public.
We are living in a time of unparalleled capture of visual data, with billions of photos shared online every day. These 2D photos collectively reveal the 3D structure and appearance of our cities. In this research, we have created new methods for automatically, quickly, and reliably building 3D models from very large community photo collections found online. Our work has addressed key problems in scalability and robustness, leading to methods that can reconstruct geometry from hundreds of thousands or millions of photos of a city. Beyond the 3D geometry of cities, we have also developed techniques to reconstruct scene appearance from photo collections, for instance the intrinsic color of each point in a scene. To make this all possible, we needed to create new computer vision algorithms that efficiently process very large collections of images. We have found that a key way to represent such image collections is as networks with links between overlapping images (akin to the links between friends in social networks). Given this representation, we use tools originally developed for analyzing complex networks to help us filter out noise and reconstruct geometry from the photo collection. The models we can build from our methods can have applications in a range of areas, from urban planning to ecology, and can also be used in education and entertainment. Much of the software and data we have developed throughout this project is available online; please see www.cs.cornell.edu/projects/bigsfm/ for pointers to all of our individual projects and for software, videos, and data.
