The goal of this project is to design a scalable and robust system for modeling and representing the spatiotemporal and semantic structure of large collections of partially geo-referenced imagery. Specifically, the project is aimed at Internet photo collections of images of famous landmarks and cities. The functionalities of the system include 3D reconstruction, browsing, summarization, location recognition, and scene segmentation. In addition, the system incorporates human-created annotations such as text and geo-tags, models scene illumination conditions, and supports incremental model updating using an incoming stream of images. This system is designed to take advantage of the redundancy inherent in community photo collections to achieve levels of robustness and scalability not attainable by existing geometric modeling approaches. The key technical innovation of the project is a novel data structure, the iconic scene graph that efficiently and compactly captures the perceptual, geometric, and semantic relationships between images in the collection.
The key methodological insight of this project is that successful representation and recognition of landmarks requires the integration of statistical recognition and geometric reconstruction approaches. The project incorporates statistical inference into all components of the landmark modeling system, and includes a significant layer of high-level semantic functionality that is implemented using recognition techniques.
Potential applications with societal impact include virtual tourism and navigation, security and surveillance, cultural heritage preservation, immersive environments and computer games, and movie special effects. Datasets and code produced in the course of the project will be made publicly available. The project includes a significant education component through undergraduate and graduate course development.
In the first phase of the project we focussed on improved modeling for the 3D reconstruction from photo collections. The challenges in the modeling that we more closely investigated were: (i) related to the difficulties related to scene symmetries, which lead to modeling disturbances; and (ii) the challenges of the heterogenous scene appearance across photos ranging from different weather conditions, different time of the day to different seasons alltogether. (i) On the first problem we developed a method to correct the output of commonly used reconstruction techniques like Structure-from-motion (SFM) as their reconstruction is often disturbed when there is ambigious scene structure. We researched a method to detect and correct the resulting reconstruction errors but not to modify already correct reconstructions. The main idea of our algorithm is to leverage the fact that scene parts, which are seen together should also be reconstructed together and conversely if parts are not seen jointly the should not be jointly reconstructed. Our experiments on large scale photo collection data demonstrate that our new algorith is robust and outperforms existing state-of-the-art methods. (ii) For the problem of dense reconstruction from heterogenous image data, we researched a novel multi-view depthmap estimation approach that carefully selects the images leveraged to obtain the depth estimate for a certain pixel. The selected pixels are then used is in a probabilistic framework to jointly model the specific views selected for the estimation and the depthmap estimation using the appearance consistency of the pixels.Our experiments on large-scale internet photo collection data underline the robustness of our approach against unstructured and heterogeneous image capture characteristics. Moreover, our method is inherently parallel, which leads to an efficient and scalable graphics card based (GPU) implementation.
