This project investigates the use of image curve fragments to augment the use of isolated features in multi-view calibration and reconstruction tasks. The research team develops infrastructure based on differential geometry that utilizes curves and surfaces beyond lines and planes to correlate structure in multiple images of a scene: a pair of corresponding curve fragments in two views initiates a candidate 3D curve fragment whose presence can be validated in additional views. This results in a 3D curve sketch. A significant advantage of the 3D curve sketch over an unorganized cloud of point reconstruction is that it can correlate with image curve structure in novel views without referring back to the original views. This allows both incremental reconstruction (incorporating one additional view at a time) and simultaneous reconstruction from numerous views. Calibration methods are also being explored by using curve fragments under a RANSAC regime by using differential geometry in three views or more, and differential geometry together with appearance in two views. In a similar vein the correlation of surface patches by matching observable intensity local form in images is being investigated as a method for reconstruction of local surface patches. The project provides a core technology for many applications in the computer vision field. The developed technology can be also applied to other fields such as archaeology and art.
This project works with imagery that is obtained from a moving video camera as one might do when walking around and recording a status when visiting a new city. It also works with multiple cameras that are stationed around a scene, for example around a football stadium or around a stage. The goal is to develop technices to extrac the semantically meaningful geometric curves that are on the objects of interest in the scene, for examples corner ridges on a building, its windows, roof, etc. This is useful when modeling a 3D scene since the curves have high information content. We have developed the technology to process a video of a scene and extract a 3D line drawing from the scene in the form of a graph connecting various 3D curve segments. It has potential application in architectural drawing from a scene, 3D world modeling for entertainment and gaming, etc. It can also work with images obtained from multiple cameras all viewing the scene at the same time. We also developed a host of tools that use differential geoemetry, for example to calibate cameras using edges, establish a dense correspondence between two images, and on the way to our goal also developed better curve exaction algorithms.