This project develops technologies to improve stereo and multi-view stereo algorithms by removing heuristics and hand-tuning using machine learning techniques. Stereo matching is the process of estimating depth of points, or 3D coordinates in a scene, and is enabled by the estimation of correspondences between pixels or other primitives in two or more images. Even the most successful current stereo matching algorithms, however, use a large number of heuristics. The developed methods from this project eliminate the heuristics from binocular and multi-view stereo matching and deliver algorithms with higher accuracy, interpretability of the results and higher portability to different settings. Stereo vision plays an important role in many applications, such as 3D modeling, augmented reality, driver assistance, autonomous navigation and human computer interaction. The educational and outreach aspects of the project focus on involving K-12 and undergraduate students in STEM education and research.
This research addresses stereo vision by training classifiers that learn from pairs, or larger sets of images, with ground truth depth to make more accurate predictions about unobserved data than those obtained by hand-crafted rules. The approach is comprehensive and tackles all stages of the binocular stereo matching process, including the matching cost function, cost aggregation, optimization and refinement. Representations for multi-view stereo based on surface patches, depth maps or occupancy grids and the corresponding algorithms are also supported by the same framework. Random forest classifiers are well suited for use in inhomogeneous feature spaces and classifier calibration can ensure that their outputs are close to the true posterior probabilities of the classes under consideration. The resulting algorithms and findings can be transferred to other computer vision problems that require pixel correspondences, such as optical flow estimation, image stitching and template matching.
