This project focuses on developing the theoretical foundation of differential ray geometry. The PI first derives a comprehensive ray geometry framework including the ray-distortion, ray-caustics, and ray-curvature theories as well as ray differential operators. This new framework is widely applicable to real-world problems. On the computer vision front, the PI explores robust and efficient schemes to infer ray structures from distortions or from caustics patterns and then recover surface geometry from ray differential attributes. This leads to a new class of specular (reflective and refractive) surface reconstruction algorithms. On the computer graphics front, the PI employs a novel normal-ray representation that converts a smooth 3D surface into a 2D ray manifold so that surface differential attributes can be directly derived from normal-ray geometry. The PI further develops new subdivision, re-meshing, and mesh simplification schemes for generating surfaces consistent with the underlying normal ray structures.
This research benefits many computer vision and graphics applications by providing a differential ray geometry model for cameras, light sources, and surfaces. It also benefits shape designs in aircraft, automobile, and many other industries, where higher-order shape consistencies are required. This project contributes to education through the development of new differential geometry courses and seminars and by involving women and under-represented students in mathematical and computer science research. The PI further seeks to build strong connections with the fields of mathematics and physics, optical engineering, and mechanical engineering through the project.
In this project, we have focused on developing theoretical foundations of ray geometry, studying common visual phenomenon such as image distortions and scattering from the ray geometry perspective, and applying the theory to resolve a range of computer vision and computer graphics problems. On the theory front, we have developed a comprehensive set of theories to directly characterize how geometric ray structures are related to defocus blurs, image distortions, light field imaging, and non-centric imaging. On the application front, we have built upon the theories a set of practical solutions for transparent object reconstruction, practical non-centric imaging, photorealistic hair modeling and rendering, and low-level imaging processing. Specfically, we have developed light field imaging based solutions that can directly recover 2D fluid wavefront of transparent flows and 3D density field of transparaent gas flows based on ray path analysis. We have also developed high quality light field imaging techniques via hybrid sensing and frequency light field analysis. For non-centric imaging, we have applied our theory to construct a practical non-centric camera called the crossed-slit or XSlit camera. We have shown that the XSlit is capable of resolving the co-planar ambiguity in perspective cameras for reliable Manhattan Scene reconstruction. In addition, we have developed a hair modeling and rendering system. The modeling system builds upon a new guide surface model and the rendering solution exploits a new ray-based scattering modeling. Finally, we have extensively studied challenging image processing problems such as denoising and deblurring from the ray space perspective. Broader Impacts. The proposed research focuses on using four-dimensional ray geometry to efficiently acquire, analyze, and model three-dimensional world. Our results have the potential to cast significant impact to computational imaging and computer vision, in particular camera designs and reconstruction frameworks. Direct educational impacts include integration of latest research results into course instructions at the University of Delaware and demos of our projects to K12 students at the annual Computer Science Researh Day. Students have had the opportunity to work on various aspects of the project through mathematical modeling, laboratory experiments, and computer simulations. Several course projects have led to publications at the premiere computer vision conferences such as CVPR and ICCV. The project has partially supported multiple graduate RAs (including 3 female students), a postdoc, and a visiting scholar. As the proposed work is multi-disciplinary, the PI has built strong connections with electrical engineering, optical engineering and mechanical engineering at the University of Delaware (UD). For example, 2D/3D fluid reconstruction system and results have been made to the UD community.
