Using image measurements to understand and acquire properties of the physical world (such as the shape of an object, the reflectance of a surface, or the lighting in a room) is a critical capability for many sciences including medicine, material fabrication, remote sensing, robotics, autonomous navigation, architectural design, computer graphics, and computer vision. At a high level, these properties can be found by taking images and using computational algorithms to infer unknown parameters from the measurements. Broadly, the inference algorithms can be classified into two categories. On the one hand, physics-based algorithms try to analytically model and then invert the physics underlying the process of how a scene of certain parameters produces measured images; these algorithms are generally accurate but require a lot of computation. On the other hand, data-driven algorithms use supervised datasets to learn how to directly map measurements to unknowns; these algorithms are computationally efficient but are not guaranteed to produce accurate predictions. This project aims to transform physical acquisition pipelines, by creating general-purpose computational tools that combine the advantages of physics-based and machine-learning-based techniques, and that are simultaneously efficient, accurate and robust. By developing the theory and computational tools for this integration of simulation and learning, the project has the potential for transformative impact in application areas like industrial quality control, material science, oceanography, and biomedical imaging. Widespread adoption of project outcomes will be encouraged by making new software publicly available, as well as by offering tutorials and workshops in computer graphics, vision, and imaging conferences. The project also includes an education and outreach program that is tightly coupled to the research objectives, and which takes the form of courses, summer workshops, and lab visits for K-12 students intended to introduce them to science at an early stage and encourage STEM education. Additionally, the project will contribute towards broadening participation in computing through targeted involvement in existing programs in the participating institutions that focus on outreach to female students, first-generation students, and students from traditionally underrepresented minorities.
This project aims to transform physical acquisition pipelines by creating general-purpose computational tools that enable efficient and robust inference. This will be achieved by coupling physics-based and learning-based approaches, in a way that combines their complementary strengths of accuracy, generality, and efficiency. Three core areas of research will contribute to this. First, the project will develop inference pipelines that synergistically combine neural networks with analysis by synthesis optimization, in order to efficiently produce high-fidelity estimates of physical parameters. Neural networks will be trained in a physics-aware manner, by using physically-accurate renderers as layers in their architecture; this will allow the neural networks to simultaneously leverage supervised information and physical knowledge when making predictions. Second, a new class of physically accurate differentiable renderers will be created, which will enable this tight integration of physics and learning without the need to sacrifice computational efficiency. Instead of images, differentiable renderers will estimate their derivatives with respect to scene parameters; this estimation will be performed in a physically accurate way, using physical simulation algorithms derived from first principles and benefiting from innovations targeting improved efficiency. Finally, the advantages of the developed inference tools will be demonstrated in a diverse range of applications such as autonomous sensing, material science and fabrication, and biomedical imaging.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
