We live in a world made of diverse materials whose variations in appearance enrich our visual experience. It is also this variability of materials that adds daunting complexity to image understanding. This research program aims to establish the theoretical and computational foundation for automatic visual understanding and recognition of real-world materials. The program tackles this challenging problem from three key aspects, namely, deriving 1) novel hybrid physically-based and data-driven representations of the spatial, angular, spectral, temporal, and scale variations of material appearance, 2) active and passive methods for estimating the values of physically-based parameters that govern material appearance, and 3) single-image material recognition methods that leverage physically-based optical parameters as priors or invariants to guide machine learning techniques. These research thrusts lead to a comprehensive set of computational tools to recognize materials in real-world images despite their complex appearance variations, such as recognizing rusted metals, discerning soft cloth from hard concrete, identifying different fat content of milks, and labeling image regions with material traits like soft, hard, rough, and heavy.
The capabilities resulting from this program are crucial to a broad range of scenarios, for instance, to enable humanoid robots to understand that it should not squeeze the soft hands of a child, autonomous vehicles to understand what regions to avoid in a rugged terrain, visual analyses of tissues to help medical diagnosis, and automated inspection systems to reliably discover sub-standard quality food to prevent ill-health. The PIs work with research groups in these specific application areas to closely integrate the results from this project into their efforts. The results from this research are also broadly disseminated via publications, websites, databases, new courses and symposiums.
