Institution: Boston College Artists are the masters of visual perception. Studying art and vision together can provide new solutions to fundamental problems in computer vision. We focus on inferring scene layout from a single image. This problem has been studied since the earliest days of Artificial Intelligence research, resulting in a host of so-called Shape-from-X methods, where X could be shading, perspective, etc. Unfortunately, each of these methods works under its own assumptions which often do not hold in real images. How these cues interact and integrate remains elusive. Painters constantly use a combination of four techniques: occlusion, perspective, shading, and form to effectively evoke a 3D percept from a 2D picture. Studying their techniques can lend insights into the computation of recovering scene layout from pixel values. The PI proposes to bring artists and vision scientists together to solve the computational problem of scene layout from pictorial cues. This project realizes it in three areas: education, experiments and computational modeling.
A new interdisciplinary course, Art and Visual Perception, has been developed at Boston College to give a comprehensive cross-examination of how art contributes to the understanding of vision, and how vision contributes to the generation and viewing of art. Students are actively engaged in both art practice and vision experiments. Learning art and vision together results in a deeper understanding than studying each discipline separately. Students' assignments also result in valuable datasets for vision research.
The computational approach to scene layout from pictorial cues in this project is to group pixels into spatially organized surfaces from a global integration of multiple pictorial cues in a spectral graph-theoretic framework. The goal is to turn artistic rendering knowledge on how these cues interact into a computational reality. The PI will study geometry (occlusion and perspective), appearance (brightness and color), and form using eye tracking and psychophysics experiments and computational models. These efforts are organized into two phases that progress from inferring the spatial layout from scenes made of planar surfaces (rooms and streets) to scenes made of curved surfaces (landscape and generic scenes).
Intellectual Merit
What is most remarkable about vision is its ability to perceive 3D spatial layout from a single 2D image. The proposed research replicates this ability in computation from a grouping perspective. Compared to statistical learning approaches, the grouping method is not only generic and thus scales well with the number of scenes, but can also produce a precise organization of surfaces in the scene. Compared to traditional Shape-from-X approaches, the grouping method examines each pictorial cue in conjunction with others. The integration of these multiple pictorial cues allows them for the first time to become applicable to real images. The PI has developed the essential grouping machinery in spectral graph theory for depth segregation. Compared to most existing formulations on this topic, it has unparalleled conceptual simplicity, computational efficiency, and guaranteed near-global optimality. The proposed research on brightness and color perception, in connection with Shape-from- Shading and surface organization, will help clarify the role of low- level and high-level mechanisms in the long-standing scientific debate between Hering and Helmholtz on color perception.
Broader Impact
This project bridges the gap between art and science not only in research but also in education by developing a new curriculum that traverses the areas of neuroscience, psychology, computer science, and visual arts, by involving students in art practice and scientific experiments, and by providing a forum for artists and scientists to exchange ideas on visual perception. These interdisciplinary efforts befit the liberal arts education tradition at Boston College. This project will not only benefit from the strong Fine Arts department on campus, but also cultivate computer science awareness and outreach to non-technical people, and promote the growth of the young Computer Science department at Boston College.
