This project investigates how to effectively learn shape features with deep neural networks from images. It has been commonly believed that features learned by deep neural networks from images include texture, color, and shape of objects. Although visualizations of learned features demonstrate that contours of objects are extracted in the process of deep learning, our preliminary results provide clear arguments that 2D shape features are not well captured by current deep neural networks. This project develops a framework for effective learning of shape features with deep neural networks. The research brings new insights to a core problem in computer vision: shape understanding, which relates to many subfields in computer vision ranging from low-level tasks, such as segmentation and image statistics, to high-level ones, such as visual retrieval and object detection in images. The project includes plan to deploy the research results directly to applications such as biodiversity study (species recognition). The project also involves high school students and undergraduates in research.
This project conducts both theoretical and experimental research to gain better understanding why shape features are not well captured by current deep neural networks. Then it develops new learning strategies specifically targeted for shape features by following two main alternatives: (1) constraining the filter learning for Convolutional Neural Networks so that they are more contour focused, and (2) designing special structures of Deep Neural Networks for learning shape representation. The project designs circular sequential networks for silhouette-based shape classification, which encode naturally contour context information while implicitly performing contour matching. It also extends these networks to sketches, which are composed of both closed and open contours. Attention models are investigated on shapes to analyze roles of parts in shape representations so as to improve further shape matching and recognition algorithms.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
