This project develops real-time object recognition algorithms that generate extensive semantic object descriptions as a side effect of recognition. This side-information includes perceived object costs, attributes (object properties), and affordances (actions afforded by objects). With these, the act of recognizing a "door knob" would automatically produce the information that this is a "flexible" object, "made of metal," which "can be grasped" and "can be twisted," but "cannot be eaten." For robotics, this information is sometimes more important than the recognition of the object itself. The project enables robots to perform zero shot learning, e.g. learn to recognize door knobs by simply being told that these are objects that "are flexible, made of metal, can be grasped and twisted but not eaten." The research has applicability in areas such as manufacturing, intelligent systems, assisted living, and homeland security. Educationally, the project provides an exciting opportunity for undergraduate research.
This research develops new methods for top-down (task-driven) regularization of deep learning algorithms, though a combination of structural and loss-based regularizers. Structural regularizers constrain object and scene recognition models to guarantee speed and automatic generation of rich mid-level semantic (MLS) descriptions as a side effect of recognition. Loss-based regularizers penalize errors in the multiple semantic outputs of these models, enabling simultaneously high performance in object recognition, MLS predictions, and zero-shot learning. The resulting learning algorithms will endow robots with human-like abilities to infer rich MLS descriptions of objects and scenes as a "side effect" of object recognition and scene classification, in real-time. These contributions will be developed in the context of a new co-robotics problem, person-following unmanned aerial vehicles, where computer vision plays a mission critical role for tasks such as control and semantic motion planning but whose requirements in terms of speed and MLS inference are far superior to what is feasible today.
