Humans can effortlessly infer functionalities of objects in a scene and interact with them. In facing a cabinet door, a human can instantly identify the location of a handle, understand how to interact with it, and predict what the interaction would cause in a scene. Understanding object functionality with full consideration of interactions and dynamics is an essential but missing capability in computing. The understanding of such object functionalities facilitates novel applications in a broad spectrum. First, the project changes how robots influences our society. Rescue robots becomes able to operate in uncontrolled, unexpected, and disastrous environments to help the lives of trapped victims. Home-robots becomes more deeply involved in our daily activities, helping us from household chores to the care of the elderly in the aging societies. The project enables to construct functional object models easily for 3D printing, a technology that is already revolutionizing the world through applications in art, fashion, human health-care, construction, and more. At the emergence of virtual reality and augmented reality, where interactions with the virtual environments are becoming more ubiquitous, this project also enables to easily add "lives" to virtual assets and create "live" virtual environments for better training and education.
This research studies a computational framework and algorithms to construct, learn, and infer functional object models. The following two key observations are at the heart of the project. First, objects are designed for optimal functionality, and careful physics-based reasoning should automatically reveal the information. For example, if the handle of a cabinet door is in the middle, the door should slide-forward to minimize the amount of necessary forces and torques. Second, functionalities are universal across object categories. For example, an act of opening is not fundamentally different from cabinet doors, laundry machines, ovens, to refrigerators. The approach discovers object functionality by integrating segmentation, motion estimation, and kinematic inference problems into a novel physics-driven formulation. The research team develops algorithms to learn and infer object functionalities from images, RGBD images, and 3D models. The project opens up a new frontier of research, functional object modeling, as well as introduces a principled new yet general computational framework incorporating physics.
