Deformable objects; that is, those objects that change shape with pressure, are ubiquitous in our lives. Humans deal with deformable objects all the time when we fold laundry, pour ourselves a drink, and peel vegetables. Although such tasks are performed effortlessly by humans, they are challenging for traditional robotics techniques which assume that all objects are rigid. Robot deformable object manipulation, such as laundry folding or cooking, could enable those who cannot perform these tasks easily for themselves to live more independent lives. This Faculty Early Career Development (CAREER) project will develop novel methods for robots to learn to manipulate deformable objects. First, robots will learn to predict how their actions will cause an object to deform; robots will then use these predictions to determine how to achieve deformable object manipulation tasks. To enable the public to better understand this research, the investigators will develop a virtual robot interface called “Teach-a-Robot" that will enable anyone in the world to experience teaching a robot a new task: users will upload a video of themselves performing a task and then watch a robot attempting to imitate them. Finally, the investigators will design new course assignments that demonstrate the interplay between robotics, computer vision, and machine learning.
The research objective of this project is to explore how self-supervised representation learning can be used by robots to achieve deformable object manipulation. Given a set of demonstrations of robots or people interacting with deformable objects, the project will enable robots to learn representations of these objects that are useful for manipulation. This effort will explore how self-supervised learning can be used to: predict the effect of a robot's actions on a deformable object; infer object properties; recognize deformable object parts under large deformations; learn to transfer policies between deformable objects of the same class; and learn visual representations that transfer well from simulation to the real world. The second research thrust will explore how robots can use these self-supervised representations to efficiently learn policies to manipulate deformable objects. This effort will develop novel methods for reinforcement learning that can perform multi-scale reasoning, enabling robots to focus on both task-relevant low-level object details as well as the global object configuration. This research effort will open new doors for robots to interact with a wide range of deformable objects, such as folding cloth, molding dough to make pastries, spreading sauce onto pizza dough, and peeling vegetables.
This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
