How can we build robots that are able to distinguish and handle the many objects located in our everyday environments? And how can we endow these robots with the ability to reason about spatial concepts such as rooms, hallways, streets, and intersections? Even though the robotics community has made tremendous progress in the development of efficient techniques for representing and dealing with noisy sensor information, current techniques do not have the expressive power to address these questions. In this project, we will develop statistical relational machine learning techniques that are able to extract high-level concepts from robotic sensor data. By transferring knowledge learned in other environments, our techniques will enable robots to recognize objects and places in previously unseen environments. Ultimately, this research will bring us closer to the dream of truly autonomous robots; robots that can interact with people and operate successfully in the complex environments we live in.
This project also includes teaching efforts and the involvement of undergraduate students in research. Furthermore, it contains collaboration with an existing NSF project to expose young African-American students to the educational and career opportunities available in computer science, robotics and artificial intelligence.
Mobile robots need to build maps, or models, of their environments in order to navigate to specific locations or to interact with their environment. For instance, executing the simple command "bring me my coffee mug from the conference room" requires a robot to recognize the coffee mug, know where the conference room is, and navigate to that room. Most previous work on robot mapping has focused on building maps that are spatially consistent and suitable for path planning, but not on generating maps that contain semantic information such as the different types of places and objects in an environment. The goal of this project was to develop the fundamental techniques needed to generate and utilize such semantic maps. The research was organized in three thrusts. The first thrust focused on building much richer, 3D representations of indoor environments. This work took advantage of the new depth cameras developed mainly for the computer gaming industry, such as Microsoft's Kinect system. By combining color and depth information provided by such cameras, we were able to generate 3D models of indoor environments that went far beyond the quality of maps being built before. We demonstrated that such maps could be built by a person walking through a building or even by an autonomous quadcopter flying through the environment. The second research thrust focused on recognizing objects in an environment. Here, we developed novel approaches that take advantage of both color and 3D shape information to better recognize everyday objects, such as cereal boxes, coffee mugs, cars, trees, and computer keyboards. In one paper, we showed how the large sets of objects stored in Google's 3D Warehouse object database can be used to further improve object recognition. The third research thrust investigated how a robot could follow directions given in natural language. By phrasing this problem as a statistical machine translation problem, our system was able to learn to parse human directions into a semantic map of an environment. For instance, we showed that a robot can build a map containing information about hallways, rooms and intersections, and follow human commands such as "turn left into the next hallway and then enter the second room on the right". Overall, this project developed novel statistical reasoning and machine learning techniques that enable robots to reason about their environments in high level terms, such as objects and places. Building on such representations, these robots can learn to understand natural language commands, thereby enabling them to interact with people in a much more intuitive way.
