The physical environment of a cyber-physical system is unboundedly complex, changing continuously in time and space. An embodied cyber-physical system, embedded in the physical world, will receive a high bandwidth stream of sensory information, and may have multiple effectors with continuous control signals. In addition to dynamic change in the world, the properties of the cyber-physical system itself ? its sensors and effectors ? change over time. How can it cope with this complexity? The hypothesis behind this proposal is that a successful cyber-physical system will need to be a learning agent, learning the properties of its sensors, effectors, and environment from its own experience, and adapting over time. Inspired by human developmental learning, the assertion is that foundational concepts such as Space, Object, Action, etc., are essential for such a learning agent to abstract and control the complexity of its world. To bridge the gap between continuous interaction with the physical environment, and discrete symbolic descriptions that support effective planning, the agent will need multiple representations for these foundational domains, linked by abstraction relations. To achieve this, the team is developing the Object Semantic Hierarchy (OSH), which shows how a learning agent can create a hierarchy of representations for objects it interacts with. The OSH shows how the ?object abstraction? factors the uncertainty in the sensor stream into object models and object trajectories. These object models then support the creation of action models, abstracting from low-level motor signals. To ensure generality across cyber-physical systems, these methods make only very generic assumptions about the nature of the sensors, effectors, and environment. However, to provide a physical test bed for rapid evaluation and refinement of our methods, the team has designed a model laboratory robotic system to be built from off-the-shelf components, including a stereo camera, a pan-tilt-translate base, and a manipulator arm. For dissemination and replication of research results, the core system will be affordable and easily duplicated at other labs. There are plans to distribute the plans, the control software, and the software for experiments, to encourage other labs to replicate and extend the work. The same system will serve as a platform for an open-ended set of undergraduate laboratory tasks, ranging from classroom exercises, to term projects, to independent study projects. There is a preliminary design for a very inexpensive version of the model cyberphysical system that can be constructed from servo motors and pan-tilt webcams, for use in collaborating high schools and middle schools, to communicate the breadth and excitement of STEM research.
A cyber-physical system exists in a physical environment that is unboundedly complex, changing continuously in time and space. The system is embedded in the dynamically changing physical world; it receives a high-bandwidth stream of sensory information; and it may have multiple effectors with continuous control signals. Our hypothesis is that, for a cyber-physical system to cope successfully with this level of complexity, it should be a learning agent, learning the properties of its sensors, effectors, and environment from its own experience, and adapting over time. To bridge the gap between continuous interaction with the physical environment, and discrete symbolic descriptions that support effective planning, the agent will need multiple representations for foundational domains such as Space, Objects, Actions, etc., linked by abstraction relations. We built on our previous successful work in representing knowledge of the spatial environment in the Spatial Semantic Hierarchy. This project supported work on sensory perception, especially vision, on learning useful conceptual organization on perceived categories of objects, and on effective and robust means of taking action in a dynamic and incompletely known environment. Grace Tsai, a doctoral student in the UM ECE department, developed methods for using robot vision to build a concise plane-based abstraction of the structure of the static background environment from a stream of visual observations during travel. Her method used local visual features to generate planar structure hypotheses, and then used Bayesian inference to converge rapidly to the correct hypothesis. Over a series of publications, she extended her original method to handle newly-revealed portions of the environment during travel, to use the set of active hypotheses to focus attention on features in the most informative regions of the image, and to separate clutter (objects not well-described by planar surfaces) from the plane-based model of the environment. She received her PhD in Fall 2014. Yu Xiang, a doctoral student in the UM ECE department, has developed methods for describing 3D foreground objects in terms of oriented planar surfaces ("aspects"), and for detecting and estimating the 3D poses of multiple objects from multiple images. By exploiting the constraint that the appearance of an object should be consistent when observed from different viewpoints, his method allows an object to be recognized in spite of occlusions, including self-occlusions, from the appearances of its visible parts. Thorough evaluation demonstrates his method to be more accurate than previous methods. Dr. Jingen Liu, a post-doctoral fellow who worked on this project, focused on attribute-based methods for human action recognition from video input. He developed a framework for selecting effective attributes for discriminating among viewed actions. He also developed and evaluated methods inspired by bilingual dictionaries for combining view-dependent features ("visual words") to build consistent models of human activities across multiple views. This resulted in two highly-successful papers at a leading conference. Dr. Liu is now working at the SRI Sarnoff Research Lab. Dr. Roni Mittelman, a post-doc in Computer Science & Engineering at UM, worked on Bayesian methods for learning the hierarchical conceptual structure of a collection of images. An uninformed agent is exposed to experience of the world in the form of a stream of instances, sensed as low-level sensory inputs (e.g., pixels, phonemes, individual words, etc.). To structure this experience, the agent must learn two different but interdependent higher-level structures, both of which are potentially infinite-dimensional. The first step uses Restricted Boltzmann Machines to learn semantically meaningful attributes from low-level sensory input. The second takes the resulting attribute vector and learns the most likely hierarchy of categories using a probabilistic method called the "Tree-Structured Stick-Breaking Process." Jong-Jin Park, a doctoral student in the UM Mechanical Engineering Department, worked on efficient and robust methods for planning safe and comfortable motion to specified goals. This work was applied to a robot wheelchair, but is more generally applicable. Starting with a novel, low-dimensional parameterization for smooth and comfortable local motion control laws, he developed a motion-planning framework capable of robustly solving extended navigation problems in complex environments in the presence of uncertainty due to static and dynamic obstacles. This project has contributed to the support of research training and experience for two post-doctoral fellows, eight doctoral students (five are still in progress and three have completed their PhDs), eight Masters students, two REU-funded undergraduate students, and two high-school students (who have gone on to study computer science and engineering at Stanford and Olin College of Engineering). Much remains to be done on the deep and fundamental problems we are addressing, but we have made important progress on sensing robustly, on learning conceptual structure, and on acting effectively. We have also helped train outstanding members of the next generation of researchers.
