This research explores the deep technical challenges of a new class of computing systems that integrate a wearable device (such as Google Glass) with cloud-based processing to guide a user step by step through a complex task. Although easy to describe, many challenges in computer systems, computer vision and human-computer interaction must be overcome for this concept to become reality. Human cognition is a remarkable feat of real-time processing. It involves the synthesis of outputs from real-time analytics on multiple sensor stream inputs. An assistive system amplifies human cognition with compute-intensive processing that is so responsive that it fits into the inner loop of the human cognitive workflow. In its most general form, cognitive assistance is a very broad and ambitious concept that could be applied to virtually all facets of everyday life. As a pioneering effort, this research is more narrowly focused on user assistance for well-defined tasks that require specialized knowledge and/or skills, and for which task state and task-relevant actions are fully accessible to computer vision algorithms.
The research is organized into four broad thrusts. The first thrust decouples and cleanly separates low-level mobile computing and cloud computing issues such as resource management, network latency, placement, provisioning, scalability, and load balancing from the task-centric foci of the other tasks. The second thrust focuses on the computer vision research necessary to address the challenges of wearable cognitive assistance. Vision is the dominant sensing modality for the kinds of tasks addressed in this research, but the validation experiments will include proof-of-concept use of other sensing modalities such as audio and location. The third thrust focuses on task description, tracking, sequencing and user guidance. Its goal is to create a set of generalizable principles and tools that can be applied to a wide range of tasks. Matching task assistance to task demands and user capabilities will be integral to this thrust. The fourth thrust involves continuous integration of research from the first three thrusts and applies it towards end-to-end validation on a series of tasks of increasing sophistication and difficulty. This thrust involves close collaboration with industry partners.
This research will advance computer science by producing scientific insights, algorithms, system designs, implementation techniques, and experimental validations at the intersection of computer systems (including mobile computing, cloud computing, virtual machines, operating systems, wireless networking, and sensor networks), vision technologies (including computer vision and machine learning), and human-computer interaction. More broadly, society will benefit from wearable cognitive application in areas such as health care training, industrial troubleshooting and consumer product assembly. From an educational viewpoint, this research offers many unique opportunities to train graduate and undergraduate students on how to approach problems from a broad cross-disciplinary viewpoint.
