Human Robot Interaction (HRI) is research that is a key component in making robots part of our everyday life. Current interface modalities such as video, keyboard, tactile, audio, and speech can all contribute to an HRI interface. However, an emerging area is the use of Brain-Computer Interfaces (BCI) for communication and information exchange between humans and robots. BCIs can provide another channel of communication with more direct access to physiological changes in the brain. BCIs vary widely in their capabilities, particularly with respect to spatial resolution, temporal resolution and noise. This project is aimed at exploring the use of multimodal BCIs for HRI. Multimodal BCIs, also referred to as hybrid BCIs (hBCI), have been shown to improve performance over single modality interfaces. This project is focused on using a novel suite of sensors (Electroencephalography (EEG), eye-tracking, pupillary size, computer vision, and functional Near Infrared Spectroscopy (fNIRS)) to improve current HRI systems. Each of these sensing modalities can reinforce and complement each other, and when used together, can address a major shortcoming of current BCIs which is the determination of the user state or situational awareness (SA). SA is a necessary component of any complex interaction between agents, as each agent has its own expectations and assumptions about the environment. Traditional BCI systems have difficulty recognizing state and context, and accordingly can become confusing and unreliable. This project will develop techniques to recognize state from multiple modalities, and will also allow the robot and human to learn about each other's state and expectations using the hBCI we are developing. The goal is to build a usable hBCI for real physical robot environments, with noise, real-time constraints, and added complexity.
The technical contributions of this project include: 1. Characterization of a novel hBCI interface for visual recognition and labeling tasks with real physical data and environments. 2. Integration of fNIRS sensing with EEG and other modalities in human robot interaction tasks. We will test our ability in the temporal domain to determine at what timescale we can correctly classify movement components that would predict a correct (rewarding) trial or non-rewarding/incorrect movement. 3. Analysis and validation of the hBCI in complex robotic tele-operation tasks with human subject operators such as open door, grasp object on table, pick up item off floor etc. 4. Use of hBCI to characterize human/robot state and create a learning method to recognize state over time. 5. Use of augmented reality for HRI decision making. 6. Further develop hBCI for tracking cognitive states related to reward, motivation, attention and value. A new class of HRI interfaces will be developed that can expand the ability of humans to work with robots; promote the use and acceptance of robot agent systems in everyday life; expand the use of hBCIs in areas other than robotics for human-machine interaction; further the development of hBCIs as our system will be tapping into reward modulated activity that will be used via reinforcement learning to autonomously update the learning machinery; and bridge the educational divide between Engineering and Neuroscience.
