Research supported by this EAGER award is developing the NeuroPhone system, the first Brain-Mobile phone Interface (BMI) that enables neural signals from consumer-level wireless electroencephalography (EEG) headsets worn by people as they go about their everyday lives to be interfaced to mobile phones and combined with existing sensor streams on the phone (e.g., accelerometers, gyroscopes, GPS) to enable new forms of interaction, communications and human behavior modeling.
Specifically, this high-risk exploratory research is to:
1) study new energy-efficient techniques and algorithms for low-cost wireless EEG headsets and mobile phones for robust sensing, processing and duty cycling of neural signals using consumer devices;
2) develop new learning and classifications algorithms for the mobile phone to extract and infer cognitively informative signals (e.g., P300, N400, and neural synchrony) from EEG headsets in noisy mobile environments;
3) deploy networked NeuroPhone systems with a focus on real-time multi-party neural synchrony and the networking, privacy and sharing of neural signals between networked NeuroPhones; and
4) evaluate networked NeuroPhones applications, specifically, measuring teacher-student engagement in the classroom and measuring group level emotional state.
This interdisciplinary research opens up opportunities in education, teaching and outreach, in part because it focuses on an educational NeuroPhone application, which contributes new insights into cognitive engagements of students in the classroom as well as engages students from the Department of Computer Science and the Department of Psychological and Brain Sciences in the project. Results from this work will transform applications across diverse domains such as education, health monitoring, and social networking.
This research project addressed the intersection of three different problems: using neural decoding to extract meaningful information from people's brain activation patterns, obtaining reliable neural data in noisy real-world environments using low-cost consumer-level EEG headsets, and using the information decoded from that neural data to interface with iPhones. The project made significant steps forward on all three of these problems. Although it will likely be many years before neural interfacing with smartphones becomes fully practical for everyday use, we believe that this project has laid the foundations for that future work. Specifically, we produced a fully working prototype system which allowed the user to choose amongst contacts in their phonebook and to dial the selected contact, entirely driven by neural signals transmitted wirelessly to the phone. We further developed and extended this work by testing the reliability of the neural signals in real-world conditions. The most difficult typical scenario is when the user is freely walking, as vibration from each footstep causes large EEG artifacts. We found that even in this scenario, information could still be extracted from the neural signals for the purposes of classification. We also developed novel algorithms for neural decoding which are able to capture commonalities across different individuals, a result which will be important in order for systems to be usable by new users with minimal person-specific training. We disseminated the results of this research via publications not only in scholarly journals and international conferences, but also in ways which will reach a wider audience. In particular, our work was featured in the New York Times, and we freely and publicly released the source code for our system so that anybody can explore our system and build upon it in order to further their own research.
