The relationship between brain activation patterns in fMRI and the semantic features of concepts have been developed into bi-directional generative mappings in previous studies: they can extrapolate beyond the training stimuli and either predict neural activation patterns of a new word based on semantic features or predict the semantic features of a new word from the neural activation patterns it evokes. The current project will develop an analogous mapping between the semantic features and the EEG signals. Because fMRI is less portable and available than EEG, imparting this brain reading capability to EEG systems is desirable for clinical practice and in-home healthcare since it can potentially provide neurally-based diagnosis or direct brain communication for a variety of cognitive or psychiatric disorders. For example, altered social concept representations can serve as a thought marker for further screening of autism and patients with locked-in syndrome can communicate with caregivers using their interpreted EEG brain signals. Thus, this project has two aims. First, it will systematically find EEG features (e.g. Event-Related Potential, Event-Related Synchronization, etc.) that encode concept semantics and develop a mapping between these EEG features and semantic features. Second, it will bootstrap the semantic prediction accuracy of EEG by simultaneously acquired fMRI. Specifically, the mutual dependencies (e.g. mutual information, correlation) between the two recording modalities will be computed and used to relate the EEG features to their precise source locations. This can fulfill a long-awaited scientific promise of simultaneous EEG-fMRI: understanding the neural processing of concept semantics with both high spatial and temporal resolutions. Furthermore, this mutual dependency pattern will be constructed into a cross- participant bootstrapping mask to up-weight EEG features that are closely correlated with fMRI activation patterns. This mask will be applied to EEG acquired without fMRI to test the prediction accuracy on new words in new participants. In sum, this project will construct a systematic mapping between EEG features and concept semantics, and bootstrap the mapping by concurrent fMRI. These efforts will lead to the development of a portable and cost-effective concept interpreter, which can serve as a platform for screening psychiatric disorders by detecting altered concept representations or be engineered into future assistive devices for patients with communication disorders. Project Summary/Abstract
This project will develop advanced methods for using electrical signals from the scalp (electroencephalography or EEG) to indicate what a person is thinking about, or to determine if their EEG patterns indicate a psychiatric disorder. The research will consistent of finding the correspondences between the meaning elements of a concept (such as whether it is a food or a tool) and the EEG signal. The outcome of the project will be an EEG-based concept interpreter, which can provide caregivers or clinicians with rich information about the contents of a person's thoughts.
