Despite significant advances in neural science, the dynamics by which neural activity propagates across cortex while we think of a word and produce it remains poorly understood. This proposal will develop novel, data-driven approaches for understanding functions and interactions of various brain regions by leveraging rare neural recordings obtained with electrocorticography (ECoG) sensors while neurosurgical patients participate in tasks involving language perception, semantic access and word production. This project will produce a set of validated novel computational tools for estimating neural representations and their dynamics as well as elucidate the cortical networks subserving perception, semantic access, and production of speech. Although these tools will be developed for ECoG data, the proposed frameworks are applicable to other neural data modalities including fMRI and EEG, and thus have broad applications in neuroscience. The ability to robustly translate between speech and its neural representations is vital to the development of speech prosthetics, which would allow patients with degenerative conditions (Amyotrophic Lateral Sclerosis) or neurological damage (locked-in syndrome) to drive a speech synthesizer via control from intact cortical structures. The network connectivity tools could shed light on the propagation dynamics of epileptic seizures as well as on how cortical communication, when impaired, gives rise to language aphasias and disconnection syndromes. Furthermore, the decoding and network connectivity tools could help develop novel language mapping approaches for brain surgery without the associated risks of electrical stimulation mapping.
The project consists of three core thrusts: developing neural decoders for language processing, developing directed connectivity models, and experimental validation. The neural decoders will be based on deep-learning architectures able to learn a transformation between neural signals and the speech heard by the patient, the speech produced by the patient, or the semantic concept represented by the stimulus word. The connectivity models will generalize and coalesce current approaches for estimating the task-dependent, time-varying directed connectivity between cortical regions. Lastly, these findings will be experimentally validated via clinical electrical stimulation data and cortico-cortico evoked potential (CCEP) stimulation experiments. Current modeling approaches of ECoG data have mostly focused on variants of linear models and on speech acoustics. This project will harness the potential of highly non-linear and deep networks for modeling neural responses to both speech acoustics and access to semantics. Additionally, tools for inferring direct connectivity and interactions among neural regions will provide a detailed characterization of the network dynamics, which is largely overlooked by most ECoG decoding studies.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
