Our society is built upon shared ideas, ideas that get from one person to another via language that is "understood." But how do brains give us the ability to understand a stream of spoken words? This is a grand challenge question in computational neuroscience. This project addresses it using mathematical models of the language understanding process. These models reflect insights from computer science as well as linguistics. They allow investigators to ask: which process model best accounts for the signals from a particular brain region, at particular moment in time? The signals come from people listening to French and English versions of the same book. By comparing across models and across languages, the project seeks to differentiate between aspects of the understanding process that are language-specific and aspects that might be common to all humans. Increasingly precise modeling of this sort paves the way for future work with individuals who have trouble using language, such as those with Autism Spectrum Disorder. It could also lead to better computer systems, ones that use language in a brain-inspired way.
Bringing together computational linguists and cognitive neuroscientists, this project pursues two specific questions: (1) what aspects of sentence structure determine our expectations for upcoming words? and (2) what is the detailed balance between memorization and composition in natural language? Using electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) the PIs examine participants' neural responses to the spoken recitation of a literary work. These neural signals are fitted by time series predictors, themselves derived from linguistically plausible grammars and other language models. The project explores a family of such models, varying the size of grammatical units as well as the propensity for such units to be simply memorized as opposed to built up, step by step. Via information-theoretical complexity metrics, these theories derive quantitative predictions about the moment-by-moment neural responses of a person hearing a story. The approach as a whole leads to computationally explicit process models that are grounded in human brain responses to naturalistic text across two languages.
A companion project is being funded by the French National Research Agency (ANR).
