Language comprehension is central to successful communication. However, to understand spoken language, people must cope with noisy environments, variability across speakers' voices and language backgrounds, and differences in the way words are pronounced. As a result, people rely on context, such as the other words in a sentence, to interpret the meaning of what they hear. This project will investigate how people understand spoken language in different contexts by studying brain responses to speech and developing computer models that recognize spoken words in context. The aim is to understand the ways that human listeners successfully communicate in a noisy world. The research may also provide insights into how computer systems designed to recognize speech can be made smarter by having them process language in ways similar to humans. This research will be integrated with the educational component of the project, which will provide students with training in advanced computational and neuroscience techniques that are vital to the STEM workforce.
The project addresses these issues by studying language comprehension in experiments using neuroscience techniques that reveal how speech is perceived by the brain in the first few hundred milliseconds of hearing a sound. By studying these early brain responses, the investigators will be able to identify what information the listener uses to distinguish spoken words and determine how these brain responses are affected by the listeners' expectation of which words they will hear. The investigators will also create neural network models that use techniques from machine learning to recognize speech and are trained in a way that mimics how children learn language. The behavior of the model will be compared with the data from human listeners to determine whether the model accurately captures the way the brain understands spoken language in different contexts. The techniques developed from this research will also be used in classroom and laboratory settings to train undergraduate and graduate students in the use of these approaches.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
