This project investigates how neural representations of speech and music in the cortex can be adapted and applied to overcome the challenge of robust perception in extremely noisy and cluttered environments, mimicking processing and capabilities of the brain. More specifically, the project will formulate algorithms inspired by the architecture of the brain to segregate and track targeted speakers or sound sources, test their performance, and relate them to state-of-the-art approaches that utilize deep artificial neural networks to accomplish these tasks. Human psychoacoustic and physiological experiments with these algorithms will be conducted to test the validity of these ideas for mimicking human abilities. This effort will spur the development of new neuromorphic computational tools modeled after the brain and its cognitive functions. In turn, these will provide a theoretical framework to guide future experiments into how complex cognitive functions originate and how they influence sensory perception and lead to robust behavioral performance.
The planned projects will be organized into two flavors. The first attempts to borrow from existing neuromorphic approaches that rely on cortical representations to develop new embeddings within the deep neural networks framework, which will in turn endow the latter with brain-like robustness in challenging unanticipated environments. Three specific efforts within this flavor will be conducted: Learning DNN embeddings using cortical representations of speech and music, exploring unsupervised clustering of cortical features using adversarial auto-encoders, and exploiting pitch and timbre representations to enhance segregation of sound. The second flavor of projects borrows from the DNN approach to build into neuromorphic algorithms the desirable performance and flexibility attained by training on available databases. Two broad areas of studies are planned: one focuses on questions of neuromorphic implementations that benefit from DNN toolboxes and ideas, especially in segregation and reconstruction. The other focuses on investigating how autoencoders can be exploited to implement feature reduction and clustering efficiently.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
