The goal of this research is to advance wearable sound sensing and feedback technology for users who are deaf or hard of hearing, along with appropriate techniques for human-computer interaction. A wearable device with multiple microphones and audio processing algorithms to automatically sense, localize, and identify sounds will be developed. Means to discreetly provide this information to the user via emerging wearable technologies such as head-mounted displays and smartwatches will also be implemented. Evaluations will include lab and field studies, everyday tasks such as noticing sounds and participating in oral conversations, and objective and subjective measures. Project outcomes will have broad impact by enabling new sound awareness options for users who are deaf or hard of hearing, thereby augmenting the wearer's existing strategies with additional, unobtrusive information. The new assistive technologies will have the potential to improve the lives of a large portion of the population, in particular the growing number of older adults with hearing loss in the United States.
To these ends, major subgoals will include: understanding user needs for wearable sound sensing and feedback, including prioritizing the importance of different sounds across a variety of contexts and based on an individual user's level of hearing loss; developing and evaluating a lightweight wearable sound sensing platform and accompanying algorithms, including both new sound scene analysis algorithms for a microphone array conformal on, or in proximity to, a complex-shaped baffle (the wearer's head) and that take into account how sound scatters off the wearer's body, along with adaptive state-of-the-art sound classification and speech recognition approaches to work with this processed audio; and developing and evaluating visual and haptic or vibrational feedback of the sensed sound via wearable prototypes.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
