The PI's goal in this project is to enable a quantum leap towards next-generation social assistive aids that enrich the lived social experiences of individuals with visual impairments. Social interaction is a central component of the human experience. The ability to communicate effectively with fellow individuals is a fundamental necessity for professional success as well as personal fulfillment. But nonverbal cues (including prosody, environment attributes, the appearance of communicators and their physical movements) account for a substantial and important part of the information conveyed during social interactions. As a consequence, the more than 1.3 million individuals in the United States (and 37 million worldwide) who are legally blind have only a limited experience of social interaction. This "social disability" often isolates them from their social environments. Existing assistive technologies are focused on problems such as navigation, reading text, and access to everyday appliances as well as to computers and the Internet, whereas little or no work has been devoted to real-time accessibility to social and behavioral cues. Providing real-time access to nonverbal communication cues to visually impaired users poses fundamental challenges in several related fields including affective computing, human communication engineering, behavioral modeling, machine learning, human-machine interaction, multimodal interfaces, usability engineering, multimedia computing, and assistive technology design and development
As a first step towards practical and viable social assistive solutions, the PI will focus in this project on the design and development of a social situational awareness assistive prototype for dyadic (one-on-one) interactions, with an emphasis on head/face-based nonverbal cues. The research will be accomplished through the following specific objectives: (1) Design and development of a dyadic interpersonal mediation interface; (2) Extraction and understanding of nonverbal communication cues; (3) Visuo-haptic sensory substitution for delivering high-bandwidth socio-behavioral data; and (4) Evaluation of the social assistive prototype system in dyadic interaction scenarios representing real-world conditions. The project draws upon intellectual synergies among the team members, who are experts in human-centered multimedia computing, human-computer interfaces and machine intelligence (Panchanathan, Computer Science); assistive technology design and usability engineering (Hedgpeth, Disability Resources Center); and human communication modeling and socio-behavioral analysis (Ramirez, Human Communication). The work will build on the team's past successes in developing assistive technologies that have been designed, prototyped, deployed and tested for individuals who are visually impaired. Project outcomes will be evaluated through the Arizona State University Disability Resource Center and the Arizona Center for the Blind and Visually Impaired.
Broader Impacts: This project will pioneer the development of next-generation social assistive aids for individuals with visual impairments and thus will have a significant impact on their lives. The research to these ends will result in the advancement of computational thinking within and at the confluence of the component disciplines, namely socio-behavioral computing (through the introduction of novel methodologies for computational analysis and evaluation of human communication dynamics in general and social behavior in dyadic interactions, specifically), human-computer interfaces (through the design of novel interfaces that deliver high-bandwidth social data), machine intelligence (through the study of algorithms that elicit various levels of interaction semantics) and assistive technology/usability (through the development and evaluation of social assistive prototypes). The concepts and technologies developed will also provide pathways to technologies for individuals with other disabilities, such as autism, dementia, and (in the most general sense) a very large portion of society. The methodologies developed will provide a wealth of data and information that will be made publicly accessible to promote and catalyze further research in the component disciplines.
