This project develops a unified multimodal and multialgorithm fusion framework to recognize facial action units, which describe complex and rich facial behaviors. The information from voice is incorporated with visual observations to effectively improve facial activity understanding since voice and facial activity are intrinsically correlated. The developed framework systematically captures the inherent interactions between the visual and audio channels in a global context of human perception of facial behavior. Advanced machine learning techniques are developed to integrate these relationships together with uncertainties associated with various visual and audio measurements in the fusion framework to achieve a robust and accurate understanding of facial activity. It is these coordinated and consistent interactions that produce a meaningful facial display.
The research work from this project fosters computer vision and machine learning technologies with applications across a wide range of fields varying from psychiatry to human-computer interaction. The new audiovisual emotional database constructed in this research facilitates benchmark evaluations and promotes new research directions, especially, in human behavior analysis. An integration of research and education promotes cutting-edge training on human-computer interactions to K-12, undergraduate, and graduate students, especially encourages the participation of women in engineering and computing.
