The PI's objective is to explore a novel way of searching large video data collections semi-automatically for instances of people expressing emotions, by engaging the user's intelligence in the search process in conjunction with an automated search assistant (agent) that would not only perform searches but also help the user phrase queries and define new search concepts. To lay the groundwork for development of such a high-level system that would support searching for complex concepts in digital video libraries beyond the reach of current search technologies, the PI will focus in this exploratory study on running extensive user studies to elicit information on how to categorize and define emotions such as happiness or fear in terms of observable features of a video scene (e.g., facial expression, pitch of voice, gestures, etc). Because considerable research has been done in the domain of expressing emotions in faces, the PI will begin by exploring the use of facial features for describing emotional states. Later on, he will investigate other aspects of human behavior relevant to emotion, such as physical motion, as well as emotions in speech, in an effort to develop an extensible vocabulary.
Broader Impacts: This research has the potential to play a key role in developing a foundation for searching abstract concepts such as emotional states in large video data collections and digital libraries, which could have profound and broad impact on numerous application domains.
