Narrative generation is the process of generating textual descriptions of action in dynamic environments such as movies, sports events and educational programs. On-line narration of a dynamic environment is beneficial in a wide range of contexts, from entertainment to training and education. For example, successfully narrating a video would allow blind and visually-impaired people to follow visual cues that are important to understanding the video. Key to attaining this goal is the ability to translate natural language into a form that is understandable by computers. A particular challenge is to do this not for a specifically chosen domain, but in a general way that is suitable for adaptation to a wide range of natural dynamic environments. This project explores new directions to tackle these extremely challenging, yet crucial, issues, undertaking exploratory research towards building essential components of a domain-adaptive framework that learns to understand and generate narratives on-line for natural dynamic environments with minimal supervision by human experts. This research explores methods to generate narratives on-line by learning the natural dynamics of the environment, automatically forming templates, and deciding when and what to mention.
Many natural language applications are concerned with recognition of paraphrases and semantic understanding. The software and data resulting from this project are potentially useful for semantic analysis in natural language processing, and is being made available for research purposes. This work is designed for significant social impact through a broad range of applications including educational, entertainment, and accessibility. A narrative generation system could be beneficial to visually-impaired people to better understand videos over the internet. In addition, such a system can help broadcasting companies to report news or sports events with customized commentaries for different users. This project also provides research and collaborative work experience to undergraduate and graduate students including under-represented and minority groups.
