This project helps speakers and teachers to measure and improve their impact on their audiences. It uses visual observations of body, head, and hand gestures of the communicator, plus recordings of brain activity and eye movements of the audience. Together, these determine which sections of a presentation elicit the most audience engagement. The project is developing new methods to capture and calibrate electroencephalogram and eye-tracking data from listeners and from students. It is determining new ways to relate this subject information to what a speaker or teacher can be seen to be doing while developing an argument or reviewing a concept. The project produces analyses of when and how the communicator is most effective. This system is being ported to the Columbia Video Network distance education facility, for their use in improving the online delivery of Columbia University Master's level technical courses. This project continues a research effort that has involved women, minorities, disabled students, and undergrads.
This research investigates the degree to which certain speaker gestures can convey significant information that are correlated to audience engagement, in speeches and in classroom lectures. The project develops and validates a catalog of gestural attributes derived from pose and movements of body, head, and hand, and automatically extracts these attributes from videos. It demonstrates correlations between gesture attributes and an objective method of measuring audience engagement: electroencephalography (EEG). The project leverages a multi-disciplinary approach, with neural engineers and computer/media scientists collaborating to build a system that identifies and tracks physiological measures of engagement, and relates these to features in the video as well as information content. It records subjects' high-density EEG, and tracks their eyes and pupillary responses while they are watching video lectures. It uses machine learning, specifically novel methods which expand upon canonical correlation analysis, to relate inter- and intra-subject correlations, between the physiological changes and the gestural features derived from the video by using enhanced computer vision techniques. These measures are further integrated with pupillary measures, which have been shown to correlate with arousal, as well as with gaze measures, which are indicative of attention. The project is producing an analysis of body, head, and hand gestures useful in persuasion and in education, and a catalog of their influence on engagement and speaker effectiveness.
