Emotion and motivation are fundamental to learning; students with high intrinsic motivation often outperform students with low motivation. Yet affect and emotion are often ignored or marginalized with respect to classroom practice. This project will help redress the emotion versus cognition imbalance. The researchers will develop Affective Learning Companions, real-time computational agents that infer emotions and leverage this knowledge to increase student performance. The goal is to determine the affective state of a student, at any point in time, and to provide appropriate support to improve student learning in the long term. Emotion recognition methods include using hardware sensors and machine learning software to identify a student's state. Five independent affective variables are targeted (frustration, motivation, self-confidence, boredom and fatigue) within a research platform consisting of four sensors (skin conductance glove, pressure mouse, face recognition camera and posture sensing devices). Emotion feedback methods include using a variety of interventions (encouraging comments, graphics of past performance) varied according to type (explanation, hints, worked examples) and timing (immediately following an answer, after some elapsed time). The interventions will be evaluated as to which best increase performance and in which contexts. Machine learning optimization algorithms search for policies that further engage individual students who are involved in different affective and cognitive states. Animated agents are enhanced with appropriate gestures and empathetic feedback in relation to student achievement level and task complexity. Approximately 500 ethnically and economically diverse students in Massachusetts and Arizona will participate.
The broader impact of this research is its potential for developing computer-based tutors that better address student diversity, including underrepresented minorities and disabled students. The solution proposed here provides alternative representations of scientific content, alternative paths through material and alternative means of interaction; thus, potentially leading to highly individualized science learning. Further, the project has the potential to advance our understanding of emotion as a predictor of individual differences in learning, unveiling the extent to which emotion, cognitive ability and gender impact different forms of learning.
Effective teachers generally address students’ emotional states and social backgrounds. Thus, for tutoring computational systems to interact more naturally and supportively with students, they need to provide an environment that not only recognizes affective behavior, but also effectively expresses socio-emotional capability to address challenges and fluctuations in individual affective states. While progress has been made on modeling students’ affect and advancing tutoring affective systems, most of the efforts had been focused on the cognitive rather than affective aspects of learning and evaluation of their impact in schools was still preliminary. The overall objective of this grant was to develop and evaluate computer agent systems that: (1) understand, at any point in time, the emotional state of a student through the use of unobtrusive sensors capable to measure emotional variations due to the interaction with the system; and (2) use that information to provide personalized support to improve student’s affective state and learning (see figure 1). Our studies incorporated an affective sensor platform into the Wayang Outpost multimedia intelligent tutor. This intelligent tutor has trained thousands of students to solve challenging geometry problems of the type that commonly appear on standardized tests and has demonstrated improved learning gains in state standard exams. By the end of this award we had manufactured and used in classrooms (with close to 2,000 students) thirty sets of sensors, which captured users’ physiological responses reflecting emotional natural responses to interactions with the given system. Each set of sensors included (figure 2): a pressure mouse, to detect the intensity of the user’s grip on the mouse; a bracelet to measure skin response (through micro-perspiration) as an indicator of arousal (e.g., frustration or engagement); a posture chair sensor; and a facial-expression camera, supplemented with software to measure head nod/shake, mouth fidgets, smiles, blink events, and pupil dilations. During the second year of this grant we focused on understanding student emotional state patterns to create models that were then validated and extended in the following years. During the third and fourth year we developed learning companions and performed studies with students to evaluate their impact on affective and cognitive outcomes. When testing the impact of the Wayang tutor system (learning companion), results showed that while all students demonstrated math learning after working with the system, low-achieving students learned more than high achieving students. Learning companions did not affect student learning directly, but did successfully induced positive student behaviors, which has been associated with learning. The beneficial effect of learning companions particularly improved confidence. Low-achieving students with learning companions improved their confidence more than students with out them, while and after using the affective tutor. It was also noted that the female (not male) companion was more accepted and improved liking and self-concept of math ability. However, learning companions did not manage to change some negative feelings and behaviors, e.g., low-achieving students did more quick-guessing and reported less interest than high-achieving students. When we compared the inclusion of the Wayang Tutor as an affective companion with a standard Math Facts Retrieval Training (MFR) in middle school students, we found that, despite limited exposure to the software (3 days), the Wayang Tutor effectively improved students’ performance on standardized test and specifically improved learning on hard questions that required increased number of steps and computation. And although MFR was highly effective at providing training irrespective to achieving level and gender, the system alone did not help students perform better at standardized test items, suggesting that this training should be supplemented with appropriate instruction on test topics. Finally we used a large data collection set of interactions between students and tutors to classify time-based student behavior based on combinations of student attributes, versions of tutors, problem topics, orders of presentation, selected hints, and times of student responses. Through visual analysis of patterns and mathematical projection algorithms, we identified distinct patterns associated with student behavior in the tutor, including the following. Game-like: not reading the problems and either skipping or making quick guesses. Frustration (guess and hints): guessing or using hints to find solution, then skipping to next problems. Not challenged: solving problems on first attempt, not using hints or guessing. Too difficult: taking time to read the problem, then using hints to find the answer. Skipping: skiping a number of problems then solving a number of problems. On task: mixture of guessing, solving problems
