The purpose of this project is to develop computational and brain-based models of emotion cue integration: people?s inferences about others? emotions based on dynamic, multimodal cues. Observers often decide how targets feel based on cues such as facial expressions, prosody, and language. Such inferences scaffold healthy social interaction, and abnormal inference both marks and exacerbates social deficits in numerous psychiatric disorders. Psychologists and neuroscientists have studied emotion inference for decades, but the vast majority of this work employs simplified social cues, such as vignettes or static images of faces. By contrast, ?real world? emotion cues are complex, dynamic, and multimodal. Cue integration?inference based on naturalistic emotion information?likely differs from simpler inference at cognitive and neural levels, but this phenomenon remains poorly understood. This means that scientists lack a clear model of how observers adaptively process complex emotion cues, and how that processing goes awry in mental illness. Especially lacking are mechanistic models that can describe the computations and brain processes involved in cue integration with sufficient precision to predict inference in new cases, observers, and samples. This project will merge tools from social psychology, computer science, and neuroscience to generate a novel and rigorous model of emotion cue integration. We have demonstrated that in the face of complex emotion cues, observers dynamically ?weight? cues from each modality (e.g., visual, linguistic) over time, a process that (i) tracks shifts in brain activity and connectivity; and (ii) can be captured using Bayesian models. Here, we will expand this work in several ways. First, we will develop precise computational tools to isolate features of emotion cues?such as facial movements, prosody, and linguistic sentiment?that track observers? use of each cue modality during integration. Second, we will develop multi-region ?signatures? of brain activity and connectivity that track emotion inference in each modality. We will use these signatures in conjunction with machine learning to predict unimodal emotion inference and cue integration in new observers and samples, based on brain data alone. Third, we will explore the context-dependence of naturalistic emotion inference by testing whether reinforcement learning can bias observers? cue integration and accompanying brain signatures. Finally, we will model computational and neural abnormalities associated with cue integration in patients with Major Depressive Disorder and Bipolar Disorder. At the level of basic science, these data will generate a fundamentally new?and more naturalistic?approach to the neuroscience of emotion inference. The computational and brain metrics we produce will also be made publically available to facilitate the open and cumulative study of emotion inference across labs. At a translational level, we will provide a mechanistic, rich account of abnormal emotion inference in mood disorders, paving the way for computational and brain markers that can be used to assess social dysfunction and treatment efficacy in these and other mental illnesses.
The proposed research will use methods from social psychology, cognitive neuroscience, and computer science to (i) precisely model people?s inferences about others? emotions based on complex, dynamic cues, (ii) generate multi-region, brain-based predictors of these inferences, and (iii) characterize abnormalities in inference among individuals with mood disorders. Several psychiatric and neurodevelopmental disorders are characterized by difficulties understanding others? emotions, which in turn worsen social functioning in patients. In addition to providing new insights about social processes?a core target within the NIMH?s research domain criteria (RDoC) framework?the proposed research will offer powerful, novel computational and neural targets through which to assess and treat difficulties in emotion inference, and to eventually reduce the social burden faced by people with mental illness on a broad scale.
