What aspects of previous experiences guide decisions? Much research concerns how the brain computes the average, over many experiences, of rewards received for an option. But such a summary - produced by prominent models of dopaminergic incremental learning- is chiefly useful for repetitive tasks. Much less is understood about how the brain can flexibly evaluate new or changing options in more realistic tasks, which must rely on less aggregated information. This application argues that this is fundamentally a function of memory, so this project looks to the brain's memories for the most individuated experiences - episodes - to seek new computational, cognitive and neural mechanisms that could support more flexible decisions. The overarching hypothesis is that episodic memory, supported by the hippocampus, plays a central role in guiding flexible decision making and complements the wellknown role of dopaminergic and striatal systems in incremental learning of value. What is the intellectual merit of the proposed activity? By connecting the computational neuroscience of decision making with the cognitive neuroscience of memory, and bringing together collaborators from each area, this project promises to shed light on both areas. This is because the neural mechanisms supporting episodic memory are well studied, but less so their contribution to adaptive behavior. Computationally, episodic memories can support a family of learning algorithms that draw on sparse, individual experiences, such as Monte Carlo and kernel methods. These suggest novel, plausible hypotheses for how the brain solves more realistic decision problems, and in particular how it implements goal-directed or model-based choices. The proposed studies aim to differentiate the contributions of incremental and episodic learning to value-based decisions, and test to what extent episodic memories contribute to decisions previously identified as model-based. Our hypotheses are tested fitting computational models to neural activity from functional MRI experiments in humans, and also to choice behavior in healthy individuals compared to patients with isolated damage to specific neural systems. This combination of computational, neuroimaging and neuropsychological approaches permits finely tracing the trial-by-trial dynamics of learning as reflected both in brain activity nd behavior, and also testing the causal role of particular brain regions in these same processes. What are the broader impacts of the proposed activity? A striking range of psychiatric and neurological disorders, including Parkinson's disease, schizophrenia and eating disorders, are accompanied by aberrant decision-making and by dysfunction in circuitry central to this proposal, such as striatal and fronto-temporal mechanisms. But understanding such dysfunction requires a better understanding of how each of these circuits separately influences decisions. A focus on untangling multiple decision systems is particularly pertinent to disorders such as drug abuse, which is hypothesized to center on the compromise of incremental reinforcement mechanisms that may support more habitual actions and underlie the compulsive nature of such diseases. At the same time, drugs may also weaken or compromise more deliberative or goal-directed choice systems that might otherwise be able to support more advantageous decisions. Formally understanding the roles played by both of these influences, and how they interact, promises to improve the conceptualization, diagnosis, and treatment of these and other disorders. The proposed program also provides unique opportunities for training and education. By integrating multiple core tools of systems and cognitive neuroscience (computational modeling, functional imaging, patient studies, behavioral analyses), students in the labs of both PIs are trained in different approaches to a unified research question, preparing them to be effective scientists in a more interdisciplinary future. Components of this training will also be extended to undergraduate and high school student populations through existing programs at both NYU and at Columbia and through outreach to New York area schools. This project will also help promote broader representation of minorities in science, including women. As a female neuroscientist with many women trainees in her laboratory, PI Shohamy serves as a role model and the collaborative project facilitates training for women in computational neuroscience, an area in which women are particularly underrepresented.
Drug abuse is acquired through learning. Much research focuses on the involvement of dopaminergic incremental learning processes. But other learning and memory systems are likely to also contribute to aspects, such as drug-seeking, not well explained by those mechanisms. By isolating the contribution of different memory systems to learned choice, this project may help to guide future treatment strategies.
| Braun, Erin Kendall; Wimmer, G Elliott; Shohamy, Daphna (2018) Retroactive and graded prioritization of memory by reward. Nat Commun 9:4886 |
| Gerraty, Raphael T; Davidow, Juliet Y; Foerde, Karin et al. (2018) Dynamic flexibility in striatal-cortical circuits supports reinforcement learning. J Neurosci : |
| Mattar, Marcelo G; Daw, Nathaniel D (2018) Prioritized memory access explains planning and hippocampal replay. Nat Neurosci 21:1609-1617 |
| Duncan, Katherine; Doll, Bradley B; Daw, Nathaniel D et al. (2018) More Than the Sum of Its Parts: A Role for the Hippocampus in Configural Reinforcement Learning. Neuron 98:645-657.e6 |
| Bornstein, Aaron M; Khaw, Mel W; Shohamy, Daphna et al. (2017) Reminders of past choices bias decisions for reward in humans. Nat Commun 8:15958 |
| Gershman, Samuel J; Daw, Nathaniel D (2017) Reinforcement Learning and Episodic Memory in Humans and Animals: An Integrative Framework. Annu Rev Psychol 68:101-128 |
| Duncan, Katherine D; Shohamy, Daphna (2016) Memory states influence value-based decisions. J Exp Psychol Gen 145:1420-1426 |
| Sharp, Madeleine E; Foerde, Karin; Daw, Nathaniel D et al. (2016) Dopamine selectively remediates 'model-based' reward learning: a computational approach. Brain 139:355-64 |
| Shadlen, Michael N; Shohamy, Daphna (2016) Decision Making and Sequential Sampling from Memory. Neuron 90:927-39 |
| Decker, Johannes H; Otto, A Ross; Daw, Nathaniel D et al. (2016) From Creatures of Habit to Goal-Directed Learners: Tracking the Developmental Emergence of Model-Based Reinforcement Learning. Psychol Sci 27:848-58 |
