Humans are unendingly curious; we spontaneously explore and manipulate our surroundings to see what we can make them do; we obtain enjoyment from making discoveries and for making things happen. We often engage in these activities for their own sakes rather than as steps toward solving practical problems. Psychologists call these intrinsically motivated behaviors because rewards are intrinsic in these activities instead of being due to the satisfaction of more primary biological needs. But what we learn during intrinsically motivated behavior is essential for our development as competent autonomous entities able to efficiently solve a wide range of practical problems as they arise. This project's objective is to develop a computational model of intrinsically motivated learning that will allow artificial agents to construct and extend hierarchies of reusable skills that are needed for competent autonomy. This project builds on existing research in machine learning, recent advances in the neuroscience of brain reward systems, and classical and contemporary psychological theories of motivation. At the core of the model are recent theoretical and algorithmic advances in computational reinforcement learning, specifically, new concepts related to skills and new learning algorithms for learning with skill hierarchies. The project develops a mathematical framework, implements the model in a series of simulated agents, and demonstrates the advances this will make possible in a series of increasingly complex environments. Intellectual Merit-Machine learning methods have become much more powerful in recent years. Despite these advances and their utility, today's learning algorithms fall far short of the possibilities for machine learning. They are typically applied to single, isolated problems for each of which they have to be hand-tuned and for which training data sets have to be carefully prepared. They do not have the generative capacity required to significantly extend their abilities beyond initially built-in representations. They do not address many of the reasons that learning is so useful in allowing animals to cope with new problems as they arise over extended periods of time. Success in this project will provide a fundamental advance in machine learning and move the field in a new direction. Although computational study related to intrinsic motivation is not entirely new, it is currently underdeveloped and does not take advantage of the highly relevant recent advances in the field of computational reinforcement learning and in the neuroscience of brain reward and motivation systems. Furthermore, computational studies do not take advantage of psychological theories of play, curiosity, surprise, and other factors involved in intrinsically motivated learning. This project addresses these shortcoming by taking an explicitly interdisciplinary approach. Broader Impacts-The new methods promise to improve our ability to control the behavior of complex systems in ways that will benefit society. Machine learning algorithms have been instrumental in a wide variety of applications in such areas as bioinformatics, manufacturing, communications, robotics, and security systems. It is important strategically, economically, and intellectually to increase the power of machine learning technologies as rapidly as possible. This project attempts to address some of these challenges. This project will strengthen interdisciplinary ties between the machine learning community of computer science and various disciplines devoted to the study of human cognitive development and education. The specific methods of concern in the proposed research have not yet been integrated. There has been very little cross-fertilization between the psychological study of intrinsic motivation and machine learning. The proposed research will remedy this situation, thereby helping to create an avenue of communication that can foster future developments in both fields. The project has the potential to contribute to our understanding of general principles underlying human cognitive development, with implications for education, where enhancing intrinsic motivation is a key factor. The educational component of the project focuses on graduate education through its training of graduate students. This includes the offering interdisciplinary graduate-level seminars at both U. of Massachusetts and U. of Michigan, to be taught by the PIs on the topic of intrinsically motivated learning. In its recruitment of graduate students, the project will take advantage of the role that U. Massachusetts plays as the lead institution in the NSF funded Northeast Alliance, which supports and mentors underrepresented minority students interested in academic careers in a science, mathematics, or engineering discipline. At U. of Michigan special effort will be made to recruit and involve undergraduates in student projects leading to summer projects funded by the Marian Sarah Parker Scholars Program, which targets female undergraduates and provides funds for summer research opportunities.
