The specification of goals in the form of utility or reward functions is a cornerstone of most approaches to developing autonomous artificial agents, and to understanding and shaping the behavior of natural biological agents - in fields ranging from control, AI, economics, psychology, and ethology. But in practice there are actually two notions of goals: the (human or evolutionary) designer's goals and the (artificial or natural) agent's goals. Should these be the same? The conventional (implicit) answer is "yes", but new work by the PIs shows that the answer may be "no" for computationally bounded agents. We define a new problem in agent design, the optimal rewards problem, whose solution is a reward function to assign to the agent so that in attempting to maximize its reward the agent best achieves the designer's goals. This project explores optimal rewards, through computational experimentation and analysis, on two broad fronts. We will develop new principles and algorithms for bounded planning agents, demonstrating increased performance over using conventional rewards, even when taking into account the cost of finding the optimal rewards. We will make significant advances in two areas of behavioral economics and ethology: the understanding of subjective utility in humans, and the understanding of foraging behavior in animals, by using optimal reward theory to rigorously derive subjective reward functions that take into account the computational limits of the natural agents. The results of this work, disseminated through published theory and software, should lead to foundational changes in the way we understand and design incentive structures for humans, animals, and artificial agents, and thus to significant practical benefits for any methods in engineering, education, economics, and health that depend on finding good incentives for desired behavioral outcomes.
For artificial agents to have autonomy their behavior must exhibit the sort of flexibility and adaptiveness exhibited by humans and other animals. One way to achieve this is for the human builders of such artificial agents to endow them with high-level goals as opposed to low-level detailed behaviors, provided these agents implement algorithms that can translate their high-level goals automatically into low-level behaviors that are adaptive to the specific circumstances of the agent. The field of Reinforcement Learning or RL offers a formal mathematical framework in which the high-level goals are represented as reward functions and provides a set of algorithms for translating reward functions into behavior when an RL-agent is placed in some environment. The research focus of this project was on the question: What reward functions should be given to an RL-agent? A starting point for the answer lies in an earlier optimization framework developed by the PIs in which the optimal reward function is one which when given to an RL-agent yields behavior that is optimal with respect to the human-designer’s preferences over the agent’s behavior. In this project, the PIs extended the optimal rewards framework in two essential ways. The first extension considered cases in which the RL-agent is to solve a sequence of tasks as expressed via a sequence of human-designer’s reward functions. A major contribution was to show that a useful reward-mapping function that maps the human-designer’s reward function to the agent’s reward function can be learned from experience. The reward-mapping function is useful in that it is of benefit to the human-designer to use it to automatically set the RL-agent’s reward function. The second extension considered multiagent settings in which two RL-agents have to cooperate and compete with each other. A major contribution was to show that automatically learned reward functions lead the two agent’s to specialize or differentiate their skills (via learning different reward functions) when that is useful in their environment and to learn similar skills (via learning the same reward functions) when that is better in their environment. In summary, research supported by this project significantly expanded our understanding of how reward functions should be designed for autonomous artificial agents.
