The rational control of aspiration in learning
-
1
University of Minnesota, Department of Computer Science and Engineering, United States
-
2
University of Minnesota, United States
One of the fundamental questions for any agent entering a novel environment is how to balance exploratory and exploitative actions. This is especially difficult when the total number of states, the number of rewarded states, and the distribution of rewards at rewarded states are initially unknown. How much to explore in an unknown environment is controlled by the agent’s beliefs about the value of unexperienced states, which we refer to as the agent’s "aspiration." To examine the effect of aspiration on exploratory behavior in humans, we used a well-known challenging test problem for exploration in reinforcement learning - the "chain game." Briefly, this task produces two reasonable policies - a small reward policy that requires little exploration to find and a large reward policy that requires more exploratory (and unrewarded) actions to find. Human strategies fell into two distinct groups; one group performed enough unrewarded exploratory actions to find the larger reward policy, while the second group under-explored and stuck with the low reward policy without ever experiencing the larger reward state. In debriefing, subjects in the former group reported finding the small local maxima quickly, but believed that higher rewards were possible and thus continued exploration. Conversely, subjects in the latter group typically reported an initial exploratory phase, but upon finding only the local maxima and otherwise only unrewarded states, determined that the optimal solution was to exploit the small local maxima. Using model-based Bayesian reinforcement-learning, an agent can be made to mimic either of these groups by manipulating the agent’s initial prior belief about the size of the state space and/or the magnitude of potential rewards. In particular, we model these prior beliefs in terms of hyperparameters of a hierarchical Dirichlet process prior on the entries in the state-action transition matrix. Interestingly, these hyperparameters can be learned from data. Given that aspiration is obviously critical in determining exploratory behavior, the question then arises: what factors determine aspiration in humans? One potential source of information is knowledge regarding the reward history of others. To test the effect of this type of knowledge on exploratory choice behavior, subjects were again placed within an environment with an easy-to-find low-reward policy and hard-to-find high-reward policy. After half the total trials (and convergence to the easy to find policy), a "high-score" sheet was shown to the subjects. One group was shown high-reward values and while the other group was shown low-reward values similar to their own score. While low-reward scores produced no change in behavior in the second half of the experiment, the group exposed to high-reward scores showed a complete reinitialization of exploration. Finally, we also tested whether subjects are capable of inferring aspiration directly from the statistics of the environment. Subjects played repeated games with similar reward structures (i.e., similar probabilities of states being rewarded, similar distributions on reward amount, etc). Choice behavior was extremely sensitive to these statistics (e.g. , in sparse environments with few rewarded states, subjects typically exploited the first rewarded state they encountered).
Conference:
Computational and Systems Neuroscience 2010, Salt Lake City, UT, United States, 25 Feb - 2 Mar, 2010.
Presentation Type:
Poster Presentation
Topic:
Poster session II
Citation:
Acuna
D,
Green
SC and
Schrater
P
(2010). The rational control of aspiration in learning.
Front. Neurosci.
Conference Abstract:
Computational and Systems Neuroscience 2010.
doi: 10.3389/conf.fnins.2010.03.00169
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
02 Mar 2010;
Published Online:
02 Mar 2010.
*
Correspondence:
Daniel Acuna, University of Minnesota, Department of Computer Science and Engineering, Minneapolis, United States, acuna002@umn.edu