A Reinforcement learning model develops causal inference and cue integration abilities
-
1
Frankfurt Institute for Advanced Studies, Germany
-
2
Honda Research Institute Europe GmbH, Germany
In recent years it has been suggested that the performance of human subjects in a large variety of perceptual tasks can be modelled using Bayesian inference (e.g. [1]). The success of these methods stems from their capacity to explicitly represent the involved uncertainties. Recently, such methods have been extended to the task of model selection where the observer not only has to integrate different cues into a single estimate, but needs to first select which causal model best describes the stimuli [2]. As an example, consider the task of orienting towards a putative object. The stimuli consist of an auditory and a visual cue. Depending on the spatial distance between the position measurements provided by the two modalities it is more probable to assume that the signals originated from the same source or from two different sources. An open problem in this area is how the brain acquires the required models and how it learns to perform the proper kind of inference. Since infants and young children have been shown not to integrate cues initially [3,4], it seems likely that extended learning processes play an important role in our developing ability to integrate cues and select appropriate models.
In the present study we investigate whether the framework of reinforcement learning (RL) could be used to study these questions. A one-dimensional version of an orienting task is considered, in which an auditory and a visual cue are placed at either the same or different positions. Each cue is corrupted by Gaussian noise with the variance of the auditory noise being larger than that of the visual, reflecting the different uncertainties in the sensory modalities. A positive reward is given if the agent orients to the true position of the object. In case the orienting movement does not target the object, we assume that an additional movement has to be carried out. The cost for each additional movement is proportional to the distance between the current position and the true position of the target. The action selection of the agent is probabilistic, using the softmax rule. Learning takes place using the SARSA algorithm [5].
The simulations show that the reinforcement learning agent is indeed capable of learning to integrate cues taking their relative reliabilities into account when this interpretation leads to a better detection of the target. Furthermore, the agent learns that if the position estimates provided by the two modalities are too far apart, it is better not to integrate the two signals but to select an action that only considers the cue with higher reliability. The displayed behaviour therefore implicitly corresponds to selection of different causal models. Our results suggest that generic reinforcement learning processes may contribute to the development of the ability to integrate different sensory cues and select causal models.
References
1. Knill&Pouget 2004, TiNS 27(12), 712-19
2. Kording et al. 2007, PloS One 2(9)
3. Nardini et al. 2008, Current Biology 18(9), 689-93
4. Gori et al. 2008, Current Biology 18(9), 694-98
5. Rummery&Niranjan 1994, Tech Rep
Conference:
Bernstein Conference on Computational Neuroscience, Frankfurt am Main, Germany, 30 Sep - 2 Oct, 2009.
Presentation Type:
Poster Presentation
Topic:
Learning and plasticity
Citation:
Weisswange
T,
Rothkopf
C,
Rodemann
T and
Triesch
J
(2009). A Reinforcement learning model develops causal inference and cue integration abilities.
Front. Comput. Neurosci.
Conference Abstract:
Bernstein Conference on Computational Neuroscience.
doi: 10.3389/conf.neuro.10.2009.14.151
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
28 Aug 2009;
Published Online:
28 Aug 2009.
*
Correspondence:
Thomas Weisswange, Frankfurt Institute for Advanced Studies, Frankfurt, Germany, thomas.weisswange@honda-ri.de