Open-ended Learning: a Conceptual Framework based on Representational Redescription
- 1Sorbonne Universités, France
- 2Computer Science and Systems Engineering Laboratory (U2IS), ENSTA ParisTech, France
- 3University of Edinburgh, United Kingdom
- 4University of A Coruña, Spain
- 5VU University Amsterdam, Netherlands
Reinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL.
But in an open-ended learning process, an agent or robot must solve an unbounded sequence of tasks that are not known in advance and the corresponding MDPs cannot be built at design time. This defines the main challenges of open-ended learning: how can the agent learn how to behave appropriately when the adequate states, actions and rewards representations are not given?
In this paper, we propose a conceptual framework to address this question. We assume an agent endowed with low-level perception and action capabilities. This agent receives an external reward when it faces a task. It must discover the state and action representations that will let it cast the tasks as MDPs in order to solve them by RL. The relevance of the action or state representation is critical for the agent to learn efficiently. Considering that the agent starts with a low level, task-agnostic state and action spaces based on its low-level perception and action capabilities, we describe open-ended learning as the challenge of building the adequate representation of states and actions, i.e. of redescribing available representations. We suggest an iterative approach to this problem based on several successive Representational Redescription processes, and highlight the corresponding challenges in which intrinsic motivations play a key role.
Keywords: developmental robotics, reinforcement learning, state representation learning, representational redescription, actions and goals, Skills
Received: 28 Apr 2018;
Accepted: 28 Aug 2018.
Edited by:Andrew Barto, University of Massachusetts Amherst, United States
Reviewed by:Robert J. Lowe, University of Gothenburg, Sweden
Georg Martius, Max-Planck-Institut für Intelligente Systeme, Germany
Eiji Uchibe, Advanced Telecommunications Research Institute International (ATR), Japan
Copyright: © 2018 Doncieux, Filliat, Diaz-Rodriguez, Hospedales, Duro, Coninx, Roijers, Girard, Perrin and Sigaud. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
* Correspondence: Prof. Stephane Doncieux, Sorbonne Universités, Paris, France, email@example.com