Open-Ended Learning: A Conceptual Framework Based on Representational Redescription
Reinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL. But in an op...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2018-09-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/article/10.3389/fnbot.2018.00059/full |
| _version_ | 1818499863691657216 |
|---|---|
| author | Stephane Doncieux David Filliat Natalia Díaz-Rodríguez Timothy Hospedales Richard Duro Alexandre Coninx Diederik M. Roijers Benoît Girard Nicolas Perrin Olivier Sigaud |
| author_facet | Stephane Doncieux David Filliat Natalia Díaz-Rodríguez Timothy Hospedales Richard Duro Alexandre Coninx Diederik M. Roijers Benoît Girard Nicolas Perrin Olivier Sigaud |
| author_sort | Stephane Doncieux |
| collection | DOAJ |
| description | Reinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL. But in an open-ended learning process, an agent or robot must solve an unbounded sequence of tasks that are not known in advance and the corresponding MDPs cannot be built at design time. This defines the main challenges of open-ended learning: how can the agent learn how to behave appropriately when the adequate states, actions and rewards representations are not given? In this paper, we propose a conceptual framework to address this question. We assume an agent endowed with low-level perception and action capabilities. This agent receives an external reward when it faces a task. It must discover the state and action representations that will let it cast the tasks as MDPs in order to solve them by RL. The relevance of the action or state representation is critical for the agent to learn efficiently. Considering that the agent starts with a low level, task-agnostic state and action spaces based on its low-level perception and action capabilities, we describe open-ended learning as the challenge of building the adequate representation of states and actions, i.e., of redescribing available representations. We suggest an iterative approach to this problem based on several successive Representational Redescription processes, and highlight the corresponding challenges in which intrinsic motivations play a key role. |
| first_indexed | 2024-12-10T20:35:24Z |
| format | Article |
| id | doaj.art-64fcc7e345784b03a0f7d0cafb347c38 |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-12-10T20:35:24Z |
| publishDate | 2018-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-64fcc7e345784b03a0f7d0cafb347c382022-12-22T01:34:33ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182018-09-011210.3389/fnbot.2018.00059390268Open-Ended Learning: A Conceptual Framework Based on Representational RedescriptionStephane Doncieux0David Filliat1Natalia Díaz-Rodríguez2Timothy Hospedales3Richard Duro4Alexandre Coninx5Diederik M. Roijers6Benoît Girard7Nicolas Perrin8Olivier Sigaud9Sorbonne Université, CNRS, ISIR, Paris, FranceU2IS, INRIA Flowers, ENSTA ParisTech, Palaiseau, FranceU2IS, INRIA Flowers, ENSTA ParisTech, Palaiseau, FranceInstitute of Perception, Action and Behaviour, University of Edinburgh, Edinburgh, United KingdomGII, Universidade da Coruña, A Coruña, SpainSorbonne Université, CNRS, ISIR, Paris, FranceDepartment of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, NetherlandsSorbonne Université, CNRS, ISIR, Paris, FranceSorbonne Université, CNRS, ISIR, Paris, FranceSorbonne Université, CNRS, ISIR, Paris, FranceReinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL. But in an open-ended learning process, an agent or robot must solve an unbounded sequence of tasks that are not known in advance and the corresponding MDPs cannot be built at design time. This defines the main challenges of open-ended learning: how can the agent learn how to behave appropriately when the adequate states, actions and rewards representations are not given? In this paper, we propose a conceptual framework to address this question. We assume an agent endowed with low-level perception and action capabilities. This agent receives an external reward when it faces a task. It must discover the state and action representations that will let it cast the tasks as MDPs in order to solve them by RL. The relevance of the action or state representation is critical for the agent to learn efficiently. Considering that the agent starts with a low level, task-agnostic state and action spaces based on its low-level perception and action capabilities, we describe open-ended learning as the challenge of building the adequate representation of states and actions, i.e., of redescribing available representations. We suggest an iterative approach to this problem based on several successive Representational Redescription processes, and highlight the corresponding challenges in which intrinsic motivations play a key role.https://www.frontiersin.org/article/10.3389/fnbot.2018.00059/fulldevelopmental roboticsreinforcement learningstate representation learningrepresentational redescriptionactions and goalsskills |
| spellingShingle | Stephane Doncieux David Filliat Natalia Díaz-Rodríguez Timothy Hospedales Richard Duro Alexandre Coninx Diederik M. Roijers Benoît Girard Nicolas Perrin Olivier Sigaud Open-Ended Learning: A Conceptual Framework Based on Representational Redescription Frontiers in Neurorobotics developmental robotics reinforcement learning state representation learning representational redescription actions and goals skills |
| title | Open-Ended Learning: A Conceptual Framework Based on Representational Redescription |
| title_full | Open-Ended Learning: A Conceptual Framework Based on Representational Redescription |
| title_fullStr | Open-Ended Learning: A Conceptual Framework Based on Representational Redescription |
| title_full_unstemmed | Open-Ended Learning: A Conceptual Framework Based on Representational Redescription |
| title_short | Open-Ended Learning: A Conceptual Framework Based on Representational Redescription |
| title_sort | open ended learning a conceptual framework based on representational redescription |
| topic | developmental robotics reinforcement learning state representation learning representational redescription actions and goals skills |
| url | https://www.frontiersin.org/article/10.3389/fnbot.2018.00059/full |
| work_keys_str_mv | AT stephanedoncieux openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT davidfilliat openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT nataliadiazrodriguez openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT timothyhospedales openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT richardduro openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT alexandreconinx openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT diederikmroijers openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT benoitgirard openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT nicolasperrin openendedlearningaconceptualframeworkbasedonrepresentationalredescription AT oliviersigaud openendedlearningaconceptualframeworkbasedonrepresentationalredescription |