Goal modelling for deep reinforcement learning agents
Goals provide a high-level abstraction of an agent’s objectives and guide its behavior in complex environments. As agents become more intelligent, it is necessary to ensure that the agent’s goals are aligned with the goals of the agent designers to avoid unexpected or unwanted agent behavior. In thi...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Conference Paper |
| Language: | English |
| Published: |
2022
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/156966 |
| _version_ | 1824454675667091456 |
|---|---|
| author | Leung, Jonathan Shen, Zhiqi Zeng, Zhiwei Miao, Chunyan |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Leung, Jonathan Shen, Zhiqi Zeng, Zhiwei Miao, Chunyan |
| author_sort | Leung, Jonathan |
| collection | NTU |
| description | Goals provide a high-level abstraction of an agent’s objectives and guide its behavior in complex environments. As agents become more intelligent, it is necessary to ensure that the agent’s goals are aligned with the goals of the agent designers to avoid unexpected or unwanted agent behavior. In this work, we propose using Goal Net, a goal-oriented agent modelling methodology, as a way for agent designers to incorporate their prior knowledge regarding the subgoals an agent needs to achieve in order to accomplish an overall goal. This knowledge is used to guide the agent’s learning process to train it to achieve goals in dynamic environments where its goal may change between episodes. We propose a model that integrates a Goal Net model and hierarchical reinforcement learning. A high-level goal selection policy selects goals according to a given Goal Net model and a low-level action selection policy selects actions based on the selected goal, both of which use deep neural networks to enable learning in complex, high-dimensional environments. The experiments demonstrate that our method is more sample efficient and can obtain higher average rewards than other related methods that incorporate prior human knowledge in similar ways. |
| first_indexed | 2025-02-19T03:26:05Z |
| format | Conference Paper |
| id | ntu-10356/156966 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:26:05Z |
| publishDate | 2022 |
| record_format | dspace |
| spelling | ntu-10356/1569662022-05-12T00:50:31Z Goal modelling for deep reinforcement learning agents Leung, Jonathan Shen, Zhiqi Zeng, Zhiwei Miao, Chunyan School of Computer Science and Engineering Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2021) Engineering::Computer science and engineering Deep Reinforcement Learning Hierarchical Reinforcement Learning Goals provide a high-level abstraction of an agent’s objectives and guide its behavior in complex environments. As agents become more intelligent, it is necessary to ensure that the agent’s goals are aligned with the goals of the agent designers to avoid unexpected or unwanted agent behavior. In this work, we propose using Goal Net, a goal-oriented agent modelling methodology, as a way for agent designers to incorporate their prior knowledge regarding the subgoals an agent needs to achieve in order to accomplish an overall goal. This knowledge is used to guide the agent’s learning process to train it to achieve goals in dynamic environments where its goal may change between episodes. We propose a model that integrates a Goal Net model and hierarchical reinforcement learning. A high-level goal selection policy selects goals according to a given Goal Net model and a low-level action selection policy selects actions based on the selected goal, both of which use deep neural networks to enable learning in complex, high-dimensional environments. The experiments demonstrate that our method is more sample efficient and can obtain higher average rewards than other related methods that incorporate prior human knowledge in similar ways. National Research Foundation (NRF) Submitted/Accepted version This research is supported, in part, by the National Research Foundation, Prime Minister’s Office, Singapore under its NRF Investigatorship Programme (NRF Award No. NRF-NRFI05-2019-0002). 2022-05-12T00:48:28Z 2022-05-12T00:48:28Z 2021 Conference Paper Leung, J., Shen, Z., Zeng, Z. & Miao, C. (2021). Goal modelling for deep reinforcement learning agents. Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2021), 12975, 271-286. https://dx.doi.org/10.1007/978-3-030-86486-6_17 9783030864859 0302-9743 https://hdl.handle.net/10356/156966 10.1007/978-3-030-86486-6_17 2-s2.0-85115448159 12975 271 286 en NRF-NRFI05-2019-0002 © 2021 Springer Nature Switzerland AG. All rights reserved. This paper was published in Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2021) and is made available with permission of Springer Nature Switzerland AG. application/pdf |
| spellingShingle | Engineering::Computer science and engineering Deep Reinforcement Learning Hierarchical Reinforcement Learning Leung, Jonathan Shen, Zhiqi Zeng, Zhiwei Miao, Chunyan Goal modelling for deep reinforcement learning agents |
| title | Goal modelling for deep reinforcement learning agents |
| title_full | Goal modelling for deep reinforcement learning agents |
| title_fullStr | Goal modelling for deep reinforcement learning agents |
| title_full_unstemmed | Goal modelling for deep reinforcement learning agents |
| title_short | Goal modelling for deep reinforcement learning agents |
| title_sort | goal modelling for deep reinforcement learning agents |
| topic | Engineering::Computer science and engineering Deep Reinforcement Learning Hierarchical Reinforcement Learning |
| url | https://hdl.handle.net/10356/156966 |
| work_keys_str_mv | AT leungjonathan goalmodellingfordeepreinforcementlearningagents AT shenzhiqi goalmodellingfordeepreinforcementlearningagents AT zengzhiwei goalmodellingfordeepreinforcementlearningagents AT miaochunyan goalmodellingfordeepreinforcementlearningagents |