Efficient novelty search through deep reinforcement learning
Novelty search, which was inspired by the nature that evolves creatures with diversity, has shown great potential in solving reinforcement learning (RL) tasks with sparse and deceptive rewards. However, most of the existing novelty search methods evolve the populations through hybrization and mutati...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2021
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/152665 |
| _version_ | 1826129506210938880 |
|---|---|
| author | Shi, Longxiang Li, Shijian Zheng, Qian Yao, Min Pan, Gang |
| author2 | School of Electrical and Electronic Engineering |
| author_facet | School of Electrical and Electronic Engineering Shi, Longxiang Li, Shijian Zheng, Qian Yao, Min Pan, Gang |
| author_sort | Shi, Longxiang |
| collection | NTU |
| description | Novelty search, which was inspired by the nature that evolves creatures with diversity, has shown great potential in solving reinforcement learning (RL) tasks with sparse and deceptive rewards. However, most of the existing novelty search methods evolve the populations through hybrization and mutation, which is inefficient in diverging populations. In this paper, we propose a method which incorporates deep RL with novelty search to improve the efficiency of diverging the populations for novelty search. We first propose a strategy that improves the novelty of individuals generated by genetic algorithm using reinforcement learning. Based on this strategy, we propose a framework that incorporates deep RL with novelty search, and then derive an algorithm to improve the search efficiency of the novelty search for continuous control tasks. Our experimental results show that our method can improve the search efficiency of novelty search and can also provide a competitive performance compared to some of the existing novelty search methods. The implementation of our method is available at: https://github.com/shilx001/NoveltySearch_Improvement. |
| first_indexed | 2024-10-01T07:41:42Z |
| format | Journal Article |
| id | ntu-10356/152665 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:41:42Z |
| publishDate | 2021 |
| record_format | dspace |
| spelling | ntu-10356/1526652021-09-08T08:57:43Z Efficient novelty search through deep reinforcement learning Shi, Longxiang Li, Shijian Zheng, Qian Yao, Min Pan, Gang School of Electrical and Electronic Engineering Engineering::Electrical and electronic engineering Reinforcement Learning Novelty Search Novelty search, which was inspired by the nature that evolves creatures with diversity, has shown great potential in solving reinforcement learning (RL) tasks with sparse and deceptive rewards. However, most of the existing novelty search methods evolve the populations through hybrization and mutation, which is inefficient in diverging populations. In this paper, we propose a method which incorporates deep RL with novelty search to improve the efficiency of diverging the populations for novelty search. We first propose a strategy that improves the novelty of individuals generated by genetic algorithm using reinforcement learning. Based on this strategy, we propose a framework that incorporates deep RL with novelty search, and then derive an algorithm to improve the search efficiency of the novelty search for continuous control tasks. Our experimental results show that our method can improve the search efficiency of novelty search and can also provide a competitive performance compared to some of the existing novelty search methods. The implementation of our method is available at: https://github.com/shilx001/NoveltySearch_Improvement. Published version 2021-09-08T08:57:42Z 2021-09-08T08:57:42Z 2020 Journal Article Shi, L., Li, S., Zheng, Q., Yao, M. & Pan, G. (2020). Efficient novelty search through deep reinforcement learning. IEEE Access, 8, 128809-128818. https://dx.doi.org/10.1109/ACCESS.2020.3008735 2169-3536 https://hdl.handle.net/10356/152665 10.1109/ACCESS.2020.3008735 8 128809 128818 en IEEE Access © 2020 IEEE. This journal is 100% open access, which means that all content is freely available without charge to users or their institutions. All articles accepted after 12 June 2019 are published under a CC BY 4.0 license, and the author retains copyright. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles, or use them for any other lawful purpose, as long as proper attribution is given. application/pdf |
| spellingShingle | Engineering::Electrical and electronic engineering Reinforcement Learning Novelty Search Shi, Longxiang Li, Shijian Zheng, Qian Yao, Min Pan, Gang Efficient novelty search through deep reinforcement learning |
| title | Efficient novelty search through deep reinforcement learning |
| title_full | Efficient novelty search through deep reinforcement learning |
| title_fullStr | Efficient novelty search through deep reinforcement learning |
| title_full_unstemmed | Efficient novelty search through deep reinforcement learning |
| title_short | Efficient novelty search through deep reinforcement learning |
| title_sort | efficient novelty search through deep reinforcement learning |
| topic | Engineering::Electrical and electronic engineering Reinforcement Learning Novelty Search |
| url | https://hdl.handle.net/10356/152665 |
| work_keys_str_mv | AT shilongxiang efficientnoveltysearchthroughdeepreinforcementlearning AT lishijian efficientnoveltysearchthroughdeepreinforcementlearning AT zhengqian efficientnoveltysearchthroughdeepreinforcementlearning AT yaomin efficientnoveltysearchthroughdeepreinforcementlearning AT pangang efficientnoveltysearchthroughdeepreinforcementlearning |