Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential Calculation
Hierarchical reinforcement learning (HRL) offers a hierarchical structure for organizing tasks, enabling agents to learn and make decisions autonomously in complex environments. However, traditional HRL approaches face limitations in effectively handling complex tasks. Reward machines, which specify...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/17/3700 |
| _version_ | 1797582648393072640 |
|---|---|
| author | Ziyun Zhou Jingwei Shang Yimang Li |
| author_facet | Ziyun Zhou Jingwei Shang Yimang Li |
| author_sort | Ziyun Zhou |
| collection | DOAJ |
| description | Hierarchical reinforcement learning (HRL) offers a hierarchical structure for organizing tasks, enabling agents to learn and make decisions autonomously in complex environments. However, traditional HRL approaches face limitations in effectively handling complex tasks. Reward machines, which specify high-level goals and associated rewards for sub-goals, have been introduced to address these limitations by facilitating the agent’s understanding and reasoning with respect to the task hierarchy. In this paper, we propose a novel approach to enhance HRL performance through topologically sorted potential calculation for reward machines. By leveraging the topological structure of the task hierarchy, our method efficiently determines potentials for different sub-goals. This topological sorting enables the agent to prioritize actions leading to the accomplishment of higher-level goals, enhancing the learning process. To assess the efficacy of our approach, we conducted experiments in the grid-world environment with OpenAI-Gym. The results showcase the superiority of our proposed method over traditional HRL techniques and reward machine-based reinforcement learning approaches in terms of learning efficiency and overall task performance. |
| first_indexed | 2024-03-10T23:24:25Z |
| format | Article |
| id | doaj.art-21d1a29946934a0a9e477929238e963f |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T23:24:25Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-21d1a29946934a0a9e477929238e963f2023-11-19T08:02:49ZengMDPI AGElectronics2079-92922023-09-011217370010.3390/electronics12173700Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential CalculationZiyun Zhou0Jingwei Shang1Yimang Li2School of Mechanical Engineering and Rail Transit, Changzhou University, Changzhou 213164, ChinaChina Electronic Product Reliability and Environmental Testing Research Institute, Guangzhou 510610, ChinaSchool of Mechanical Engineering and Rail Transit, Changzhou University, Changzhou 213164, ChinaHierarchical reinforcement learning (HRL) offers a hierarchical structure for organizing tasks, enabling agents to learn and make decisions autonomously in complex environments. However, traditional HRL approaches face limitations in effectively handling complex tasks. Reward machines, which specify high-level goals and associated rewards for sub-goals, have been introduced to address these limitations by facilitating the agent’s understanding and reasoning with respect to the task hierarchy. In this paper, we propose a novel approach to enhance HRL performance through topologically sorted potential calculation for reward machines. By leveraging the topological structure of the task hierarchy, our method efficiently determines potentials for different sub-goals. This topological sorting enables the agent to prioritize actions leading to the accomplishment of higher-level goals, enhancing the learning process. To assess the efficacy of our approach, we conducted experiments in the grid-world environment with OpenAI-Gym. The results showcase the superiority of our proposed method over traditional HRL techniques and reward machine-based reinforcement learning approaches in terms of learning efficiency and overall task performance.https://www.mdpi.com/2079-9292/12/17/3700reinforcement learningreward learningtopological sorting |
| spellingShingle | Ziyun Zhou Jingwei Shang Yimang Li Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential Calculation Electronics reinforcement learning reward learning topological sorting |
| title | Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential Calculation |
| title_full | Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential Calculation |
| title_fullStr | Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential Calculation |
| title_full_unstemmed | Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential Calculation |
| title_short | Enhancing Efficiency in Hierarchical Reinforcement Learning through Topological-Sorted Potential Calculation |
| title_sort | enhancing efficiency in hierarchical reinforcement learning through topological sorted potential calculation |
| topic | reinforcement learning reward learning topological sorting |
| url | https://www.mdpi.com/2079-9292/12/17/3700 |
| work_keys_str_mv | AT ziyunzhou enhancingefficiencyinhierarchicalreinforcementlearningthroughtopologicalsortedpotentialcalculation AT jingweishang enhancingefficiencyinhierarchicalreinforcementlearningthroughtopologicalsortedpotentialcalculation AT yimangli enhancingefficiencyinhierarchicalreinforcementlearningthroughtopologicalsortedpotentialcalculation |