A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments
Multi-agent reinforcement learning excels at addressing group intelligent decision-making problems involving sequential decision-making. In particular, in complex, high-dimensional state and action spaces, it imposes higher demands on the reliability, stability, and adaptability of decision algorith...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/23/9520 |
| _version_ | 1797399539152322560 |
|---|---|
| author | Yang Yang Jiang Li Jinyong Hou Ye Wang Huadong Zhao |
| author_facet | Yang Yang Jiang Li Jinyong Hou Ye Wang Huadong Zhao |
| author_sort | Yang Yang |
| collection | DOAJ |
| description | Multi-agent reinforcement learning excels at addressing group intelligent decision-making problems involving sequential decision-making. In particular, in complex, high-dimensional state and action spaces, it imposes higher demands on the reliability, stability, and adaptability of decision algorithms. The reinforcement learning algorithm based on the multi-agent deep strategy gradient incorporates a function approximation method using discriminant networks. However, this can lead to estimation errors when agents evaluate action values, thereby reducing model reliability and stability and resulting in challenging convergence. With the increasing complexity of the environment, there is a decline in the quality of experience collected by the experience playback pool, resulting in low efficiency of the sampling stage and difficulties in algorithm convergence. To address these challenges, we propose an innovative approach called the empirical clustering layer-based multi-agent dual dueling policy gradient (ECL-MAD3PG) algorithm. Experimental results demonstrate that our ECL-MAD3PG algorithm outperforms other methods in various complex environments, demonstrating a remarkable 9.1% improvement in mission completion compared to MADDPG within the context of complex UAV cooperative combat scenarios. |
| first_indexed | 2024-03-09T01:41:40Z |
| format | Article |
| id | doaj.art-07733c7127e342a0b269be0c566e5203 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T01:41:40Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-07733c7127e342a0b269be0c566e52032023-12-08T15:26:17ZengMDPI AGSensors1424-82202023-11-012323952010.3390/s23239520A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex EnvironmentsYang Yang0Jiang Li1Jinyong Hou2Ye Wang3Huadong Zhao4Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, ChinaChangchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, ChinaUnit 32802 of the Chinese People’s Liberation Army, Beijing 100191, ChinaChangchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, ChinaChangchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, ChinaMulti-agent reinforcement learning excels at addressing group intelligent decision-making problems involving sequential decision-making. In particular, in complex, high-dimensional state and action spaces, it imposes higher demands on the reliability, stability, and adaptability of decision algorithms. The reinforcement learning algorithm based on the multi-agent deep strategy gradient incorporates a function approximation method using discriminant networks. However, this can lead to estimation errors when agents evaluate action values, thereby reducing model reliability and stability and resulting in challenging convergence. With the increasing complexity of the environment, there is a decline in the quality of experience collected by the experience playback pool, resulting in low efficiency of the sampling stage and difficulties in algorithm convergence. To address these challenges, we propose an innovative approach called the empirical clustering layer-based multi-agent dual dueling policy gradient (ECL-MAD3PG) algorithm. Experimental results demonstrate that our ECL-MAD3PG algorithm outperforms other methods in various complex environments, demonstrating a remarkable 9.1% improvement in mission completion compared to MADDPG within the context of complex UAV cooperative combat scenarios.https://www.mdpi.com/1424-8220/23/23/9520deep deterministic policy gradientplayback of experiencegroup decision-makingoverestimation of value function |
| spellingShingle | Yang Yang Jiang Li Jinyong Hou Ye Wang Huadong Zhao A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments Sensors deep deterministic policy gradient playback of experience group decision-making overestimation of value function |
| title | A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments |
| title_full | A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments |
| title_fullStr | A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments |
| title_full_unstemmed | A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments |
| title_short | A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments |
| title_sort | policy gradient algorithm to alleviate the multi agent value overestimation problem in complex environments |
| topic | deep deterministic policy gradient playback of experience group decision-making overestimation of value function |
| url | https://www.mdpi.com/1424-8220/23/23/9520 |
| work_keys_str_mv | AT yangyang apolicygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT jiangli apolicygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT jinyonghou apolicygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT yewang apolicygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT huadongzhao apolicygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT yangyang policygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT jiangli policygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT jinyonghou policygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT yewang policygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments AT huadongzhao policygradientalgorithmtoalleviatethemultiagentvalueoverestimationproblemincomplexenvironments |