Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments
Autonomous motion planning (AMP) of unmanned aerial vehicles (UAVs) is aimed at enabling a UAV to safely fly to the target without human intervention. Recently, several emerging deep reinforcement learning (DRL) methods have been employed to address the AMP problem in some simplified environments, a...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/20/7/1890 |
| _version_ | 1797626393001984000 |
|---|---|
| author | Zijian Hu Kaifang Wan Xiaoguang Gao Yiwei Zhai Qianglong Wang |
| author_facet | Zijian Hu Kaifang Wan Xiaoguang Gao Yiwei Zhai Qianglong Wang |
| author_sort | Zijian Hu |
| collection | DOAJ |
| description | Autonomous motion planning (AMP) of unmanned aerial vehicles (UAVs) is aimed at enabling a UAV to safely fly to the target without human intervention. Recently, several emerging deep reinforcement learning (DRL) methods have been employed to address the AMP problem in some simplified environments, and these methods have yielded good results. This paper proposes a multiple experience pools (MEPs) framework leveraging human expert experiences for DRL to speed up the learning process. Based on the deep deterministic policy gradient (DDPG) algorithm, a MEP–DDPG algorithm was designed using model predictive control and simulated annealing to generate expert experiences. On applying this algorithm to a complex unknown simulation environment constructed based on the parameters of the real UAV, the training experiment results showed that the novel DRL algorithm resulted in a performance improvement exceeding 20% as compared with the state-of-the-art DDPG. The results of the experimental testing indicate that UAVs trained using MEP–DDPG can stably complete a variety of tasks in complex, unknown environments. |
| first_indexed | 2024-03-11T10:09:47Z |
| format | Article |
| id | doaj.art-542c9cc6b3784b39b6e00301f55bd957 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T10:09:47Z |
| publishDate | 2020-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-542c9cc6b3784b39b6e00301f55bd9572023-11-16T14:34:50ZengMDPI AGSensors1424-82202020-03-01207189010.3390/s20071890Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown EnvironmentsZijian Hu0Kaifang Wan1Xiaoguang Gao2Yiwei Zhai3Qianglong Wang4School of Electronic and Information, Northwestern Polytechnical University, Xi’an 710129, ChinaSchool of Electronic and Information, Northwestern Polytechnical University, Xi’an 710129, ChinaSchool of Electronic and Information, Northwestern Polytechnical University, Xi’an 710129, ChinaSchool of Electronic and Information, Northwestern Polytechnical University, Xi’an 710129, ChinaSchool of Electronic and Information, Northwestern Polytechnical University, Xi’an 710129, ChinaAutonomous motion planning (AMP) of unmanned aerial vehicles (UAVs) is aimed at enabling a UAV to safely fly to the target without human intervention. Recently, several emerging deep reinforcement learning (DRL) methods have been employed to address the AMP problem in some simplified environments, and these methods have yielded good results. This paper proposes a multiple experience pools (MEPs) framework leveraging human expert experiences for DRL to speed up the learning process. Based on the deep deterministic policy gradient (DDPG) algorithm, a MEP–DDPG algorithm was designed using model predictive control and simulated annealing to generate expert experiences. On applying this algorithm to a complex unknown simulation environment constructed based on the parameters of the real UAV, the training experiment results showed that the novel DRL algorithm resulted in a performance improvement exceeding 20% as compared with the state-of-the-art DDPG. The results of the experimental testing indicate that UAVs trained using MEP–DDPG can stably complete a variety of tasks in complex, unknown environments.https://www.mdpi.com/1424-8220/20/7/1890UAVmotion planningdeep reinforcement learningmultiple experience pools |
| spellingShingle | Zijian Hu Kaifang Wan Xiaoguang Gao Yiwei Zhai Qianglong Wang Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments Sensors UAV motion planning deep reinforcement learning multiple experience pools |
| title | Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments |
| title_full | Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments |
| title_fullStr | Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments |
| title_full_unstemmed | Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments |
| title_short | Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments |
| title_sort | deep reinforcement learning approach with multiple experience pools for uav s autonomous motion planning in complex unknown environments |
| topic | UAV motion planning deep reinforcement learning multiple experience pools |
| url | https://www.mdpi.com/1424-8220/20/7/1890 |
| work_keys_str_mv | AT zijianhu deepreinforcementlearningapproachwithmultipleexperiencepoolsforuavsautonomousmotionplanningincomplexunknownenvironments AT kaifangwan deepreinforcementlearningapproachwithmultipleexperiencepoolsforuavsautonomousmotionplanningincomplexunknownenvironments AT xiaoguanggao deepreinforcementlearningapproachwithmultipleexperiencepoolsforuavsautonomousmotionplanningincomplexunknownenvironments AT yiweizhai deepreinforcementlearningapproachwithmultipleexperiencepoolsforuavsautonomousmotionplanningincomplexunknownenvironments AT qianglongwang deepreinforcementlearningapproachwithmultipleexperiencepoolsforuavsautonomousmotionplanningincomplexunknownenvironments |