Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning
Due to the unknown motion model and the complexity of the environment, the problem of target tracking for autonomous underwater vehicles (AUVs) became one of the major difficulties in model-based controllers. Therefore, the target tracking task of AUV is modeled as a Markov decision process (MDP) wi...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-10-01
|
| Series: | Journal of Marine Science and Engineering |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2077-1312/10/10/1406 |
| _version_ | 1797472370765594624 |
|---|---|
| author | Jiaxiang Shi Jianer Fang Qizhong Zhang Qiuxuan Wu Botao Zhang Farong Gao |
| author_facet | Jiaxiang Shi Jianer Fang Qizhong Zhang Qiuxuan Wu Botao Zhang Farong Gao |
| author_sort | Jiaxiang Shi |
| collection | DOAJ |
| description | Due to the unknown motion model and the complexity of the environment, the problem of target tracking for autonomous underwater vehicles (AUVs) became one of the major difficulties in model-based controllers. Therefore, the target tracking task of AUV is modeled as a Markov decision process (MDP) with unknown state transition probabilities. Based on actor–critic framework and experience replay technique, a model-free reinforcement learning algorithm is proposed to realize the dynamic target tracking of AUVs. In order to improve the performance of the algorithm, an adaptive experience replay scheme is further proposed. Specifically, the proposed algorithm utilizes the experience replay buffer to store and disrupt the samples, so that the time series samples can be used for training the neural network. Then, the sample priority is arranged according to the temporal difference error, while the adaptive parameters are introduced in the sample priority calculation, thus improving the experience replay rules. The results confirm the quick and stable learning of the proposed algorithm, when tracking the dynamic targets in various motion states. Additionally, the results also demonstrate good control performance regarding both stability and computational complexity, thus indicating the effectiveness of the proposed algorithm in target tracking tasks. |
| first_indexed | 2024-03-09T20:00:53Z |
| format | Article |
| id | doaj.art-5d71b4dd7ca74294a897e7f5c22d3f3c |
| institution | Directory Open Access Journal |
| issn | 2077-1312 |
| language | English |
| last_indexed | 2024-03-09T20:00:53Z |
| publishDate | 2022-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Journal of Marine Science and Engineering |
| spelling | doaj.art-5d71b4dd7ca74294a897e7f5c22d3f3c2023-11-24T00:43:45ZengMDPI AGJournal of Marine Science and Engineering2077-13122022-10-011010140610.3390/jmse10101406Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement LearningJiaxiang Shi0Jianer Fang1Qizhong Zhang2Qiuxuan Wu3Botao Zhang4Farong Gao5School of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaDue to the unknown motion model and the complexity of the environment, the problem of target tracking for autonomous underwater vehicles (AUVs) became one of the major difficulties in model-based controllers. Therefore, the target tracking task of AUV is modeled as a Markov decision process (MDP) with unknown state transition probabilities. Based on actor–critic framework and experience replay technique, a model-free reinforcement learning algorithm is proposed to realize the dynamic target tracking of AUVs. In order to improve the performance of the algorithm, an adaptive experience replay scheme is further proposed. Specifically, the proposed algorithm utilizes the experience replay buffer to store and disrupt the samples, so that the time series samples can be used for training the neural network. Then, the sample priority is arranged according to the temporal difference error, while the adaptive parameters are introduced in the sample priority calculation, thus improving the experience replay rules. The results confirm the quick and stable learning of the proposed algorithm, when tracking the dynamic targets in various motion states. Additionally, the results also demonstrate good control performance regarding both stability and computational complexity, thus indicating the effectiveness of the proposed algorithm in target tracking tasks.https://www.mdpi.com/2077-1312/10/10/1406autonomous underwater vehicledynamic target trackingdeep reinforcement learningexperience replay technique |
| spellingShingle | Jiaxiang Shi Jianer Fang Qizhong Zhang Qiuxuan Wu Botao Zhang Farong Gao Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning Journal of Marine Science and Engineering autonomous underwater vehicle dynamic target tracking deep reinforcement learning experience replay technique |
| title | Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning |
| title_full | Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning |
| title_fullStr | Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning |
| title_full_unstemmed | Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning |
| title_short | Dynamic Target Tracking of Autonomous Underwater Vehicle Based on Deep Reinforcement Learning |
| title_sort | dynamic target tracking of autonomous underwater vehicle based on deep reinforcement learning |
| topic | autonomous underwater vehicle dynamic target tracking deep reinforcement learning experience replay technique |
| url | https://www.mdpi.com/2077-1312/10/10/1406 |
| work_keys_str_mv | AT jiaxiangshi dynamictargettrackingofautonomousunderwatervehiclebasedondeepreinforcementlearning AT jianerfang dynamictargettrackingofautonomousunderwatervehiclebasedondeepreinforcementlearning AT qizhongzhang dynamictargettrackingofautonomousunderwatervehiclebasedondeepreinforcementlearning AT qiuxuanwu dynamictargettrackingofautonomousunderwatervehiclebasedondeepreinforcementlearning AT botaozhang dynamictargettrackingofautonomousunderwatervehiclebasedondeepreinforcementlearning AT faronggao dynamictargettrackingofautonomousunderwatervehiclebasedondeepreinforcementlearning |