A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case
Autonomous surfaces vehicles (ASVs) excel at monitoring and measuring aquatic nutrients due to their autonomy, mobility, and relatively low cost. When planning paths for such vehicles, the task of patrolling with multiple agents is usually addressed with heuristics approaches, such as Reinforcement...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9330612/ |
| _version_ | 1818737372284583936 |
|---|---|
| author | Samuel Yanes Luis Daniel Gutierrez Reina Sergio L. Toral Marin |
| author_facet | Samuel Yanes Luis Daniel Gutierrez Reina Sergio L. Toral Marin |
| author_sort | Samuel Yanes Luis |
| collection | DOAJ |
| description | Autonomous surfaces vehicles (ASVs) excel at monitoring and measuring aquatic nutrients due to their autonomy, mobility, and relatively low cost. When planning paths for such vehicles, the task of patrolling with multiple agents is usually addressed with heuristics approaches, such as Reinforcement Learning (RL), because of the complexity and high dimensionality of the problem. Not only do efficient paths have to be designed, but addressing disturbances in movement or the battery's performance is mandatory. For this multiagent patrolling task, the proposed approach is based on a centralized Convolutional Deep Q-Network, designed with a final independent dense layer for every agent to deal with scalability, with the hypothesis/assumption that every agent has the same properties and capabilities. For this purpose, a tailored reward function is created which penalizes illegal actions (such as collisions) and rewards visiting idle cells (cells that remains unvisited for a long time). A comparison with various multiagent Reinforcement Learning (MARL) algorithms has been done (Independent Q-Learning, Dueling Q-Network and multiagent Double Deep Q-Learning) in a case-study scenario like the Ypacaraí lake in Asunción (Paraguay). The training results in multiagent policy leads to an average improvement of 15% compared to lawn mower trajectories and a 6% improvement over the IDQL for the case-study considered. When evaluating the training speed, the proposed approach runs three times faster than the independent algorithm. |
| first_indexed | 2024-12-18T00:52:00Z |
| format | Article |
| id | doaj.art-701164caec1843988573ccdff89f81c3 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-18T00:52:00Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-701164caec1843988573ccdff89f81c32022-12-21T21:26:39ZengIEEEIEEE Access2169-35362021-01-019170841709910.1109/ACCESS.2021.30533489330612A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling CaseSamuel Yanes Luis0https://orcid.org/0000-0002-7796-3599Daniel Gutierrez Reina1Sergio L. Toral Marin2https://orcid.org/0000-0003-2612-0388Department of Electronic Engineering, Technical School of Engineering of Seville, Seville, SpainDepartment of Electronic Engineering, Technical School of Engineering of Seville, Seville, SpainDepartment of Electronic Engineering, Technical School of Engineering of Seville, Seville, SpainAutonomous surfaces vehicles (ASVs) excel at monitoring and measuring aquatic nutrients due to their autonomy, mobility, and relatively low cost. When planning paths for such vehicles, the task of patrolling with multiple agents is usually addressed with heuristics approaches, such as Reinforcement Learning (RL), because of the complexity and high dimensionality of the problem. Not only do efficient paths have to be designed, but addressing disturbances in movement or the battery's performance is mandatory. For this multiagent patrolling task, the proposed approach is based on a centralized Convolutional Deep Q-Network, designed with a final independent dense layer for every agent to deal with scalability, with the hypothesis/assumption that every agent has the same properties and capabilities. For this purpose, a tailored reward function is created which penalizes illegal actions (such as collisions) and rewards visiting idle cells (cells that remains unvisited for a long time). A comparison with various multiagent Reinforcement Learning (MARL) algorithms has been done (Independent Q-Learning, Dueling Q-Network and multiagent Double Deep Q-Learning) in a case-study scenario like the Ypacaraí lake in Asunción (Paraguay). The training results in multiagent policy leads to an average improvement of 15% compared to lawn mower trajectories and a 6% improvement over the IDQL for the case-study considered. When evaluating the training speed, the proposed approach runs three times faster than the independent algorithm.https://ieeexplore.ieee.org/document/9330612/Deep reinforcement learningmultiagent learningmonitoringpath planningautonomous surface vehiclepatrolling |
| spellingShingle | Samuel Yanes Luis Daniel Gutierrez Reina Sergio L. Toral Marin A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case IEEE Access Deep reinforcement learning multiagent learning monitoring path planning autonomous surface vehicle patrolling |
| title | A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case |
| title_full | A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case |
| title_fullStr | A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case |
| title_full_unstemmed | A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case |
| title_short | A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacaraí Lake Patrolling Case |
| title_sort | multiagent deep reinforcement learning approach for path planning in autonomous surface vehicles the ypacara x00ed lake patrolling case |
| topic | Deep reinforcement learning multiagent learning monitoring path planning autonomous surface vehicle patrolling |
| url | https://ieeexplore.ieee.org/document/9330612/ |
| work_keys_str_mv | AT samuelyanesluis amultiagentdeepreinforcementlearningapproachforpathplanninginautonomoussurfacevehiclestheypacarax00edlakepatrollingcase AT danielgutierrezreina amultiagentdeepreinforcementlearningapproachforpathplanninginautonomoussurfacevehiclestheypacarax00edlakepatrollingcase AT sergioltoralmarin amultiagentdeepreinforcementlearningapproachforpathplanninginautonomoussurfacevehiclestheypacarax00edlakepatrollingcase AT samuelyanesluis multiagentdeepreinforcementlearningapproachforpathplanninginautonomoussurfacevehiclestheypacarax00edlakepatrollingcase AT danielgutierrezreina multiagentdeepreinforcementlearningapproachforpathplanninginautonomoussurfacevehiclestheypacarax00edlakepatrollingcase AT sergioltoralmarin multiagentdeepreinforcementlearningapproachforpathplanninginautonomoussurfacevehiclestheypacarax00edlakepatrollingcase |