Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning
Harvesting data from distributed Internet of Things (IoT) devices with multiple autonomous unmanned aerial vehicles (UAVs) is a challenging problem requiring flexible path planning methods. We propose a multi-agent reinforcement learning (MARL) approach that, in contrast to previous work, can adapt...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Open Journal of the Communications Society |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9437338/ |
| _version_ | 1818647855206760448 |
|---|---|
| author | Harald Bayerlein Mirco Theile Marco Caccamo David Gesbert |
| author_facet | Harald Bayerlein Mirco Theile Marco Caccamo David Gesbert |
| author_sort | Harald Bayerlein |
| collection | DOAJ |
| description | Harvesting data from distributed Internet of Things (IoT) devices with multiple autonomous unmanned aerial vehicles (UAVs) is a challenging problem requiring flexible path planning methods. We propose a multi-agent reinforcement learning (MARL) approach that, in contrast to previous work, can adapt to profound changes in the scenario parameters defining the data harvesting mission, such as the number of deployed UAVs, number, position and data amount of IoT devices, or the maximum flying time, without the need to perform expensive recomputations or relearn control policies. We formulate the path planning problem for a cooperative, non-communicating, and homogeneous team of UAVs tasked with maximizing collected data from distributed IoT sensor nodes subject to flying time and collision avoidance constraints. The path planning problem is translated into a decentralized partially observable Markov decision process (Dec-POMDP), which we solve through a deep reinforcement learning (DRL) approach, approximating the optimal UAV control policy without prior knowledge of the challenging wireless channel characteristics in dense urban environments. By exploiting a combination of centered global and local map representations of the environment that are fed into convolutional layers of the agents, we show that our proposed network architecture enables the agents to cooperate effectively by carefully dividing the data collection task among themselves, adapt to large complex environments and state spaces, and make movement decisions that balance data collection goals, flight-time efficiency, and navigation constraints. Finally, learning a control policy that generalizes over the scenario parameter space enables us to analyze the influence of individual parameters on collection performance and provide some intuition about system-level benefits. |
| first_indexed | 2024-12-17T01:09:10Z |
| format | Article |
| id | doaj.art-a79e3f33ce504bd6bfde408a7909cc93 |
| institution | Directory Open Access Journal |
| issn | 2644-125X |
| language | English |
| last_indexed | 2024-12-17T01:09:10Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of the Communications Society |
| spelling | doaj.art-a79e3f33ce504bd6bfde408a7909cc932022-12-21T22:09:11ZengIEEEIEEE Open Journal of the Communications Society2644-125X2021-01-0121171118710.1109/OJCOMS.2021.30819969437338Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement LearningHarald Bayerlein0https://orcid.org/0000-0002-1851-9455Mirco Theile1https://orcid.org/0000-0003-1574-8858Marco Caccamo2David Gesbert3Communication Systems Department, EURECOM, Sophia Antipolis, FranceTUM Department of Mechanical Engineering, Technical University of Munich, Munich, GermanyTUM Department of Mechanical Engineering, Technical University of Munich, Munich, GermanyCommunication Systems Department, EURECOM, Sophia Antipolis, FranceHarvesting data from distributed Internet of Things (IoT) devices with multiple autonomous unmanned aerial vehicles (UAVs) is a challenging problem requiring flexible path planning methods. We propose a multi-agent reinforcement learning (MARL) approach that, in contrast to previous work, can adapt to profound changes in the scenario parameters defining the data harvesting mission, such as the number of deployed UAVs, number, position and data amount of IoT devices, or the maximum flying time, without the need to perform expensive recomputations or relearn control policies. We formulate the path planning problem for a cooperative, non-communicating, and homogeneous team of UAVs tasked with maximizing collected data from distributed IoT sensor nodes subject to flying time and collision avoidance constraints. The path planning problem is translated into a decentralized partially observable Markov decision process (Dec-POMDP), which we solve through a deep reinforcement learning (DRL) approach, approximating the optimal UAV control policy without prior knowledge of the challenging wireless channel characteristics in dense urban environments. By exploiting a combination of centered global and local map representations of the environment that are fed into convolutional layers of the agents, we show that our proposed network architecture enables the agents to cooperate effectively by carefully dividing the data collection task among themselves, adapt to large complex environments and state spaces, and make movement decisions that balance data collection goals, flight-time efficiency, and navigation constraints. Finally, learning a control policy that generalizes over the scenario parameter space enables us to analyze the influence of individual parameters on collection performance and provide some intuition about system-level benefits.https://ieeexplore.ieee.org/document/9437338/Internet of Things (IoT)map-based planningmulti-agent reinforcement learning (MARL)trajectory planningunmanned aerial vehicle (UAV) |
| spellingShingle | Harald Bayerlein Mirco Theile Marco Caccamo David Gesbert Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning IEEE Open Journal of the Communications Society Internet of Things (IoT) map-based planning multi-agent reinforcement learning (MARL) trajectory planning unmanned aerial vehicle (UAV) |
| title | Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning |
| title_full | Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning |
| title_fullStr | Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning |
| title_full_unstemmed | Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning |
| title_short | Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning |
| title_sort | multi uav path planning for wireless data harvesting with deep reinforcement learning |
| topic | Internet of Things (IoT) map-based planning multi-agent reinforcement learning (MARL) trajectory planning unmanned aerial vehicle (UAV) |
| url | https://ieeexplore.ieee.org/document/9437338/ |
| work_keys_str_mv | AT haraldbayerlein multiuavpathplanningforwirelessdataharvestingwithdeepreinforcementlearning AT mircotheile multiuavpathplanningforwirelessdataharvestingwithdeepreinforcementlearning AT marcocaccamo multiuavpathplanningforwirelessdataharvestingwithdeepreinforcementlearning AT davidgesbert multiuavpathplanningforwirelessdataharvestingwithdeepreinforcementlearning |