Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning
Mobile crowdsensing (MCS) is attracting considerable attention in the past few years as a new paradigm for large-scale information sensing. Unmanned aerial vehicles (UAVs) have played a significant role in MCS tasks and served as crucial nodes in the newly-proposed space-air-ground integrated networ...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/24/5/638 |
| _version_ | 1797500015500853248 |
|---|---|
| author | Yixiang Ren Zhenhui Ye Guanghua Song Xiaohong Jiang |
| author_facet | Yixiang Ren Zhenhui Ye Guanghua Song Xiaohong Jiang |
| author_sort | Yixiang Ren |
| collection | DOAJ |
| description | Mobile crowdsensing (MCS) is attracting considerable attention in the past few years as a new paradigm for large-scale information sensing. Unmanned aerial vehicles (UAVs) have played a significant role in MCS tasks and served as crucial nodes in the newly-proposed space-air-ground integrated network (SAGIN). In this paper, we incorporate SAGIN into MCS task and present a <i>Space-Air-Ground integrated Mobile CrowdSensing</i> (SAG-MCS) problem. Based on multi-source observations from embedded sensors and satellites, an aerial UAV swarm is required to carry out energy-efficient data collection and recharging tasks. Up to date, few studies have explored such multi-task MCS problem with the cooperation of UAV swarm and satellites. To address this multi-agent problem, we propose a novel deep reinforcement learning (DRL) based method called <i>Multi-Scale Soft Deep Recurrent Graph Network</i> (ms-SDRGN). Our ms-SDRGN approach incorporates a multi-scale convolutional encoder to process multi-source raw observations for better feature exploitation. We also use a graph attention mechanism to model inter-UAV communications and aggregate extra neighboring information, and utilize a gated recurrent unit for long-term performance. In addition, a stochastic policy can be learned through a maximum-entropy method with an adjustable temperature parameter. Specifically, we design a heuristic reward function to encourage the agents to achieve global cooperation under partial observability. We train the model to convergence and conduct a series of case studies. Evaluation results show statistical significance and that ms-SDRGN outperforms three state-of-the-art DRL baselines in SAG-MCS. Compared with the best-performing baseline, ms-SDRGN improves 29.0% reward and 3.8% CFE score. We also investigate the scalability and robustness of ms-SDRGN towards DRL environments with diverse observation scales or demanding communication conditions. |
| first_indexed | 2024-03-10T03:55:46Z |
| format | Article |
| id | doaj.art-109bd5c47336435c910509d4f586a2ab |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-10T03:55:46Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-109bd5c47336435c910509d4f586a2ab2023-11-23T10:54:59ZengMDPI AGEntropy1099-43002022-05-0124563810.3390/e24050638Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement LearningYixiang Ren0Zhenhui Ye1Guanghua Song2Xiaohong Jiang3School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, ChinaCollege of Computer Science and Technology, Zhejiang University, Hangzhou 310027, ChinaSchool of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, ChinaCollege of Computer Science and Technology, Zhejiang University, Hangzhou 310027, ChinaMobile crowdsensing (MCS) is attracting considerable attention in the past few years as a new paradigm for large-scale information sensing. Unmanned aerial vehicles (UAVs) have played a significant role in MCS tasks and served as crucial nodes in the newly-proposed space-air-ground integrated network (SAGIN). In this paper, we incorporate SAGIN into MCS task and present a <i>Space-Air-Ground integrated Mobile CrowdSensing</i> (SAG-MCS) problem. Based on multi-source observations from embedded sensors and satellites, an aerial UAV swarm is required to carry out energy-efficient data collection and recharging tasks. Up to date, few studies have explored such multi-task MCS problem with the cooperation of UAV swarm and satellites. To address this multi-agent problem, we propose a novel deep reinforcement learning (DRL) based method called <i>Multi-Scale Soft Deep Recurrent Graph Network</i> (ms-SDRGN). Our ms-SDRGN approach incorporates a multi-scale convolutional encoder to process multi-source raw observations for better feature exploitation. We also use a graph attention mechanism to model inter-UAV communications and aggregate extra neighboring information, and utilize a gated recurrent unit for long-term performance. In addition, a stochastic policy can be learned through a maximum-entropy method with an adjustable temperature parameter. Specifically, we design a heuristic reward function to encourage the agents to achieve global cooperation under partial observability. We train the model to convergence and conduct a series of case studies. Evaluation results show statistical significance and that ms-SDRGN outperforms three state-of-the-art DRL baselines in SAG-MCS. Compared with the best-performing baseline, ms-SDRGN improves 29.0% reward and 3.8% CFE score. We also investigate the scalability and robustness of ms-SDRGN towards DRL environments with diverse observation scales or demanding communication conditions.https://www.mdpi.com/1099-4300/24/5/638mobile crowdsensingdeep reinforcement learningUAV controlgraph networkmaximum-entropy learning |
| spellingShingle | Yixiang Ren Zhenhui Ye Guanghua Song Xiaohong Jiang Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning Entropy mobile crowdsensing deep reinforcement learning UAV control graph network maximum-entropy learning |
| title | Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning |
| title_full | Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning |
| title_fullStr | Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning |
| title_full_unstemmed | Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning |
| title_short | Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning |
| title_sort | space air ground integrated mobile crowdsensing for partially observable data collection by multi scale convolutional graph reinforcement learning |
| topic | mobile crowdsensing deep reinforcement learning UAV control graph network maximum-entropy learning |
| url | https://www.mdpi.com/1099-4300/24/5/638 |
| work_keys_str_mv | AT yixiangren spaceairgroundintegratedmobilecrowdsensingforpartiallyobservabledatacollectionbymultiscaleconvolutionalgraphreinforcementlearning AT zhenhuiye spaceairgroundintegratedmobilecrowdsensingforpartiallyobservabledatacollectionbymultiscaleconvolutionalgraphreinforcementlearning AT guanghuasong spaceairgroundintegratedmobilecrowdsensingforpartiallyobservabledatacollectionbymultiscaleconvolutionalgraphreinforcementlearning AT xiaohongjiang spaceairgroundintegratedmobilecrowdsensingforpartiallyobservabledatacollectionbymultiscaleconvolutionalgraphreinforcementlearning |