Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator
In recent years, reinforcement learning (RL) has been widely used to solve multi-agent navigation tasks, and a high-fidelity level for the simulator is critical to narrow the gap between simulation and real-world tasks. However, high-fidelity simulators have high sampling costs and bottleneck the tr...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9445074/ |
| _version_ | 1818601035517657088 |
|---|---|
| author | Jiantao Qiu Chao Yu Weiling Liu Tianxiang Yang Jincheng Yu Yu Wang Huazhong Yang |
| author_facet | Jiantao Qiu Chao Yu Weiling Liu Tianxiang Yang Jincheng Yu Yu Wang Huazhong Yang |
| author_sort | Jiantao Qiu |
| collection | DOAJ |
| description | In recent years, reinforcement learning (RL) has been widely used to solve multi-agent navigation tasks, and a high-fidelity level for the simulator is critical to narrow the gap between simulation and real-world tasks. However, high-fidelity simulators have high sampling costs and bottleneck the training model-free RL algorithms. Hence, we propose a Multi-Fidelity Simulator framework to train Multi-Agent Reinforcement Learning (MFS-MARL), reducing the total data cost with samples generated by a low-fidelity simulator. We apply the depth-first search to obtain local feasible policies on the low-fidelity simulator as expert policies to help the original reinforcement learning algorithm explore. We built a multi-vehicle simulator with variable fidelity levels to test the proposed method and compared it with the vanilla Soft Actor-Critic (SAC) and expert actor methods. The results show that our method can effectively obtain local feasible policies and can achieve a 23% cost reduction in multi-agent navigation tasks. |
| first_indexed | 2024-12-16T12:44:59Z |
| format | Article |
| id | doaj.art-00262cce78a64649a8011d404b48d90b |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-16T12:44:59Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-00262cce78a64649a8011d404b48d90b2022-12-21T22:31:19ZengIEEEIEEE Access2169-35362021-01-019847738478210.1109/ACCESS.2021.30853289445074Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity SimulatorJiantao Qiu0https://orcid.org/0000-0002-1328-2639Chao Yu1Weiling Liu2Tianxiang Yang3Jincheng Yu4https://orcid.org/0000-0002-6556-7468Yu Wang5https://orcid.org/0000-0001-6108-5157Huazhong Yang6https://orcid.org/0000-0003-2421-353XDepartment of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, ChinaDepartment of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, ChinaDepartment of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, ChinaDepartment of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, ChinaDepartment of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, ChinaDepartment of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, ChinaDepartment of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, ChinaIn recent years, reinforcement learning (RL) has been widely used to solve multi-agent navigation tasks, and a high-fidelity level for the simulator is critical to narrow the gap between simulation and real-world tasks. However, high-fidelity simulators have high sampling costs and bottleneck the training model-free RL algorithms. Hence, we propose a Multi-Fidelity Simulator framework to train Multi-Agent Reinforcement Learning (MFS-MARL), reducing the total data cost with samples generated by a low-fidelity simulator. We apply the depth-first search to obtain local feasible policies on the low-fidelity simulator as expert policies to help the original reinforcement learning algorithm explore. We built a multi-vehicle simulator with variable fidelity levels to test the proposed method and compared it with the vanilla Soft Actor-Critic (SAC) and expert actor methods. The results show that our method can effectively obtain local feasible policies and can achieve a 23% cost reduction in multi-agent navigation tasks.https://ieeexplore.ieee.org/document/9445074/Deep reinforcement learningintelligent robotsmulti-robot systemsmulti-fidelity simulators |
| spellingShingle | Jiantao Qiu Chao Yu Weiling Liu Tianxiang Yang Jincheng Yu Yu Wang Huazhong Yang Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator IEEE Access Deep reinforcement learning intelligent robots multi-robot systems multi-fidelity simulators |
| title | Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator |
| title_full | Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator |
| title_fullStr | Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator |
| title_full_unstemmed | Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator |
| title_short | Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator |
| title_sort | low cost multi agent navigation via reinforcement learning with multi fidelity simulator |
| topic | Deep reinforcement learning intelligent robots multi-robot systems multi-fidelity simulators |
| url | https://ieeexplore.ieee.org/document/9445074/ |
| work_keys_str_mv | AT jiantaoqiu lowcostmultiagentnavigationviareinforcementlearningwithmultifidelitysimulator AT chaoyu lowcostmultiagentnavigationviareinforcementlearningwithmultifidelitysimulator AT weilingliu lowcostmultiagentnavigationviareinforcementlearningwithmultifidelitysimulator AT tianxiangyang lowcostmultiagentnavigationviareinforcementlearningwithmultifidelitysimulator AT jinchengyu lowcostmultiagentnavigationviareinforcementlearningwithmultifidelitysimulator AT yuwang lowcostmultiagentnavigationviareinforcementlearningwithmultifidelitysimulator AT huazhongyang lowcostmultiagentnavigationviareinforcementlearningwithmultifidelitysimulator |