A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments
This paper addresses the problem of navigating decentralized multi-agent systems in partially cluttered environments and proposes a new machine-learning-based approach to solve it. On the basis of this approach, a new robust and flexible Q-learning-based model is proposed to handle a continuous spac...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9745030/ |
| _version_ | 1819047426936274944 |
|---|---|
| author | Vahid Babaei Ajabshir Mehmet Serdar Guzel Erkan Bostanci |
| author_facet | Vahid Babaei Ajabshir Mehmet Serdar Guzel Erkan Bostanci |
| author_sort | Vahid Babaei Ajabshir |
| collection | DOAJ |
| description | This paper addresses the problem of navigating decentralized multi-agent systems in partially cluttered environments and proposes a new machine-learning-based approach to solve it. On the basis of this approach, a new robust and flexible Q-learning-based model is proposed to handle a continuous space problem. As in reinforcement learning (RL) algorithms, Q-learning does not require a model of the environment. Additionally, Q-Learning (QL) has the advantages of being fast and easy to design. However, one disadvantage of QL is that it needs a massive amount of memory, and it grows exponentially with each extra feature introduced to the state space. In this research, we introduce an agent-level decentralized collision avoidance low-cost model for solving a continuous space problem in partially cluttered environments, followed by introducing a method to merge non-overlapping QL features in order to reduce its size significantly by about 70% and make it possible to solve more complicated scenarios with the same memory size. Additionally, another method is proposed for minimizing the sensory data that is used by the controller. A combination of these methods is able to handle swarm navigation low memory cost with at least18 number of robots. These methods can also be adapted for deep q-learning architectures so as to increase their approximation performance and also decrease their learning time process. Experiments reveal that the proposed method also achieves a high degree of accuracy for multi-agent systems in complex scenarios. |
| first_indexed | 2024-12-21T11:00:11Z |
| format | Article |
| id | doaj.art-76220c71b53b4d3999de22dea8e8194d |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-21T11:00:11Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-76220c71b53b4d3999de22dea8e8194d2022-12-21T19:06:22ZengIEEEIEEE Access2169-35362022-01-0110352873530110.1109/ACCESS.2022.31633939745030A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered EnvironmentsVahid Babaei Ajabshir0Mehmet Serdar Guzel1https://orcid.org/0000-0002-3408-0083Erkan Bostanci2https://orcid.org/0000-0001-8547-7569Computer Engineering Department, Ankara University, Ankara, TurkeyComputer Engineering Department, Ankara University, Ankara, TurkeyComputer Engineering Department, Ankara University, Ankara, TurkeyThis paper addresses the problem of navigating decentralized multi-agent systems in partially cluttered environments and proposes a new machine-learning-based approach to solve it. On the basis of this approach, a new robust and flexible Q-learning-based model is proposed to handle a continuous space problem. As in reinforcement learning (RL) algorithms, Q-learning does not require a model of the environment. Additionally, Q-Learning (QL) has the advantages of being fast and easy to design. However, one disadvantage of QL is that it needs a massive amount of memory, and it grows exponentially with each extra feature introduced to the state space. In this research, we introduce an agent-level decentralized collision avoidance low-cost model for solving a continuous space problem in partially cluttered environments, followed by introducing a method to merge non-overlapping QL features in order to reduce its size significantly by about 70% and make it possible to solve more complicated scenarios with the same memory size. Additionally, another method is proposed for minimizing the sensory data that is used by the controller. A combination of these methods is able to handle swarm navigation low memory cost with at least18 number of robots. These methods can also be adapted for deep q-learning architectures so as to increase their approximation performance and also decrease their learning time process. Experiments reveal that the proposed method also achieves a high degree of accuracy for multi-agent systems in complex scenarios.https://ieeexplore.ieee.org/document/9745030/Adaptive algorithmcontinuous space problemmulti-agent systemsQ-learning |
| spellingShingle | Vahid Babaei Ajabshir Mehmet Serdar Guzel Erkan Bostanci A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments IEEE Access Adaptive algorithm continuous space problem multi-agent systems Q-learning |
| title | A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments |
| title_full | A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments |
| title_fullStr | A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments |
| title_full_unstemmed | A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments |
| title_short | A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments |
| title_sort | low cost q learning based approach to handle continuous space problems for decentralized multi agent robot navigation in cluttered environments |
| topic | Adaptive algorithm continuous space problem multi-agent systems Q-learning |
| url | https://ieeexplore.ieee.org/document/9745030/ |
| work_keys_str_mv | AT vahidbabaeiajabshir alowcostqlearningbasedapproachtohandlecontinuousspaceproblemsfordecentralizedmultiagentrobotnavigationinclutteredenvironments AT mehmetserdarguzel alowcostqlearningbasedapproachtohandlecontinuousspaceproblemsfordecentralizedmultiagentrobotnavigationinclutteredenvironments AT erkanbostanci alowcostqlearningbasedapproachtohandlecontinuousspaceproblemsfordecentralizedmultiagentrobotnavigationinclutteredenvironments AT vahidbabaeiajabshir lowcostqlearningbasedapproachtohandlecontinuousspaceproblemsfordecentralizedmultiagentrobotnavigationinclutteredenvironments AT mehmetserdarguzel lowcostqlearningbasedapproachtohandlecontinuousspaceproblemsfordecentralizedmultiagentrobotnavigationinclutteredenvironments AT erkanbostanci lowcostqlearningbasedapproachtohandlecontinuousspaceproblemsfordecentralizedmultiagentrobotnavigationinclutteredenvironments |