Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram
Abstract To address the challenges of traffic congestion and suboptimal operational efficiency in the context of large-scale applications like production plants and warehouses that utilize multiple automatic guided vehicles (multi-AGVs), this article proposed using an Improved Q-learning (IQL) algor...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2024-01-01
|
| Series: | Complex & Intelligent Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s40747-023-01278-y |
| _version_ | 1797233168470769664 |
|---|---|
| author | Xiumei Zhang Wensong Li Hui Li Yue Liu Fang Liu |
| author_facet | Xiumei Zhang Wensong Li Hui Li Yue Liu Fang Liu |
| author_sort | Xiumei Zhang |
| collection | DOAJ |
| description | Abstract To address the challenges of traffic congestion and suboptimal operational efficiency in the context of large-scale applications like production plants and warehouses that utilize multiple automatic guided vehicles (multi-AGVs), this article proposed using an Improved Q-learning (IQL) algorithm and Macroscopic Fundamental Diagram (MFD) for the purposes of load balancing and congestion discrimination on road networks. Traditional Q-learning converges slowly, which is why we have proposed the use of an updated Q value of the previous iteration step as the maximum Q value of the next state to reduce the number of Q value comparisons and improve the algorithm’s convergence speed. When calculating the cost of AGV operation, the traditional Q-learning algorithm only considers the evaluation function of a single distance and introduces an improved reward and punishment mechanism to combine the operating distance of AGV and the road network load, which finally equalizes the road network load. MFD is the basic property of road networks and is based on MFD, which is combined with the Markov Chain (MC) model. Road network traffic congestion state discrimination method was proposed to classify the congestion state according to the detected number of vehicles on the road network. The MC model accurately discriminated the range near the critical point. Finally, the scale of the road network and the load factor were changed for several simulations. The findings indicated that the improved algorithm showed a notable ability to achieve equilibrium in the load distribution of the road network. This led to a substantial enhancement in AGV operational efficiency. |
| first_indexed | 2024-04-24T16:11:53Z |
| format | Article |
| id | doaj.art-32bc6eee56884d55b0f54809cb05137b |
| institution | Directory Open Access Journal |
| issn | 2199-4536 2198-6053 |
| language | English |
| last_indexed | 2024-04-24T16:11:53Z |
| publishDate | 2024-01-01 |
| publisher | Springer |
| record_format | Article |
| series | Complex & Intelligent Systems |
| spelling | doaj.art-32bc6eee56884d55b0f54809cb05137b2024-03-31T11:39:29ZengSpringerComplex & Intelligent Systems2199-45362198-60532024-01-011023025303910.1007/s40747-023-01278-yLoad balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagramXiumei Zhang0Wensong Li1Hui Li2Yue Liu3Fang Liu4School of Electrical and Electronic Engineering, Changchun University of TechnologySchool of Electrical and Electronic Engineering, Changchun University of TechnologySchool of Electrical and Electronic Engineering, Changchun University of TechnologySchool of Electrical and Electronic Engineering, Changchun University of TechnologySchool of Electrical and Electronic Engineering, Changchun University of TechnologyAbstract To address the challenges of traffic congestion and suboptimal operational efficiency in the context of large-scale applications like production plants and warehouses that utilize multiple automatic guided vehicles (multi-AGVs), this article proposed using an Improved Q-learning (IQL) algorithm and Macroscopic Fundamental Diagram (MFD) for the purposes of load balancing and congestion discrimination on road networks. Traditional Q-learning converges slowly, which is why we have proposed the use of an updated Q value of the previous iteration step as the maximum Q value of the next state to reduce the number of Q value comparisons and improve the algorithm’s convergence speed. When calculating the cost of AGV operation, the traditional Q-learning algorithm only considers the evaluation function of a single distance and introduces an improved reward and punishment mechanism to combine the operating distance of AGV and the road network load, which finally equalizes the road network load. MFD is the basic property of road networks and is based on MFD, which is combined with the Markov Chain (MC) model. Road network traffic congestion state discrimination method was proposed to classify the congestion state according to the detected number of vehicles on the road network. The MC model accurately discriminated the range near the critical point. Finally, the scale of the road network and the load factor were changed for several simulations. The findings indicated that the improved algorithm showed a notable ability to achieve equilibrium in the load distribution of the road network. This led to a substantial enhancement in AGV operational efficiency.https://doi.org/10.1007/s40747-023-01278-yMulti-AGVsImproved Q-learningMacroscopic fundamental diagramCongestion state discriminationLoad balancing |
| spellingShingle | Xiumei Zhang Wensong Li Hui Li Yue Liu Fang Liu Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram Complex & Intelligent Systems Multi-AGVs Improved Q-learning Macroscopic fundamental diagram Congestion state discrimination Load balancing |
| title | Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram |
| title_full | Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram |
| title_fullStr | Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram |
| title_full_unstemmed | Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram |
| title_short | Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram |
| title_sort | load balancing of multi agv road network based on improved q learning algorithm and macroscopic fundamental diagram |
| topic | Multi-AGVs Improved Q-learning Macroscopic fundamental diagram Congestion state discrimination Load balancing |
| url | https://doi.org/10.1007/s40747-023-01278-y |
| work_keys_str_mv | AT xiumeizhang loadbalancingofmultiagvroadnetworkbasedonimprovedqlearningalgorithmandmacroscopicfundamentaldiagram AT wensongli loadbalancingofmultiagvroadnetworkbasedonimprovedqlearningalgorithmandmacroscopicfundamentaldiagram AT huili loadbalancingofmultiagvroadnetworkbasedonimprovedqlearningalgorithmandmacroscopicfundamentaldiagram AT yueliu loadbalancingofmultiagvroadnetworkbasedonimprovedqlearningalgorithmandmacroscopicfundamentaldiagram AT fangliu loadbalancingofmultiagvroadnetworkbasedonimprovedqlearningalgorithmandmacroscopicfundamentaldiagram |