Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning
The prevailing variable speed limit (VSL) systems as an effective strategy for traffic control on motorways have the disadvantage that they only work with static VSL zones. Under changing traffic conditions, VSL systems with static VSL zones may perform suboptimally. Therefore, the adaptive design o...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-11-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/9/23/3081 |
| _version_ | 1797507439278424064 |
|---|---|
| author | Krešimir Kušić Edouard Ivanjko Filip Vrbanić Martin Gregurić Ivana Dusparic |
| author_facet | Krešimir Kušić Edouard Ivanjko Filip Vrbanić Martin Gregurić Ivana Dusparic |
| author_sort | Krešimir Kušić |
| collection | DOAJ |
| description | The prevailing variable speed limit (VSL) systems as an effective strategy for traffic control on motorways have the disadvantage that they only work with static VSL zones. Under changing traffic conditions, VSL systems with static VSL zones may perform suboptimally. Therefore, the adaptive design of VSL zones is required in traffic scenarios where congestion characteristics vary widely over space and time. To address this problem, we propose a novel distributed spatial-temporal multi-agent VSL (DWL-ST-VSL) approach capable of dynamically adjusting the length and position of VSL zones to complement the adjustment of speed limits in current VSL control systems. To model DWL-ST-VSL, distributed W-learning (DWL), a reinforcement learning (RL)-based algorithm for collaborative agent-based self-optimization toward multiple policies, is used. Each agent uses RL to learn local policies, thereby maximizing travel speed and eliminating congestion. In addition to local policies, through the concept of remote policies, agents learn how their actions affect their immediate neighbours and which policy or action is preferred in a given situation. To assess the impact of deploying additional agents in the control loop and the different cooperation levels on the control process, DWL-ST-VSL is evaluated in a four-agent configuration (DWL4-ST-VSL). This evaluation is done via SUMO microscopic simulations using collaborative agents controlling four segments upstream of the congestion in traffic scenarios with medium and high traffic loads. DWL also allows for heterogeneity in agents’ policies; cooperating agents in DWL4-ST-VSL implement two speed limit sets with different granularity. DWL4-ST-VSL outperforms all baselines (W-learning-based VSL and simple proportional speed control), which use static VSL zones. Finally, our experiments yield insights into the new concept of VSL control. This may trigger further research on using advanced learning-based technology to design a new generation of adaptive traffic control systems to meet the requirements of operating in a nonstationary environment and at the leading edge of emerging connected and autonomous vehicles in general. |
| first_indexed | 2024-03-10T04:48:30Z |
| format | Article |
| id | doaj.art-8ba32815b6b44e3fbd66a50e9a801847 |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-10T04:48:30Z |
| publishDate | 2021-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-8ba32815b6b44e3fbd66a50e9a8018472023-11-23T02:45:46ZengMDPI AGMathematics2227-73902021-11-01923308110.3390/math9233081Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement LearningKrešimir Kušić0Edouard Ivanjko1Filip Vrbanić2Martin Gregurić3Ivana Dusparic4Faculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, CroatiaFaculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, CroatiaFaculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, CroatiaFaculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, CroatiaSchool of Computer Science and Statistics, Trinity College Dublin, Dublin 2, IrelandThe prevailing variable speed limit (VSL) systems as an effective strategy for traffic control on motorways have the disadvantage that they only work with static VSL zones. Under changing traffic conditions, VSL systems with static VSL zones may perform suboptimally. Therefore, the adaptive design of VSL zones is required in traffic scenarios where congestion characteristics vary widely over space and time. To address this problem, we propose a novel distributed spatial-temporal multi-agent VSL (DWL-ST-VSL) approach capable of dynamically adjusting the length and position of VSL zones to complement the adjustment of speed limits in current VSL control systems. To model DWL-ST-VSL, distributed W-learning (DWL), a reinforcement learning (RL)-based algorithm for collaborative agent-based self-optimization toward multiple policies, is used. Each agent uses RL to learn local policies, thereby maximizing travel speed and eliminating congestion. In addition to local policies, through the concept of remote policies, agents learn how their actions affect their immediate neighbours and which policy or action is preferred in a given situation. To assess the impact of deploying additional agents in the control loop and the different cooperation levels on the control process, DWL-ST-VSL is evaluated in a four-agent configuration (DWL4-ST-VSL). This evaluation is done via SUMO microscopic simulations using collaborative agents controlling four segments upstream of the congestion in traffic scenarios with medium and high traffic loads. DWL also allows for heterogeneity in agents’ policies; cooperating agents in DWL4-ST-VSL implement two speed limit sets with different granularity. DWL4-ST-VSL outperforms all baselines (W-learning-based VSL and simple proportional speed control), which use static VSL zones. Finally, our experiments yield insights into the new concept of VSL control. This may trigger further research on using advanced learning-based technology to design a new generation of adaptive traffic control systems to meet the requirements of operating in a nonstationary environment and at the leading edge of emerging connected and autonomous vehicles in general.https://www.mdpi.com/2227-7390/9/23/3081intelligent transport systemstraffic controlspatial-temporal variable speed limitmulti-agent systemsreinforcement learningdistributed W-learning |
| spellingShingle | Krešimir Kušić Edouard Ivanjko Filip Vrbanić Martin Gregurić Ivana Dusparic Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning Mathematics intelligent transport systems traffic control spatial-temporal variable speed limit multi-agent systems reinforcement learning distributed W-learning |
| title | Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning |
| title_full | Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning |
| title_fullStr | Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning |
| title_full_unstemmed | Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning |
| title_short | Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning |
| title_sort | spatial temporal traffic flow control on motorways using distributed multi agent reinforcement learning |
| topic | intelligent transport systems traffic control spatial-temporal variable speed limit multi-agent systems reinforcement learning distributed W-learning |
| url | https://www.mdpi.com/2227-7390/9/23/3081 |
| work_keys_str_mv | AT kresimirkusic spatialtemporaltrafficflowcontrolonmotorwaysusingdistributedmultiagentreinforcementlearning AT edouardivanjko spatialtemporaltrafficflowcontrolonmotorwaysusingdistributedmultiagentreinforcementlearning AT filipvrbanic spatialtemporaltrafficflowcontrolonmotorwaysusingdistributedmultiagentreinforcementlearning AT martingreguric spatialtemporaltrafficflowcontrolonmotorwaysusingdistributedmultiagentreinforcementlearning AT ivanadusparic spatialtemporaltrafficflowcontrolonmotorwaysusingdistributedmultiagentreinforcementlearning |