Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation Learning
Traffic signal control has long been considered as a critical topic in intelligent transportation systems. Most existing related methods either suffer from inefficient training or mainly focus on isolated intersections. This article aims at the cooperative control for multi-intersection traffic sign...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9241814/ |
| _version_ | 1818662501510807552 |
|---|---|
| author | Yusen Huo Qinghua Tao Jianming Hu |
| author_facet | Yusen Huo Qinghua Tao Jianming Hu |
| author_sort | Yusen Huo |
| collection | DOAJ |
| description | Traffic signal control has long been considered as a critical topic in intelligent transportation systems. Most existing related methods either suffer from inefficient training or mainly focus on isolated intersections. This article aims at the cooperative control for multi-intersection traffic signal, in which a novel end-to-end learning model is established and an efficient training method is proposed analogously, which is capable of adapting to large-scale scenarios. In the proposed method, the input traffic status in multi-intersection are expressed by a tensor without information loss, which significantly reduces model complexity than using a huge matrix, since additional convolutional layers can be required to extract features from a huge matrix. For the output, a multidimensional boolean vector is employed to simplify the control policy with abiding the practical phase changing rules, and then a multi-task learning structure is used to get the cooperative policy. Instead of only using the reinforcement learning to train the model, we employ imitation learning to integrate a rule based model to do the pre-training, which greatly accelerates the convergence. Afterwards, the reinforcement learning method is adopted to continue the fine training, where proximal policy optimization algorithm is incorporated to solve the policy collapse problem in multi-dimensional output situation. Numerical experiments demonstrate the distinctive advantages of the proposed method with comparison to the efficiency and accuracy of the related state-of-the-art methods. |
| first_indexed | 2024-12-17T05:01:57Z |
| format | Article |
| id | doaj.art-7b734978160b434db13930f781753957 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-17T05:01:57Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-7b734978160b434db13930f7817539572022-12-21T22:02:32ZengIEEEIEEE Access2169-35362020-01-01819957319958510.1109/ACCESS.2020.30344199241814Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation LearningYusen Huo0https://orcid.org/0000-0002-4823-9710Qinghua Tao1https://orcid.org/0000-0001-9705-7748Jianming Hu2Department of Automation, Tsinghua University, Beijing, ChinaDepartment of Automation, Tsinghua University, Beijing, ChinaDepartment of Automation, Tsinghua University, Beijing, ChinaTraffic signal control has long been considered as a critical topic in intelligent transportation systems. Most existing related methods either suffer from inefficient training or mainly focus on isolated intersections. This article aims at the cooperative control for multi-intersection traffic signal, in which a novel end-to-end learning model is established and an efficient training method is proposed analogously, which is capable of adapting to large-scale scenarios. In the proposed method, the input traffic status in multi-intersection are expressed by a tensor without information loss, which significantly reduces model complexity than using a huge matrix, since additional convolutional layers can be required to extract features from a huge matrix. For the output, a multidimensional boolean vector is employed to simplify the control policy with abiding the practical phase changing rules, and then a multi-task learning structure is used to get the cooperative policy. Instead of only using the reinforcement learning to train the model, we employ imitation learning to integrate a rule based model to do the pre-training, which greatly accelerates the convergence. Afterwards, the reinforcement learning method is adopted to continue the fine training, where proximal policy optimization algorithm is incorporated to solve the policy collapse problem in multi-dimensional output situation. Numerical experiments demonstrate the distinctive advantages of the proposed method with comparison to the efficiency and accuracy of the related state-of-the-art methods.https://ieeexplore.ieee.org/document/9241814/Deep reinforcement learningimitation learningmulti-intersectionproximal policy optimizationtensor |
| spellingShingle | Yusen Huo Qinghua Tao Jianming Hu Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation Learning IEEE Access Deep reinforcement learning imitation learning multi-intersection proximal policy optimization tensor |
| title | Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation Learning |
| title_full | Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation Learning |
| title_fullStr | Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation Learning |
| title_full_unstemmed | Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation Learning |
| title_short | Cooperative Control for Multi-Intersection Traffic Signal Based on Deep Reinforcement Learning and Imitation Learning |
| title_sort | cooperative control for multi intersection traffic signal based on deep reinforcement learning and imitation learning |
| topic | Deep reinforcement learning imitation learning multi-intersection proximal policy optimization tensor |
| url | https://ieeexplore.ieee.org/document/9241814/ |
| work_keys_str_mv | AT yusenhuo cooperativecontrolformultiintersectiontrafficsignalbasedondeepreinforcementlearningandimitationlearning AT qinghuatao cooperativecontrolformultiintersectiontrafficsignalbasedondeepreinforcementlearningandimitationlearning AT jianminghu cooperativecontrolformultiintersectiontrafficsignalbasedondeepreinforcementlearningandimitationlearning |