Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces
This paper proposes a vehicle-parking trajectory planning method that addresses the issues of a long trajectory planning time and difficult training convergence during automatic parking. The process involves two stages: finding a parking space and parking planning. The first stage uses model predict...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-08-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/16/7124 |
| _version_ | 1797583283449495552 |
|---|---|
| author | Junren Shi Kexin Li Changhao Piao Jun Gao Lizhi Chen |
| author_facet | Junren Shi Kexin Li Changhao Piao Jun Gao Lizhi Chen |
| author_sort | Junren Shi |
| collection | DOAJ |
| description | This paper proposes a vehicle-parking trajectory planning method that addresses the issues of a long trajectory planning time and difficult training convergence during automatic parking. The process involves two stages: finding a parking space and parking planning. The first stage uses model predictive control (MPC) for trajectory tracking from the initial position of the vehicle to the starting point of the parking operation. The second stage employs the proximal policy optimization (PPO) algorithm to transform the parking behavior into a reinforcement learning process. A four-dimensional reward function is set to evaluate the strategy based on a formal reward, guiding the adjustment of neural network parameters and reducing the exploration of invalid actions. Finally, a simulation environment is built for the parking scene, and a network framework is designed. The proposed method is compared with the deep deterministic policy gradient and double-delay deep deterministic policy gradient algorithms in the same scene. Results confirm that the MPC controller accurately performs trajectory-tracking control with minimal steering wheel angle changes and smooth, continuous movement. The PPO-based reinforcement learning method achieves shorter learning times, totaling only 30% and 37.5% of the deep deterministic policy gradient (DDPG) and twin-delayed deep deterministic policy gradient (TD3), and the number of iterations to reach convergence for the PPO algorithm with the introduction of the four-dimensional evaluation metrics is 75% and 68% shorter compared to the DDPG and TD3 algorithms, respectively. This study demonstrates the effectiveness of the proposed method in addressing a slow convergence and long training times in parking trajectory planning, improving parking timeliness. |
| first_indexed | 2024-03-10T23:35:42Z |
| format | Article |
| id | doaj.art-7d68f2f21f3e40c78cd0fe01e11dd2be |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T23:35:42Z |
| publishDate | 2023-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-7d68f2f21f3e40c78cd0fe01e11dd2be2023-11-19T02:57:07ZengMDPI AGSensors1424-82202023-08-012316712410.3390/s23167124Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking SpacesJunren Shi0Kexin Li1Changhao Piao2Jun Gao3Lizhi Chen4School of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, ChinaSchool of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, ChinaSchool of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, ChinaSchool of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, ChinaSchool of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, ChinaThis paper proposes a vehicle-parking trajectory planning method that addresses the issues of a long trajectory planning time and difficult training convergence during automatic parking. The process involves two stages: finding a parking space and parking planning. The first stage uses model predictive control (MPC) for trajectory tracking from the initial position of the vehicle to the starting point of the parking operation. The second stage employs the proximal policy optimization (PPO) algorithm to transform the parking behavior into a reinforcement learning process. A four-dimensional reward function is set to evaluate the strategy based on a formal reward, guiding the adjustment of neural network parameters and reducing the exploration of invalid actions. Finally, a simulation environment is built for the parking scene, and a network framework is designed. The proposed method is compared with the deep deterministic policy gradient and double-delay deep deterministic policy gradient algorithms in the same scene. Results confirm that the MPC controller accurately performs trajectory-tracking control with minimal steering wheel angle changes and smooth, continuous movement. The PPO-based reinforcement learning method achieves shorter learning times, totaling only 30% and 37.5% of the deep deterministic policy gradient (DDPG) and twin-delayed deep deterministic policy gradient (TD3), and the number of iterations to reach convergence for the PPO algorithm with the introduction of the four-dimensional evaluation metrics is 75% and 68% shorter compared to the DDPG and TD3 algorithms, respectively. This study demonstrates the effectiveness of the proposed method in addressing a slow convergence and long training times in parking trajectory planning, improving parking timeliness.https://www.mdpi.com/1424-8220/23/16/7124reinforcement learningPPOMPCautoparking |
| spellingShingle | Junren Shi Kexin Li Changhao Piao Jun Gao Lizhi Chen Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces Sensors reinforcement learning PPO MPC autoparking |
| title | Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces |
| title_full | Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces |
| title_fullStr | Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces |
| title_full_unstemmed | Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces |
| title_short | Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces |
| title_sort | model based predictive control and reinforcement learning for planning vehicle parking trajectories for vertical parking spaces |
| topic | reinforcement learning PPO MPC autoparking |
| url | https://www.mdpi.com/1424-8220/23/16/7124 |
| work_keys_str_mv | AT junrenshi modelbasedpredictivecontrolandreinforcementlearningforplanningvehicleparkingtrajectoriesforverticalparkingspaces AT kexinli modelbasedpredictivecontrolandreinforcementlearningforplanningvehicleparkingtrajectoriesforverticalparkingspaces AT changhaopiao modelbasedpredictivecontrolandreinforcementlearningforplanningvehicleparkingtrajectoriesforverticalparkingspaces AT jungao modelbasedpredictivecontrolandreinforcementlearningforplanningvehicleparkingtrajectoriesforverticalparkingspaces AT lizhichen modelbasedpredictivecontrolandreinforcementlearningforplanningvehicleparkingtrajectoriesforverticalparkingspaces |