Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic
Lane-free traffic is a novel research domain, in which vehicles no longer adhere to the notion of lanes, and consider the whole lateral space within the road boundaries. This constitutes an entirely different problem domain for autonomous driving compared to lane-based traffic, as there is no leader...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-03-01
|
| Series: | Systems |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-8954/11/3/134 |
| _version_ | 1797608864936361984 |
|---|---|
| author | Athanasia Karalakou Dimitrios Troullinos Georgios Chalkiadakis Markos Papageorgiou |
| author_facet | Athanasia Karalakou Dimitrios Troullinos Georgios Chalkiadakis Markos Papageorgiou |
| author_sort | Athanasia Karalakou |
| collection | DOAJ |
| description | Lane-free traffic is a novel research domain, in which vehicles no longer adhere to the notion of lanes, and consider the whole lateral space within the road boundaries. This constitutes an entirely different problem domain for autonomous driving compared to lane-based traffic, as there is no leader vehicle or lane-changing operation. Therefore, the observations of the vehicles need to properly accommodate the lane-free environment without carrying over bias from lane-based approaches. The recent successes of deep reinforcement learning (DRL) for lane-based approaches, along with emerging work for lane-free traffic environments, render DRL for lane-free traffic an interesting endeavor to investigate. In this paper, we provide an extensive look at the DRL formulation, focusing on the reward function of a lane-free autonomous driving agent. Our main interest is designing an effective reward function, as the reward model is crucial in determining the overall efficiency of the resulting policy. Specifically, we construct different components of reward functions tied to the environment at various levels of information. Then, we combine and collate the aforementioned components, and focus on attaining a reward function that results in a policy that manages to both reduce the collisions among vehicles and address their requirement of maintaining a desired speed. Additionally, we employ two popular DRL algorithms—namely, deep Q-networks (enhanced with some commonly used extensions), and deep deterministic policy gradient (DDPG), which results in better policies. Our experiments provide a thorough investigative study on the effectiveness of different combinations among the various reward components we propose, and confirm that our DRL-employing autonomous vehicle is able to gradually learn effective policies in environments with varying levels of difficulty, especially when all of the proposed rewards components are properly combined. |
| first_indexed | 2024-03-11T05:50:30Z |
| format | Article |
| id | doaj.art-b108bf04d0e04dc1b14b4ff84f5de6d1 |
| institution | Directory Open Access Journal |
| issn | 2079-8954 |
| language | English |
| last_indexed | 2024-03-11T05:50:30Z |
| publishDate | 2023-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Systems |
| spelling | doaj.art-b108bf04d0e04dc1b14b4ff84f5de6d12023-11-17T14:11:11ZengMDPI AGSystems2079-89542023-03-0111313410.3390/systems11030134Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free TrafficAthanasia Karalakou0Dimitrios Troullinos1Georgios Chalkiadakis2Markos Papageorgiou3School of Electrical and Computer Engineering, Technical University of Crete, 73100 Chania, GreeceSchool of Production Engineering and Management, Technical University of Crete, 73100 Chania, GreeceSchool of Electrical and Computer Engineering, Technical University of Crete, 73100 Chania, GreeceSchool of Production Engineering and Management, Technical University of Crete, 73100 Chania, GreeceLane-free traffic is a novel research domain, in which vehicles no longer adhere to the notion of lanes, and consider the whole lateral space within the road boundaries. This constitutes an entirely different problem domain for autonomous driving compared to lane-based traffic, as there is no leader vehicle or lane-changing operation. Therefore, the observations of the vehicles need to properly accommodate the lane-free environment without carrying over bias from lane-based approaches. The recent successes of deep reinforcement learning (DRL) for lane-based approaches, along with emerging work for lane-free traffic environments, render DRL for lane-free traffic an interesting endeavor to investigate. In this paper, we provide an extensive look at the DRL formulation, focusing on the reward function of a lane-free autonomous driving agent. Our main interest is designing an effective reward function, as the reward model is crucial in determining the overall efficiency of the resulting policy. Specifically, we construct different components of reward functions tied to the environment at various levels of information. Then, we combine and collate the aforementioned components, and focus on attaining a reward function that results in a policy that manages to both reduce the collisions among vehicles and address their requirement of maintaining a desired speed. Additionally, we employ two popular DRL algorithms—namely, deep Q-networks (enhanced with some commonly used extensions), and deep deterministic policy gradient (DDPG), which results in better policies. Our experiments provide a thorough investigative study on the effectiveness of different combinations among the various reward components we propose, and confirm that our DRL-employing autonomous vehicle is able to gradually learn effective policies in environments with varying levels of difficulty, especially when all of the proposed rewards components are properly combined.https://www.mdpi.com/2079-8954/11/3/134deep reinforcement learninglane-free trafficautonomous driving |
| spellingShingle | Athanasia Karalakou Dimitrios Troullinos Georgios Chalkiadakis Markos Papageorgiou Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic Systems deep reinforcement learning lane-free traffic autonomous driving |
| title | Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic |
| title_full | Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic |
| title_fullStr | Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic |
| title_full_unstemmed | Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic |
| title_short | Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic |
| title_sort | deep reinforcement learning reward function design for autonomous driving in lane free traffic |
| topic | deep reinforcement learning lane-free traffic autonomous driving |
| url | https://www.mdpi.com/2079-8954/11/3/134 |
| work_keys_str_mv | AT athanasiakaralakou deepreinforcementlearningrewardfunctiondesignforautonomousdrivinginlanefreetraffic AT dimitriostroullinos deepreinforcementlearningrewardfunctiondesignforautonomousdrivinginlanefreetraffic AT georgioschalkiadakis deepreinforcementlearningrewardfunctiondesignforautonomousdrivinginlanefreetraffic AT markospapageorgiou deepreinforcementlearningrewardfunctiondesignforautonomousdrivinginlanefreetraffic |