GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving
Deep reinforcement learning (DRL) has been demonstrated to be effective for several complex decision-making applications, such as autonomous driving and robotics. However, DRL is notoriously limited by its high sample complexity and its lack of stability. Prior knowledge, e.g., as expert demonstrati...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Robotics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2218-6581/12/5/127 |
| _version_ | 1797572348390408192 |
|---|---|
| author | Raphael Chekroun Marin Toromanoff Sascha Hornauer Fabien Moutarde |
| author_facet | Raphael Chekroun Marin Toromanoff Sascha Hornauer Fabien Moutarde |
| author_sort | Raphael Chekroun |
| collection | DOAJ |
| description | Deep reinforcement learning (DRL) has been demonstrated to be effective for several complex decision-making applications, such as autonomous driving and robotics. However, DRL is notoriously limited by its high sample complexity and its lack of stability. Prior knowledge, e.g., as expert demonstrations, is often available but challenging to leverage to mitigate these issues. In this paper, we propose General Reinforced Imitation (GRI), a novel method which combines benefits from exploration and expert data and is straightforward to implement over any off-policy RL algorithm. We make one simplifying hypothesis: expert demonstrations can be seen as perfect data whose underlying policy gets a constant high reward. Based on this assumption, GRI introduces the notion of offline demonstration agent. This agent sends expert data which are processed both concurrently and indistinguishably with the experiences coming from the online RL exploration agent. We show that our approach enables major improvements on camera-based autonomous driving in urban environments. We further validate the GRI method on Mujoco continuous control tasks with different off-policy RL algorithms. Our method ranked first on the CARLA Leaderboard and outperforms World on Rails, the previous state-of-the-art method, by 17%. |
| first_indexed | 2024-03-10T20:54:59Z |
| format | Article |
| id | doaj.art-b1048a8c866d48639b380ceedde5efc9 |
| institution | Directory Open Access Journal |
| issn | 2218-6581 |
| language | English |
| last_indexed | 2024-03-10T20:54:59Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Robotics |
| spelling | doaj.art-b1048a8c866d48639b380ceedde5efc92023-11-19T18:01:11ZengMDPI AGRobotics2218-65812023-09-0112512710.3390/robotics12050127GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous DrivingRaphael Chekroun0Marin Toromanoff1Sascha Hornauer2Fabien Moutarde3Center for Robotics, Mines Paris, PSL University, 75006 Paris, FranceValeo Driving Assistant Research, 75017 Paris, FranceCenter for Robotics, Mines Paris, PSL University, 75006 Paris, FranceCenter for Robotics, Mines Paris, PSL University, 75006 Paris, FranceDeep reinforcement learning (DRL) has been demonstrated to be effective for several complex decision-making applications, such as autonomous driving and robotics. However, DRL is notoriously limited by its high sample complexity and its lack of stability. Prior knowledge, e.g., as expert demonstrations, is often available but challenging to leverage to mitigate these issues. In this paper, we propose General Reinforced Imitation (GRI), a novel method which combines benefits from exploration and expert data and is straightforward to implement over any off-policy RL algorithm. We make one simplifying hypothesis: expert demonstrations can be seen as perfect data whose underlying policy gets a constant high reward. Based on this assumption, GRI introduces the notion of offline demonstration agent. This agent sends expert data which are processed both concurrently and indistinguishably with the experiences coming from the online RL exploration agent. We show that our approach enables major improvements on camera-based autonomous driving in urban environments. We further validate the GRI method on Mujoco continuous control tasks with different off-policy RL algorithms. Our method ranked first on the CARLA Leaderboard and outperforms World on Rails, the previous state-of-the-art method, by 17%.https://www.mdpi.com/2218-6581/12/5/127deep reinforcement learningimitation learningautonomous drivingMujoco |
| spellingShingle | Raphael Chekroun Marin Toromanoff Sascha Hornauer Fabien Moutarde GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving Robotics deep reinforcement learning imitation learning autonomous driving Mujoco |
| title | GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving |
| title_full | GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving |
| title_fullStr | GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving |
| title_full_unstemmed | GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving |
| title_short | GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving |
| title_sort | gri general reinforced imitation and its application to vision based autonomous driving |
| topic | deep reinforcement learning imitation learning autonomous driving Mujoco |
| url | https://www.mdpi.com/2218-6581/12/5/127 |
| work_keys_str_mv | AT raphaelchekroun grigeneralreinforcedimitationanditsapplicationtovisionbasedautonomousdriving AT marintoromanoff grigeneralreinforcedimitationanditsapplicationtovisionbasedautonomousdriving AT saschahornauer grigeneralreinforcedimitationanditsapplicationtovisionbasedautonomousdriving AT fabienmoutarde grigeneralreinforcedimitationanditsapplicationtovisionbasedautonomousdriving |