Learning Sim-to-Real Robot Parkour from RGB Images
Advancements in quadrupedal robot locomotion have yielded impressive results, achieving dynamic maneuvers like climbing, ducking, and jumping. These successes are largely attributed to depth-based visual locomotion policies, known for their robust transferability between simulated and real-world env...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2024
|
| Online Access: | https://hdl.handle.net/1721.1/156972 |
| _version_ | 1824458266391871488 |
|---|---|
| author | Jenkins, Andrew |
| author2 | Agrawal, Pulkit |
| author_facet | Agrawal, Pulkit Jenkins, Andrew |
| author_sort | Jenkins, Andrew |
| collection | MIT |
| description | Advancements in quadrupedal robot locomotion have yielded impressive results, achieving dynamic maneuvers like climbing, ducking, and jumping. These successes are largely attributed to depth-based visual locomotion policies, known for their robust transferability between simulated and real-world environments (sim-to-real). However, depth information inherently lacks the semantic information present in RGB images. This thesis investigates the application of an RGB visual locomotion policy for navigating complex environments, specifically focusing on extreme parkour terrain. While RGB data offers a deeper understanding of the scene through semantic cues, it presents challenges in sim-to-real transfer due to large domain gaps. This work proposes a novel approach for training an RGB parkour policy and demonstrates that it achieves performance comparable to depth-based approaches in simulation. Furthermore, we successfully deploy and evaluate our RGB policy on real-world parkour obstacles, signifying its potential for practical applications. |
| first_indexed | 2025-02-19T04:23:09Z |
| format | Thesis |
| id | mit-1721.1/156972 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2025-02-19T04:23:09Z |
| publishDate | 2024 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1569722024-09-25T03:26:28Z Learning Sim-to-Real Robot Parkour from RGB Images Jenkins, Andrew Agrawal, Pulkit Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Advancements in quadrupedal robot locomotion have yielded impressive results, achieving dynamic maneuvers like climbing, ducking, and jumping. These successes are largely attributed to depth-based visual locomotion policies, known for their robust transferability between simulated and real-world environments (sim-to-real). However, depth information inherently lacks the semantic information present in RGB images. This thesis investigates the application of an RGB visual locomotion policy for navigating complex environments, specifically focusing on extreme parkour terrain. While RGB data offers a deeper understanding of the scene through semantic cues, it presents challenges in sim-to-real transfer due to large domain gaps. This work proposes a novel approach for training an RGB parkour policy and demonstrates that it achieves performance comparable to depth-based approaches in simulation. Furthermore, we successfully deploy and evaluate our RGB policy on real-world parkour obstacles, signifying its potential for practical applications. M.Eng. 2024-09-24T18:23:54Z 2024-09-24T18:23:54Z 2024-05 2024-07-11T15:30:33.981Z Thesis https://hdl.handle.net/1721.1/156972 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Jenkins, Andrew Learning Sim-to-Real Robot Parkour from RGB Images |
| title | Learning Sim-to-Real Robot Parkour from RGB Images |
| title_full | Learning Sim-to-Real Robot Parkour from RGB Images |
| title_fullStr | Learning Sim-to-Real Robot Parkour from RGB Images |
| title_full_unstemmed | Learning Sim-to-Real Robot Parkour from RGB Images |
| title_short | Learning Sim-to-Real Robot Parkour from RGB Images |
| title_sort | learning sim to real robot parkour from rgb images |
| url | https://hdl.handle.net/1721.1/156972 |
| work_keys_str_mv | AT jenkinsandrew learningsimtorealrobotparkourfromrgbimages |