Leveraging Engineering Expertise in Deep Reinforcement Learning
Deep reinforcement learning has been used to craft robust and performant control policies for legged robotics. However, the engineering processes to create these policies are often plagued by long training times that slow down engineering iteration. This thesis suggests that model-based controllers...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/147435 |
| _version_ | 1826201103726804992 |
|---|---|
| author | Ackerman, Liam J. |
| author2 | Kim, Sangbae |
| author_facet | Kim, Sangbae Ackerman, Liam J. |
| author_sort | Ackerman, Liam J. |
| collection | MIT |
| description | Deep reinforcement learning has been used to craft robust and performant control policies for legged robotics. However, the engineering processes to create these policies are often plagued by long training times that slow down engineering iteration. This thesis suggests that model-based controllers offer a wealth of successful computation that may be used within reinforcement learning control pipelines to improve learning efficiency. Two ideas incorporate this engineering expertise to increase reinforcement learning efficiency. First, successful model-based computations are pre-processed and incorporated directly into network observations. Introducing these terms into the reinforcement learning architecture is shown to increase learning speeds and policy performance dramatically. Next, inspired by model-based task hierarchies, more structure is added to the reinforcement learning objective function to activate and deactivate reward terms based on an agent’s state. This structure is intended to avoid local minima which impede learning. This reward restructure is shown to avoid local minima during training but degrades final policy performance at edge-cases. |
| first_indexed | 2024-09-23T11:46:37Z |
| format | Thesis |
| id | mit-1721.1/147435 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T11:46:37Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1474352023-01-20T03:37:06Z Leveraging Engineering Expertise in Deep Reinforcement Learning Ackerman, Liam J. Kim, Sangbae Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Deep reinforcement learning has been used to craft robust and performant control policies for legged robotics. However, the engineering processes to create these policies are often plagued by long training times that slow down engineering iteration. This thesis suggests that model-based controllers offer a wealth of successful computation that may be used within reinforcement learning control pipelines to improve learning efficiency. Two ideas incorporate this engineering expertise to increase reinforcement learning efficiency. First, successful model-based computations are pre-processed and incorporated directly into network observations. Introducing these terms into the reinforcement learning architecture is shown to increase learning speeds and policy performance dramatically. Next, inspired by model-based task hierarchies, more structure is added to the reinforcement learning objective function to activate and deactivate reward terms based on an agent’s state. This structure is intended to avoid local minima which impede learning. This reward restructure is shown to avoid local minima during training but degrades final policy performance at edge-cases. M.Eng. 2023-01-19T19:50:12Z 2023-01-19T19:50:12Z 2022-09 2022-09-16T20:23:57.251Z Thesis https://hdl.handle.net/1721.1/147435 In Copyright - Educational Use Permitted Copyright MIT http://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Ackerman, Liam J. Leveraging Engineering Expertise in Deep Reinforcement Learning |
| title | Leveraging Engineering Expertise in Deep Reinforcement Learning |
| title_full | Leveraging Engineering Expertise in Deep Reinforcement Learning |
| title_fullStr | Leveraging Engineering Expertise in Deep Reinforcement Learning |
| title_full_unstemmed | Leveraging Engineering Expertise in Deep Reinforcement Learning |
| title_short | Leveraging Engineering Expertise in Deep Reinforcement Learning |
| title_sort | leveraging engineering expertise in deep reinforcement learning |
| url | https://hdl.handle.net/1721.1/147435 |
| work_keys_str_mv | AT ackermanliamj leveragingengineeringexpertiseindeepreinforcementlearning |