Powderworld: A Platform for Understanding Generalization via Rich Task Distributions
One of the grand challenges of reinforcement learning is the ability to generalize to new tasks. However, general agents require a set of rich, diverse tasks to train on. Designing a ‘foundation environment’ for such tasks is tricky – the ideal environment would support a range of emergent phenomena...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/151635 |
| _version_ | 1826211240101281792 |
|---|---|
| author | Frans, Kevin |
| author2 | Isola, Phillip |
| author_facet | Isola, Phillip Frans, Kevin |
| author_sort | Frans, Kevin |
| collection | MIT |
| description | One of the grand challenges of reinforcement learning is the ability to generalize to new tasks. However, general agents require a set of rich, diverse tasks to train on. Designing a ‘foundation environment’ for such tasks is tricky – the ideal environment would support a range of emergent phenomena, an expressive task space, and fast runtime. To take a step towards addressing this research bottleneck, this work presents Powderworld, a lightweight yet expressive simulation environment running directly on the GPU. Within Powderworld, two motivating challenges are presented, one for world-modelling and one for reinforcement learning. Each contains hand-designed test tasks to examine generalization. Experiments indicate that increasing the environment’s complexity improves generalization for world models and certain reinforcement learning agents, yet may inhibit learning in high-variance environments. Powderworld aims to support the study of generalization by providing a source of diverse tasks arising from the same core rules. |
| first_indexed | 2024-09-23T15:02:47Z |
| format | Thesis |
| id | mit-1721.1/151635 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T15:02:47Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1516352023-08-01T03:16:50Z Powderworld: A Platform for Understanding Generalization via Rich Task Distributions Frans, Kevin Isola, Phillip Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science One of the grand challenges of reinforcement learning is the ability to generalize to new tasks. However, general agents require a set of rich, diverse tasks to train on. Designing a ‘foundation environment’ for such tasks is tricky – the ideal environment would support a range of emergent phenomena, an expressive task space, and fast runtime. To take a step towards addressing this research bottleneck, this work presents Powderworld, a lightweight yet expressive simulation environment running directly on the GPU. Within Powderworld, two motivating challenges are presented, one for world-modelling and one for reinforcement learning. Each contains hand-designed test tasks to examine generalization. Experiments indicate that increasing the environment’s complexity improves generalization for world models and certain reinforcement learning agents, yet may inhibit learning in high-variance environments. Powderworld aims to support the study of generalization by providing a source of diverse tasks arising from the same core rules. M.Eng. 2023-07-31T19:54:42Z 2023-07-31T19:54:42Z 2023-06 2023-06-06T16:34:54.162Z Thesis https://hdl.handle.net/1721.1/151635 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Frans, Kevin Powderworld: A Platform for Understanding Generalization via Rich Task Distributions |
| title | Powderworld: A Platform for Understanding Generalization via Rich Task Distributions |
| title_full | Powderworld: A Platform for Understanding Generalization via Rich Task Distributions |
| title_fullStr | Powderworld: A Platform for Understanding Generalization via Rich Task Distributions |
| title_full_unstemmed | Powderworld: A Platform for Understanding Generalization via Rich Task Distributions |
| title_short | Powderworld: A Platform for Understanding Generalization via Rich Task Distributions |
| title_sort | powderworld a platform for understanding generalization via rich task distributions |
| url | https://hdl.handle.net/1721.1/151635 |
| work_keys_str_mv | AT franskevin powderworldaplatformforunderstandinggeneralizationviarichtaskdistributions |