Learning to Simulate Dynamic Environments With GameGAN
© 2020 IEEE. Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by s...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137598 |
| _version_ | 1811087227300610048 |
|---|---|
| author | Kim, Seung Wook Zhou, Yuhao Philion, Jonah Torralba, Antonio Fidler, Sanja |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Kim, Seung Wook Zhou, Yuhao Philion, Jonah Torralba, Antonio Fidler, Sanja |
| author_sort | Kim, Seung Wook |
| collection | MIT |
| description | © 2020 IEEE. Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN 'renders' the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist. We will release the code and trained model, enabling human players to play generated games and their variations with our GameGAN. |
| first_indexed | 2024-09-23T13:42:05Z |
| format | Article |
| id | mit-1721.1/137598 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:42:05Z |
| publishDate | 2021 |
| publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| record_format | dspace |
| spelling | mit-1721.1/1375982023-06-20T17:17:48Z Learning to Simulate Dynamic Environments With GameGAN Kim, Seung Wook Zhou, Yuhao Philion, Jonah Torralba, Antonio Fidler, Sanja Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © 2020 IEEE. Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN 'renders' the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist. We will release the code and trained model, enabling human players to play generated games and their variations with our GameGAN. 2021-11-05T19:31:21Z 2021-11-05T19:31:21Z 2020 2021-01-28T15:48:06Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137598 Kim, Seung Wook, Zhou, Yuhao, Philion, Jonah, Torralba, Antonio and Fidler, Sanja. 2020. "Learning to Simulate Dynamic Environments With GameGAN." Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. en 10.1109/CVPR42600.2020.00131 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute of Electrical and Electronics Engineers (IEEE) arXiv |
| spellingShingle | Kim, Seung Wook Zhou, Yuhao Philion, Jonah Torralba, Antonio Fidler, Sanja Learning to Simulate Dynamic Environments With GameGAN |
| title | Learning to Simulate Dynamic Environments With GameGAN |
| title_full | Learning to Simulate Dynamic Environments With GameGAN |
| title_fullStr | Learning to Simulate Dynamic Environments With GameGAN |
| title_full_unstemmed | Learning to Simulate Dynamic Environments With GameGAN |
| title_short | Learning to Simulate Dynamic Environments With GameGAN |
| title_sort | learning to simulate dynamic environments with gamegan |
| url | https://hdl.handle.net/1721.1/137598 |
| work_keys_str_mv | AT kimseungwook learningtosimulatedynamicenvironmentswithgamegan AT zhouyuhao learningtosimulatedynamicenvironmentswithgamegan AT philionjonah learningtosimulatedynamicenvironmentswithgamegan AT torralbaantonio learningtosimulatedynamicenvironmentswithgamegan AT fidlersanja learningtosimulatedynamicenvironmentswithgamegan |