Generating the Future with Adversarial Transformers
We learn models to generate the immediate future in video. This problem has two main challenges. Firstly, since the future is uncertain, models should be multi-modal, which can be difficult to learn. Secondly, since the future is similar to the past, models store low-level details, which complicates...
| Main Authors: | , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020
|
| Online Access: | https://hdl.handle.net/1721.1/123483 |
| _version_ | 1826196062989189120 |
|---|---|
| author | Vondrick, Carl Martin Torralba, Antonio |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Vondrick, Carl Martin Torralba, Antonio |
| author_sort | Vondrick, Carl Martin |
| collection | MIT |
| description | We learn models to generate the immediate future in video. This problem has two main challenges. Firstly, since the future is uncertain, models should be multi-modal, which can be difficult to learn. Secondly, since the future is similar to the past, models store low-level details, which complicates learning of high-level semantics. We propose a framework to tackle both of these challenges. We present a model that generates the future by transforming pixels in the past. Our approach explicitly disentangles the model's memory from the prediction, which helps the model learn desirable invariances. Experiments suggest that this model can generate short videos of plausible futures. We believe predictive models have many applications in robotics, health-care, and video understanding. Keywords: predictive models; generators; visualization; network architecture; spatial resolution; semantics; robots |
| first_indexed | 2024-09-23T10:20:16Z |
| format | Article |
| id | mit-1721.1/123483 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T10:20:16Z |
| publishDate | 2020 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1234832022-09-30T20:28:57Z Generating the Future with Adversarial Transformers Vondrick, Carl Martin Torralba, Antonio Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science We learn models to generate the immediate future in video. This problem has two main challenges. Firstly, since the future is uncertain, models should be multi-modal, which can be difficult to learn. Secondly, since the future is similar to the past, models store low-level details, which complicates learning of high-level semantics. We propose a framework to tackle both of these challenges. We present a model that generates the future by transforming pixels in the past. Our approach explicitly disentangles the model's memory from the prediction, which helps the model learn desirable invariances. Experiments suggest that this model can generate short videos of plausible futures. We believe predictive models have many applications in robotics, health-care, and video understanding. Keywords: predictive models; generators; visualization; network architecture; spatial resolution; semantics; robots 2020-01-21T15:24:24Z 2020-01-21T15:24:24Z 2017-11-09 2017-07 2019-07-11T16:28:34Z Article http://purl.org/eprint/type/ConferencePaper 9781538604571 9781538604588 1063-6919 https://hdl.handle.net/1721.1/123483 Vondrick, Carl, and Antonio Torralba. "Generating the Future with Adversarial Transformers." 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, Honolulu, Hawaii, USA, IEEE, 2017 en http://dx.doi.org/10.1109/cvpr.2017.319 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf IEEE other univ website |
| spellingShingle | Vondrick, Carl Martin Torralba, Antonio Generating the Future with Adversarial Transformers |
| title | Generating the Future with Adversarial Transformers |
| title_full | Generating the Future with Adversarial Transformers |
| title_fullStr | Generating the Future with Adversarial Transformers |
| title_full_unstemmed | Generating the Future with Adversarial Transformers |
| title_short | Generating the Future with Adversarial Transformers |
| title_sort | generating the future with adversarial transformers |
| url | https://hdl.handle.net/1721.1/123483 |
| work_keys_str_mv | AT vondrickcarlmartin generatingthefuturewithadversarialtransformers AT torralbaantonio generatingthefuturewithadversarialtransformers |