Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
© 2016 NIPS Foundation - All Rights Reserved. We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabil...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137466 |
| _version_ | 1826212772049846272 |
|---|---|
| author | Xue, Tianfan Wu, Jiajun Bouman, Katherine L. Freeman, William T. |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Xue, Tianfan Wu, Jiajun Bouman, Katherine L. Freeman, William T. |
| author_sort | Xue, Tianfan |
| collection | MIT |
| description | © 2016 NIPS Foundation - All Rights Reserved. We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-world video frames. We also show that our model can be applied to visual analogy-making, and present an analysis of the learned network representations. |
| first_indexed | 2024-09-23T15:37:40Z |
| format | Article |
| id | mit-1721.1/137466 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T15:37:40Z |
| publishDate | 2021 |
| record_format | dspace |
| spelling | mit-1721.1/1374662023-02-10T19:53:57Z Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks Xue, Tianfan Wu, Jiajun Bouman, Katherine L. Freeman, William T. Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science © 2016 NIPS Foundation - All Rights Reserved. We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-world video frames. We also show that our model can be applied to visual analogy-making, and present an analysis of the learned network representations. 2021-11-05T14:06:32Z 2021-11-05T14:06:32Z 2016 2019-05-28T12:58:33Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137466 Xue, Tianfan, Wu, Jiajun, Bouman, Katherine L. and Freeman, William T. 2016. "Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks." en https://papers.nips.cc/paper/6552-visual-dynamics-probabilistic-future-frame-synthesis-via-cross-convolutional-networks Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Neural Information Processing Systems (NIPS) |
| spellingShingle | Xue, Tianfan Wu, Jiajun Bouman, Katherine L. Freeman, William T. Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks |
| title | Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks |
| title_full | Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks |
| title_fullStr | Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks |
| title_full_unstemmed | Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks |
| title_short | Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks |
| title_sort | visual dynamics probabilistic future frame synthesis via cross convolutional networks |
| url | https://hdl.handle.net/1721.1/137466 |
| work_keys_str_mv | AT xuetianfan visualdynamicsprobabilisticfutureframesynthesisviacrossconvolutionalnetworks AT wujiajun visualdynamicsprobabilisticfutureframesynthesisviacrossconvolutionalnetworks AT boumankatherinel visualdynamicsprobabilisticfutureframesynthesisviacrossconvolutionalnetworks AT freemanwilliamt visualdynamicsprobabilisticfutureframesynthesisviacrossconvolutionalnetworks |