Painting many pasts: Synthesizing time lapse videos of paintings
We introduce a new video synthesis task: synthesizing time lapse videos depicting how a given painting might have been created. Artists paint using unique combinations of brushes, strokes, and colors. There are often many possible ways to create a given painting. Our goal is to learn to capture this...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021
|
| Online Access: | https://hdl.handle.net/1721.1/129682 |
| _version_ | 1811069132779552768 |
|---|---|
| author | Zhao, Amy (Xiaoyu Amy) Balakrishnan, Guha Lewis, Kathleen M.(Kathleen Marie) Durand, Frederic Guttag, John V Dalca, Adrian Vasile |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Zhao, Amy (Xiaoyu Amy) Balakrishnan, Guha Lewis, Kathleen M.(Kathleen Marie) Durand, Frederic Guttag, John V Dalca, Adrian Vasile |
| author_sort | Zhao, Amy (Xiaoyu Amy) |
| collection | MIT |
| description | We introduce a new video synthesis task: synthesizing time lapse videos depicting how a given painting might have been created. Artists paint using unique combinations of brushes, strokes, and colors. There are often many possible ways to create a given painting. Our goal is to learn to capture this rich range of possibilities. Creating distributions of long-term videos is a challenge for learning-based video synthesis methods. We present a probabilistic model that, given a single image of a completed painting, recurrently synthesizes steps of the painting process. We implement this model as a convolutional neural network, and introduce a novel training scheme to enable learning from a limited dataset of painting time lapses. We demonstrate that this model can be used to sample many time steps, enabling long-term stochastic video synthesis. We evaluate our method on digital and watercolor paintings collected from video websites, and show that human raters find our synthetic videos to be similar to time lapse videos produced by real artists. Our code is available at https://xamyzhao.github.io/timecraft. |
| first_indexed | 2024-09-23T08:06:14Z |
| format | Article |
| id | mit-1721.1/129682 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T08:06:14Z |
| publishDate | 2021 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1296822022-09-23T10:55:33Z Painting many pasts: Synthesizing time lapse videos of paintings Zhao, Amy (Xiaoyu Amy) Balakrishnan, Guha Lewis, Kathleen M.(Kathleen Marie) Durand, Frederic Guttag, John V Dalca, Adrian Vasile Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science We introduce a new video synthesis task: synthesizing time lapse videos depicting how a given painting might have been created. Artists paint using unique combinations of brushes, strokes, and colors. There are often many possible ways to create a given painting. Our goal is to learn to capture this rich range of possibilities. Creating distributions of long-term videos is a challenge for learning-based video synthesis methods. We present a probabilistic model that, given a single image of a completed painting, recurrently synthesizes steps of the painting process. We implement this model as a convolutional neural network, and introduce a novel training scheme to enable learning from a limited dataset of painting time lapses. We demonstrate that this model can be used to sample many time steps, enabling long-term stochastic video synthesis. We evaluate our method on digital and watercolor paintings collected from video websites, and show that human raters find our synthetic videos to be similar to time lapse videos produced by real artists. Our code is available at https://xamyzhao.github.io/timecraft. 2021-02-05T13:21:07Z 2021-02-05T13:21:07Z 2020-06 2020-12-11T17:39:40Z Article http://purl.org/eprint/type/ConferencePaper 9781728171685 https://hdl.handle.net/1721.1/129682 Zhao, Amy et al. “Painting many pasts: Synthesizing time lapse videos of paintings.” Paper in the Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, 13-19 June 2020, IEEE: 8789–8797 © 2020 The Author(s) en 10.1109/CVPR42600.2020.00846 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf IEEE arXiv |
| spellingShingle | Zhao, Amy (Xiaoyu Amy) Balakrishnan, Guha Lewis, Kathleen M.(Kathleen Marie) Durand, Frederic Guttag, John V Dalca, Adrian Vasile Painting many pasts: Synthesizing time lapse videos of paintings |
| title | Painting many pasts: Synthesizing time lapse videos of paintings |
| title_full | Painting many pasts: Synthesizing time lapse videos of paintings |
| title_fullStr | Painting many pasts: Synthesizing time lapse videos of paintings |
| title_full_unstemmed | Painting many pasts: Synthesizing time lapse videos of paintings |
| title_short | Painting many pasts: Synthesizing time lapse videos of paintings |
| title_sort | painting many pasts synthesizing time lapse videos of paintings |
| url | https://hdl.handle.net/1721.1/129682 |
| work_keys_str_mv | AT zhaoamyxiaoyuamy paintingmanypastssynthesizingtimelapsevideosofpaintings AT balakrishnanguha paintingmanypastssynthesizingtimelapsevideosofpaintings AT lewiskathleenmkathleenmarie paintingmanypastssynthesizingtimelapsevideosofpaintings AT durandfrederic paintingmanypastssynthesizingtimelapsevideosofpaintings AT guttagjohnv paintingmanypastssynthesizingtimelapsevideosofpaintings AT dalcaadrianvasile paintingmanypastssynthesizingtimelapsevideosofpaintings |