Video Enhancement with Task-Oriented Flow
© 2019, Springer Science+Business Media, LLC, part of Springer Nature. Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video pro...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer Science and Business Media LLC
2021
|
| Online Access: | https://hdl.handle.net/1721.1/135146 |
| _version_ | 1826208208014802944 |
|---|---|
| author | Xue, Tianfan Chen, Baian Wu, Jiajun Wei, Donglai Freeman, William T |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Xue, Tianfan Chen, Baian Wu, Jiajun Wei, Donglai Freeman, William T |
| author_sort | Xue, Tianfan |
| collection | MIT |
| description | © 2019, Springer Science+Business Media, LLC, part of Springer Nature. Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution. |
| first_indexed | 2024-09-23T14:02:14Z |
| format | Article |
| id | mit-1721.1/135146 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T14:02:14Z |
| publishDate | 2021 |
| publisher | Springer Science and Business Media LLC |
| record_format | dspace |
| spelling | mit-1721.1/1351462023-09-12T20:24:41Z Video Enhancement with Task-Oriented Flow Xue, Tianfan Chen, Baian Wu, Jiajun Wei, Donglai Freeman, William T Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science © 2019, Springer Science+Business Media, LLC, part of Springer Nature. Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution. 2021-10-27T20:10:56Z 2021-10-27T20:10:56Z 2019 2019-05-23T15:32:48Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/135146 en 10.1007/s11263-018-01144-2 International Journal of Computer Vision Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Springer Science and Business Media LLC arXiv |
| spellingShingle | Xue, Tianfan Chen, Baian Wu, Jiajun Wei, Donglai Freeman, William T Video Enhancement with Task-Oriented Flow |
| title | Video Enhancement with Task-Oriented Flow |
| title_full | Video Enhancement with Task-Oriented Flow |
| title_fullStr | Video Enhancement with Task-Oriented Flow |
| title_full_unstemmed | Video Enhancement with Task-Oriented Flow |
| title_short | Video Enhancement with Task-Oriented Flow |
| title_sort | video enhancement with task oriented flow |
| url | https://hdl.handle.net/1721.1/135146 |
| work_keys_str_mv | AT xuetianfan videoenhancementwithtaskorientedflow AT chenbaian videoenhancementwithtaskorientedflow AT wujiajun videoenhancementwithtaskorientedflow AT weidonglai videoenhancementwithtaskorientedflow AT freemanwilliamt videoenhancementwithtaskorientedflow |