Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks
Although deep learning-based approaches for video processing have been extensively investigated, the lack of generality in network construction makes it challenging for practical applications, particularly in video restoration. As a result, this paper presents a universal video restoration model tha...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/18/10001 |
| _version_ | 1797581450846928896 |
|---|---|
| author | Yuanfeng Zheng Yuchen Yan Hao Jiang |
| author_facet | Yuanfeng Zheng Yuchen Yan Hao Jiang |
| author_sort | Yuanfeng Zheng |
| collection | DOAJ |
| description | Although deep learning-based approaches for video processing have been extensively investigated, the lack of generality in network construction makes it challenging for practical applications, particularly in video restoration. As a result, this paper presents a universal video restoration model that can simultaneously tackle video inpainting and super-resolution tasks. The network, called Video-Restoration-Net (VRN), consists of four components: (1) an encoder to extract features from each frame, (2) a non-local network that recombines features from adjacent frames or different locations of a given frame, (3) a decoder to restore the coarse video from the output of a non-local block, and (4) a refinement network to refine the coarse video on the frame level. The framework is trained in a three-step pipeline to improve training stability for both tasks. Specifically, we first suggest an automated technique to generate full video datasets for super-resolution reconstruction and another complete-incomplete video dataset for inpainting, respectively. A VRN is then trained to inpaint the incomplete videos. Meanwhile, the full video datasets are adopted to train another VRN frame-wisely and validate it against authoritative datasets. We show quantitative comparisons with several baseline models, achieving 40.5042 dB/0.99473 on PSNR/SSIM in the inpainting task, while during the SR task we obtained 28.41 dB/0.7953 and 27.25/0.8152 on BSD100 and Urban100, respectively. The qualitative comparisons demonstrate that our proposed model is able to complete masked regions and implement super-resolution reconstruction in videos of high quality. Furthermore, the above results show that our method has greater versatility both in video inpainting and super-resolution tasks compared to recent models. |
| first_indexed | 2024-03-10T23:05:38Z |
| format | Article |
| id | doaj.art-948da5024aa941258535d9cc860c760d |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T23:05:38Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-948da5024aa941258535d9cc860c760d2023-11-19T09:21:38ZengMDPI AGApplied Sciences2076-34172023-09-0113181000110.3390/app131810001Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution TasksYuanfeng Zheng0Yuchen Yan1Hao Jiang2School of Electronic Information, Wuhan University, Wuhan 430072, ChinaSchool of Electronic Information, Wuhan University, Wuhan 430072, ChinaSchool of Electronic Information, Wuhan University, Wuhan 430072, ChinaAlthough deep learning-based approaches for video processing have been extensively investigated, the lack of generality in network construction makes it challenging for practical applications, particularly in video restoration. As a result, this paper presents a universal video restoration model that can simultaneously tackle video inpainting and super-resolution tasks. The network, called Video-Restoration-Net (VRN), consists of four components: (1) an encoder to extract features from each frame, (2) a non-local network that recombines features from adjacent frames or different locations of a given frame, (3) a decoder to restore the coarse video from the output of a non-local block, and (4) a refinement network to refine the coarse video on the frame level. The framework is trained in a three-step pipeline to improve training stability for both tasks. Specifically, we first suggest an automated technique to generate full video datasets for super-resolution reconstruction and another complete-incomplete video dataset for inpainting, respectively. A VRN is then trained to inpaint the incomplete videos. Meanwhile, the full video datasets are adopted to train another VRN frame-wisely and validate it against authoritative datasets. We show quantitative comparisons with several baseline models, achieving 40.5042 dB/0.99473 on PSNR/SSIM in the inpainting task, while during the SR task we obtained 28.41 dB/0.7953 and 27.25/0.8152 on BSD100 and Urban100, respectively. The qualitative comparisons demonstrate that our proposed model is able to complete masked regions and implement super-resolution reconstruction in videos of high quality. Furthermore, the above results show that our method has greater versatility both in video inpainting and super-resolution tasks compared to recent models.https://www.mdpi.com/2076-3417/13/18/10001video restorationvideo inpaintingsuper-resolutionnon-local |
| spellingShingle | Yuanfeng Zheng Yuchen Yan Hao Jiang Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks Applied Sciences video restoration video inpainting super-resolution non-local |
| title | Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks |
| title_full | Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks |
| title_fullStr | Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks |
| title_full_unstemmed | Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks |
| title_short | Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks |
| title_sort | video restoration net deep generative model with non local network for inpainting and super resolution tasks |
| topic | video restoration video inpainting super-resolution non-local |
| url | https://www.mdpi.com/2076-3417/13/18/10001 |
| work_keys_str_mv | AT yuanfengzheng videorestorationnetdeepgenerativemodelwithnonlocalnetworkforinpaintingandsuperresolutiontasks AT yuchenyan videorestorationnetdeepgenerativemodelwithnonlocalnetworkforinpaintingandsuperresolutiontasks AT haojiang videorestorationnetdeepgenerativemodelwithnonlocalnetworkforinpaintingandsuperresolutiontasks |