Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network
Recently, deep neural networks have shown surprising results in solving most of the traditional image processing problems. However, the video frame interpolation field does not show relatively good performance because the receptive field requires a vast spatio-temporal range. To reduce the computati...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-08-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/11/16/2553 |
| _version_ | 1797445876312965120 |
|---|---|
| author | Jeonghwan Heo Jechang Jeong |
| author_facet | Jeonghwan Heo Jechang Jeong |
| author_sort | Jeonghwan Heo |
| collection | DOAJ |
| description | Recently, deep neural networks have shown surprising results in solving most of the traditional image processing problems. However, the video frame interpolation field does not show relatively good performance because the receptive field requires a vast spatio-temporal range. To reduce the computational complexity, in most frame interpolation studies, motion is first calculated with the optical flow, then interpolated frames are generated through backward warping. However, while the backward warping process is simple to implement, the interpolated image contains mixed motion and ghosting defects. Therefore, we propose a new network that does not use the backward warping method through the proposed max-min warping. Since max-min warping generates a clear warping image in advance according to the size of the motion and the network is configured to select the warping result according to the warped layer, using the proposed method, it is possible to optimize the computational complexity while selecting a contextually appropriate image. The video interpolation method using the proposed method showed 34.847 PSNR in the Vimeo90k dataset and 0.13 PSNR improvement compared to the Quadratic Video Interpolation method, showing that it is an efficient frame interpolation self-supervised learning. |
| first_indexed | 2024-03-09T13:32:06Z |
| format | Article |
| id | doaj.art-59f800032e1e427293f5df157c598782 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-09T13:32:06Z |
| publishDate | 2022-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-59f800032e1e427293f5df157c5987822023-11-30T21:16:40ZengMDPI AGElectronics2079-92922022-08-011116255310.3390/electronics11162553Forward Warping-Based Video Frame Interpolation Using a Motion Selective NetworkJeonghwan Heo0Jechang Jeong1Department of Electronic Engineering, Hanyang University, Seoul 04763, KoreaDepartment of Electronic Engineering, Hanyang University, Seoul 04763, KoreaRecently, deep neural networks have shown surprising results in solving most of the traditional image processing problems. However, the video frame interpolation field does not show relatively good performance because the receptive field requires a vast spatio-temporal range. To reduce the computational complexity, in most frame interpolation studies, motion is first calculated with the optical flow, then interpolated frames are generated through backward warping. However, while the backward warping process is simple to implement, the interpolated image contains mixed motion and ghosting defects. Therefore, we propose a new network that does not use the backward warping method through the proposed max-min warping. Since max-min warping generates a clear warping image in advance according to the size of the motion and the network is configured to select the warping result according to the warped layer, using the proposed method, it is possible to optimize the computational complexity while selecting a contextually appropriate image. The video interpolation method using the proposed method showed 34.847 PSNR in the Vimeo90k dataset and 0.13 PSNR improvement compared to the Quadratic Video Interpolation method, showing that it is an efficient frame interpolation self-supervised learning.https://www.mdpi.com/2079-9292/11/16/2553frame rate up-conversionoptical flowflow warpingdeep learning |
| spellingShingle | Jeonghwan Heo Jechang Jeong Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network Electronics frame rate up-conversion optical flow flow warping deep learning |
| title | Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network |
| title_full | Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network |
| title_fullStr | Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network |
| title_full_unstemmed | Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network |
| title_short | Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network |
| title_sort | forward warping based video frame interpolation using a motion selective network |
| topic | frame rate up-conversion optical flow flow warping deep learning |
| url | https://www.mdpi.com/2079-9292/11/16/2553 |
| work_keys_str_mv | AT jeonghwanheo forwardwarpingbasedvideoframeinterpolationusingamotionselectivenetwork AT jechangjeong forwardwarpingbasedvideoframeinterpolationusingamotionselectivenetwork |