Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference
Video super-resolution aims to generate high-resolution frames from low-resolution counterparts. It can be regarded as a specialized application of image super-resolution, serving various purposes, such as video display and surveillance. This paper proposes a novel method for real-time video super-r...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/18/7880 |
| _version_ | 1797576989320675328 |
|---|---|
| author | Wenhao Wang Zhenbing Liu Haoxiang Lu Rushi Lan Zhaoyuan Zhang |
| author_facet | Wenhao Wang Zhenbing Liu Haoxiang Lu Rushi Lan Zhaoyuan Zhang |
| author_sort | Wenhao Wang |
| collection | DOAJ |
| description | Video super-resolution aims to generate high-resolution frames from low-resolution counterparts. It can be regarded as a specialized application of image super-resolution, serving various purposes, such as video display and surveillance. This paper proposes a novel method for real-time video super-resolution. It effectively exploits spatial information by utilizing the capabilities of an image super-resolution model and leverages the temporal information inherent in videos. Specifically, the method incorporates a pre-trained image super-resolution network as its foundational framework, allowing it to leverage existing expertise for super-resolution. A fast temporal information aggregation module is presented to further aggregate temporal cues across frames. By using deformable convolution to align features of neighboring frames, this module takes advantage of inter-frame dependency. In addition, it employs a hierarchical fast spatial offset feature extraction and a channel attention-based temporal fusion. A redundancy-aware inference algorithm is developed to reduce computational redundancy by reusing intermediate features, achieving real-time inferring speed. Extensive experiments on several benchmarks demonstrate that the proposed method can reconstruct satisfactory results with strong quantitative performance and visual qualities. The real-time inferring ability makes it suitable for real-world deployment. |
| first_indexed | 2024-03-10T22:02:31Z |
| format | Article |
| id | doaj.art-987ece9b78ef43009f0aaef5e8e9ec63 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T22:02:31Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-987ece9b78ef43009f0aaef5e8e9ec632023-11-19T12:55:30ZengMDPI AGSensors1424-82202023-09-012318788010.3390/s23187880Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware InferenceWenhao Wang0Zhenbing Liu1Haoxiang Lu2Rushi Lan3Zhaoyuan Zhang4School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, ChinaSchool of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, ChinaSchool of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, ChinaSchool of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, ChinaSchool of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, ChinaVideo super-resolution aims to generate high-resolution frames from low-resolution counterparts. It can be regarded as a specialized application of image super-resolution, serving various purposes, such as video display and surveillance. This paper proposes a novel method for real-time video super-resolution. It effectively exploits spatial information by utilizing the capabilities of an image super-resolution model and leverages the temporal information inherent in videos. Specifically, the method incorporates a pre-trained image super-resolution network as its foundational framework, allowing it to leverage existing expertise for super-resolution. A fast temporal information aggregation module is presented to further aggregate temporal cues across frames. By using deformable convolution to align features of neighboring frames, this module takes advantage of inter-frame dependency. In addition, it employs a hierarchical fast spatial offset feature extraction and a channel attention-based temporal fusion. A redundancy-aware inference algorithm is developed to reduce computational redundancy by reusing intermediate features, achieving real-time inferring speed. Extensive experiments on several benchmarks demonstrate that the proposed method can reconstruct satisfactory results with strong quantitative performance and visual qualities. The real-time inferring ability makes it suitable for real-world deployment.https://www.mdpi.com/1424-8220/23/18/7880video super-resolutiontemporal aggregationdeformable convolutionredundancy-aware inferencedeep learning |
| spellingShingle | Wenhao Wang Zhenbing Liu Haoxiang Lu Rushi Lan Zhaoyuan Zhang Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference Sensors video super-resolution temporal aggregation deformable convolution redundancy-aware inference deep learning |
| title | Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference |
| title_full | Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference |
| title_fullStr | Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference |
| title_full_unstemmed | Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference |
| title_short | Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference |
| title_sort | real time video super resolution with spatio temporal modeling and redundancy aware inference |
| topic | video super-resolution temporal aggregation deformable convolution redundancy-aware inference deep learning |
| url | https://www.mdpi.com/1424-8220/23/18/7880 |
| work_keys_str_mv | AT wenhaowang realtimevideosuperresolutionwithspatiotemporalmodelingandredundancyawareinference AT zhenbingliu realtimevideosuperresolutionwithspatiotemporalmodelingandredundancyawareinference AT haoxianglu realtimevideosuperresolutionwithspatiotemporalmodelingandredundancyawareinference AT rushilan realtimevideosuperresolutionwithspatiotemporalmodelingandredundancyawareinference AT zhaoyuanzhang realtimevideosuperresolutionwithspatiotemporalmodelingandredundancyawareinference |