Frequency-Based Enhancement Network for Efficient Super-Resolution
Recently, deep convolutional neural networks (CNNs) have provided outstanding performance in single image super-resolution (SISR). Despite their remarkable performance, the lack of high-frequency information in the recovered images remains a core problem. Moreover, as the networks increase in depth...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9778017/ |
| _version_ | 1817975872556105728 |
|---|---|
| author | Parichehr Behjati Pau Rodriguez Carles Fernandez Tena Armin Mehri F. Xavier Roca Seiichi Ozawa Jordi Gonzalez |
| author_facet | Parichehr Behjati Pau Rodriguez Carles Fernandez Tena Armin Mehri F. Xavier Roca Seiichi Ozawa Jordi Gonzalez |
| author_sort | Parichehr Behjati |
| collection | DOAJ |
| description | Recently, deep convolutional neural networks (CNNs) have provided outstanding performance in single image super-resolution (SISR). Despite their remarkable performance, the lack of high-frequency information in the recovered images remains a core problem. Moreover, as the networks increase in depth and width, deep CNN-based SR methods are faced with the challenge of computational complexity in practice. A promising and under-explored solution is to adapt the amount of compute based on the different frequency bands of the input. To this end, we present a novel Frequency-based Enhancement Block (FEB) which explicitly enhances the information of high frequencies while forwarding low-frequencies to the output. In particular, this block efficiently decomposes features into low- and high-frequency and assigns more computation to high-frequency ones. Thus, it can help the network generate more discriminative representations by explicitly recovering finer details. Our FEB design is simple and generic and can be used as a direct replacement of commonly used SR blocks with no need to change network architectures. We experimentally show that when replacing SR blocks with FEB we consistently improve the reconstruction error, while reducing the number of parameters in the model. Moreover, we propose a lightweight SR model — Frequency-based Enhancement Network (FENet) — based on FEB that matches the performance of larger models. Extensive experiments demonstrate that our proposal performs favorably against the state-of-the-art SR algorithms in terms of visual quality, memory footprint, and inference time. The code is available at <uri>https://github.com/pbehjatii/FENet</uri> |
| first_indexed | 2024-04-13T21:56:01Z |
| format | Article |
| id | doaj.art-eee36a27d907449f80b965886182366b |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-13T21:56:01Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-eee36a27d907449f80b965886182366b2022-12-22T02:28:16ZengIEEEIEEE Access2169-35362022-01-0110573835739710.1109/ACCESS.2022.31764419778017Frequency-Based Enhancement Network for Efficient Super-ResolutionParichehr Behjati0https://orcid.org/0000-0003-4266-545XPau Rodriguez1https://orcid.org/0000-0002-1689-8084Carles Fernandez Tena2https://orcid.org/0000-0001-6185-3427Armin Mehri3https://orcid.org/0000-0003-3472-2530F. Xavier Roca4https://orcid.org/0000-0002-7043-7334Seiichi Ozawa5https://orcid.org/0000-0002-0965-0064Jordi Gonzalez6https://orcid.org/0000-0001-8033-0306Computer Vision Center, Universitat Autònoma de Barcelona, Bellaterra, SpainServiceNow Research, Montreal, QC, CanadaOxolo GmbH, Hamburg, GermanyComputer Vision Center, Universitat Autònoma de Barcelona, Bellaterra, SpainComputer Vision Center, Universitat Autònoma de Barcelona, Bellaterra, SpainCenter for Mathematical and Data Sciences, Kobe University, Kobe, JapanComputer Vision Center, Universitat Autònoma de Barcelona, Bellaterra, SpainRecently, deep convolutional neural networks (CNNs) have provided outstanding performance in single image super-resolution (SISR). Despite their remarkable performance, the lack of high-frequency information in the recovered images remains a core problem. Moreover, as the networks increase in depth and width, deep CNN-based SR methods are faced with the challenge of computational complexity in practice. A promising and under-explored solution is to adapt the amount of compute based on the different frequency bands of the input. To this end, we present a novel Frequency-based Enhancement Block (FEB) which explicitly enhances the information of high frequencies while forwarding low-frequencies to the output. In particular, this block efficiently decomposes features into low- and high-frequency and assigns more computation to high-frequency ones. Thus, it can help the network generate more discriminative representations by explicitly recovering finer details. Our FEB design is simple and generic and can be used as a direct replacement of commonly used SR blocks with no need to change network architectures. We experimentally show that when replacing SR blocks with FEB we consistently improve the reconstruction error, while reducing the number of parameters in the model. Moreover, we propose a lightweight SR model — Frequency-based Enhancement Network (FENet) — based on FEB that matches the performance of larger models. Extensive experiments demonstrate that our proposal performs favorably against the state-of-the-art SR algorithms in terms of visual quality, memory footprint, and inference time. The code is available at <uri>https://github.com/pbehjatii/FENet</uri>https://ieeexplore.ieee.org/document/9778017/Deep learningfrequency-based methodslightweight architecturessingle image super-resolution |
| spellingShingle | Parichehr Behjati Pau Rodriguez Carles Fernandez Tena Armin Mehri F. Xavier Roca Seiichi Ozawa Jordi Gonzalez Frequency-Based Enhancement Network for Efficient Super-Resolution IEEE Access Deep learning frequency-based methods lightweight architectures single image super-resolution |
| title | Frequency-Based Enhancement Network for Efficient Super-Resolution |
| title_full | Frequency-Based Enhancement Network for Efficient Super-Resolution |
| title_fullStr | Frequency-Based Enhancement Network for Efficient Super-Resolution |
| title_full_unstemmed | Frequency-Based Enhancement Network for Efficient Super-Resolution |
| title_short | Frequency-Based Enhancement Network for Efficient Super-Resolution |
| title_sort | frequency based enhancement network for efficient super resolution |
| topic | Deep learning frequency-based methods lightweight architectures single image super-resolution |
| url | https://ieeexplore.ieee.org/document/9778017/ |
| work_keys_str_mv | AT parichehrbehjati frequencybasedenhancementnetworkforefficientsuperresolution AT paurodriguez frequencybasedenhancementnetworkforefficientsuperresolution AT carlesfernandeztena frequencybasedenhancementnetworkforefficientsuperresolution AT arminmehri frequencybasedenhancementnetworkforefficientsuperresolution AT fxavierroca frequencybasedenhancementnetworkforefficientsuperresolution AT seiichiozawa frequencybasedenhancementnetworkforefficientsuperresolution AT jordigonzalez frequencybasedenhancementnetworkforefficientsuperresolution |