An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution
Convolutional neural networks and the per-pixel loss function have shown their potential to be the best combination for super-resolving severely degraded images. However, there are still challenges, such as the massive number of parameters requiring prohibitive memory and vast computing and storage...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-10-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/10/19/2434 |
| _version_ | 1797516660916092928 |
|---|---|
| author | Jucheng Yang Feng Wei Yaxin Bai Meiran Zuo Xiao Sun Yarui Chen |
| author_facet | Jucheng Yang Feng Wei Yaxin Bai Meiran Zuo Xiao Sun Yarui Chen |
| author_sort | Jucheng Yang |
| collection | DOAJ |
| description | Convolutional neural networks and the per-pixel loss function have shown their potential to be the best combination for super-resolving severely degraded images. However, there are still challenges, such as the massive number of parameters requiring prohibitive memory and vast computing and storage resources as well as time-consuming training and testing. What is more, the per-pixel loss measured by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mn>2</mn></msub></semantics></math></inline-formula> and the Peak Signal-to-Noise Ratio do not correlate well with human perception of image quality, since <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mn>2</mn></msub></semantics></math></inline-formula> simply does not capture the intricate characteristics of human visual systems. To address these issues, we propose an effective two-stage hourglass network with multi-task co-optimization, which enables the entire network to focus on training and testing time and inherent image patterns such as local luminance, contrast, structure and data distribution. Moreover, to avoid overwhelming memory overheads, our model is capable of performing real-time single image multi-scale super-resolution, so it is memory-friendly, meaning that memory space is utilized efficiently. In addition, in order to best use the underlying structure and perception of image quality and the intermediate estimates during the inference process, we introduce a cross-scale training strategy with 2×, 3× and 4× image super-resolution. This effective multi-task two-stage network with the cross-scale strategy for multi-scale image super-resolution is named EMTCM. Quantitative and qualitative experiment results show that the proposed EMTCM network outperforms state-of-the-art methods in recovering high-quality images. |
| first_indexed | 2024-03-10T07:03:59Z |
| format | Article |
| id | doaj.art-975e2050448f4edfa4fa78afd6e8fa1e |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T07:03:59Z |
| publishDate | 2021-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-975e2050448f4edfa4fa78afd6e8fa1e2023-11-22T15:57:40ZengMDPI AGElectronics2079-92922021-10-011019243410.3390/electronics10192434An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super ResolutionJucheng Yang0Feng Wei1Yaxin Bai2Meiran Zuo3Xiao Sun4Yarui Chen5College of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin 300457, ChinaCollege of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin 300457, ChinaCollege of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin 300457, ChinaCollege of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin 300457, ChinaCollege of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin 300457, ChinaCollege of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin 300457, ChinaConvolutional neural networks and the per-pixel loss function have shown their potential to be the best combination for super-resolving severely degraded images. However, there are still challenges, such as the massive number of parameters requiring prohibitive memory and vast computing and storage resources as well as time-consuming training and testing. What is more, the per-pixel loss measured by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mn>2</mn></msub></semantics></math></inline-formula> and the Peak Signal-to-Noise Ratio do not correlate well with human perception of image quality, since <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mn>2</mn></msub></semantics></math></inline-formula> simply does not capture the intricate characteristics of human visual systems. To address these issues, we propose an effective two-stage hourglass network with multi-task co-optimization, which enables the entire network to focus on training and testing time and inherent image patterns such as local luminance, contrast, structure and data distribution. Moreover, to avoid overwhelming memory overheads, our model is capable of performing real-time single image multi-scale super-resolution, so it is memory-friendly, meaning that memory space is utilized efficiently. In addition, in order to best use the underlying structure and perception of image quality and the intermediate estimates during the inference process, we introduce a cross-scale training strategy with 2×, 3× and 4× image super-resolution. This effective multi-task two-stage network with the cross-scale strategy for multi-scale image super-resolution is named EMTCM. Quantitative and qualitative experiment results show that the proposed EMTCM network outperforms state-of-the-art methods in recovering high-quality images.https://www.mdpi.com/2079-9292/10/19/2434CNNper-pixel lossHVSEMTCMmulti-task co-optimizationcross-scale training |
| spellingShingle | Jucheng Yang Feng Wei Yaxin Bai Meiran Zuo Xiao Sun Yarui Chen An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution Electronics CNN per-pixel loss HVS EMTCM multi-task co-optimization cross-scale training |
| title | An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution |
| title_full | An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution |
| title_fullStr | An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution |
| title_full_unstemmed | An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution |
| title_short | An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution |
| title_sort | effective multi task two stage network with the cross scale training strategy for multi scale image super resolution |
| topic | CNN per-pixel loss HVS EMTCM multi-task co-optimization cross-scale training |
| url | https://www.mdpi.com/2079-9292/10/19/2434 |
| work_keys_str_mv | AT juchengyang aneffectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT fengwei aneffectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT yaxinbai aneffectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT meiranzuo aneffectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT xiaosun aneffectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT yaruichen aneffectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT juchengyang effectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT fengwei effectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT yaxinbai effectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT meiranzuo effectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT xiaosun effectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution AT yaruichen effectivemultitasktwostagenetworkwiththecrossscaletrainingstrategyformultiscaleimagesuperresolution |