Efficient diffusion model for image restoration by residual shifting
While diffusion-based image restoration (IR) methods have achieved remarkable success, they are still limited by the low inference speed attributed to the necessity of executing hundreds or even thousands of sampling steps. Existing acceleration sampling techniques, though seeking to expedite the pr...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/181036 https://arxiv.org/abs/2403.07319 |
| _version_ | 1826110071659036672 |
|---|---|
| author | Yue, Zongsheng Wang, Jianyi Loy, Chen Change |
| author2 | College of Computing and Data Science |
| author_facet | College of Computing and Data Science Yue, Zongsheng Wang, Jianyi Loy, Chen Change |
| author_sort | Yue, Zongsheng |
| collection | NTU |
| description | While diffusion-based image restoration (IR) methods have achieved remarkable success, they are still limited by the low inference speed attributed to the necessity of executing hundreds or even thousands of sampling steps. Existing acceleration sampling techniques, though seeking to expedite the process, inevitably sacrifice performance to some extent, resulting in over-blurry restored outcomes. To address this issue, this study proposes a novel and efficient diffusion model for IR that significantly reduces the required number of diffusion steps. Our method avoids the need for post-acceleration during inference, thereby avoiding the associated performance deterioration. Specifically, our proposed method establishes a Markov chain that facilitates the transitions between the high-quality and low-quality images by shifting their residuals, substantially improving the transition efficiency. A carefully formulated noise schedule is devised to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experimental evaluations demonstrate that the proposed method achieves superior or comparable performance to current state-of-the-art methods on four classical IR tasks, namely image super-resolution, image inpainting, blind face restoration, and image deblurring, even only with four sampling steps. Our code and model are publicly available at https://github.com/zsyOAOA/ResShift. |
| first_indexed | 2025-03-09T09:57:54Z |
| format | Journal Article |
| id | ntu-10356/181036 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-03-09T09:57:54Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1810362025-02-11T03:27:24Z Efficient diffusion model for image restoration by residual shifting Yue, Zongsheng Wang, Jianyi Loy, Chen Change College of Computing and Data Science S-Lab Computer and Information Science Markov chain Noise schedule While diffusion-based image restoration (IR) methods have achieved remarkable success, they are still limited by the low inference speed attributed to the necessity of executing hundreds or even thousands of sampling steps. Existing acceleration sampling techniques, though seeking to expedite the process, inevitably sacrifice performance to some extent, resulting in over-blurry restored outcomes. To address this issue, this study proposes a novel and efficient diffusion model for IR that significantly reduces the required number of diffusion steps. Our method avoids the need for post-acceleration during inference, thereby avoiding the associated performance deterioration. Specifically, our proposed method establishes a Markov chain that facilitates the transitions between the high-quality and low-quality images by shifting their residuals, substantially improving the transition efficiency. A carefully formulated noise schedule is devised to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experimental evaluations demonstrate that the proposed method achieves superior or comparable performance to current state-of-the-art methods on four classical IR tasks, namely image super-resolution, image inpainting, blind face restoration, and image deblurring, even only with four sampling steps. Our code and model are publicly available at https://github.com/zsyOAOA/ResShift. Submitted/Accepted version 2024-11-12T01:40:52Z 2024-11-12T01:40:52Z 2024 Journal Article Yue, Z., Wang, J. & Loy, C. C. (2024). Efficient diffusion model for image restoration by residual shifting. IEEE Transactions On Pattern Analysis and Machine Intelligence, 3461721-. https://dx.doi.org/10.1109/TPAMI.2024.3461721 0162-8828 https://hdl.handle.net/10356/181036 10.1109/TPAMI.2024.3461721 https://arxiv.org/abs/2403.07319 2-s2.0-85204472118 3461721 en IEEE Transactions on Pattern Analysis and Machine Intelligence 10.21979/N9/VYPJ0O © 2024 IEEE. All rights reserved. application/pdf |
| spellingShingle | Computer and Information Science Markov chain Noise schedule Yue, Zongsheng Wang, Jianyi Loy, Chen Change Efficient diffusion model for image restoration by residual shifting |
| title | Efficient diffusion model for image restoration by residual shifting |
| title_full | Efficient diffusion model for image restoration by residual shifting |
| title_fullStr | Efficient diffusion model for image restoration by residual shifting |
| title_full_unstemmed | Efficient diffusion model for image restoration by residual shifting |
| title_short | Efficient diffusion model for image restoration by residual shifting |
| title_sort | efficient diffusion model for image restoration by residual shifting |
| topic | Computer and Information Science Markov chain Noise schedule |
| url | https://hdl.handle.net/10356/181036 https://arxiv.org/abs/2403.07319 |
| work_keys_str_mv | AT yuezongsheng efficientdiffusionmodelforimagerestorationbyresidualshifting AT wangjianyi efficientdiffusionmodelforimagerestorationbyresidualshifting AT loychenchange efficientdiffusionmodelforimagerestorationbyresidualshifting |