SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting
Recently, deep learning has enabled a huge leap forward in image inpainting. However, due to the memory and computational limitation, most existing methods are able to handle only low-resolution inputs, typically less than 1 K. With the improvement of Internet transmission capacity and mobile device...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-08-01
|
| Series: | Algorithms |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-4893/14/8/236 |
| _version_ | 1827686235604254720 |
|---|---|
| author | Haoran Xu Xinya Li Kaiyi Zhang Yanbai He Haoran Fan Sijiang Liu Chuanyan Hao Bo Jiang |
| author_facet | Haoran Xu Xinya Li Kaiyi Zhang Yanbai He Haoran Fan Sijiang Liu Chuanyan Hao Bo Jiang |
| author_sort | Haoran Xu |
| collection | DOAJ |
| description | Recently, deep learning has enabled a huge leap forward in image inpainting. However, due to the memory and computational limitation, most existing methods are able to handle only low-resolution inputs, typically less than 1 K. With the improvement of Internet transmission capacity and mobile device cameras, the resolution of image and video sources available to users via the cloud or locally is increasing. For high-resolution images, the common inpainting methods simply upsample the inpainted result of the shrinked image to yield a blurry result. In recent years, there is an urgent need to reconstruct the missing high-frequency information in high-resolution images and generate sharp texture details. Hence, we propose a general deep learning framework for high-resolution image inpainting, which first hallucinates a semantically continuous blurred result using low-resolution inpainting and suppresses computational overhead. Then the sharp high-frequency details with original resolution are reconstructed using super-resolution refinement. Experimentally, our method achieves inspiring inpainting quality on 2K and 4K resolution images, ahead of the state-of-the-art high-resolution inpainting technique. This framework is expected to be popularized for high-resolution image editing tasks on personal computers and mobile devices in the future. |
| first_indexed | 2024-03-10T09:05:29Z |
| format | Article |
| id | doaj.art-9c4d3458fae045348e86564b277e0369 |
| institution | Directory Open Access Journal |
| issn | 1999-4893 |
| language | English |
| last_indexed | 2024-03-10T09:05:29Z |
| publishDate | 2021-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Algorithms |
| spelling | doaj.art-9c4d3458fae045348e86564b277e03692023-11-22T06:27:43ZengMDPI AGAlgorithms1999-48932021-08-0114823610.3390/a14080236SR-Inpaint: A General Deep Learning Framework for High Resolution Image InpaintingHaoran Xu0Xinya Li1Kaiyi Zhang2Yanbai He3Haoran Fan4Sijiang Liu5Chuanyan Hao6Bo Jiang7School of Electronic and Optical Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210049, ChinaSchool of Educational Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing 210049, ChinaSchool of Overseas Education, Nanjing University of Posts and Telecommunications, Nanjing 210049, ChinaState Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing 100875, ChinaSchool of Electronic and Optical Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210049, ChinaSchool of Educational Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing 210049, ChinaSchool of Educational Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing 210049, ChinaSchool of Educational Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing 210049, ChinaRecently, deep learning has enabled a huge leap forward in image inpainting. However, due to the memory and computational limitation, most existing methods are able to handle only low-resolution inputs, typically less than 1 K. With the improvement of Internet transmission capacity and mobile device cameras, the resolution of image and video sources available to users via the cloud or locally is increasing. For high-resolution images, the common inpainting methods simply upsample the inpainted result of the shrinked image to yield a blurry result. In recent years, there is an urgent need to reconstruct the missing high-frequency information in high-resolution images and generate sharp texture details. Hence, we propose a general deep learning framework for high-resolution image inpainting, which first hallucinates a semantically continuous blurred result using low-resolution inpainting and suppresses computational overhead. Then the sharp high-frequency details with original resolution are reconstructed using super-resolution refinement. Experimentally, our method achieves inspiring inpainting quality on 2K and 4K resolution images, ahead of the state-of-the-art high-resolution inpainting technique. This framework is expected to be popularized for high-resolution image editing tasks on personal computers and mobile devices in the future.https://www.mdpi.com/1999-4893/14/8/236deep learningimage inpaintingsuper-resolutionhigh-resolutionhigh-frequency information reconstruction |
| spellingShingle | Haoran Xu Xinya Li Kaiyi Zhang Yanbai He Haoran Fan Sijiang Liu Chuanyan Hao Bo Jiang SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting Algorithms deep learning image inpainting super-resolution high-resolution high-frequency information reconstruction |
| title | SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting |
| title_full | SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting |
| title_fullStr | SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting |
| title_full_unstemmed | SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting |
| title_short | SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting |
| title_sort | sr inpaint a general deep learning framework for high resolution image inpainting |
| topic | deep learning image inpainting super-resolution high-resolution high-frequency information reconstruction |
| url | https://www.mdpi.com/1999-4893/14/8/236 |
| work_keys_str_mv | AT haoranxu srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting AT xinyali srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting AT kaiyizhang srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting AT yanbaihe srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting AT haoranfan srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting AT sijiangliu srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting AT chuanyanhao srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting AT bojiang srinpaintageneraldeeplearningframeworkforhighresolutionimageinpainting |