Pluralistic free-form image completion
Image completion involves filling plausible contents to missing regions in images. Current image completion methods produce only one result for a given masked image, although there may be many reasonable possibilities. In this paper, we present an approach for pluralistic image completion—the task o...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/172648 |
| _version_ | 1826125240963432448 |
|---|---|
| author | Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei |
| author_sort | Zheng, Chuanxia |
| collection | NTU |
| description | Image completion involves filling plausible contents to missing regions in images. Current image completion methods produce only one result for a given masked image, although there may be many reasonable possibilities. In this paper, we present an approach for pluralistic image completion—the task of generating multiple and diverse plausible solutions for free-form image completion. A major challenge faced by learning-based approaches is that usually only one ground truth training instance per label for this multi-output problem. To overcome this, we propose a novel and probabilistically principled framework with two parallel paths. One is a reconstructive path that utilizes the only one ground truth to get prior distribution of missing patches and rebuild the original image from this distribution. The other is a generative path for which the conditional prior is coupled to the distribution obtained in the reconstructive path. Both are supported by adversarial learning. We then introduce a new short+long term patch attention layer that exploits distant relations among decoder and encoder features, to improve appearance consistency between the original visible and the generated new regions. Experiments show that our method not only yields better results in various datasets than existing state-of-the-art methods, but also provides multiple and diverse outputs. |
| first_indexed | 2024-10-01T06:33:10Z |
| format | Journal Article |
| id | ntu-10356/172648 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:33:10Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | ntu-10356/1726482023-12-19T01:56:28Z Pluralistic free-form image completion Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei School of Computer Science and Engineering Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Image Completion Image Generation Image completion involves filling plausible contents to missing regions in images. Current image completion methods produce only one result for a given masked image, although there may be many reasonable possibilities. In this paper, we present an approach for pluralistic image completion—the task of generating multiple and diverse plausible solutions for free-form image completion. A major challenge faced by learning-based approaches is that usually only one ground truth training instance per label for this multi-output problem. To overcome this, we propose a novel and probabilistically principled framework with two parallel paths. One is a reconstructive path that utilizes the only one ground truth to get prior distribution of missing patches and rebuild the original image from this distribution. The other is a generative path for which the conditional prior is coupled to the distribution obtained in the reconstructive path. Both are supported by adversarial learning. We then introduce a new short+long term patch attention layer that exploits distant relations among decoder and encoder features, to improve appearance consistency between the original visible and the generated new regions. Experiments show that our method not only yields better results in various datasets than existing state-of-the-art methods, but also provides multiple and diverse outputs. National Research Foundation (NRF) This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAFICP) Funding Initiative, as well as cash and in-kind contribution from Singapore Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU). This research is also supported by the Monash FIT Start-up Grant. 2023-12-19T01:56:27Z 2023-12-19T01:56:27Z 2021 Journal Article Zheng, C., Cham, T. & Cai, J. (2021). Pluralistic free-form image completion. International Journal of Computer Vision, 129(10), 2786-2805. https://dx.doi.org/10.1007/s11263-021-01502-7 0920-5691 https://hdl.handle.net/10356/172648 10.1007/s11263-021-01502-7 2-s2.0-85111509265 10 129 2786 2805 en IAFICP International Journal of Computer Vision © 2021 The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature. All rights reserved. |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Image Completion Image Generation Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei Pluralistic free-form image completion |
| title | Pluralistic free-form image completion |
| title_full | Pluralistic free-form image completion |
| title_fullStr | Pluralistic free-form image completion |
| title_full_unstemmed | Pluralistic free-form image completion |
| title_short | Pluralistic free-form image completion |
| title_sort | pluralistic free form image completion |
| topic | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Image Completion Image Generation |
| url | https://hdl.handle.net/10356/172648 |
| work_keys_str_mv | AT zhengchuanxia pluralisticfreeformimagecompletion AT chamtatjen pluralisticfreeformimagecompletion AT caijianfei pluralisticfreeformimagecompletion |