ManiCLIP: multi-attribute face manipulation from text
In this paper we present a novel multi-attribute face manipulation method based on textual descriptions. Previous text-based image editing methods either require test-time optimization for each individual image or are restricted to single attribute editing. Extending these methods to multi-attribute...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/179547 |
| _version_ | 1826114551334043648 |
|---|---|
| author | Wang, Hao Lin, Guosheng del Molino, Ana García Wang, Anran Feng, Jiashi Shen, Zhiqi |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Wang, Hao Lin, Guosheng del Molino, Ana García Wang, Anran Feng, Jiashi Shen, Zhiqi |
| author_sort | Wang, Hao |
| collection | NTU |
| description | In this paper we present a novel multi-attribute face manipulation method based on textual descriptions. Previous text-based image editing methods either require test-time optimization for each individual image or are restricted to single attribute editing. Extending these methods to multi-attribute face image editing scenarios will introduce undesired excessive attribute change, e.g., text-relevant attributes are overly manipulated and text-irrelevant attributes are also changed. In order to address these challenges and achieve natural editing over multiple face attributes, we propose a new decoupling training scheme where we use group sampling to get text segments from same attribute categories, instead of whole complex sentences. Further, to preserve other existing face attributes, we encourage the model to edit the latent code of each attribute separately via an entropy constraint. During the inference phase, our model is able to edit new face images without any test-time optimization, even from complex textual prompts. We show extensive experiments and analysis to demonstrate the efficacy of our method, which generates natural manipulated faces with minimal text-irrelevant attribute editing. Code and pre-trained model are available at https://github.com/hwang1996/ManiCLIP. |
| first_indexed | 2024-10-01T03:40:58Z |
| format | Journal Article |
| id | ntu-10356/179547 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T03:40:58Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1795472024-08-07T06:49:08Z ManiCLIP: multi-attribute face manipulation from text Wang, Hao Lin, Guosheng del Molino, Ana García Wang, Anran Feng, Jiashi Shen, Zhiqi School of Computer Science and Engineering Computer and Information Science Image generation Image editing In this paper we present a novel multi-attribute face manipulation method based on textual descriptions. Previous text-based image editing methods either require test-time optimization for each individual image or are restricted to single attribute editing. Extending these methods to multi-attribute face image editing scenarios will introduce undesired excessive attribute change, e.g., text-relevant attributes are overly manipulated and text-irrelevant attributes are also changed. In order to address these challenges and achieve natural editing over multiple face attributes, we propose a new decoupling training scheme where we use group sampling to get text segments from same attribute categories, instead of whole complex sentences. Further, to preserve other existing face attributes, we encourage the model to edit the latent code of each attribute separately via an entropy constraint. During the inference phase, our model is able to edit new face images without any test-time optimization, even from complex textual prompts. We show extensive experiments and analysis to demonstrate the efficacy of our method, which generates natural manipulated faces with minimal text-irrelevant attribute editing. Code and pre-trained model are available at https://github.com/hwang1996/ManiCLIP. Nanyang Technological University National Research Foundation (NRF) This research is supported, in part, by the Education Bureau of Guangzhou Municipality and the Guangzhou-HKUST (GZ) Joint Funding Program (Grant No. 2023A03J0008). This research is supported, in part, by the Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), Nanyang Technological University, Singapore; and National Research Foundation, Prime Minister’s Office, Singapore under its NRF Investigatorship Programme (NRFI Award No. NRF-NRFI05-2019-0002). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. 2024-08-07T06:48:04Z 2024-08-07T06:48:04Z 2024 Journal Article Wang, H., Lin, G., del Molino, A. G., Wang, A., Feng, J. & Shen, Z. (2024). ManiCLIP: multi-attribute face manipulation from text. International Journal of Computer Vision. https://dx.doi.org/10.1007/s11263-024-02088-6 0920-5691 https://hdl.handle.net/10356/179547 10.1007/s11263-024-02088-6 2-s2.0-85193738087 en NRF-NRFI05-2019-0002 International Journal of Computer Vision © 2024 The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature. All rights reserved. |
| spellingShingle | Computer and Information Science Image generation Image editing Wang, Hao Lin, Guosheng del Molino, Ana García Wang, Anran Feng, Jiashi Shen, Zhiqi ManiCLIP: multi-attribute face manipulation from text |
| title | ManiCLIP: multi-attribute face manipulation from text |
| title_full | ManiCLIP: multi-attribute face manipulation from text |
| title_fullStr | ManiCLIP: multi-attribute face manipulation from text |
| title_full_unstemmed | ManiCLIP: multi-attribute face manipulation from text |
| title_short | ManiCLIP: multi-attribute face manipulation from text |
| title_sort | maniclip multi attribute face manipulation from text |
| topic | Computer and Information Science Image generation Image editing |
| url | https://hdl.handle.net/10356/179547 |
| work_keys_str_mv | AT wanghao maniclipmultiattributefacemanipulationfromtext AT linguosheng maniclipmultiattributefacemanipulationfromtext AT delmolinoanagarcia maniclipmultiattributefacemanipulationfromtext AT wanganran maniclipmultiattributefacemanipulationfromtext AT fengjiashi maniclipmultiattributefacemanipulationfromtext AT shenzhiqi maniclipmultiattributefacemanipulationfromtext |