AgileGAN: stylizing portraits by inversion-consistent transfer learning
Portraiture as an art form has evolved from realistic depiction into a plethora of creative styles. While substantial progress has been made in automated stylization, generating high quality stylistic portraits is still a challenge, and even the recent popular Toonify suffers from several artifacts...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/172645 |
| _version_ | 1811693296008822784 |
|---|---|
| author | Song, Guoxian Luo, Linjie Liu, Jing Ma, Wan-Chun Lai, Chunpong Zheng, Chuanxia Cham, Tat-Jen |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Song, Guoxian Luo, Linjie Liu, Jing Ma, Wan-Chun Lai, Chunpong Zheng, Chuanxia Cham, Tat-Jen |
| author_sort | Song, Guoxian |
| collection | NTU |
| description | Portraiture as an art form has evolved from realistic depiction into a plethora of creative styles. While substantial progress has been made in automated stylization, generating high quality stylistic portraits is still a challenge, and even the recent popular Toonify suffers from several artifacts when used on real input images. Such StyleGAN-based methods have focused on finding the best latent inversion mapping for reconstructing input images; however, our key insight is that this does not lead to good generalization to different portrait styles. Hence we propose AgileGAN, a framework that can generate high quality stylistic portraits via inversion-consistent transfer learning. We introduce a novel hierarchical variational autoencoder to ensure the inverse mapped distribution conforms to the original latent Gaussian distribution, while augmenting the original space to a multi-resolution latent space so as to better encode different levels of detail. To better capture attribute-dependent stylization of facial features, we also present an attribute-aware generator and adopt an early stopping strategy to avoid overfitting small training datasets. Our approach provides greater agility in creating high quality and high resolution (1024×1024) portrait stylization models, requiring only a limited number of style exemplars (∼100) and short training time (∼1 hour). We collected several style datasets for evaluation including 3D cartoons, comics, oil paintings and celebrities. We show that we can achieve superior portrait stylization quality to previous state-of-the-art methods, with comparisons done qualitatively, quantitatively and through a perceptual user study. We also demonstrate two applications of our method, image editing and motion retargeting. |
| first_indexed | 2024-10-01T06:49:25Z |
| format | Journal Article |
| id | ntu-10356/172645 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:49:25Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | ntu-10356/1726452023-12-19T01:32:44Z AgileGAN: stylizing portraits by inversion-consistent transfer learning Song, Guoxian Luo, Linjie Liu, Jing Ma, Wan-Chun Lai, Chunpong Zheng, Chuanxia Cham, Tat-Jen School of Computer Science and Engineering Engineering::Computer science and engineering::Computing methodologies::Computer graphics Portrait Generation Stylization Portraiture as an art form has evolved from realistic depiction into a plethora of creative styles. While substantial progress has been made in automated stylization, generating high quality stylistic portraits is still a challenge, and even the recent popular Toonify suffers from several artifacts when used on real input images. Such StyleGAN-based methods have focused on finding the best latent inversion mapping for reconstructing input images; however, our key insight is that this does not lead to good generalization to different portrait styles. Hence we propose AgileGAN, a framework that can generate high quality stylistic portraits via inversion-consistent transfer learning. We introduce a novel hierarchical variational autoencoder to ensure the inverse mapped distribution conforms to the original latent Gaussian distribution, while augmenting the original space to a multi-resolution latent space so as to better encode different levels of detail. To better capture attribute-dependent stylization of facial features, we also present an attribute-aware generator and adopt an early stopping strategy to avoid overfitting small training datasets. Our approach provides greater agility in creating high quality and high resolution (1024×1024) portrait stylization models, requiring only a limited number of style exemplars (∼100) and short training time (∼1 hour). We collected several style datasets for evaluation including 3D cartoons, comics, oil paintings and celebrities. We show that we can achieve superior portrait stylization quality to previous state-of-the-art methods, with comparisons done qualitatively, quantitatively and through a perceptual user study. We also demonstrate two applications of our method, image editing and motion retargeting. 2023-12-19T01:32:44Z 2023-12-19T01:32:44Z 2021 Journal Article Song, G., Luo, L., Liu, J., Ma, W., Lai, C., Zheng, C. & Cham, T. (2021). AgileGAN: stylizing portraits by inversion-consistent transfer learning. ACM Transactions On Graphics, 40(4), 117-. https://dx.doi.org/10.1145/3450626.3459771 0730-0301 https://hdl.handle.net/10356/172645 10.1145/3450626.3459771 2-s2.0-85111321823 4 40 117 en ACM Transactions on Graphics © 2021 Copyright held by the owner/author(s). All rights reserved. |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Computer graphics Portrait Generation Stylization Song, Guoxian Luo, Linjie Liu, Jing Ma, Wan-Chun Lai, Chunpong Zheng, Chuanxia Cham, Tat-Jen AgileGAN: stylizing portraits by inversion-consistent transfer learning |
| title | AgileGAN: stylizing portraits by inversion-consistent transfer learning |
| title_full | AgileGAN: stylizing portraits by inversion-consistent transfer learning |
| title_fullStr | AgileGAN: stylizing portraits by inversion-consistent transfer learning |
| title_full_unstemmed | AgileGAN: stylizing portraits by inversion-consistent transfer learning |
| title_short | AgileGAN: stylizing portraits by inversion-consistent transfer learning |
| title_sort | agilegan stylizing portraits by inversion consistent transfer learning |
| topic | Engineering::Computer science and engineering::Computing methodologies::Computer graphics Portrait Generation Stylization |
| url | https://hdl.handle.net/10356/172645 |
| work_keys_str_mv | AT songguoxian agileganstylizingportraitsbyinversionconsistenttransferlearning AT luolinjie agileganstylizingportraitsbyinversionconsistenttransferlearning AT liujing agileganstylizingportraitsbyinversionconsistenttransferlearning AT mawanchun agileganstylizingportraitsbyinversionconsistenttransferlearning AT laichunpong agileganstylizingportraitsbyinversionconsistenttransferlearning AT zhengchuanxia agileganstylizingportraitsbyinversionconsistenttransferlearning AT chamtatjen agileganstylizingportraitsbyinversionconsistenttransferlearning |