MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image Synthesis
The Generative Adversarial Network (GAN) has recently experienced great progress in compositional image synthesis. Unfortunately, the models proposed in the literature usually require a set of pre-defined local generators and use a separate generator to model each part object. This makes the model i...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/13/2927 |
| _version_ | 1797591826254790656 |
|---|---|
| author | Zongtao Wang Jiajie Peng Zhiming Liu |
| author_facet | Zongtao Wang Jiajie Peng Zhiming Liu |
| author_sort | Zongtao Wang |
| collection | DOAJ |
| description | The Generative Adversarial Network (GAN) has recently experienced great progress in compositional image synthesis. Unfortunately, the models proposed in the literature usually require a set of pre-defined local generators and use a separate generator to model each part object. This makes the model inflexible and also limits its scalability. Inspired by humans’ structured memory system, we propose MemoryGAN to eliminate these disadvantages. MemoryGAN uses a single generator as a shared memory to hold the heterogeneous information of the parts, and it uses a recurrent neural network to model the dependency between the parts and provide the query code for the memory. The shared memory structure and the query and feedback mechanism make MemoryGAN flexible and scalable. Our experiment shows that although MemoryGAN only uses a single generator for all the parts, it achieves comparable performance with the state-of-the-art, which uses multiple generators, in terms of synthesized image quality, compositional ability and disentanglement property. We believe that our result of using the generator of the GAN as a memory model will inspire future work of both bio-friendly models and memory-augmented models. |
| first_indexed | 2024-03-11T01:42:52Z |
| format | Article |
| id | doaj.art-52be6fc713944db99d1107a2a201dae9 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-11T01:42:52Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-52be6fc713944db99d1107a2a201dae92023-11-18T16:25:27ZengMDPI AGElectronics2079-92922023-07-011213292710.3390/electronics12132927MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image SynthesisZongtao Wang0Jiajie Peng1Zhiming Liu2School of Computer & Information Science, Southwest University, Chongqing 400715, ChinaSchool of Computer Science, Northwestern Polytechnical University, Xi’an 710072, ChinaSchool of Computer & Information Science, Southwest University, Chongqing 400715, ChinaThe Generative Adversarial Network (GAN) has recently experienced great progress in compositional image synthesis. Unfortunately, the models proposed in the literature usually require a set of pre-defined local generators and use a separate generator to model each part object. This makes the model inflexible and also limits its scalability. Inspired by humans’ structured memory system, we propose MemoryGAN to eliminate these disadvantages. MemoryGAN uses a single generator as a shared memory to hold the heterogeneous information of the parts, and it uses a recurrent neural network to model the dependency between the parts and provide the query code for the memory. The shared memory structure and the query and feedback mechanism make MemoryGAN flexible and scalable. Our experiment shows that although MemoryGAN only uses a single generator for all the parts, it achieves comparable performance with the state-of-the-art, which uses multiple generators, in terms of synthesized image quality, compositional ability and disentanglement property. We believe that our result of using the generator of the GAN as a memory model will inspire future work of both bio-friendly models and memory-augmented models.https://www.mdpi.com/2079-9292/12/13/2927Generative Adversarial Networkcompositional image synthesismemorydisentanglement |
| spellingShingle | Zongtao Wang Jiajie Peng Zhiming Liu MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image Synthesis Electronics Generative Adversarial Network compositional image synthesis memory disentanglement |
| title | MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image Synthesis |
| title_full | MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image Synthesis |
| title_fullStr | MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image Synthesis |
| title_full_unstemmed | MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image Synthesis |
| title_short | MemoryGAN: GAN Generator as Heterogeneous Memory for Compositional Image Synthesis |
| title_sort | memorygan gan generator as heterogeneous memory for compositional image synthesis |
| topic | Generative Adversarial Network compositional image synthesis memory disentanglement |
| url | https://www.mdpi.com/2079-9292/12/13/2927 |
| work_keys_str_mv | AT zongtaowang memorygangangeneratorasheterogeneousmemoryforcompositionalimagesynthesis AT jiajiepeng memorygangangeneratorasheterogeneousmemoryforcompositionalimagesynthesis AT zhimingliu memorygangangeneratorasheterogeneousmemoryforcompositionalimagesynthesis |