Effective Dynamics of Generative Adversarial Networks
Generative adversarial networks (GANs) are a class of machine-learning models that use adversarial training to generate new samples with the same (potentially very complex) statistics as the training samples. One major form of training failure, known as mode collapse, involves the generator failing...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Physical Society
2023-10-01
|
| Series: | Physical Review X |
| Online Access: | http://doi.org/10.1103/PhysRevX.13.041004 |
| _version_ | 1797665779329531904 |
|---|---|
| author | Steven Durr Youssef Mroueh Yuhai Tu Shenshen Wang |
| author_facet | Steven Durr Youssef Mroueh Yuhai Tu Shenshen Wang |
| author_sort | Steven Durr |
| collection | DOAJ |
| description | Generative adversarial networks (GANs) are a class of machine-learning models that use adversarial training to generate new samples with the same (potentially very complex) statistics as the training samples. One major form of training failure, known as mode collapse, involves the generator failing to reproduce the full diversity of modes in the target probability distribution. Here, we present an effective model of GAN training, which captures the learning dynamics by replacing the generator neural network with a collection of particles in the output space; particles are coupled by a universal kernel valid for certain wide neural networks and high-dimensional inputs. The generality of our simplified model allows us to study the conditions under which mode collapse occurs. Indeed, experiments which vary the effective kernel of the generator reveal a mode collapse transition, the shape of which can be related to the type of discriminator through the frequency principle. Further, we find that gradient regularizers of intermediate strengths can optimally yield convergence through critical damping of the generator dynamics. Our effective GAN model thus provides an interpretable physical framework for understanding and improving adversarial training. |
| first_indexed | 2024-03-11T19:48:56Z |
| format | Article |
| id | doaj.art-e7b014f0ee48490dbc72717d07b041be |
| institution | Directory Open Access Journal |
| issn | 2160-3308 |
| language | English |
| last_indexed | 2024-03-11T19:48:56Z |
| publishDate | 2023-10-01 |
| publisher | American Physical Society |
| record_format | Article |
| series | Physical Review X |
| spelling | doaj.art-e7b014f0ee48490dbc72717d07b041be2023-10-05T14:01:33ZengAmerican Physical SocietyPhysical Review X2160-33082023-10-0113404100410.1103/PhysRevX.13.041004Effective Dynamics of Generative Adversarial NetworksSteven DurrYoussef MrouehYuhai TuShenshen WangGenerative adversarial networks (GANs) are a class of machine-learning models that use adversarial training to generate new samples with the same (potentially very complex) statistics as the training samples. One major form of training failure, known as mode collapse, involves the generator failing to reproduce the full diversity of modes in the target probability distribution. Here, we present an effective model of GAN training, which captures the learning dynamics by replacing the generator neural network with a collection of particles in the output space; particles are coupled by a universal kernel valid for certain wide neural networks and high-dimensional inputs. The generality of our simplified model allows us to study the conditions under which mode collapse occurs. Indeed, experiments which vary the effective kernel of the generator reveal a mode collapse transition, the shape of which can be related to the type of discriminator through the frequency principle. Further, we find that gradient regularizers of intermediate strengths can optimally yield convergence through critical damping of the generator dynamics. Our effective GAN model thus provides an interpretable physical framework for understanding and improving adversarial training.http://doi.org/10.1103/PhysRevX.13.041004 |
| spellingShingle | Steven Durr Youssef Mroueh Yuhai Tu Shenshen Wang Effective Dynamics of Generative Adversarial Networks Physical Review X |
| title | Effective Dynamics of Generative Adversarial Networks |
| title_full | Effective Dynamics of Generative Adversarial Networks |
| title_fullStr | Effective Dynamics of Generative Adversarial Networks |
| title_full_unstemmed | Effective Dynamics of Generative Adversarial Networks |
| title_short | Effective Dynamics of Generative Adversarial Networks |
| title_sort | effective dynamics of generative adversarial networks |
| url | http://doi.org/10.1103/PhysRevX.13.041004 |
| work_keys_str_mv | AT stevendurr effectivedynamicsofgenerativeadversarialnetworks AT youssefmroueh effectivedynamicsofgenerativeadversarialnetworks AT yuhaitu effectivedynamicsofgenerativeadversarialnetworks AT shenshenwang effectivedynamicsofgenerativeadversarialnetworks |