A Minimax Approach for Learning Gaussian Mixtures
Generative adversarial networks (GANs) learn the distribution of observed samples through a zero-sum game between two machine players, a generator and a discriminator. While GANs achieve great success in learning the complex distribution of image, sound, and text data, they perform suboptimally in l...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2022
|
| Online Access: | https://hdl.handle.net/1721.1/139991 |
| _version_ | 1826188846261338112 |
|---|---|
| author | Wang, William Wei |
| author2 | Jadbabaie, Ali |
| author_facet | Jadbabaie, Ali Wang, William Wei |
| author_sort | Wang, William Wei |
| collection | MIT |
| description | Generative adversarial networks (GANs) learn the distribution of observed samples through a zero-sum game between two machine players, a generator and a discriminator. While GANs achieve great success in learning the complex distribution of image, sound, and text data, they perform suboptimally in learning multi-modal distributionlearning benchmarks including Gaussian mixture models (GMMs). In this thesis, we propose Generative Adversarial Training for Gaussian Mixture Models (GATGMM), a minimax GAN framework for learning GMMs. Motivated by optimal transport theory, we design the zero-sum game in GAT-GMM using a random linear generator and a softmax-based quadratic discriminator architecture, which leads to a non-convex concave minimax optimization problem. We show that a Gradient Descent Ascent (GDA) method converges to an approximate stationary minimax point of the GAT-GMM optimization problem. In the benchmark case of a mixture of two symmetric, well-separated Gaussians, we further show this stationary point recovers the true parameters of the underlying GMM. We discuss the application of the proposed GAT-GMM framework for learning GMMs in the distributed federated learning setting, where the widely-used expectation-maximization (EM) algorithm can incur great computational and communication costs. On the other hand, we show that GAT-GMM provides a scalable learning approach and a distributed GDA algorithm can still solve the GAT-GMM minimax problem without incurring extra computation costs. We numerically support our theoretical results by performing experiments which show that our minimax framework is successful in centralized learning tasks and can outperform standard EM-type algorithms in the federated setting. |
| first_indexed | 2024-09-23T08:05:55Z |
| format | Thesis |
| id | mit-1721.1/139991 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T08:05:55Z |
| publishDate | 2022 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1399912022-02-08T04:07:15Z A Minimax Approach for Learning Gaussian Mixtures Wang, William Wei Jadbabaie, Ali Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Generative adversarial networks (GANs) learn the distribution of observed samples through a zero-sum game between two machine players, a generator and a discriminator. While GANs achieve great success in learning the complex distribution of image, sound, and text data, they perform suboptimally in learning multi-modal distributionlearning benchmarks including Gaussian mixture models (GMMs). In this thesis, we propose Generative Adversarial Training for Gaussian Mixture Models (GATGMM), a minimax GAN framework for learning GMMs. Motivated by optimal transport theory, we design the zero-sum game in GAT-GMM using a random linear generator and a softmax-based quadratic discriminator architecture, which leads to a non-convex concave minimax optimization problem. We show that a Gradient Descent Ascent (GDA) method converges to an approximate stationary minimax point of the GAT-GMM optimization problem. In the benchmark case of a mixture of two symmetric, well-separated Gaussians, we further show this stationary point recovers the true parameters of the underlying GMM. We discuss the application of the proposed GAT-GMM framework for learning GMMs in the distributed federated learning setting, where the widely-used expectation-maximization (EM) algorithm can incur great computational and communication costs. On the other hand, we show that GAT-GMM provides a scalable learning approach and a distributed GDA algorithm can still solve the GAT-GMM minimax problem without incurring extra computation costs. We numerically support our theoretical results by performing experiments which show that our minimax framework is successful in centralized learning tasks and can outperform standard EM-type algorithms in the federated setting. S.M. 2022-02-07T15:17:35Z 2022-02-07T15:17:35Z 2021-09 2021-09-21T19:54:19.248Z Thesis https://hdl.handle.net/1721.1/139991 0000-0002-7993-5145 In Copyright - Educational Use Permitted Copyright MIT http://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Wang, William Wei A Minimax Approach for Learning Gaussian Mixtures |
| title | A Minimax Approach for Learning Gaussian Mixtures |
| title_full | A Minimax Approach for Learning Gaussian Mixtures |
| title_fullStr | A Minimax Approach for Learning Gaussian Mixtures |
| title_full_unstemmed | A Minimax Approach for Learning Gaussian Mixtures |
| title_short | A Minimax Approach for Learning Gaussian Mixtures |
| title_sort | minimax approach for learning gaussian mixtures |
| url | https://hdl.handle.net/1721.1/139991 |
| work_keys_str_mv | AT wangwilliamwei aminimaxapproachforlearninggaussianmixtures AT wangwilliamwei minimaxapproachforlearninggaussianmixtures |