Semi-Amortized Variational Autoencoders
© CURRAN-CONFERENCE. All rights reserved. Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network. While AVI has enabled efficient training of deep generative models such as variational autoencoders (VAE), recent empirical work suggests that i...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137476 |
| _version_ | 1826207770582450176 |
|---|---|
| author | Kim, Yoon Wiseman, Sam Miller, Andrew C. Sontag, David Rush, Alexander M. |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Kim, Yoon Wiseman, Sam Miller, Andrew C. Sontag, David Rush, Alexander M. |
| author_sort | Kim, Yoon |
| collection | MIT |
| description | © CURRAN-CONFERENCE. All rights reserved. Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network. While AVI has enabled efficient training of deep generative models such as variational autoencoders (VAE), recent empirical work suggests that inference networks can produce suboptimal variational parameters. We propose a hybrid approach, to use AVI to initialize the variational parameters and run stochastic variational inference (SVI) to refine them. Crucially, the local SVI procedure is itself differentiable, so the inference network and generative model can be trained end-to-end with gradient-based optimization. This semi-amortized approach enables the use of rich generative models without experiencing the posterior-collapse phenomenon common in training VAEs for problems like text generation. Experiments show this approach outperforms strong autoregressive and variational baselines on standard text and image datasets. |
| first_indexed | 2024-09-23T13:54:41Z |
| format | Article |
| id | mit-1721.1/137476 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:54:41Z |
| publishDate | 2021 |
| record_format | dspace |
| spelling | mit-1721.1/1374762022-01-14T21:35:28Z Semi-Amortized Variational Autoencoders Kim, Yoon Wiseman, Sam Miller, Andrew C. Sontag, David Rush, Alexander M. Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Institute for Medical Engineering & Science © CURRAN-CONFERENCE. All rights reserved. Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network. While AVI has enabled efficient training of deep generative models such as variational autoencoders (VAE), recent empirical work suggests that inference networks can produce suboptimal variational parameters. We propose a hybrid approach, to use AVI to initialize the variational parameters and run stochastic variational inference (SVI) to refine them. Crucially, the local SVI procedure is itself differentiable, so the inference network and generative model can be trained end-to-end with gradient-based optimization. This semi-amortized approach enables the use of rich generative models without experiencing the posterior-collapse phenomenon common in training VAEs for problems like text generation. Experiments show this approach outperforms strong autoregressive and variational baselines on standard text and image datasets. 2021-11-05T14:22:51Z 2021-11-05T14:22:51Z 2018 2021-04-09T14:27:01Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137476 Kim, Yoon, Wiseman, Sam, Miller, Andrew C., Sontag, David and Rush, Alexander M. 2018. "Semi-Amortized Variational Autoencoders." 35th International Conference on Machine Learning, ICML 2018, 6. en http://proceedings.mlr.press/v80/kim18e.html 35th International Conference on Machine Learning, ICML 2018 Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Proceedings of Machine Learning Research |
| spellingShingle | Kim, Yoon Wiseman, Sam Miller, Andrew C. Sontag, David Rush, Alexander M. Semi-Amortized Variational Autoencoders |
| title | Semi-Amortized Variational Autoencoders |
| title_full | Semi-Amortized Variational Autoencoders |
| title_fullStr | Semi-Amortized Variational Autoencoders |
| title_full_unstemmed | Semi-Amortized Variational Autoencoders |
| title_short | Semi-Amortized Variational Autoencoders |
| title_sort | semi amortized variational autoencoders |
| url | https://hdl.handle.net/1721.1/137476 |
| work_keys_str_mv | AT kimyoon semiamortizedvariationalautoencoders AT wisemansam semiamortizedvariationalautoencoders AT millerandrewc semiamortizedvariationalautoencoders AT sontagdavid semiamortizedvariationalautoencoders AT rushalexanderm semiamortizedvariationalautoencoders |