Tighter variational bounds are not necessarily better
We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter E...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Proceedings of Machine Learning Research
2018
|
| _version_ | 1797053824312016896 |
|---|---|
| author | Rainforth, T Kosiorek, A Le, T Maddison, C Igl, M Wood, F Teh, Y |
| author_facet | Rainforth, T Kosiorek, A Le, T Maddison, C Igl, M Wood, F Teh, Y |
| author_sort | Rainforth, T |
| collection | OXFORD |
| description | We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted autoencoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Furthermore, our results suggest that PIWAE may be able to deliver simultaneous improvements in the training of both the inference and generative networks. |
| first_indexed | 2024-03-06T18:48:59Z |
| format | Conference item |
| id | oxford-uuid:0f862a36-5288-422c-b39a-46feaf717a9c |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T18:48:59Z |
| publishDate | 2018 |
| publisher | Proceedings of Machine Learning Research |
| record_format | dspace |
| spelling | oxford-uuid:0f862a36-5288-422c-b39a-46feaf717a9c2022-03-26T09:51:40ZTighter variational bounds are not necessarily betterConference itemhttp://purl.org/coar/resource_type/c_5794uuid:0f862a36-5288-422c-b39a-46feaf717a9cEnglishSymplectic Elements at OxfordProceedings of Machine Learning Research2018Rainforth, TKosiorek, ALe, TMaddison, CIgl, MWood, FTeh, YWe provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted autoencoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Furthermore, our results suggest that PIWAE may be able to deliver simultaneous improvements in the training of both the inference and generative networks. |
| spellingShingle | Rainforth, T Kosiorek, A Le, T Maddison, C Igl, M Wood, F Teh, Y Tighter variational bounds are not necessarily better |
| title | Tighter variational bounds are not necessarily better |
| title_full | Tighter variational bounds are not necessarily better |
| title_fullStr | Tighter variational bounds are not necessarily better |
| title_full_unstemmed | Tighter variational bounds are not necessarily better |
| title_short | Tighter variational bounds are not necessarily better |
| title_sort | tighter variational bounds are not necessarily better |
| work_keys_str_mv | AT rainfortht tightervariationalboundsarenotnecessarilybetter AT kosioreka tightervariationalboundsarenotnecessarilybetter AT let tightervariationalboundsarenotnecessarilybetter AT maddisonc tightervariationalboundsarenotnecessarilybetter AT iglm tightervariationalboundsarenotnecessarilybetter AT woodf tightervariationalboundsarenotnecessarilybetter AT tehy tightervariationalboundsarenotnecessarilybetter |