Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation
Variational inference is a powerful framework, used to approximate intractable posteriors through variational distributions. The de facto standard is to rely on Gaussian variational families, which come with numerous advantages: they are easy to sample from, simple to parametrize, and many expectati...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-07-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/23/8/990 |
| _version_ | 1797523915820498944 |
|---|---|
| author | Théo Galy-Fajou Valerio Perrone Manfred Opper |
| author_facet | Théo Galy-Fajou Valerio Perrone Manfred Opper |
| author_sort | Théo Galy-Fajou |
| collection | DOAJ |
| description | Variational inference is a powerful framework, used to approximate intractable posteriors through variational distributions. The de facto standard is to rely on Gaussian variational families, which come with numerous advantages: they are easy to sample from, simple to parametrize, and many expectations are known in closed-form or readily computed by quadrature. In this paper, we view the Gaussian variational approximation problem through the lens of gradient flows. We introduce a flexible and efficient algorithm based on a linear flow leading to a particle-based approximation. We prove that, with a sufficient number of particles, our algorithm converges linearly to the exact solution for Gaussian targets, and a low-rank approximation otherwise. In addition to the theoretical analysis, we show, on a set of synthetic and real-world high-dimensional problems, that our algorithm outperforms existing methods with Gaussian targets while performing on a par with non-Gaussian targets. |
| first_indexed | 2024-03-10T08:49:55Z |
| format | Article |
| id | doaj.art-9477fd3509134d62807bb6d0cf170b3b |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-10T08:49:55Z |
| publishDate | 2021-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-9477fd3509134d62807bb6d0cf170b3b2023-11-22T07:34:42ZengMDPI AGEntropy1099-43002021-07-0123899010.3390/e23080990Flexible and Efficient Inference with Particles for the Variational Gaussian ApproximationThéo Galy-Fajou0Valerio Perrone1Manfred Opper2Artificial Intelligence Group, Technische Universität Berlin, 10623 Berlin, GermanyAmazon Web Services, 10969 Berlin, GermanyArtificial Intelligence Group, Technische Universität Berlin, 10623 Berlin, GermanyVariational inference is a powerful framework, used to approximate intractable posteriors through variational distributions. The de facto standard is to rely on Gaussian variational families, which come with numerous advantages: they are easy to sample from, simple to parametrize, and many expectations are known in closed-form or readily computed by quadrature. In this paper, we view the Gaussian variational approximation problem through the lens of gradient flows. We introduce a flexible and efficient algorithm based on a linear flow leading to a particle-based approximation. We prove that, with a sufficient number of particles, our algorithm converges linearly to the exact solution for Gaussian targets, and a low-rank approximation otherwise. In addition to the theoretical analysis, we show, on a set of synthetic and real-world high-dimensional problems, that our algorithm outperforms existing methods with Gaussian targets while performing on a par with non-Gaussian targets.https://www.mdpi.com/1099-4300/23/8/990variational inferenceGaussianparticle flowvariable flow |
| spellingShingle | Théo Galy-Fajou Valerio Perrone Manfred Opper Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation Entropy variational inference Gaussian particle flow variable flow |
| title | Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation |
| title_full | Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation |
| title_fullStr | Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation |
| title_full_unstemmed | Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation |
| title_short | Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation |
| title_sort | flexible and efficient inference with particles for the variational gaussian approximation |
| topic | variational inference Gaussian particle flow variable flow |
| url | https://www.mdpi.com/1099-4300/23/8/990 |
| work_keys_str_mv | AT theogalyfajou flexibleandefficientinferencewithparticlesforthevariationalgaussianapproximation AT valerioperrone flexibleandefficientinferencewithparticlesforthevariationalgaussianapproximation AT manfredopper flexibleandefficientinferencewithparticlesforthevariationalgaussianapproximation |