Covariances, robustness, and variational Bayes
Mean-field Variational Bayes (MFVB) is an approximate Bayesian posterior inference technique that is increasingly popular due to its fast runtimes on large-scale data sets. However, even when MFVB provides accurate posterior means for certain parameters, it often mis-estimates variances and covarian...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
MIT Press
2020
|
| Online Access: | https://hdl.handle.net/1721.1/128780 |
| _version_ | 1826210336197312512 |
|---|---|
| author | Broderick, Tamara A |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Broderick, Tamara A |
| author_sort | Broderick, Tamara A |
| collection | MIT |
| description | Mean-field Variational Bayes (MFVB) is an approximate Bayesian posterior inference technique that is increasingly popular due to its fast runtimes on large-scale data sets. However, even when MFVB provides accurate posterior means for certain parameters, it often mis-estimates variances and covariances. Furthermore, prior robustness measures have remained undeveloped for MFVB. By deriving a simple formula for the effect of infinitesimal model perturbations on MFVB posterior means, we provide both improved covariance estimates and local robustness measures for MFVB, thus greatly expanding the practical usefulness of MFVB posterior approximations. The estimates for MFVB posterior covariances rely on a result from the classical Bayesian robustness literature that relates derivatives of posterior expectations to posterior covariances and includes the Laplace approximation as a special case. Our key condition is that the MFVB approximation provides good estimates of a select subset of posterior means-an assumption that has been shown to hold in many practical settings. In our experiments, we demonstrate that our methods are simple, general, and fast, providing accurate posterior uncertainty estimates and robustness measures with runtimes that can be an order of magnitude faster than MCMC. |
| first_indexed | 2024-09-23T14:47:54Z |
| format | Article |
| id | mit-1721.1/128780 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T14:47:54Z |
| publishDate | 2020 |
| publisher | MIT Press |
| record_format | dspace |
| spelling | mit-1721.1/1287802022-10-01T22:33:56Z Covariances, robustness, and variational Bayes Broderick, Tamara A Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Mean-field Variational Bayes (MFVB) is an approximate Bayesian posterior inference technique that is increasingly popular due to its fast runtimes on large-scale data sets. However, even when MFVB provides accurate posterior means for certain parameters, it often mis-estimates variances and covariances. Furthermore, prior robustness measures have remained undeveloped for MFVB. By deriving a simple formula for the effect of infinitesimal model perturbations on MFVB posterior means, we provide both improved covariance estimates and local robustness measures for MFVB, thus greatly expanding the practical usefulness of MFVB posterior approximations. The estimates for MFVB posterior covariances rely on a result from the classical Bayesian robustness literature that relates derivatives of posterior expectations to posterior covariances and includes the Laplace approximation as a special case. Our key condition is that the MFVB approximation provides good estimates of a select subset of posterior means-an assumption that has been shown to hold in many practical settings. In our experiments, we demonstrate that our methods are simple, general, and fast, providing accurate posterior uncertainty estimates and robustness measures with runtimes that can be an order of magnitude faster than MCMC. 2020-12-10T21:31:01Z 2020-12-10T21:31:01Z 2018-08 2018-07 2020-12-03T18:15:20Z Article http://purl.org/eprint/type/JournalArticle 1533-7928 1532-4435 https://hdl.handle.net/1721.1/128780 Giordano, Ryan, Tamara Broderick and Michael I. Jordan. “Covariances, robustness, and variational Bayes.” Journal of Machine Learning Research, 19 (August 2018): 1-49 © 2018 The Author(s) en https://jmlr.org/papers/v19/17-670.html Journal of Machine Learning Research Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf MIT Press Journal of Machine Learning Research |
| spellingShingle | Broderick, Tamara A Covariances, robustness, and variational Bayes |
| title | Covariances, robustness, and variational Bayes |
| title_full | Covariances, robustness, and variational Bayes |
| title_fullStr | Covariances, robustness, and variational Bayes |
| title_full_unstemmed | Covariances, robustness, and variational Bayes |
| title_short | Covariances, robustness, and variational Bayes |
| title_sort | covariances robustness and variational bayes |
| url | https://hdl.handle.net/1721.1/128780 |
| work_keys_str_mv | AT brodericktamaraa covariancesrobustnessandvariationalbayes |