Linear response methods for accurate covariance estimates from mean field variational bayes
Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, a well known failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable c...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Neural Information Processing Systems Foundation
2017
|
| Online Access: | http://hdl.handle.net/1721.1/110786 https://orcid.org/0000-0003-4704-5196 |
| _version_ | 1811083420429713408 |
|---|---|
| author | Giordano, Ryan Jordan, Michael Broderick, Tamara A |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Giordano, Ryan Jordan, Michael Broderick, Tamara A |
| author_sort | Giordano, Ryan |
| collection | MIT |
| description | Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, a well known failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We generalize linear response methods from statistical physics to deliver accurate uncertainty estimates for model variables---both for individual variables and coherently across variables. We call our method linear response variational Bayes (LRVB). When the MFVB posterior approximation is in the exponential family, LRVB has a simple, analytic form, even for non-conjugate models. Indeed, we make no assumptions about the form of the true posterior. We demonstrate the accuracy and scalability of our method on a range of models for both simulated and real data. |
| first_indexed | 2024-09-23T12:32:47Z |
| format | Article |
| id | mit-1721.1/110786 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T12:32:47Z |
| publishDate | 2017 |
| publisher | Neural Information Processing Systems Foundation |
| record_format | dspace |
| spelling | mit-1721.1/1107862022-09-28T08:31:14Z Linear response methods for accurate covariance estimates from mean field variational bayes Giordano, Ryan Jordan, Michael Broderick, Tamara A Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Broderick, Tamara A Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, a well known failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We generalize linear response methods from statistical physics to deliver accurate uncertainty estimates for model variables---both for individual variables and coherently across variables. We call our method linear response variational Bayes (LRVB). When the MFVB posterior approximation is in the exponential family, LRVB has a simple, analytic form, even for non-conjugate models. Indeed, we make no assumptions about the form of the true posterior. We demonstrate the accuracy and scalability of our method on a range of models for both simulated and real data. 2017-07-20T14:53:40Z 2017-07-20T14:53:40Z 2015-12 Article http://purl.org/eprint/type/ConferencePaper http://hdl.handle.net/1721.1/110786 Giordano, Ryan, Tamara Broderick, Tamara and Michael Jordan. "Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes." Advances in Neural Information Processing Systems 28 (NIPS 2015), https://orcid.org/0000-0003-4704-5196 en_US https://papers.nips.cc/book/advances-in-neural-information-processing-systems-28-2015 Advances in Neural Information Processing Systems 28 (NIPS 2015) Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Neural Information Processing Systems Foundation Neural Information Processing Systems (NIPS) |
| spellingShingle | Giordano, Ryan Jordan, Michael Broderick, Tamara A Linear response methods for accurate covariance estimates from mean field variational bayes |
| title | Linear response methods for accurate covariance estimates from mean field variational bayes |
| title_full | Linear response methods for accurate covariance estimates from mean field variational bayes |
| title_fullStr | Linear response methods for accurate covariance estimates from mean field variational bayes |
| title_full_unstemmed | Linear response methods for accurate covariance estimates from mean field variational bayes |
| title_short | Linear response methods for accurate covariance estimates from mean field variational bayes |
| title_sort | linear response methods for accurate covariance estimates from mean field variational bayes |
| url | http://hdl.handle.net/1721.1/110786 https://orcid.org/0000-0003-4704-5196 |
| work_keys_str_mv | AT giordanoryan linearresponsemethodsforaccuratecovarianceestimatesfrommeanfieldvariationalbayes AT jordanmichael linearresponsemethodsforaccuratecovarianceestimatesfrommeanfieldvariationalbayes AT brodericktamaraa linearresponsemethodsforaccuratecovarianceestimatesfrommeanfieldvariationalbayes |