Relative entropy minimization over Hilbert spaces via Robbins-Monro
One way of getting insight into non-Gaussian measures is to first obtain good Gaussian approximations. These best fit Gaussians can then provide a sense of the mean and variance of the distribution of interest. They can also be used to accelerate sampling algorithms. This begs the question of how on...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
AIMS Press
2019-03-01
|
| Series: | AIMS Mathematics |
| Subjects: | |
| Online Access: | https://www.aimspress.com/article/10.3934/math.2019.3.359/fulltext.html |
| _version_ | 1819100046048624640 |
|---|---|
| author | Gideon Simpson Daniel Watkins |
| author_facet | Gideon Simpson Daniel Watkins |
| author_sort | Gideon Simpson |
| collection | DOAJ |
| description | One way of getting insight into non-Gaussian measures is to first obtain good Gaussian approximations. These best fit Gaussians can then provide a sense of the mean and variance of the distribution of interest. They can also be used to accelerate sampling algorithms. This begs the question of how one should measure optimality, and how to then obtain this optimal approximation. Here, we consider the problem of minimizing the distance between a family of Gaussians and the target measure with respect to relative entropy, or Kullback-Leibler divergence. As we are interested in applications in the infinite dimensional setting, it is desirable to have convergent algorithms that are well posed on abstract Hilbert spaces. We examine this minimization problem by seeking roots of the first variation of relative entropy, taken with respect to the mean of the Gaussian, leaving the covariance fixed. We prove the convergence of Robbins-Monro type root finding algorithms in this context, highlighting the assumptions necessary for convergence to relative entropy minimizers. Numerical examples are included to illustrate the algorithms. |
| first_indexed | 2024-12-22T00:56:32Z |
| format | Article |
| id | doaj.art-f06fb259fc6941e08c9f879a8f392aa1 |
| institution | Directory Open Access Journal |
| issn | 2473-6988 |
| language | English |
| last_indexed | 2024-12-22T00:56:32Z |
| publishDate | 2019-03-01 |
| publisher | AIMS Press |
| record_format | Article |
| series | AIMS Mathematics |
| spelling | doaj.art-f06fb259fc6941e08c9f879a8f392aa12022-12-21T18:44:18ZengAIMS PressAIMS Mathematics2473-69882019-03-014335938310.3934/math.2019.3.359Relative entropy minimization over Hilbert spaces via Robbins-MonroGideon Simpson0Daniel Watkins11 Department of Mathematics, Drexel University, Philadelphia, PA 19104, USA2 Department of Ocean, Earth, and Atmospheric Sciences, Oregon State University, Corvallis, OR 97331, USAOne way of getting insight into non-Gaussian measures is to first obtain good Gaussian approximations. These best fit Gaussians can then provide a sense of the mean and variance of the distribution of interest. They can also be used to accelerate sampling algorithms. This begs the question of how one should measure optimality, and how to then obtain this optimal approximation. Here, we consider the problem of minimizing the distance between a family of Gaussians and the target measure with respect to relative entropy, or Kullback-Leibler divergence. As we are interested in applications in the infinite dimensional setting, it is desirable to have convergent algorithms that are well posed on abstract Hilbert spaces. We examine this minimization problem by seeking roots of the first variation of relative entropy, taken with respect to the mean of the Gaussian, leaving the covariance fixed. We prove the convergence of Robbins-Monro type root finding algorithms in this context, highlighting the assumptions necessary for convergence to relative entropy minimizers. Numerical examples are included to illustrate the algorithms.https://www.aimspress.com/article/10.3934/math.2019.3.359/fulltext.htmlRobbins-Monrorelative entropyHilbert space |
| spellingShingle | Gideon Simpson Daniel Watkins Relative entropy minimization over Hilbert spaces via Robbins-Monro AIMS Mathematics Robbins-Monro relative entropy Hilbert space |
| title | Relative entropy minimization over Hilbert spaces via Robbins-Monro |
| title_full | Relative entropy minimization over Hilbert spaces via Robbins-Monro |
| title_fullStr | Relative entropy minimization over Hilbert spaces via Robbins-Monro |
| title_full_unstemmed | Relative entropy minimization over Hilbert spaces via Robbins-Monro |
| title_short | Relative entropy minimization over Hilbert spaces via Robbins-Monro |
| title_sort | relative entropy minimization over hilbert spaces via robbins monro |
| topic | Robbins-Monro relative entropy Hilbert space |
| url | https://www.aimspress.com/article/10.3934/math.2019.3.359/fulltext.html |
| work_keys_str_mv | AT gideonsimpson relativeentropyminimizationoverhilbertspacesviarobbinsmonro AT danielwatkins relativeentropyminimizationoverhilbertspacesviarobbinsmonro |