AdaGeo: Adaptive geometric learning for optimization and sampling
Gradient-based optimization and Markov Chain Monte Carlo sampling can be found at the heart of a multitude of machine learning methods. In high-dimensional settings, well-known issues such as slow-mixing, non-convexity and correlations can hinder the algorithms’ efficiency. In order to overcome thes...
| Những tác giả chính: | , , , |
|---|---|
| Định dạng: | Conference item |
| Được phát hành: |
Proceedings of Machine Learning Research
2018
|
| Tóm tắt: | Gradient-based optimization and Markov Chain Monte Carlo sampling can be found at the heart of a multitude of machine learning methods. In high-dimensional settings, well-known issues such as slow-mixing, non-convexity and correlations can hinder the algorithms’ efficiency. In order to overcome these difficulties, we propose AdaGeo, a preconditioning framework for adaptively learning the geometry of parameter space during optimization or sampling. We use the Gaussian Process latent variable model (GP-LVM) to represent a lower-dimensional embedding of the parameters, identifying the underlying Riemannian manifold on which the optimization or sampling are taking place. Samples or optimization steps are consequently proposed based on the geometry of the manifold. We apply our framework to stochastic gradient descent and stochastic gradient Langevin dynamics and show performance improvements for both optimization and sampling. |
|---|