| Resumo: | Gradient-based optimization and Markov Chain Monte Carlo sampling can be found at the heart of a multitude of machine learning methods. In high-dimensional settings, well-known issues such as slow-mixing, non-convexity and correlations can hinder the algorithms’ efficiency. In order to overcome these difficulties, we propose AdaGeo, a preconditioning framework for adaptively learning the geometry of parameter space during optimization or sampling. We use the Gaussian Process latent variable model (GP-LVM) to represent a lower-dimensional embedding of the parameters, identifying the underlying Riemannian manifold on which the optimization or sampling are taking place. Samples or optimization steps are consequently proposed based on the geometry of the manifold. We apply our framework to stochastic gradient descent and stochastic gradient Langevin dynamics and show performance improvements for both optimization and sampling.
|
|---|