AdaGeo: Adaptive geometric learning for optimization and sampling
Gradient-based optimization and Markov Chain Monte Carlo sampling can be found at the heart of a multitude of machine learning methods. In high-dimensional settings, well-known issues such as slow-mixing, non-convexity and correlations can hinder the algorithms’ efficiency. In order to overcome thes...
| Main Authors: | , , , |
|---|---|
| Format: | Conference item |
| Published: |
Proceedings of Machine Learning Research
2018
|
| _version_ | 1797071115807358976 |
|---|---|
| author | Abbati, G Tosi, A Osborne, M Flaxman, S |
| author_facet | Abbati, G Tosi, A Osborne, M Flaxman, S |
| author_sort | Abbati, G |
| collection | OXFORD |
| description | Gradient-based optimization and Markov Chain Monte Carlo sampling can be found at the heart of a multitude of machine learning methods. In high-dimensional settings, well-known issues such as slow-mixing, non-convexity and correlations can hinder the algorithms’ efficiency. In order to overcome these difficulties, we propose AdaGeo, a preconditioning framework for adaptively learning the geometry of parameter space during optimization or sampling. We use the Gaussian Process latent variable model (GP-LVM) to represent a lower-dimensional embedding of the parameters, identifying the underlying Riemannian manifold on which the optimization or sampling are taking place. Samples or optimization steps are consequently proposed based on the geometry of the manifold. We apply our framework to stochastic gradient descent and stochastic gradient Langevin dynamics and show performance improvements for both optimization and sampling. |
| first_indexed | 2024-03-06T22:48:36Z |
| format | Conference item |
| id | oxford-uuid:5e0c3640-183b-409d-a2df-d56fb8e3b8c1 |
| institution | University of Oxford |
| last_indexed | 2024-03-06T22:48:36Z |
| publishDate | 2018 |
| publisher | Proceedings of Machine Learning Research |
| record_format | dspace |
| spelling | oxford-uuid:5e0c3640-183b-409d-a2df-d56fb8e3b8c12022-03-26T17:38:04ZAdaGeo: Adaptive geometric learning for optimization and samplingConference itemhttp://purl.org/coar/resource_type/c_5794uuid:5e0c3640-183b-409d-a2df-d56fb8e3b8c1Symplectic Elements at OxfordProceedings of Machine Learning Research2018Abbati, GTosi, AOsborne, MFlaxman, SGradient-based optimization and Markov Chain Monte Carlo sampling can be found at the heart of a multitude of machine learning methods. In high-dimensional settings, well-known issues such as slow-mixing, non-convexity and correlations can hinder the algorithms’ efficiency. In order to overcome these difficulties, we propose AdaGeo, a preconditioning framework for adaptively learning the geometry of parameter space during optimization or sampling. We use the Gaussian Process latent variable model (GP-LVM) to represent a lower-dimensional embedding of the parameters, identifying the underlying Riemannian manifold on which the optimization or sampling are taking place. Samples or optimization steps are consequently proposed based on the geometry of the manifold. We apply our framework to stochastic gradient descent and stochastic gradient Langevin dynamics and show performance improvements for both optimization and sampling. |
| spellingShingle | Abbati, G Tosi, A Osborne, M Flaxman, S AdaGeo: Adaptive geometric learning for optimization and sampling |
| title | AdaGeo: Adaptive geometric learning for optimization and sampling |
| title_full | AdaGeo: Adaptive geometric learning for optimization and sampling |
| title_fullStr | AdaGeo: Adaptive geometric learning for optimization and sampling |
| title_full_unstemmed | AdaGeo: Adaptive geometric learning for optimization and sampling |
| title_short | AdaGeo: Adaptive geometric learning for optimization and sampling |
| title_sort | adageo adaptive geometric learning for optimization and sampling |
| work_keys_str_mv | AT abbatig adageoadaptivegeometriclearningforoptimizationandsampling AT tosia adageoadaptivegeometriclearningforoptimizationandsampling AT osbornem adageoadaptivegeometriclearningforoptimizationandsampling AT flaxmans adageoadaptivegeometriclearningforoptimizationandsampling |