Optimal transport based simulation methods for deep probabilistic models
<p>Deep probabilistic models have emerged as state-of-the-art for high-dimensional, multi-modal data synthesis and density estimation tasks. By combining abstract probabilistic formulations with the expressivity and scalability of neural networks, deep probabilistic models have become a fundam...
| Hovedforfatter: | |
|---|---|
| Andre forfattere: | |
| Format: | Thesis |
| Sprog: | English |
| Udgivet: |
2023
|
| Fag: |
| _version_ | 1826313273868288000 |
|---|---|
| author | Thornton, J |
| author2 | Deligiannidis, G |
| author_facet | Deligiannidis, G Thornton, J |
| author_sort | Thornton, J |
| collection | OXFORD |
| description | <p>Deep probabilistic models have emerged as state-of-the-art for high-dimensional,
multi-modal data synthesis and density estimation tasks. By combining abstract
probabilistic formulations with the expressivity and scalability of neural networks,
deep probabilistic models have become a fundamental component of the machine
learning toolbox. Such models still have a number of limitations however. For
example, deep probabilistic models are often limited to gradient based training and
hence struggle to incorporate non-differentiable operations; they are expensive to
train and sample from; and often deep probabilistic models do not leverage prior
geometric and problem-specific structural knowledge.</p>
<br>
<p>This thesis consists of four contributing pieces of work and advances the field of
deep probabilistic models through optimal transport based simulation methods.
First, by using regularized optimal transport via the Sinkhorn algorithm, we
provide a theoretically grounded and differentiable approximation to resampling
within particle filtering. This allows one to perform gradient based training of
state space models, a class of sequential probabilistic model, with end-to-end
differentiable particle filtering. Next, we explore initialization strategies for the
Sinkhorn algorithm to address speed issues. We show that careful initializations
result in dramatic acceleration of the Sinkhorn algorithm. This has applications in
differentiable sorting; clustering within the latent space of a variational autoencoder;
and within particle filtering. The remaining two works contribute to the field of
diffusion based generative modelling through the Schrödinger Bridge. First, we
connect diffusion models to the Schrödinger Bridge, coined the Diffusion Schrödinger
Bridge. This methodology enables accelerated sampling; data-to-data simulation,
and a novel way to compute regularized optimal transport for high dimensional,
continuous state-space problems. Finally, we extend the <i>Diffusion Schrödinger
Bridge</i> to the Riemannian manifold setting. This allows one to incorporate prior
geometric knowledge and hence enable more efficient training and inference for
diffusion models on Riemannian manifold valued data. This has applications
in climate and Earth science.</p> |
| first_indexed | 2024-09-25T04:10:29Z |
| format | Thesis |
| id | oxford-uuid:6411bcfe-b31f-41bd-8558-357cb5e3a076 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-09-25T04:10:29Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | oxford-uuid:6411bcfe-b31f-41bd-8558-357cb5e3a0762024-06-21T15:21:57ZOptimal transport based simulation methods for deep probabilistic modelsThesishttp://purl.org/coar/resource_type/c_db06uuid:6411bcfe-b31f-41bd-8558-357cb5e3a076Computational statisticsOptimal transportMachine learningEnglishHyrax Deposit2023Thornton, JDeligiannidis, GDoucet, A<p>Deep probabilistic models have emerged as state-of-the-art for high-dimensional, multi-modal data synthesis and density estimation tasks. By combining abstract probabilistic formulations with the expressivity and scalability of neural networks, deep probabilistic models have become a fundamental component of the machine learning toolbox. Such models still have a number of limitations however. For example, deep probabilistic models are often limited to gradient based training and hence struggle to incorporate non-differentiable operations; they are expensive to train and sample from; and often deep probabilistic models do not leverage prior geometric and problem-specific structural knowledge.</p> <br> <p>This thesis consists of four contributing pieces of work and advances the field of deep probabilistic models through optimal transport based simulation methods. First, by using regularized optimal transport via the Sinkhorn algorithm, we provide a theoretically grounded and differentiable approximation to resampling within particle filtering. This allows one to perform gradient based training of state space models, a class of sequential probabilistic model, with end-to-end differentiable particle filtering. Next, we explore initialization strategies for the Sinkhorn algorithm to address speed issues. We show that careful initializations result in dramatic acceleration of the Sinkhorn algorithm. This has applications in differentiable sorting; clustering within the latent space of a variational autoencoder; and within particle filtering. The remaining two works contribute to the field of diffusion based generative modelling through the Schrödinger Bridge. First, we connect diffusion models to the Schrödinger Bridge, coined the Diffusion Schrödinger Bridge. This methodology enables accelerated sampling; data-to-data simulation, and a novel way to compute regularized optimal transport for high dimensional, continuous state-space problems. Finally, we extend the <i>Diffusion Schrödinger Bridge</i> to the Riemannian manifold setting. This allows one to incorporate prior geometric knowledge and hence enable more efficient training and inference for diffusion models on Riemannian manifold valued data. This has applications in climate and Earth science.</p> |
| spellingShingle | Computational statistics Optimal transport Machine learning Thornton, J Optimal transport based simulation methods for deep probabilistic models |
| title | Optimal transport based simulation methods for deep probabilistic models |
| title_full | Optimal transport based simulation methods for deep probabilistic models |
| title_fullStr | Optimal transport based simulation methods for deep probabilistic models |
| title_full_unstemmed | Optimal transport based simulation methods for deep probabilistic models |
| title_short | Optimal transport based simulation methods for deep probabilistic models |
| title_sort | optimal transport based simulation methods for deep probabilistic models |
| topic | Computational statistics Optimal transport Machine learning |
| work_keys_str_mv | AT thorntonj optimaltransportbasedsimulationmethodsfordeepprobabilisticmodels |