HoloDiffusion: training a 3D diffusion model using 2D Images
Diffusion models have emerged as the best approach for generative modeling of 2D images. Part of their success is due to the possibility of training them on millions if not billions of images with a stable learning objective. However, extending these models to 3D remains difficult for two reasons. F...
| Prif Awduron: | , , , |
|---|---|
| Fformat: | Conference item |
| Iaith: | English |
| Cyhoeddwyd: |
IEEE
2023
|
| _version_ | 1826310871283924992 |
|---|---|
| author | Karnewar, A Vedaldi, A Novotny, D Mitra, NJ |
| author_facet | Karnewar, A Vedaldi, A Novotny, D Mitra, NJ |
| author_sort | Karnewar, A |
| collection | OXFORD |
| description | Diffusion models have emerged as the best approach for
generative modeling of 2D images. Part of their success is
due to the possibility of training them on millions if not billions of images with a stable learning objective. However,
extending these models to 3D remains difficult for two reasons. First, finding a large quantity of 3D training data is
much more complex than for 2D images. Second, while it is
conceptually trivial to extend the models to operate on 3D
rather than 2D grids, the associated cubic growth in memory and compute complexity makes this infeasible. We address the first challenge by introducing a new diffusion setup
that can be trained, end-to-end, with only posed 2D images
for supervision; and the second challenge by proposing an
image formation model that decouples model memory from
spatial memory. We evaluate our method on real-world
data, using the CO3D dataset which has not been used to
train 3D generative models before. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.
|
| first_indexed | 2024-03-07T07:59:54Z |
| format | Conference item |
| id | oxford-uuid:0632fa8a-0331-45ea-9f75-ac10054b45fe |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:59:54Z |
| publishDate | 2023 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:0632fa8a-0331-45ea-9f75-ac10054b45fe2023-09-21T07:34:56ZHoloDiffusion: training a 3D diffusion model using 2D ImagesConference itemhttp://purl.org/coar/resource_type/c_5794uuid:0632fa8a-0331-45ea-9f75-ac10054b45feEnglishSymplectic ElementsIEEE2023Karnewar, AVedaldi, ANovotny, DMitra, NJDiffusion models have emerged as the best approach for generative modeling of 2D images. Part of their success is due to the possibility of training them on millions if not billions of images with a stable learning objective. However, extending these models to 3D remains difficult for two reasons. First, finding a large quantity of 3D training data is much more complex than for 2D images. Second, while it is conceptually trivial to extend the models to operate on 3D rather than 2D grids, the associated cubic growth in memory and compute complexity makes this infeasible. We address the first challenge by introducing a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision; and the second challenge by proposing an image formation model that decouples model memory from spatial memory. We evaluate our method on real-world data, using the CO3D dataset which has not been used to train 3D generative models before. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling. |
| spellingShingle | Karnewar, A Vedaldi, A Novotny, D Mitra, NJ HoloDiffusion: training a 3D diffusion model using 2D Images |
| title | HoloDiffusion: training a 3D diffusion model using 2D Images |
| title_full | HoloDiffusion: training a 3D diffusion model using 2D Images |
| title_fullStr | HoloDiffusion: training a 3D diffusion model using 2D Images |
| title_full_unstemmed | HoloDiffusion: training a 3D diffusion model using 2D Images |
| title_short | HoloDiffusion: training a 3D diffusion model using 2D Images |
| title_sort | holodiffusion training a 3d diffusion model using 2d images |
| work_keys_str_mv | AT karnewara holodiffusiontraininga3ddiffusionmodelusing2dimages AT vedaldia holodiffusiontraininga3ddiffusionmodelusing2dimages AT novotnyd holodiffusiontraininga3ddiffusionmodelusing2dimages AT mitranj holodiffusiontraininga3ddiffusionmodelusing2dimages |