GOEnFusion: gradient origin encodings for 3D forward diffusion models
The recently introduced Forward-Diffusion method allows to train a 3D diffusion model using only 2D images for supervision. However, it does not easily generalise to different 3D representations and requires a computationally expensive auto-regressive sampling process to generate the underlying 3D s...
| Main Authors: | , , , |
|---|---|
| Format: | Internet publication |
| Language: | English |
| Published: |
2023
|
| _version_ | 1826317573739773952 |
|---|---|
| author | Karnewar, A Vedaldi, A Mitra, NJ Novotny, D |
| author_facet | Karnewar, A Vedaldi, A Mitra, NJ Novotny, D |
| author_sort | Karnewar, A |
| collection | OXFORD |
| description | The recently introduced Forward-Diffusion method allows to train a 3D diffusion model using only 2D images for supervision. However, it does not easily generalise to different 3D representations and requires a computationally expensive auto-regressive sampling process to generate the underlying 3D scenes. In this paper, we propose GOEn: Gradient Origin Encoding (pronounced "gone"). GOEn can encode input images into any type of 3D representation without the need to use a pre-trained image feature extractor. It can also handle single, multiple or no source view(s) alike, by design, and tries to maximise the information transfer from the views to the encodings. Our proposed GOEnFusion model pairs GOEn encodings with a realisation of the Forward-Diffusion model which addresses the limitations of the vanilla Forward-Diffusion realisation. We evaluate how much information the GOEn mechanism transfers to the encoded representations, and how well it captures the prior distribution over the underlying 3D scenes, through the lens of a partial AutoEncoder. Lastly, the efficacy of the GOEnFusion model is evaluated on the recently proposed OmniObject3D dataset while comparing to the state-of-the-art Forward and non-Forward-Diffusion models and other 3D generative models.
|
| first_indexed | 2025-03-11T16:56:03Z |
| format | Internet publication |
| id | oxford-uuid:fff3ece1-a249-4b85-bb7d-01f16b0c8e8f |
| institution | University of Oxford |
| language | English |
| last_indexed | 2025-03-11T16:56:03Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | oxford-uuid:fff3ece1-a249-4b85-bb7d-01f16b0c8e8f2025-02-19T16:12:48ZGOEnFusion: gradient origin encodings for 3D forward diffusion modelsInternet publicationhttp://purl.org/coar/resource_type/c_7ad9uuid:fff3ece1-a249-4b85-bb7d-01f16b0c8e8fEnglishSymplectic Elements2023Karnewar, AVedaldi, AMitra, NJNovotny, DThe recently introduced Forward-Diffusion method allows to train a 3D diffusion model using only 2D images for supervision. However, it does not easily generalise to different 3D representations and requires a computationally expensive auto-regressive sampling process to generate the underlying 3D scenes. In this paper, we propose GOEn: Gradient Origin Encoding (pronounced "gone"). GOEn can encode input images into any type of 3D representation without the need to use a pre-trained image feature extractor. It can also handle single, multiple or no source view(s) alike, by design, and tries to maximise the information transfer from the views to the encodings. Our proposed GOEnFusion model pairs GOEn encodings with a realisation of the Forward-Diffusion model which addresses the limitations of the vanilla Forward-Diffusion realisation. We evaluate how much information the GOEn mechanism transfers to the encoded representations, and how well it captures the prior distribution over the underlying 3D scenes, through the lens of a partial AutoEncoder. Lastly, the efficacy of the GOEnFusion model is evaluated on the recently proposed OmniObject3D dataset while comparing to the state-of-the-art Forward and non-Forward-Diffusion models and other 3D generative models. |
| spellingShingle | Karnewar, A Vedaldi, A Mitra, NJ Novotny, D GOEnFusion: gradient origin encodings for 3D forward diffusion models |
| title | GOEnFusion: gradient origin encodings for 3D forward diffusion models |
| title_full | GOEnFusion: gradient origin encodings for 3D forward diffusion models |
| title_fullStr | GOEnFusion: gradient origin encodings for 3D forward diffusion models |
| title_full_unstemmed | GOEnFusion: gradient origin encodings for 3D forward diffusion models |
| title_short | GOEnFusion: gradient origin encodings for 3D forward diffusion models |
| title_sort | goenfusion gradient origin encodings for 3d forward diffusion models |
| work_keys_str_mv | AT karnewara goenfusiongradientoriginencodingsfor3dforwarddiffusionmodels AT vedaldia goenfusiongradientoriginencodingsfor3dforwarddiffusionmodels AT mitranj goenfusiongradientoriginencodingsfor3dforwarddiffusionmodels AT novotnyd goenfusiongradientoriginencodingsfor3dforwarddiffusionmodels |