DreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transfer
3D scene understanding for robotic applications exhibits a unique set of requirements including real-time inference, object-centric latent representation learning, accurate 6D pose estimation and 3D reconstruction of objects. Current methods for scene understanding typically rely on a combination of...
| Main Authors: | , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
IEEE
2024
|
| _version_ | 1797112834203582464 |
|---|---|
| author | Wu, Y Sáez de Ocáriz Borde, H Collins, J Jones, OP Posner, I |
| author_facet | Wu, Y Sáez de Ocáriz Borde, H Collins, J Jones, OP Posner, I |
| author_sort | Wu, Y |
| collection | OXFORD |
| description | 3D scene understanding for robotic applications exhibits a unique set of requirements including real-time inference, object-centric latent representation learning, accurate 6D pose estimation and 3D reconstruction of objects. Current methods for scene understanding typically rely on a combination of trained models paired with either an explicit or learnt volumetric representation, all of which have their own drawbacks and limitations. We introduce DreamUp3D , a novel Object-Centric Generative Model (OCGM) designed explicitly to perform inference on a 3D scene informed only by a single RGB-D image. DreamUp3D is a self-supervised model, trained end-to-end, and is capable of segmenting objects, providing 3D object reconstructions, generating object-centric latent representations and accurate per-object 6D pose estimates. We compare DreamUp3D to baselines including NeRFs, pre-trained CLIP-features, ObSurf, and ObPose, in a range of tasks including 3D scene reconstruction, object-matching and object pose estimation. Our experiments show that our model outperforms all baselines by a significant margin in real-world scenarios displaying its applicability for 3D scene understanding tasks while meeting the strict demands exhibited in robotics applications. |
| first_indexed | 2024-03-07T08:29:30Z |
| format | Journal article |
| id | oxford-uuid:a2fe43e1-c726-4ee5-a015-8166c42d19e2 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T08:29:30Z |
| publishDate | 2024 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:a2fe43e1-c726-4ee5-a015-8166c42d19e22024-03-06T09:45:15ZDreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transferJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:a2fe43e1-c726-4ee5-a015-8166c42d19e2EnglishSymplectic ElementsIEEE2024Wu, YSáez de Ocáriz Borde, HCollins, JJones, OPPosner, I3D scene understanding for robotic applications exhibits a unique set of requirements including real-time inference, object-centric latent representation learning, accurate 6D pose estimation and 3D reconstruction of objects. Current methods for scene understanding typically rely on a combination of trained models paired with either an explicit or learnt volumetric representation, all of which have their own drawbacks and limitations. We introduce DreamUp3D , a novel Object-Centric Generative Model (OCGM) designed explicitly to perform inference on a 3D scene informed only by a single RGB-D image. DreamUp3D is a self-supervised model, trained end-to-end, and is capable of segmenting objects, providing 3D object reconstructions, generating object-centric latent representations and accurate per-object 6D pose estimates. We compare DreamUp3D to baselines including NeRFs, pre-trained CLIP-features, ObSurf, and ObPose, in a range of tasks including 3D scene reconstruction, object-matching and object pose estimation. Our experiments show that our model outperforms all baselines by a significant margin in real-world scenarios displaying its applicability for 3D scene understanding tasks while meeting the strict demands exhibited in robotics applications. |
| spellingShingle | Wu, Y Sáez de Ocáriz Borde, H Collins, J Jones, OP Posner, I DreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transfer |
| title | DreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transfer |
| title_full | DreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transfer |
| title_fullStr | DreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transfer |
| title_full_unstemmed | DreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transfer |
| title_short | DreamUp3D: object-centric generative models for single-view 3D scene understanding and real-to-sim transfer |
| title_sort | dreamup3d object centric generative models for single view 3d scene understanding and real to sim transfer |
| work_keys_str_mv | AT wuy dreamup3dobjectcentricgenerativemodelsforsingleview3dsceneunderstandingandrealtosimtransfer AT saezdeocarizbordeh dreamup3dobjectcentricgenerativemodelsforsingleview3dsceneunderstandingandrealtosimtransfer AT collinsj dreamup3dobjectcentricgenerativemodelsforsingleview3dsceneunderstandingandrealtosimtransfer AT jonesop dreamup3dobjectcentricgenerativemodelsforsingleview3dsceneunderstandingandrealtosimtransfer AT posneri dreamup3dobjectcentricgenerativemodelsforsingleview3dsceneunderstandingandrealtosimtransfer |