MagicPony: learning articulated 3D animals in the wild
We consider the problem of learning a function that can estimate the 3D shape, articulation, viewpoint, texture, and lighting of an articulated animal like a horse, given a single test image. We present a new method, dubbed MagicPony, that learns this function purely from in-the-wild single-view ima...
| Main Authors: | , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
IEEE
2023
|
| _version_ | 1797110548857356288 |
|---|---|
| author | Wu, S Li, R Jakab, T Rupprecht, C Vedaldi, A |
| author_facet | Wu, S Li, R Jakab, T Rupprecht, C Vedaldi, A |
| author_sort | Wu, S |
| collection | OXFORD |
| description | We consider the problem of learning a function that can
estimate the 3D shape, articulation, viewpoint, texture, and
lighting of an articulated animal like a horse, given a single
test image. We present a new method, dubbed MagicPony,
that learns this function purely from in-the-wild single-view
images of the object category, with minimal assumptions
about the topology of deformation. At its core is an implicitexplicit representation of articulated shape and appearance, combining the strengths of neural fields and meshes.
In order to help the model understand an object’s shape
and pose, we distil the knowledge captured by an off-theshelf self-supervised vision transformer and fuse it into the
3D model. To overcome common local optima in viewpoint
estimation, we further introduce a new viewpoint sampling
scheme that comes at no added training cost. Compared to
prior works, we show significant quantitative and qualitative improvements on this challenging task. The model also
demonstrates excellent generalisation in reconstructing abstract drawings and artefacts, despite the fact that it is only
trained on real images. |
| first_indexed | 2024-03-07T07:56:20Z |
| format | Conference item |
| id | oxford-uuid:9d1b1cac-2b1c-4de1-9f3c-c2e128af9c29 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:56:20Z |
| publishDate | 2023 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:9d1b1cac-2b1c-4de1-9f3c-c2e128af9c292023-08-23T08:14:45ZMagicPony: learning articulated 3D animals in the wildConference itemhttp://purl.org/coar/resource_type/c_5794uuid:9d1b1cac-2b1c-4de1-9f3c-c2e128af9c29EnglishSymplectic ElementsIEEE2023Wu, SLi, RJakab, TRupprecht, CVedaldi, AWe consider the problem of learning a function that can estimate the 3D shape, articulation, viewpoint, texture, and lighting of an articulated animal like a horse, given a single test image. We present a new method, dubbed MagicPony, that learns this function purely from in-the-wild single-view images of the object category, with minimal assumptions about the topology of deformation. At its core is an implicitexplicit representation of articulated shape and appearance, combining the strengths of neural fields and meshes. In order to help the model understand an object’s shape and pose, we distil the knowledge captured by an off-theshelf self-supervised vision transformer and fuse it into the 3D model. To overcome common local optima in viewpoint estimation, we further introduce a new viewpoint sampling scheme that comes at no added training cost. Compared to prior works, we show significant quantitative and qualitative improvements on this challenging task. The model also demonstrates excellent generalisation in reconstructing abstract drawings and artefacts, despite the fact that it is only trained on real images. |
| spellingShingle | Wu, S Li, R Jakab, T Rupprecht, C Vedaldi, A MagicPony: learning articulated 3D animals in the wild |
| title | MagicPony: learning articulated 3D animals in the wild |
| title_full | MagicPony: learning articulated 3D animals in the wild |
| title_fullStr | MagicPony: learning articulated 3D animals in the wild |
| title_full_unstemmed | MagicPony: learning articulated 3D animals in the wild |
| title_short | MagicPony: learning articulated 3D animals in the wild |
| title_sort | magicpony learning articulated 3d animals in the wild |
| work_keys_str_mv | AT wus magicponylearningarticulated3danimalsinthewild AT lir magicponylearningarticulated3danimalsinthewild AT jakabt magicponylearningarticulated3danimalsinthewild AT rupprechtc magicponylearningarticulated3danimalsinthewild AT vedaldia magicponylearningarticulated3danimalsinthewild |