3D hand shape and pose from images in the wild
We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint det...
| Main Authors: | , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers
2020
|
| _version_ | 1797059652596269056 |
|---|---|
| author | Boukhayma, A de Bem, R Torr, PHS |
| author_facet | Boukhayma, A de Bem, R Torr, PHS |
| author_sort | Boukhayma, A |
| collection | OXFORD |
| description | We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters that projects the generated hand into the image domain. We show that using the shape and pose prior knowledge encoded in the hand model within a deep learning framework yields state-of-the-art performance in 3D pose prediction from images on standard benchmarks, and produces geometrically valid and plausible 3D reconstructions. Additionally, we show that training with weak supervision in the form of 2D joint annotations on datasets of images in the wild, in conjunction with full supervision in the form of 3D joint annotations on limited available datasets allows for good generalization to 3D shape and pose predictions on images in the wild. |
| first_indexed | 2024-03-06T20:07:19Z |
| format | Conference item |
| id | oxford-uuid:2955c437-0c93-436a-86f6-d44f8da67a90 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T20:07:19Z |
| publishDate | 2020 |
| publisher | Institute of Electrical and Electronics Engineers |
| record_format | dspace |
| spelling | oxford-uuid:2955c437-0c93-436a-86f6-d44f8da67a902022-03-26T12:18:32Z3D hand shape and pose from images in the wildConference itemhttp://purl.org/coar/resource_type/c_5794uuid:2955c437-0c93-436a-86f6-d44f8da67a90EnglishSymplectic ElementsInstitute of Electrical and Electronics Engineers2020Boukhayma, Ade Bem, RTorr, PHSWe present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters that projects the generated hand into the image domain. We show that using the shape and pose prior knowledge encoded in the hand model within a deep learning framework yields state-of-the-art performance in 3D pose prediction from images on standard benchmarks, and produces geometrically valid and plausible 3D reconstructions. Additionally, we show that training with weak supervision in the form of 2D joint annotations on datasets of images in the wild, in conjunction with full supervision in the form of 3D joint annotations on limited available datasets allows for good generalization to 3D shape and pose predictions on images in the wild. |
| spellingShingle | Boukhayma, A de Bem, R Torr, PHS 3D hand shape and pose from images in the wild |
| title | 3D hand shape and pose from images in the wild |
| title_full | 3D hand shape and pose from images in the wild |
| title_fullStr | 3D hand shape and pose from images in the wild |
| title_full_unstemmed | 3D hand shape and pose from images in the wild |
| title_short | 3D hand shape and pose from images in the wild |
| title_sort | 3d hand shape and pose from images in the wild |
| work_keys_str_mv | AT boukhaymaa 3dhandshapeandposefromimagesinthewild AT debemr 3dhandshapeandposefromimagesinthewild AT torrphs 3dhandshapeandposefromimagesinthewild |