An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based Landmarks
We propose an efficient and novel architecture for 3D articulated human pose retrieval and reconstruction from 2D landmarks extracted from a 2D synthetic image, an annotated 2D image, an <i>in-the-wild</i> real RGB image or even a hand-drawn sketch. Given 2D joint positions in a single i...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-04-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/7/2415 |
| _version_ | 1827696313123209216 |
|---|---|
| author | Hashim Yasin Björn Krüger |
| author_facet | Hashim Yasin Björn Krüger |
| author_sort | Hashim Yasin |
| collection | DOAJ |
| description | We propose an efficient and novel architecture for 3D articulated human pose retrieval and reconstruction from 2D landmarks extracted from a 2D synthetic image, an annotated 2D image, an <i>in-the-wild</i> real RGB image or even a hand-drawn sketch. Given 2D joint positions in a single image, we devise a data-driven framework to infer the corresponding 3D human pose. To this end, we first normalize 3D human poses from Motion Capture (MoCap) dataset by eliminating translation, orientation, and the skeleton size discrepancies from the poses and then build a <i>knowledge-base</i> by projecting a subset of joints of the normalized 3D poses onto 2D image-planes by fully exploiting a variety of virtual cameras. With this approach, we not only transform 3D pose space to the normalized 2D pose space but also resolve the 2D-3D cross-domain retrieval task efficiently. The proposed architecture searches for poses from a MoCap dataset that are near to a given 2D query pose in a definite feature space made up of specific joint sets. These retrieved poses are then used to construct a weak perspective camera and a final 3D posture under the camera model that minimizes the reconstruction error. To estimate unknown camera parameters, we introduce a nonlinear, two-fold method. We exploit the retrieved similar poses and the viewing directions at which the MoCap dataset was sampled to minimize the projection error. Finally, we evaluate our approach thoroughly on a large number of heterogeneous 2D examples generated synthetically, 2D images with ground-truth, a variety of real <i>in-the-wild</i> internet images, and a proof of concept using 2D hand-drawn sketches of human poses. We conduct a pool of experiments to perform a quantitative study on PARSE dataset. We also show that the proposed system yields competitive, convincing results in comparison to other state-of-the-art methods. |
| first_indexed | 2024-03-10T12:43:57Z |
| format | Article |
| id | doaj.art-ddc55e3847c140a280931126b94a3858 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T12:43:57Z |
| publishDate | 2021-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-ddc55e3847c140a280931126b94a38582023-11-21T13:41:25ZengMDPI AGSensors1424-82202021-04-01217241510.3390/s21072415An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based LandmarksHashim Yasin0Björn Krüger1School of Computing, National University of Computer and Emerging Sciences, Islamabad 44000, PakistanGokhale Method Institute, Stanford, CA 94305, USAWe propose an efficient and novel architecture for 3D articulated human pose retrieval and reconstruction from 2D landmarks extracted from a 2D synthetic image, an annotated 2D image, an <i>in-the-wild</i> real RGB image or even a hand-drawn sketch. Given 2D joint positions in a single image, we devise a data-driven framework to infer the corresponding 3D human pose. To this end, we first normalize 3D human poses from Motion Capture (MoCap) dataset by eliminating translation, orientation, and the skeleton size discrepancies from the poses and then build a <i>knowledge-base</i> by projecting a subset of joints of the normalized 3D poses onto 2D image-planes by fully exploiting a variety of virtual cameras. With this approach, we not only transform 3D pose space to the normalized 2D pose space but also resolve the 2D-3D cross-domain retrieval task efficiently. The proposed architecture searches for poses from a MoCap dataset that are near to a given 2D query pose in a definite feature space made up of specific joint sets. These retrieved poses are then used to construct a weak perspective camera and a final 3D posture under the camera model that minimizes the reconstruction error. To estimate unknown camera parameters, we introduce a nonlinear, two-fold method. We exploit the retrieved similar poses and the viewing directions at which the MoCap dataset was sampled to minimize the projection error. Finally, we evaluate our approach thoroughly on a large number of heterogeneous 2D examples generated synthetically, 2D images with ground-truth, a variety of real <i>in-the-wild</i> internet images, and a proof of concept using 2D hand-drawn sketches of human poses. We conduct a pool of experiments to perform a quantitative study on PARSE dataset. We also show that the proposed system yields competitive, convincing results in comparison to other state-of-the-art methods.https://www.mdpi.com/1424-8220/21/7/2415motion capturefeature sets3D human pose retrievalknowledge-base3D articulated pose estimationoptimization |
| spellingShingle | Hashim Yasin Björn Krüger An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based Landmarks Sensors motion capture feature sets 3D human pose retrieval knowledge-base 3D articulated pose estimation optimization |
| title | An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based Landmarks |
| title_full | An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based Landmarks |
| title_fullStr | An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based Landmarks |
| title_full_unstemmed | An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based Landmarks |
| title_short | An Efficient 3D Human Pose Retrieval and Reconstruction from 2D Image-Based Landmarks |
| title_sort | efficient 3d human pose retrieval and reconstruction from 2d image based landmarks |
| topic | motion capture feature sets 3D human pose retrieval knowledge-base 3D articulated pose estimation optimization |
| url | https://www.mdpi.com/1424-8220/21/7/2415 |
| work_keys_str_mv | AT hashimyasin anefficient3dhumanposeretrievalandreconstructionfrom2dimagebasedlandmarks AT bjornkruger anefficient3dhumanposeretrievalandreconstructionfrom2dimagebasedlandmarks AT hashimyasin efficient3dhumanposeretrievalandreconstructionfrom2dimagebasedlandmarks AT bjornkruger efficient3dhumanposeretrievalandreconstructionfrom2dimagebasedlandmarks |