3D hand pose and shape estimation from monocular RGB via efficient 2D cues
Abstract Estimating 3D hand shape from a single-view RGB image is important for many applications. However, the diversity of hand shapes and postures, depth ambiguity, and occlusion may result in pose errors and noisy hand meshes. Making full use of 2D cues such as 2D pose can effectively improve th...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-11-01
|
| Series: | Computational Visual Media |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s41095-023-0346-4 |
| _version_ | 1827388295780237312 |
|---|---|
| author | Fenghao Zhang Lin Zhao Shengling Li Wanjuan Su Liman Liu Wenbing Tao |
| author_facet | Fenghao Zhang Lin Zhao Shengling Li Wanjuan Su Liman Liu Wenbing Tao |
| author_sort | Fenghao Zhang |
| collection | DOAJ |
| description | Abstract Estimating 3D hand shape from a single-view RGB image is important for many applications. However, the diversity of hand shapes and postures, depth ambiguity, and occlusion may result in pose errors and noisy hand meshes. Making full use of 2D cues such as 2D pose can effectively improve the quality of 3D human hand shape estimation. In this paper, we use 2D joint heatmaps to obtain spatial details for robust pose estimation. We also introduce a depth-independent 2D mesh to avoid depth ambiguity in mesh regression for efficient hand-image alignment. Our method has four cascaded stages: 2D cue extraction, pose feature encoding, initial reconstruction, and reconstruction refinement. Specifically, we first encode the image to determine semantic features during 2D cue extraction; this is also used to predict hand joints and for segmentation. Then, during the pose feature encoding stage, we use a hand joints encoder to learn spatial information from the joint heatmaps. Next, a coarse 3D hand mesh and 2D mesh are obtained in the initial reconstruction step; a mesh squeeze-and-excitation block is used to fuse different hand features to enhance perception of 3D hand structures. Finally, a global mesh refinement stage learns non-local relations between vertices of the hand mesh from the predicted 2D mesh, to predict an offset hand mesh to fine-tune the reconstruction results. Quantitative and qualitative results on the FreiHAND benchmark dataset demonstrate that our approach achieves state-of-the-art performance. |
| first_indexed | 2024-03-08T16:15:40Z |
| format | Article |
| id | doaj.art-d1bcd8a212644d7c96762a6f98d1555c |
| institution | Directory Open Access Journal |
| issn | 2096-0433 2096-0662 |
| language | English |
| last_indexed | 2024-03-08T16:15:40Z |
| publishDate | 2023-11-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Computational Visual Media |
| spelling | doaj.art-d1bcd8a212644d7c96762a6f98d1555c2024-01-07T12:38:59ZengSpringerOpenComputational Visual Media2096-04332096-06622023-11-01101799610.1007/s41095-023-0346-43D hand pose and shape estimation from monocular RGB via efficient 2D cuesFenghao Zhang0Lin Zhao1Shengling Li2Wanjuan Su3Liman Liu4Wenbing Tao5Hubei Key Laboratory of Medical Information Analysis and Tumor Diagnosis & Treatment, School of Biomedical Engineering, South Central Minzu UniversityNational Key Laboratory of Science and Technology of Multi-spectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and TechnologyHubei Key Laboratory of Medical Information Analysis and Tumor Diagnosis & Treatment, School of Biomedical Engineering, South Central Minzu UniversityNational Key Laboratory of Science and Technology of Multi-spectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and TechnologyHubei Key Laboratory of Medical Information Analysis and Tumor Diagnosis & Treatment, School of Biomedical Engineering, South Central Minzu UniversityNational Key Laboratory of Science and Technology of Multi-spectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and TechnologyAbstract Estimating 3D hand shape from a single-view RGB image is important for many applications. However, the diversity of hand shapes and postures, depth ambiguity, and occlusion may result in pose errors and noisy hand meshes. Making full use of 2D cues such as 2D pose can effectively improve the quality of 3D human hand shape estimation. In this paper, we use 2D joint heatmaps to obtain spatial details for robust pose estimation. We also introduce a depth-independent 2D mesh to avoid depth ambiguity in mesh regression for efficient hand-image alignment. Our method has four cascaded stages: 2D cue extraction, pose feature encoding, initial reconstruction, and reconstruction refinement. Specifically, we first encode the image to determine semantic features during 2D cue extraction; this is also used to predict hand joints and for segmentation. Then, during the pose feature encoding stage, we use a hand joints encoder to learn spatial information from the joint heatmaps. Next, a coarse 3D hand mesh and 2D mesh are obtained in the initial reconstruction step; a mesh squeeze-and-excitation block is used to fuse different hand features to enhance perception of 3D hand structures. Finally, a global mesh refinement stage learns non-local relations between vertices of the hand mesh from the predicted 2D mesh, to predict an offset hand mesh to fine-tune the reconstruction results. Quantitative and qualitative results on the FreiHAND benchmark dataset demonstrate that our approach achieves state-of-the-art performance.https://doi.org/10.1007/s41095-023-0346-4hand3D reconstructiondeep learningimage features3D mesh |
| spellingShingle | Fenghao Zhang Lin Zhao Shengling Li Wanjuan Su Liman Liu Wenbing Tao 3D hand pose and shape estimation from monocular RGB via efficient 2D cues Computational Visual Media hand 3D reconstruction deep learning image features 3D mesh |
| title | 3D hand pose and shape estimation from monocular RGB via efficient 2D cues |
| title_full | 3D hand pose and shape estimation from monocular RGB via efficient 2D cues |
| title_fullStr | 3D hand pose and shape estimation from monocular RGB via efficient 2D cues |
| title_full_unstemmed | 3D hand pose and shape estimation from monocular RGB via efficient 2D cues |
| title_short | 3D hand pose and shape estimation from monocular RGB via efficient 2D cues |
| title_sort | 3d hand pose and shape estimation from monocular rgb via efficient 2d cues |
| topic | hand 3D reconstruction deep learning image features 3D mesh |
| url | https://doi.org/10.1007/s41095-023-0346-4 |
| work_keys_str_mv | AT fenghaozhang 3dhandposeandshapeestimationfrommonocularrgbviaefficient2dcues AT linzhao 3dhandposeandshapeestimationfrommonocularrgbviaefficient2dcues AT shenglingli 3dhandposeandshapeestimationfrommonocularrgbviaefficient2dcues AT wanjuansu 3dhandposeandshapeestimationfrommonocularrgbviaefficient2dcues AT limanliu 3dhandposeandshapeestimationfrommonocularrgbviaefficient2dcues AT wenbingtao 3dhandposeandshapeestimationfrommonocularrgbviaefficient2dcues |