Visibility constrained generative model for depth-based 3D facial pose tracking
In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations. Specifically, we introduce a statistical 3D morphable model that fl...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2020
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/138265 |
| _version_ | 1826124585598189568 |
|---|---|
| author | Sheng, Lu Cai, Jianfei Cham, Tat-Jen Pavlovic, Vladimir Ngan, King Ngi |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Sheng, Lu Cai, Jianfei Cham, Tat-Jen Pavlovic, Vladimir Ngan, King Ngi |
| author_sort | Sheng, Lu |
| collection | NTU |
| description | In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations. Specifically, we introduce a statistical 3D morphable model that flexibly describes the distribution of points on the surface of the face model, with an efficient switchable online adaptation that gradually captures the identity of the tracked subject and rapidly constructs a suitable face model when the subject changes. Moreover, unlike prior art that employed ICP-based facial pose estimation, to improve robustness to occlusions, we propose a ray visibility constraint that regularizes the pose based on the face model's visibility with respect to the input point cloud. Ablation studies and experimental results on Biwi and ICT-3DHP datasets demonstrate that the proposed framework is effective and outperforms completing state-of-the-art depth-based methods. |
| first_indexed | 2024-10-01T06:23:06Z |
| format | Journal Article |
| id | ntu-10356/138265 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:23:06Z |
| publishDate | 2020 |
| record_format | dspace |
| spelling | ntu-10356/1382652020-04-30T01:35:55Z Visibility constrained generative model for depth-based 3D facial pose tracking Sheng, Lu Cai, Jianfei Cham, Tat-Jen Pavlovic, Vladimir Ngan, King Ngi School of Computer Science and Engineering Institute for Media Innovation (IMI) Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision 3D Facial Pose Tracking Generative Model In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations. Specifically, we introduce a statistical 3D morphable model that flexibly describes the distribution of points on the surface of the face model, with an efficient switchable online adaptation that gradually captures the identity of the tracked subject and rapidly constructs a suitable face model when the subject changes. Moreover, unlike prior art that employed ICP-based facial pose estimation, to improve robustness to occlusions, we propose a ray visibility constraint that regularizes the pose based on the face model's visibility with respect to the input point cloud. Ablation studies and experimental results on Biwi and ICT-3DHP datasets demonstrate that the proposed framework is effective and outperforms completing state-of-the-art depth-based methods. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore) 2020-04-30T01:35:54Z 2020-04-30T01:35:54Z 2019 Journal Article Sheng, L., Cai, J., Cham, T.-J., Pavlovic, V., & Ngan, K. N. (2019). Visibility constrained generative model for depth-based 3D facial pose tracking. IEEE transactions on pattern analysis and machine intelligence, 41(8), 1994-2007. doi:10.1109/TPAMI.2018.2877675 0162-8828 https://hdl.handle.net/10356/138265 10.1109/TPAMI.2018.2877675 30369437 2-s2.0-85055681100 8 41 1994 2007 en IEEE transactions on pattern analysis and machine intelligence © 2018 IEEE. All rights reserved. |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision 3D Facial Pose Tracking Generative Model Sheng, Lu Cai, Jianfei Cham, Tat-Jen Pavlovic, Vladimir Ngan, King Ngi Visibility constrained generative model for depth-based 3D facial pose tracking |
| title | Visibility constrained generative model for depth-based 3D facial pose tracking |
| title_full | Visibility constrained generative model for depth-based 3D facial pose tracking |
| title_fullStr | Visibility constrained generative model for depth-based 3D facial pose tracking |
| title_full_unstemmed | Visibility constrained generative model for depth-based 3D facial pose tracking |
| title_short | Visibility constrained generative model for depth-based 3D facial pose tracking |
| title_sort | visibility constrained generative model for depth based 3d facial pose tracking |
| topic | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision 3D Facial Pose Tracking Generative Model |
| url | https://hdl.handle.net/10356/138265 |
| work_keys_str_mv | AT shenglu visibilityconstrainedgenerativemodelfordepthbased3dfacialposetracking AT caijianfei visibilityconstrainedgenerativemodelfordepthbased3dfacialposetracking AT chamtatjen visibilityconstrainedgenerativemodelfordepthbased3dfacialposetracking AT pavlovicvladimir visibilityconstrainedgenerativemodelfordepthbased3dfacialposetracking AT ngankingngi visibilityconstrainedgenerativemodelfordepthbased3dfacialposetracking |