Facial optical flow estimation via neural non-rigid registration
Abstract Optical flow estimation in human facial video, which provides 2D correspondences between adjacent frames, is a fundamental pre-processing step for many applications, like facial expression capture and recognition. However, it is quite challenging as human facial images contain large areas o...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2022-10-01
|
| Series: | Computational Visual Media |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s41095-021-0267-z |
| _version_ | 1811336237822246912 |
|---|---|
| author | Zhuang Peng Boyi Jiang Haofei Xu Wanquan Feng Juyong Zhang |
| author_facet | Zhuang Peng Boyi Jiang Haofei Xu Wanquan Feng Juyong Zhang |
| author_sort | Zhuang Peng |
| collection | DOAJ |
| description | Abstract Optical flow estimation in human facial video, which provides 2D correspondences between adjacent frames, is a fundamental pre-processing step for many applications, like facial expression capture and recognition. However, it is quite challenging as human facial images contain large areas of similar textures, rich expressions, and large rotations. These characteristics also result in the scarcity of large, annotated real-world datasets. We propose a robust and accurate method to learn facial optical flow in a self-supervised manner. Specifically, we utilize various shape priors, including face depth, landmarks, and parsing, to guide the self-supervised learning task via a differentiable nonrigid registration framework. Extensive experiments demonstrate that our method achieves remarkable improvements for facial optical flow estimation in the presence of significant expressions and large rotations. |
| first_indexed | 2024-04-13T17:36:32Z |
| format | Article |
| id | doaj.art-fc4131d9163e440f983c219e7ba8979f |
| institution | Directory Open Access Journal |
| issn | 2096-0433 2096-0662 |
| language | English |
| last_indexed | 2024-04-13T17:36:32Z |
| publishDate | 2022-10-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Computational Visual Media |
| spelling | doaj.art-fc4131d9163e440f983c219e7ba8979f2022-12-22T02:37:21ZengSpringerOpenComputational Visual Media2096-04332096-06622022-10-019110912210.1007/s41095-021-0267-zFacial optical flow estimation via neural non-rigid registrationZhuang Peng0Boyi Jiang1Haofei Xu2Wanquan Feng3Juyong Zhang4School of Mathematical Sciences, University of Science and Technology of ChinaSchool of Mathematical Sciences, University of Science and Technology of ChinaSchool of Mathematical Sciences, University of Science and Technology of ChinaSchool of Mathematical Sciences, University of Science and Technology of ChinaSchool of Mathematical Sciences, University of Science and Technology of ChinaAbstract Optical flow estimation in human facial video, which provides 2D correspondences between adjacent frames, is a fundamental pre-processing step for many applications, like facial expression capture and recognition. However, it is quite challenging as human facial images contain large areas of similar textures, rich expressions, and large rotations. These characteristics also result in the scarcity of large, annotated real-world datasets. We propose a robust and accurate method to learn facial optical flow in a self-supervised manner. Specifically, we utilize various shape priors, including face depth, landmarks, and parsing, to guide the self-supervised learning task via a differentiable nonrigid registration framework. Extensive experiments demonstrate that our method achieves remarkable improvements for facial optical flow estimation in the presence of significant expressions and large rotations.https://doi.org/10.1007/s41095-021-0267-zhuman faceoptical flowself-supervisednon-rigid registrationneural networksfacial priors |
| spellingShingle | Zhuang Peng Boyi Jiang Haofei Xu Wanquan Feng Juyong Zhang Facial optical flow estimation via neural non-rigid registration Computational Visual Media human face optical flow self-supervised non-rigid registration neural networks facial priors |
| title | Facial optical flow estimation via neural non-rigid registration |
| title_full | Facial optical flow estimation via neural non-rigid registration |
| title_fullStr | Facial optical flow estimation via neural non-rigid registration |
| title_full_unstemmed | Facial optical flow estimation via neural non-rigid registration |
| title_short | Facial optical flow estimation via neural non-rigid registration |
| title_sort | facial optical flow estimation via neural non rigid registration |
| topic | human face optical flow self-supervised non-rigid registration neural networks facial priors |
| url | https://doi.org/10.1007/s41095-021-0267-z |
| work_keys_str_mv | AT zhuangpeng facialopticalflowestimationvianeuralnonrigidregistration AT boyijiang facialopticalflowestimationvianeuralnonrigidregistration AT haofeixu facialopticalflowestimationvianeuralnonrigidregistration AT wanquanfeng facialopticalflowestimationvianeuralnonrigidregistration AT juyongzhang facialopticalflowestimationvianeuralnonrigidregistration |