Recovering facial reflectance and geometry from multi-view images
While the problem of estimating shapes and diffuse reflectances of human faces from images has been extensively studied, there is relatively less work done on recovering the specular albedo. This paper presents a lightweight solution for inferring photorealistic facial reflectance and geometry. Our...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/172643 |
| _version_ | 1826112512736624640 |
|---|---|
| author | Song, Guoxian Zheng, Jianmin Cai, Jianfei Cham, Tat-Jen |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Song, Guoxian Zheng, Jianmin Cai, Jianfei Cham, Tat-Jen |
| author_sort | Song, Guoxian |
| collection | NTU |
| description | While the problem of estimating shapes and diffuse reflectances of human faces from images has been extensively studied, there is relatively less work done on recovering the specular albedo. This paper presents a lightweight solution for inferring photorealistic facial reflectance and geometry. Our system processes video streams from two views of a subject, and outputs two reflectance maps for diffuse and specular albedos, as well as a vector map of surface normals. A model-based optimization approach is used, consisting of the three stages of multi-view face model fitting, facial reflectance inference and facial geometry refinement. Our approach is based on a novel formulation built upon the 3D morphable model (3DMM) for representing 3D textured faces in conjunction with the Blinn-Phong reflection model. It has the advantage of requiring only a simple setup with two video streams, and is able to exploit the interaction between the diffuse and specular reflections across multiple views as well as time frames. As a result, the method is able to reliably recover high-fidelity facial reflectance and geometry, which facilitates various applications such as generating photorealistic facial images under new viewpoints or illumination conditions. |
| first_indexed | 2024-10-01T03:08:16Z |
| format | Journal Article |
| id | ntu-10356/172643 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T03:08:16Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | ntu-10356/1726432023-12-19T01:23:12Z Recovering facial reflectance and geometry from multi-view images Song, Guoxian Zheng, Jianmin Cai, Jianfei Cham, Tat-Jen School of Computer Science and Engineering Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision 3D Facial Reconstruction Specular Estimation While the problem of estimating shapes and diffuse reflectances of human faces from images has been extensively studied, there is relatively less work done on recovering the specular albedo. This paper presents a lightweight solution for inferring photorealistic facial reflectance and geometry. Our system processes video streams from two views of a subject, and outputs two reflectance maps for diffuse and specular albedos, as well as a vector map of surface normals. A model-based optimization approach is used, consisting of the three stages of multi-view face model fitting, facial reflectance inference and facial geometry refinement. Our approach is based on a novel formulation built upon the 3D morphable model (3DMM) for representing 3D textured faces in conjunction with the Blinn-Phong reflection model. It has the advantage of requiring only a simple setup with two video streams, and is able to exploit the interaction between the diffuse and specular reflections across multiple views as well as time frames. As a result, the method is able to reliably recover high-fidelity facial reflectance and geometry, which facilitates various applications such as generating photorealistic facial images under new viewpoints or illumination conditions. This research is supported by Singtel Cognitive and Artificial Intelligence Lab for Enterprises at NTU. 2023-12-19T01:23:12Z 2023-12-19T01:23:12Z 2020 Journal Article Song, G., Zheng, J., Cai, J. & Cham, T. (2020). Recovering facial reflectance and geometry from multi-view images. Image and Vision Computing, 96, 103897-. https://dx.doi.org/10.1016/j.imavis.2020.103897 0262-8856 https://hdl.handle.net/10356/172643 10.1016/j.imavis.2020.103897 2-s2.0-85081129546 96 103897 en Image and Vision Computing © 2020 Elsevier B.V. All rights reserved. |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision 3D Facial Reconstruction Specular Estimation Song, Guoxian Zheng, Jianmin Cai, Jianfei Cham, Tat-Jen Recovering facial reflectance and geometry from multi-view images |
| title | Recovering facial reflectance and geometry from multi-view images |
| title_full | Recovering facial reflectance and geometry from multi-view images |
| title_fullStr | Recovering facial reflectance and geometry from multi-view images |
| title_full_unstemmed | Recovering facial reflectance and geometry from multi-view images |
| title_short | Recovering facial reflectance and geometry from multi-view images |
| title_sort | recovering facial reflectance and geometry from multi view images |
| topic | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision 3D Facial Reconstruction Specular Estimation |
| url | https://hdl.handle.net/10356/172643 |
| work_keys_str_mv | AT songguoxian recoveringfacialreflectanceandgeometryfrommultiviewimages AT zhengjianmin recoveringfacialreflectanceandgeometryfrommultiviewimages AT caijianfei recoveringfacialreflectanceandgeometryfrommultiviewimages AT chamtatjen recoveringfacialreflectanceandgeometryfrommultiviewimages |