A Deeper Analysis of Volumetric Relightiable Faces
Abstract Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer US
2023
|
| Online Access: | https://hdl.handle.net/1721.1/152909 |
| _version_ | 1826197202862604288 |
|---|---|
| author | Rao, Pramod Mallikarjun, B. R. Fox, Gereon Weyrich, Tim Bickel, Bernd Pfister, Hanspeter Matusik, Wojciech Zhan, Fangneng Tewari, Ayush Theobalt, Christian Elgharib, Mohamed |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Rao, Pramod Mallikarjun, B. R. Fox, Gereon Weyrich, Tim Bickel, Bernd Pfister, Hanspeter Matusik, Wojciech Zhan, Fangneng Tewari, Ayush Theobalt, Christian Elgharib, Mohamed |
| author_sort | Rao, Pramod |
| collection | MIT |
| description | Abstract
Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handling both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learned in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illumination. This work is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide extensive evaluation and ablative studies of our model and also provide an application, where any face can be relighted using textual input. |
| first_indexed | 2024-09-23T10:44:05Z |
| format | Article |
| id | mit-1721.1/152909 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T10:44:05Z |
| publishDate | 2023 |
| publisher | Springer US |
| record_format | dspace |
| spelling | mit-1721.1/1529092024-01-24T18:52:25Z A Deeper Analysis of Volumetric Relightiable Faces Rao, Pramod Mallikarjun, B. R. Fox, Gereon Weyrich, Tim Bickel, Bernd Pfister, Hanspeter Matusik, Wojciech Zhan, Fangneng Tewari, Ayush Theobalt, Christian Elgharib, Mohamed Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Abstract Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handling both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learned in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illumination. This work is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide extensive evaluation and ablative studies of our model and also provide an application, where any face can be relighted using textual input. 2023-11-06T16:33:58Z 2023-11-06T16:33:58Z 2023-10-31 2023-11-05T04:12:09Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/152909 Rao, Pramod, Mallikarjun, B. R., Fox, Gereon, Weyrich, Tim, Bickel, Bernd et al. 2023. "A Deeper Analysis of Volumetric Relightiable Faces." PUBLISHER_CC en https://doi.org/10.1007/s11263-023-01899-3 Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ The Author(s) application/pdf Springer US Springer US |
| spellingShingle | Rao, Pramod Mallikarjun, B. R. Fox, Gereon Weyrich, Tim Bickel, Bernd Pfister, Hanspeter Matusik, Wojciech Zhan, Fangneng Tewari, Ayush Theobalt, Christian Elgharib, Mohamed A Deeper Analysis of Volumetric Relightiable Faces |
| title | A Deeper Analysis of Volumetric Relightiable Faces |
| title_full | A Deeper Analysis of Volumetric Relightiable Faces |
| title_fullStr | A Deeper Analysis of Volumetric Relightiable Faces |
| title_full_unstemmed | A Deeper Analysis of Volumetric Relightiable Faces |
| title_short | A Deeper Analysis of Volumetric Relightiable Faces |
| title_sort | deeper analysis of volumetric relightiable faces |
| url | https://hdl.handle.net/1721.1/152909 |
| work_keys_str_mv | AT raopramod adeeperanalysisofvolumetricrelightiablefaces AT mallikarjunbr adeeperanalysisofvolumetricrelightiablefaces AT foxgereon adeeperanalysisofvolumetricrelightiablefaces AT weyrichtim adeeperanalysisofvolumetricrelightiablefaces AT bickelbernd adeeperanalysisofvolumetricrelightiablefaces AT pfisterhanspeter adeeperanalysisofvolumetricrelightiablefaces AT matusikwojciech adeeperanalysisofvolumetricrelightiablefaces AT zhanfangneng adeeperanalysisofvolumetricrelightiablefaces AT tewariayush adeeperanalysisofvolumetricrelightiablefaces AT theobaltchristian adeeperanalysisofvolumetricrelightiablefaces AT elgharibmohamed adeeperanalysisofvolumetricrelightiablefaces AT raopramod deeperanalysisofvolumetricrelightiablefaces AT mallikarjunbr deeperanalysisofvolumetricrelightiablefaces AT foxgereon deeperanalysisofvolumetricrelightiablefaces AT weyrichtim deeperanalysisofvolumetricrelightiablefaces AT bickelbernd deeperanalysisofvolumetricrelightiablefaces AT pfisterhanspeter deeperanalysisofvolumetricrelightiablefaces AT matusikwojciech deeperanalysisofvolumetricrelightiablefaces AT zhanfangneng deeperanalysisofvolumetricrelightiablefaces AT tewariayush deeperanalysisofvolumetricrelightiablefaces AT theobaltchristian deeperanalysisofvolumetricrelightiablefaces AT elgharibmohamed deeperanalysisofvolumetricrelightiablefaces |