FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces
3D rendering of dynamic face captures is a challenging problem, and it demands improvements on several fronts---photorealism, efficiency, compatibility, and configurability. We present a novel representation that enables high-quality volumetric rendering of an actor's dynamic facial performance...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Association for Computing Machinery
2024
|
| Online Access: | https://hdl.handle.net/1721.1/155198 |
| _version_ | 1824458040446812160 |
|---|---|
| author | Medin, Safa C. Li, Gengyan Du, Ruofei Garbin, Stephan Davidson, Philip Wornell, Gregory W. Beeler, Thabo Meka, Abhimitra |
| author2 | Massachusetts Institute of Technology. Signals, Information and Algorithms Laboratory |
| author_facet | Massachusetts Institute of Technology. Signals, Information and Algorithms Laboratory Medin, Safa C. Li, Gengyan Du, Ruofei Garbin, Stephan Davidson, Philip Wornell, Gregory W. Beeler, Thabo Meka, Abhimitra |
| author_sort | Medin, Safa C. |
| collection | MIT |
| description | 3D rendering of dynamic face captures is a challenging problem, and it demands improvements on several fronts---photorealism, efficiency, compatibility, and configurability. We present a novel representation that enables high-quality volumetric rendering of an actor's dynamic facial performances with minimal compute and memory footprint. It runs natively on commodity graphics soft- and hardware, and allows for a graceful trade-off between quality and efficiency. Our method utilizes recent advances in neural rendering, particularly learning discrete radiance manifolds to sparsely sample the scene to model volumetric effects. We achieve efficient modeling by learning a single set of manifolds for the entire dynamic sequence, while implicitly modeling appearance changes as temporal canonical texture. We export a single layered mesh and view-independent RGBA texture video that is compatible with legacy graphics renderers without additional ML integration. We demonstrate our method by rendering dynamic face captures of real actors in a game engine, at comparable photorealism to state-of-the-art neural rendering techniques at previously unseen frame rates. |
| first_indexed | 2024-09-23T10:45:03Z |
| format | Article |
| id | mit-1721.1/155198 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2025-02-19T04:19:34Z |
| publishDate | 2024 |
| publisher | Association for Computing Machinery |
| record_format | dspace |
| spelling | mit-1721.1/1551982024-12-21T06:05:02Z FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces Medin, Safa C. Li, Gengyan Du, Ruofei Garbin, Stephan Davidson, Philip Wornell, Gregory W. Beeler, Thabo Meka, Abhimitra Massachusetts Institute of Technology. Signals, Information and Algorithms Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 3D rendering of dynamic face captures is a challenging problem, and it demands improvements on several fronts---photorealism, efficiency, compatibility, and configurability. We present a novel representation that enables high-quality volumetric rendering of an actor's dynamic facial performances with minimal compute and memory footprint. It runs natively on commodity graphics soft- and hardware, and allows for a graceful trade-off between quality and efficiency. Our method utilizes recent advances in neural rendering, particularly learning discrete radiance manifolds to sparsely sample the scene to model volumetric effects. We achieve efficient modeling by learning a single set of manifolds for the entire dynamic sequence, while implicitly modeling appearance changes as temporal canonical texture. We export a single layered mesh and view-independent RGBA texture video that is compatible with legacy graphics renderers without additional ML integration. We demonstrate our method by rendering dynamic face captures of real actors in a game engine, at comparable photorealism to state-of-the-art neural rendering techniques at previously unseen frame rates. 2024-06-05T16:23:00Z 2024-06-05T16:23:00Z 2024-05-11 2024-06-01T07:56:46Z Article http://purl.org/eprint/type/JournalArticle 2577-6193 https://hdl.handle.net/1721.1/155198 Medin, Safa C., Li, Gengyan, Du, Ruofei, Garbin, Stephan, Davidson, Philip et al. 2024. "FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces." Proceedings of the ACM on Computer Graphics and Interactive Techniques, 7 (1). PUBLISHER_CC en 10.1145/3651304 Proceedings of the ACM on Computer Graphics and Interactive Techniques Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ The author(s) application/pdf Association for Computing Machinery Association for Computing Machinery |
| spellingShingle | Medin, Safa C. Li, Gengyan Du, Ruofei Garbin, Stephan Davidson, Philip Wornell, Gregory W. Beeler, Thabo Meka, Abhimitra FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces |
| title | FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces |
| title_full | FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces |
| title_fullStr | FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces |
| title_full_unstemmed | FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces |
| title_short | FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces |
| title_sort | facefolds meshed radiance manifolds for efficient volumetric rendering of dynamic faces |
| url | https://hdl.handle.net/1721.1/155198 |
| work_keys_str_mv | AT medinsafac facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces AT ligengyan facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces AT duruofei facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces AT garbinstephan facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces AT davidsonphilip facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces AT wornellgregoryw facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces AT beelerthabo facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces AT mekaabhimitra facefoldsmeshedradiancemanifoldsforefficientvolumetricrenderingofdynamicfaces |