Light Stage Super-Resolution: Continuous High-Frequency Relighting
The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based rel...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Association for Computing Machinery
2025
|
| Online Access: | https://hdl.handle.net/1721.1/158233 |
| _version_ | 1824458234492092416 |
|---|---|
| author | Sun, Tiancheng Xu, Zexiang Zhang, Xiuming Fanello, Sean Rhemann, Christoph Debevec, Paul Tsai, Yun-Ta Barron, Jonathan Ramamoorthi, Ravi |
| author_facet | Sun, Tiancheng Xu, Zexiang Zhang, Xiuming Fanello, Sean Rhemann, Christoph Debevec, Paul Tsai, Yun-Ta Barron, Jonathan Ramamoorthi, Ravi |
| author_sort | Sun, Tiancheng |
| collection | MIT |
| description | The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects. Our super-resolution approach enables more accurate renderings of human subjects under detailed environment maps, or the construction of simpler light stages that contain fewer light sources while still yielding comparable quality renderings as light stages with more densely sampled lights. |
| first_indexed | 2025-02-19T04:22:39Z |
| format | Article |
| id | mit-1721.1/158233 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2025-02-19T04:22:39Z |
| publishDate | 2025 |
| publisher | Association for Computing Machinery |
| record_format | dspace |
| spelling | mit-1721.1/1582332025-02-18T18:18:25Z Light Stage Super-Resolution: Continuous High-Frequency Relighting Sun, Tiancheng Xu, Zexiang Zhang, Xiuming Fanello, Sean Rhemann, Christoph Debevec, Paul Tsai, Yun-Ta Barron, Jonathan Ramamoorthi, Ravi The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects. Our super-resolution approach enables more accurate renderings of human subjects under detailed environment maps, or the construction of simpler light stages that contain fewer light sources while still yielding comparable quality renderings as light stages with more densely sampled lights. 2025-02-18T18:18:23Z 2025-02-18T18:18:23Z 2020-11-26 2025-02-01T08:50:23Z Article http://purl.org/eprint/type/JournalArticle 978-1-4503-8107-9 https://hdl.handle.net/1721.1/158233 Sun, Tiancheng, Xu, Zexiang, Zhang, Xiuming, Fanello, Sean, Rhemann, Christoph et al. 2020. "Light Stage Super-Resolution: Continuous High-Frequency Relighting." ACM Transactions on Graphics, 39 (6). PUBLISHER_POLICY en https://doi.org/10.1145/3414685.3417821 ACM Transactions on Graphics Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. The author(s) application/pdf Association for Computing Machinery Association for Computing Machinery |
| spellingShingle | Sun, Tiancheng Xu, Zexiang Zhang, Xiuming Fanello, Sean Rhemann, Christoph Debevec, Paul Tsai, Yun-Ta Barron, Jonathan Ramamoorthi, Ravi Light Stage Super-Resolution: Continuous High-Frequency Relighting |
| title | Light Stage Super-Resolution: Continuous High-Frequency Relighting |
| title_full | Light Stage Super-Resolution: Continuous High-Frequency Relighting |
| title_fullStr | Light Stage Super-Resolution: Continuous High-Frequency Relighting |
| title_full_unstemmed | Light Stage Super-Resolution: Continuous High-Frequency Relighting |
| title_short | Light Stage Super-Resolution: Continuous High-Frequency Relighting |
| title_sort | light stage super resolution continuous high frequency relighting |
| url | https://hdl.handle.net/1721.1/158233 |
| work_keys_str_mv | AT suntiancheng lightstagesuperresolutioncontinuoushighfrequencyrelighting AT xuzexiang lightstagesuperresolutioncontinuoushighfrequencyrelighting AT zhangxiuming lightstagesuperresolutioncontinuoushighfrequencyrelighting AT fanellosean lightstagesuperresolutioncontinuoushighfrequencyrelighting AT rhemannchristoph lightstagesuperresolutioncontinuoushighfrequencyrelighting AT debevecpaul lightstagesuperresolutioncontinuoushighfrequencyrelighting AT tsaiyunta lightstagesuperresolutioncontinuoushighfrequencyrelighting AT barronjonathan lightstagesuperresolutioncontinuoushighfrequencyrelighting AT ramamoorthiravi lightstagesuperresolutioncontinuoushighfrequencyrelighting |