Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering
Inverse rendering, the process of inferring scene properties from images, is a challenging inverse problem. The task is ill-posed, as many different scene configurations can give rise to the same image. Most existing solutions incorporate priors into the inverse-rendering pipeline to encourage plaus...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
ACM
2024
|
| Online Access: | https://hdl.handle.net/1721.1/153281 |
| _version_ | 1826202770484494336 |
|---|---|
| author | Lyu, Linjie Tewari, Ayush Habermann, Marc Saito, Shunsuke Zollh?fer, Michael Leimk?hler, Thomas Theobalt, Christian |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Lyu, Linjie Tewari, Ayush Habermann, Marc Saito, Shunsuke Zollh?fer, Michael Leimk?hler, Thomas Theobalt, Christian |
| author_sort | Lyu, Linjie |
| collection | MIT |
| description | Inverse rendering, the process of inferring scene properties from images, is a challenging inverse problem. The task is ill-posed, as many different scene configurations can give rise to the same image. Most existing solutions incorporate priors into the inverse-rendering pipeline to encourage plausible solutions, but they do not consider the inherent ambiguities and the multi-modal distribution of possible decompositions. In this work, we propose a novel scheme that integrates a denoising diffusion probabilistic model pre-trained on natural illumination maps into an optimization framework involving a differentiable path tracer. The proposed method allows sampling from combinations of illumination and spatially-varying surface materials that are, both, natural and explain the image observations. We further conduct an extensive comparative study of different priors on illumination used in previous work on inverse rendering. Our method excels in recovering materials and producing highly realistic and diverse environment map samples that faithfully explain the illumination of the input images. |
| first_indexed | 2024-09-23T12:18:15Z |
| format | Article |
| id | mit-1721.1/153281 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T12:18:15Z |
| publishDate | 2024 |
| publisher | ACM |
| record_format | dspace |
| spelling | mit-1721.1/1532812024-07-11T19:18:17Z Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering Lyu, Linjie Tewari, Ayush Habermann, Marc Saito, Shunsuke Zollh?fer, Michael Leimk?hler, Thomas Theobalt, Christian Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Inverse rendering, the process of inferring scene properties from images, is a challenging inverse problem. The task is ill-posed, as many different scene configurations can give rise to the same image. Most existing solutions incorporate priors into the inverse-rendering pipeline to encourage plausible solutions, but they do not consider the inherent ambiguities and the multi-modal distribution of possible decompositions. In this work, we propose a novel scheme that integrates a denoising diffusion probabilistic model pre-trained on natural illumination maps into an optimization framework involving a differentiable path tracer. The proposed method allows sampling from combinations of illumination and spatially-varying surface materials that are, both, natural and explain the image observations. We further conduct an extensive comparative study of different priors on illumination used in previous work on inverse rendering. Our method excels in recovering materials and producing highly realistic and diverse environment map samples that faithfully explain the illumination of the input images. 2024-01-04T16:55:11Z 2024-01-04T16:55:11Z 2023-12-04 2024-01-01T08:49:46Z Article http://purl.org/eprint/type/JournalArticle 0730-0301 https://hdl.handle.net/1721.1/153281 Lyu, Linjie, Tewari, Ayush, Habermann, Marc, Saito, Shunsuke, Zollh?fer, Michael et al. 2023. "Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering." ACM Transactions on Graphics, 42 (6). PUBLISHER_POLICY en https://doi.org/10.1145/3618357 ACM Transactions on Graphics Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. The author(s) application/pdf ACM Association for Computing Machinery |
| spellingShingle | Lyu, Linjie Tewari, Ayush Habermann, Marc Saito, Shunsuke Zollh?fer, Michael Leimk?hler, Thomas Theobalt, Christian Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering |
| title | Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering |
| title_full | Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering |
| title_fullStr | Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering |
| title_full_unstemmed | Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering |
| title_short | Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering |
| title_sort | diffusion posterior illumination for ambiguity aware inverse rendering |
| url | https://hdl.handle.net/1721.1/153281 |
| work_keys_str_mv | AT lyulinjie diffusionposteriorilluminationforambiguityawareinverserendering AT tewariayush diffusionposteriorilluminationforambiguityawareinverserendering AT habermannmarc diffusionposteriorilluminationforambiguityawareinverserendering AT saitoshunsuke diffusionposteriorilluminationforambiguityawareinverserendering AT zollhfermichael diffusionposteriorilluminationforambiguityawareinverserendering AT leimkhlerthomas diffusionposteriorilluminationforambiguityawareinverserendering AT theobaltchristian diffusionposteriorilluminationforambiguityawareinverserendering |