ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field
Neural Radiance Field (NeRF) is a popular method in representing 3D scenes by optimising a continuous volumetric scene function. Its large success which lies in applying volumetric rendering (VR) is also its Achilles' heel in producing view-dependent effects. As a consequence, glossy and transp...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Conference Paper |
| Language: | English |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/172666 |
| _version_ | 1811677505870888960 |
|---|---|
| author | Tang, Zhe Jun Cham, Tat-Jen Zhao, Haiyu |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Tang, Zhe Jun Cham, Tat-Jen Zhao, Haiyu |
| author_sort | Tang, Zhe Jun |
| collection | NTU |
| description | Neural Radiance Field (NeRF) is a popular method in representing 3D scenes by optimising a continuous volumetric scene function. Its large success which lies in applying volumetric rendering (VR) is also its Achilles' heel in producing view-dependent effects. As a consequence, glossy and transparent surfaces often appear murky. A remedy to reduce these artefacts is to constrain this VR equation by excluding volumes with back-facing normal. While this approach has some success in rendering glossy surfaces, translucent objects are still poorly represented. In this paper, we present an alternative to the physics-based VR approach by introducing a self-attention-based framework on volumes along a ray. In addition, inspired by modern game engines which utilise Light Probes to store local lighting passing through the scene, we incorporate Learnable Embeddings to capture view dependent effects within the scene. Our method, which we call ABLE-NeRF, significantly reduces ‘blurry’ glossy surfaces in rendering and produces realistic translucent surfaces which lack in prior art. In the Blender dataset, ABLE-NeRF achieves SOTA results and surpasses Ref-NeRF in all 3 image quality metrics PSNR, SSIM, LPIPS. |
| first_indexed | 2024-10-01T02:38:26Z |
| format | Conference Paper |
| id | ntu-10356/172666 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T02:38:26Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | ntu-10356/1726662023-12-19T06:15:00Z ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field Tang, Zhe Jun Cham, Tat-Jen Zhao, Haiyu School of Computer Science and Engineering 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) S-Lab for Advanced Intelligence Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Visualization Three-Dimensional Displays Neural Radiance Field (NeRF) is a popular method in representing 3D scenes by optimising a continuous volumetric scene function. Its large success which lies in applying volumetric rendering (VR) is also its Achilles' heel in producing view-dependent effects. As a consequence, glossy and transparent surfaces often appear murky. A remedy to reduce these artefacts is to constrain this VR equation by excluding volumes with back-facing normal. While this approach has some success in rendering glossy surfaces, translucent objects are still poorly represented. In this paper, we present an alternative to the physics-based VR approach by introducing a self-attention-based framework on volumes along a ray. In addition, inspired by modern game engines which utilise Light Probes to store local lighting passing through the scene, we incorporate Learnable Embeddings to capture view dependent effects within the scene. Our method, which we call ABLE-NeRF, significantly reduces ‘blurry’ glossy surfaces in rendering and produces realistic translucent surfaces which lack in prior art. In the Blender dataset, ABLE-NeRF achieves SOTA results and surpasses Ref-NeRF in all 3 image quality metrics PSNR, SSIM, LPIPS. This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). 2023-12-19T06:15:00Z 2023-12-19T06:15:00Z 2023 Conference Paper Tang, Z. J., Cham, T. & Zhao, H. (2023). ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16559-16568. https://dx.doi.org/10.1109/CVPR52729.2023.01589 979-8-3503-0129-8 https://hdl.handle.net/10356/172666 10.1109/CVPR52729.2023.01589 16559 16568 en IAF-ICP © 2023 IEEE. All rights reserved. |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Visualization Three-Dimensional Displays Tang, Zhe Jun Cham, Tat-Jen Zhao, Haiyu ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field |
| title | ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field |
| title_full | ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field |
| title_fullStr | ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field |
| title_full_unstemmed | ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field |
| title_short | ABLE-NeRF: attention-based rendering with learnable embeddings for neural radiance field |
| title_sort | able nerf attention based rendering with learnable embeddings for neural radiance field |
| topic | Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Visualization Three-Dimensional Displays |
| url | https://hdl.handle.net/10356/172666 |
| work_keys_str_mv | AT tangzhejun ablenerfattentionbasedrenderingwithlearnableembeddingsforneuralradiancefield AT chamtatjen ablenerfattentionbasedrenderingwithlearnableembeddingsforneuralradiancefield AT zhaohaiyu ablenerfattentionbasedrenderingwithlearnableembeddingsforneuralradiancefield |