Depth-Based Dynamic Sampling of Neural Radiation Fields
Although the NeRF approach can achieve outstanding view synthesis, it is limited in practical use because it requires many views (hundreds) for training. With only a few input views, the Depth-DYN NeRF that we propose can accurately match the shape. First, we adopted the ip_basic depth-completion me...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/4/1053 |
| _version_ | 1797621328615833600 |
|---|---|
| author | Jie Wang Jiangjian Xiao Xiaolu Zhang Xiaolin Xu Tianxing Jin Zhijia Jin |
| author_facet | Jie Wang Jiangjian Xiao Xiaolu Zhang Xiaolin Xu Tianxing Jin Zhijia Jin |
| author_sort | Jie Wang |
| collection | DOAJ |
| description | Although the NeRF approach can achieve outstanding view synthesis, it is limited in practical use because it requires many views (hundreds) for training. With only a few input views, the Depth-DYN NeRF that we propose can accurately match the shape. First, we adopted the ip_basic depth-completion method, which can recover the complete depth map from sparse radar depth data. Then, we further designed the Depth-DYN MLP network architecture, which uses a dense depth prior to constraining the NeRF optimization and combines the depthloss to supervise the Depth-DYN MLP network. When compared to the color-only supervised-based NeRF, the Depth-DYN MLP network can better recover the geometric structure of the model and reduce the appearance of shadows. To further ensure that the depth depicted along the rays intersecting these 3D points is close to the measured depth, we dynamically modified the sample space based on the depth of each pixel point. Depth-DYN NeRF considerably outperforms depth NeRF and other sparse view versions when there are a few input views. Using only 10–20 photos to render high-quality images on the new view, our strategy was tested and confirmed on a variety of benchmark datasets. Compared with NeRF, we obtained better image quality (NeRF average at 22.47 dB vs. our 27.296 dB). |
| first_indexed | 2024-03-11T08:54:17Z |
| format | Article |
| id | doaj.art-e20288ca76e64f999c1c718243baeba9 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-11T08:54:17Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-e20288ca76e64f999c1c718243baeba92023-11-16T20:14:14ZengMDPI AGElectronics2079-92922023-02-01124105310.3390/electronics12041053Depth-Based Dynamic Sampling of Neural Radiation FieldsJie Wang0Jiangjian Xiao1Xiaolu Zhang2Xiaolin Xu3Tianxing Jin4Zhijia Jin5Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo 315201, ChinaFaculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo 315201, ChinaNingbo Institute of Industrial Technology, Chinese Academy of Sciences, Ningbo 315201, ChinaFaculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo 315201, ChinaFaculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo 315201, ChinaFaculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo 315201, ChinaAlthough the NeRF approach can achieve outstanding view synthesis, it is limited in practical use because it requires many views (hundreds) for training. With only a few input views, the Depth-DYN NeRF that we propose can accurately match the shape. First, we adopted the ip_basic depth-completion method, which can recover the complete depth map from sparse radar depth data. Then, we further designed the Depth-DYN MLP network architecture, which uses a dense depth prior to constraining the NeRF optimization and combines the depthloss to supervise the Depth-DYN MLP network. When compared to the color-only supervised-based NeRF, the Depth-DYN MLP network can better recover the geometric structure of the model and reduce the appearance of shadows. To further ensure that the depth depicted along the rays intersecting these 3D points is close to the measured depth, we dynamically modified the sample space based on the depth of each pixel point. Depth-DYN NeRF considerably outperforms depth NeRF and other sparse view versions when there are a few input views. Using only 10–20 photos to render high-quality images on the new view, our strategy was tested and confirmed on a variety of benchmark datasets. Compared with NeRF, we obtained better image quality (NeRF average at 22.47 dB vs. our 27.296 dB).https://www.mdpi.com/2079-9292/12/4/1053NeRFscene representationview synthesisimage-based renderingvolume rendering |
| spellingShingle | Jie Wang Jiangjian Xiao Xiaolu Zhang Xiaolin Xu Tianxing Jin Zhijia Jin Depth-Based Dynamic Sampling of Neural Radiation Fields Electronics NeRF scene representation view synthesis image-based rendering volume rendering |
| title | Depth-Based Dynamic Sampling of Neural Radiation Fields |
| title_full | Depth-Based Dynamic Sampling of Neural Radiation Fields |
| title_fullStr | Depth-Based Dynamic Sampling of Neural Radiation Fields |
| title_full_unstemmed | Depth-Based Dynamic Sampling of Neural Radiation Fields |
| title_short | Depth-Based Dynamic Sampling of Neural Radiation Fields |
| title_sort | depth based dynamic sampling of neural radiation fields |
| topic | NeRF scene representation view synthesis image-based rendering volume rendering |
| url | https://www.mdpi.com/2079-9292/12/4/1053 |
| work_keys_str_mv | AT jiewang depthbaseddynamicsamplingofneuralradiationfields AT jiangjianxiao depthbaseddynamicsamplingofneuralradiationfields AT xiaoluzhang depthbaseddynamicsamplingofneuralradiationfields AT xiaolinxu depthbaseddynamicsamplingofneuralradiationfields AT tianxingjin depthbaseddynamicsamplingofneuralradiationfields AT zhijiajin depthbaseddynamicsamplingofneuralradiationfields |