A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTION
Neural Radiance Fields (NeRFs) are trained using a set of camera poses and associated images as input to estimate density and color values for each position. The position-dependent density learning is of particular interest for photogrammetry, enabling 3D reconstruction by querying and filtering the...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2023-05-01
|
| Series: | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Online Access: | https://isprs-archives.copernicus.org/articles/XLVIII-1-W1-2023/207/2023/isprs-archives-XLVIII-1-W1-2023-207-2023.pdf |
| _version_ | 1797820126057201664 |
|---|---|
| author | M. Jäger P. Hübner D. Haitz B. Jutzi |
| author_facet | M. Jäger P. Hübner D. Haitz B. Jutzi |
| author_sort | M. Jäger |
| collection | DOAJ |
| description | Neural Radiance Fields (NeRFs) are trained using a set of camera poses and associated images as input to estimate density and color values for each position. The position-dependent density learning is of particular interest for photogrammetry, enabling 3D reconstruction by querying and filtering the NeRF coordinate system based on the object density. While traditional methods like Structure from Motion are commonly used for camera pose calculation in pre-processing for NeRFs, the HoloLens offers an interesting interface for extracting the required input data directly. We present a workflow for high-resolution 3D reconstructions almost directly from HoloLens data using NeRFs. Thereby, different investigations are considered: Internal camera poses from the HoloLens trajectory via a server application, and external camera poses from Structure from Motion, both with an enhanced variant applied through pose refinement. Results show that the internal camera poses lead to NeRF convergence with a PSNR of 25 dB with a simple rotation around the x-axis and enable a 3D reconstruction. Pose refinement enables comparable quality compared to external camera poses, resulting in improved training process with a PSNR of 27 dB and a better 3D reconstruction. Overall, NeRF reconstructions outperform the conventional photogrammetric dense reconstruction using Multi-View Stereo in terms of completeness and level of detail. |
| first_indexed | 2024-03-13T09:33:47Z |
| format | Article |
| id | doaj.art-c7f424d61e2a458b92bd99716ac7cd38 |
| institution | Directory Open Access Journal |
| issn | 1682-1750 2194-9034 |
| language | English |
| last_indexed | 2024-03-13T09:33:47Z |
| publishDate | 2023-05-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| spelling | doaj.art-c7f424d61e2a458b92bd99716ac7cd382023-05-25T18:50:25ZengCopernicus PublicationsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences1682-17502194-90342023-05-01XLVIII-1-W1-202320721310.5194/isprs-archives-XLVIII-1-W1-2023-207-2023A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTIONM. Jäger0P. Hübner1D. Haitz2B. Jutzi3Institute of Photogrammetry and Remote Sensing (IPF), Karlsruhe Institute of Technology (KIT), Karlsruhe, GermanyInstitute of Photogrammetry and Remote Sensing (IPF), Karlsruhe Institute of Technology (KIT), Karlsruhe, GermanyInstitute of Photogrammetry and Remote Sensing (IPF), Karlsruhe Institute of Technology (KIT), Karlsruhe, GermanyInstitute of Photogrammetry and Remote Sensing (IPF), Karlsruhe Institute of Technology (KIT), Karlsruhe, GermanyNeural Radiance Fields (NeRFs) are trained using a set of camera poses and associated images as input to estimate density and color values for each position. The position-dependent density learning is of particular interest for photogrammetry, enabling 3D reconstruction by querying and filtering the NeRF coordinate system based on the object density. While traditional methods like Structure from Motion are commonly used for camera pose calculation in pre-processing for NeRFs, the HoloLens offers an interesting interface for extracting the required input data directly. We present a workflow for high-resolution 3D reconstructions almost directly from HoloLens data using NeRFs. Thereby, different investigations are considered: Internal camera poses from the HoloLens trajectory via a server application, and external camera poses from Structure from Motion, both with an enhanced variant applied through pose refinement. Results show that the internal camera poses lead to NeRF convergence with a PSNR of 25 dB with a simple rotation around the x-axis and enable a 3D reconstruction. Pose refinement enables comparable quality compared to external camera poses, resulting in improved training process with a PSNR of 27 dB and a better 3D reconstruction. Overall, NeRF reconstructions outperform the conventional photogrammetric dense reconstruction using Multi-View Stereo in terms of completeness and level of detail.https://isprs-archives.copernicus.org/articles/XLVIII-1-W1-2023/207/2023/isprs-archives-XLVIII-1-W1-2023-207-2023.pdf |
| spellingShingle | M. Jäger P. Hübner D. Haitz B. Jutzi A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTION The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| title | A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTION |
| title_full | A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTION |
| title_fullStr | A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTION |
| title_full_unstemmed | A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTION |
| title_short | A COMPARATIVE NEURAL RADIANCE FIELD (NERF) 3D ANALYSIS OF CAMERA POSES FROM HOLOLENS TRAJECTORIES AND STRUCTURE FROM MOTION |
| title_sort | comparative neural radiance field nerf 3d analysis of camera poses from hololens trajectories and structure from motion |
| url | https://isprs-archives.copernicus.org/articles/XLVIII-1-W1-2023/207/2023/isprs-archives-XLVIII-1-W1-2023-207-2023.pdf |
| work_keys_str_mv | AT mjager acomparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion AT phubner acomparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion AT dhaitz acomparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion AT bjutzi acomparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion AT mjager comparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion AT phubner comparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion AT dhaitz comparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion AT bjutzi comparativeneuralradiancefieldnerf3danalysisofcameraposesfromhololenstrajectoriesandstructurefrommotion |