DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDS
Neural Radiance Fields (NeRFs) are non-convolutional neural models that learn 3D scene structure and color to produce novel images of a given scene from a new view point. NeRFs are closely related to such photogrammetric problems as camera pose estimation and bundle adjustment. NeRF takes a number o...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2023-05-01
|
| Series: | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Online Access: | https://isprs-archives.copernicus.org/articles/XLVIII-2-W3-2023/115/2023/isprs-archives-XLVIII-2-W3-2023-115-2023.pdf |
| _version_ | 1797828434647318528 |
|---|---|
| author | V. V. Kniaz V. V. Kniaz V. A. Knyaz V. A. Knyaz A. Bordodymov P. Moshkantsev D. Novikov S. Barylnik |
| author_facet | V. V. Kniaz V. V. Kniaz V. A. Knyaz V. A. Knyaz A. Bordodymov P. Moshkantsev D. Novikov S. Barylnik |
| author_sort | V. V. Kniaz |
| collection | DOAJ |
| description | Neural Radiance Fields (NeRFs) are non-convolutional neural models that learn 3D scene structure and color to produce novel images of a given scene from a new view point. NeRFs are closely related to such photogrammetric problems as camera pose estimation and bundle adjustment. NeRF takes a number of oriented cameras and photos as an input and learns a function that maps a 5D pose vector to an RGB color and volume destiny at point. The estimated function can be used to draw an image using a volume rendering pipeline. Still NeRF have a major limitation: they can not be used for dynamic scene synthesis. We propose a modified NeRF framework that can represent a dynamic scene as a superposition of two or more neural radiance fields. We consider a simple dynamic scene consisting of a static background scene and moving object with a static shape. We implemented our DoubleNeRF model using TensorFlow library. The results of evaluation are encouraging and demonstrate that our DoubleNeRF model achieves and surpasses the state of the art in the dynamic scene synthesis. Our framework includes two neural radiance fields for a background scene and dynamic objects. The evaluation of the model demonstrates that it can be effectively used for synthesis of photorealistic dynamic image sequence and videos. |
| first_indexed | 2024-04-09T13:04:16Z |
| format | Article |
| id | doaj.art-f8a7c1f10c7f4b3781aa560ad9dfd2fa |
| institution | Directory Open Access Journal |
| issn | 1682-1750 2194-9034 |
| language | English |
| last_indexed | 2024-04-09T13:04:16Z |
| publishDate | 2023-05-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| spelling | doaj.art-f8a7c1f10c7f4b3781aa560ad9dfd2fa2023-05-12T17:30:25ZengCopernicus PublicationsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences1682-17502194-90342023-05-01XLVIII-2-W3-202311512010.5194/isprs-archives-XLVIII-2-W3-2023-115-2023DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDSV. V. Kniaz0V. V. Kniaz1V. A. Knyaz2V. A. Knyaz3A. Bordodymov4P. Moshkantsev5D. Novikov6S. Barylnik7State Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaMoscow Institute of Physics and Technology (MIPT), RussiaState Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaMoscow Institute of Physics and Technology (MIPT), RussiaState Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaState Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaState Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaState Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaNeural Radiance Fields (NeRFs) are non-convolutional neural models that learn 3D scene structure and color to produce novel images of a given scene from a new view point. NeRFs are closely related to such photogrammetric problems as camera pose estimation and bundle adjustment. NeRF takes a number of oriented cameras and photos as an input and learns a function that maps a 5D pose vector to an RGB color and volume destiny at point. The estimated function can be used to draw an image using a volume rendering pipeline. Still NeRF have a major limitation: they can not be used for dynamic scene synthesis. We propose a modified NeRF framework that can represent a dynamic scene as a superposition of two or more neural radiance fields. We consider a simple dynamic scene consisting of a static background scene and moving object with a static shape. We implemented our DoubleNeRF model using TensorFlow library. The results of evaluation are encouraging and demonstrate that our DoubleNeRF model achieves and surpasses the state of the art in the dynamic scene synthesis. Our framework includes two neural radiance fields for a background scene and dynamic objects. The evaluation of the model demonstrates that it can be effectively used for synthesis of photorealistic dynamic image sequence and videos.https://isprs-archives.copernicus.org/articles/XLVIII-2-W3-2023/115/2023/isprs-archives-XLVIII-2-W3-2023-115-2023.pdf |
| spellingShingle | V. V. Kniaz V. V. Kniaz V. A. Knyaz V. A. Knyaz A. Bordodymov P. Moshkantsev D. Novikov S. Barylnik DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDS The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| title | DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDS |
| title_full | DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDS |
| title_fullStr | DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDS |
| title_full_unstemmed | DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDS |
| title_short | DOUBLE NERF: REPRESENTING DYNAMIC SCENES AS NEURAL RADIANCE FIELDS |
| title_sort | double nerf representing dynamic scenes as neural radiance fields |
| url | https://isprs-archives.copernicus.org/articles/XLVIII-2-W3-2023/115/2023/isprs-archives-XLVIII-2-W3-2023-115-2023.pdf |
| work_keys_str_mv | AT vvkniaz doublenerfrepresentingdynamicscenesasneuralradiancefields AT vvkniaz doublenerfrepresentingdynamicscenesasneuralradiancefields AT vaknyaz doublenerfrepresentingdynamicscenesasneuralradiancefields AT vaknyaz doublenerfrepresentingdynamicscenesasneuralradiancefields AT abordodymov doublenerfrepresentingdynamicscenesasneuralradiancefields AT pmoshkantsev doublenerfrepresentingdynamicscenesasneuralradiancefields AT dnovikov doublenerfrepresentingdynamicscenesasneuralradiancefields AT sbarylnik doublenerfrepresentingdynamicscenesasneuralradiancefields |