GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION
Fast but precise 3D reconstructions of cultural heritage scenes are becoming very requested in the archaeology and architecture. While modern multi-image 3D reconstruction approaches provide impressive results in terms of textured surface models, it is often the need to create a 3D model for which o...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2019-01-01
|
| Series: | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Online Access: | https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-2-W9/403/2019/isprs-archives-XLII-2-W9-403-2019.pdf |
| _version_ | 1818021620316372992 |
|---|---|
| author | V. V. Kniaz V. V. Kniaz F. Remondino V. A. Knyaz V. A. Knyaz |
| author_facet | V. V. Kniaz V. V. Kniaz F. Remondino V. A. Knyaz V. A. Knyaz |
| author_sort | V. V. Kniaz |
| collection | DOAJ |
| description | Fast but precise 3D reconstructions of cultural heritage scenes are becoming very requested in the archaeology and architecture. While modern multi-image 3D reconstruction approaches provide impressive results in terms of textured surface models, it is often the need to create a 3D model for which only a single photo (or few sparse) is available. This paper focuses on the single photo 3D reconstruction problem for lost cultural objects for which only a few images are remaining. We use image-to-voxel translation network (Z-GAN) as a starting point. Z-GAN network utilizes the skip connections in the generator network to transfer 2D features to a 3D voxel model effectively (Figure 1). Therefore, the network can generate voxel models of previously unseen objects using object silhouettes present on the input image and the knowledge obtained during a training stage. In order to train our Z-GAN network, we created a large dataset that includes aligned sets of images and corresponding voxel models of an ancient Greek temple. We evaluated the Z-GAN network for single photo reconstruction on complex structures like temples as well as on lost heritage still available in crowdsourced images. Comparison of the reconstruction results with state-of-the-art methods are also presented and commented. |
| first_indexed | 2024-04-14T08:20:30Z |
| format | Article |
| id | doaj.art-e875c07f91d74ba9bff601187167331b |
| institution | Directory Open Access Journal |
| issn | 1682-1750 2194-9034 |
| language | English |
| last_indexed | 2024-04-14T08:20:30Z |
| publishDate | 2019-01-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| spelling | doaj.art-e875c07f91d74ba9bff601187167331b2022-12-22T02:04:14ZengCopernicus PublicationsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences1682-17502194-90342019-01-01XLII-2-W940340810.5194/isprs-archives-XLII-2-W9-403-2019GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTIONV. V. Kniaz0V. V. Kniaz1F. Remondino2V. A. Knyaz3V. A. Knyaz4State Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaMoscow Institute of Physics and Technology (MIPT), Russia3D Optical Metrology (3DOM) unit, Bruno Kessler Foundation (FBK), Trento, ItalyState Res. Institute of Aviation Systems (GosNIIAS), 125319, 7, Victorenko str., Moscow, RussiaMoscow Institute of Physics and Technology (MIPT), RussiaFast but precise 3D reconstructions of cultural heritage scenes are becoming very requested in the archaeology and architecture. While modern multi-image 3D reconstruction approaches provide impressive results in terms of textured surface models, it is often the need to create a 3D model for which only a single photo (or few sparse) is available. This paper focuses on the single photo 3D reconstruction problem for lost cultural objects for which only a few images are remaining. We use image-to-voxel translation network (Z-GAN) as a starting point. Z-GAN network utilizes the skip connections in the generator network to transfer 2D features to a 3D voxel model effectively (Figure 1). Therefore, the network can generate voxel models of previously unseen objects using object silhouettes present on the input image and the knowledge obtained during a training stage. In order to train our Z-GAN network, we created a large dataset that includes aligned sets of images and corresponding voxel models of an ancient Greek temple. We evaluated the Z-GAN network for single photo reconstruction on complex structures like temples as well as on lost heritage still available in crowdsourced images. Comparison of the reconstruction results with state-of-the-art methods are also presented and commented.https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-2-W9/403/2019/isprs-archives-XLII-2-W9-403-2019.pdf |
| spellingShingle | V. V. Kniaz V. V. Kniaz F. Remondino V. A. Knyaz V. A. Knyaz GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| title | GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION |
| title_full | GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION |
| title_fullStr | GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION |
| title_full_unstemmed | GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION |
| title_short | GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION |
| title_sort | generative adversarial networks for single photo 3d reconstruction |
| url | https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-2-W9/403/2019/isprs-archives-XLII-2-W9-403-2019.pdf |
| work_keys_str_mv | AT vvkniaz generativeadversarialnetworksforsinglephoto3dreconstruction AT vvkniaz generativeadversarialnetworksforsinglephoto3dreconstruction AT fremondino generativeadversarialnetworksforsinglephoto3dreconstruction AT vaknyaz generativeadversarialnetworksforsinglephoto3dreconstruction AT vaknyaz generativeadversarialnetworksforsinglephoto3dreconstruction |