COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING
Current 2D and 3D semantic segmentation frameworks are developed and trained on specific benchmark datasets, often rich of synthetic data, and when they are applied to complex and real-world heritage scenarios they offer much lower accuracy than expected. In this work, we present and demonstrate an...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2024-02-01
|
| Series: | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Online Access: | https://isprs-archives.copernicus.org/articles/XLVIII-2-W4-2024/49/2024/isprs-archives-XLVIII-2-W4-2024-49-2024.pdf |
| _version_ | 1827349993670836224 |
|---|---|
| author | M. Bassier G. Mazzacca G. Mazzacca R. Battisti S. Malek F. Remondino |
| author_facet | M. Bassier G. Mazzacca G. Mazzacca R. Battisti S. Malek F. Remondino |
| author_sort | M. Bassier |
| collection | DOAJ |
| description | Current 2D and 3D semantic segmentation frameworks are developed and trained on specific benchmark datasets, often rich of synthetic data, and when they are applied to complex and real-world heritage scenarios they offer much lower accuracy than expected. In this work, we present and demonstrate an early and late fusion of methods for semantic segmentation in cultural heritage applications. We rely on image datasets, point clouds and BIM models. The early fusion utilizes multi-view rendering to generate RGBD imagery of the scene. In contrast, the late fusion approach merges image-based segmentation with a Point Transformer applied to point clouds. Two scenarios are considered and inference results show that predictions are primarily influenced by whether the scene has a predominantly geometric or texture-based signature, underscoring the necessity of fusion methods. |
| first_indexed | 2024-03-08T00:51:58Z |
| format | Article |
| id | doaj.art-707bac6354014c1a937410af24a357da |
| institution | Directory Open Access Journal |
| issn | 1682-1750 2194-9034 |
| language | English |
| last_indexed | 2024-03-08T00:51:58Z |
| publishDate | 2024-02-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| spelling | doaj.art-707bac6354014c1a937410af24a357da2024-02-14T22:29:13ZengCopernicus PublicationsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences1682-17502194-90342024-02-01XLVIII-2-W4-2024495610.5194/isprs-archives-XLVIII-2-W4-2024-49-2024COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDINGM. Bassier0G. Mazzacca1G. Mazzacca2R. Battisti3S. Malek4F. Remondino5Dept. of Civil Engineering – Geomatics, KU Leuven – Faculty of Engineering Technology, Ghent, Belgium3D Optical Metrology (3DOM) unit, Bruno Kessler Foundation (FBK), Trento, ItalyDept. Mathematics, Computer Science and Physics, University of Udine, Italy3D Optical Metrology (3DOM) unit, Bruno Kessler Foundation (FBK), Trento, Italy3D Optical Metrology (3DOM) unit, Bruno Kessler Foundation (FBK), Trento, Italy3D Optical Metrology (3DOM) unit, Bruno Kessler Foundation (FBK), Trento, ItalyCurrent 2D and 3D semantic segmentation frameworks are developed and trained on specific benchmark datasets, often rich of synthetic data, and when they are applied to complex and real-world heritage scenarios they offer much lower accuracy than expected. In this work, we present and demonstrate an early and late fusion of methods for semantic segmentation in cultural heritage applications. We rely on image datasets, point clouds and BIM models. The early fusion utilizes multi-view rendering to generate RGBD imagery of the scene. In contrast, the late fusion approach merges image-based segmentation with a Point Transformer applied to point clouds. Two scenarios are considered and inference results show that predictions are primarily influenced by whether the scene has a predominantly geometric or texture-based signature, underscoring the necessity of fusion methods.https://isprs-archives.copernicus.org/articles/XLVIII-2-W4-2024/49/2024/isprs-archives-XLVIII-2-W4-2024-49-2024.pdf |
| spellingShingle | M. Bassier G. Mazzacca G. Mazzacca R. Battisti S. Malek F. Remondino COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| title | COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING |
| title_full | COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING |
| title_fullStr | COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING |
| title_full_unstemmed | COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING |
| title_short | COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING |
| title_sort | combining image and point cloud segmentation to improve heritage understanding |
| url | https://isprs-archives.copernicus.org/articles/XLVIII-2-W4-2024/49/2024/isprs-archives-XLVIII-2-W4-2024-49-2024.pdf |
| work_keys_str_mv | AT mbassier combiningimageandpointcloudsegmentationtoimproveheritageunderstanding AT gmazzacca combiningimageandpointcloudsegmentationtoimproveheritageunderstanding AT gmazzacca combiningimageandpointcloudsegmentationtoimproveheritageunderstanding AT rbattisti combiningimageandpointcloudsegmentationtoimproveheritageunderstanding AT smalek combiningimageandpointcloudsegmentationtoimproveheritageunderstanding AT fremondino combiningimageandpointcloudsegmentationtoimproveheritageunderstanding |