3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques
Semantic augmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionised image segmentation and classification, its impact on point cloud is an active research field. In this paper, we propose an instance segmentation and augment...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-09-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/13/18/3647 |
| _version_ | 1797517324083789824 |
|---|---|
| author | Ghizlane Karara Rafika Hajji Florent Poux |
| author_facet | Ghizlane Karara Rafika Hajji Florent Poux |
| author_sort | Ghizlane Karara |
| collection | DOAJ |
| description | Semantic augmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionised image segmentation and classification, its impact on point cloud is an active research field. In this paper, we propose an instance segmentation and augmentation of 3D point clouds using deep learning architectures. We show the potential of an indirect approach using 2D images and a Mask R-CNN (Region-Based Convolution Neural Network). Our method consists of four core steps. We first project the point cloud onto panoramic 2D images using three types of projections: spherical, cylindrical, and cubic. Next, we homogenise the resulting images to correct the artefacts and the empty pixels to be comparable to images available in common training libraries. These images are then used as input to the Mask R-CNN neural network, designed for 2D instance segmentation. Finally, the obtained predictions are reprojected to the point cloud to obtain the segmentation results. We link the results to a context-aware neural network to augment the semantics. Several tests were performed on different datasets to test the adequacy of the method and its potential for generalisation. The developed algorithm uses only the attributes X, Y, Z, and a projection centre (virtual camera) position as inputs. |
| first_indexed | 2024-03-10T07:15:04Z |
| format | Article |
| id | doaj.art-1159ffe9de25457dbd5633b0b72f49f7 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T07:15:04Z |
| publishDate | 2021-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-1159ffe9de25457dbd5633b0b72f49f72023-11-22T15:06:16ZengMDPI AGRemote Sensing2072-42922021-09-011318364710.3390/rs131836473D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning TechniquesGhizlane Karara0Rafika Hajji1Florent Poux2College of Geomatic Sciences and Surveying Engineering, Institute of Agronomy and Veterinary Medicine, Rabat 10112, MoroccoCollege of Geomatic Sciences and Surveying Engineering, Institute of Agronomy and Veterinary Medicine, Rabat 10112, MoroccoGeomatics Unit, University of Liège (ULiège), 4000 Liège, BelgiumSemantic augmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionised image segmentation and classification, its impact on point cloud is an active research field. In this paper, we propose an instance segmentation and augmentation of 3D point clouds using deep learning architectures. We show the potential of an indirect approach using 2D images and a Mask R-CNN (Region-Based Convolution Neural Network). Our method consists of four core steps. We first project the point cloud onto panoramic 2D images using three types of projections: spherical, cylindrical, and cubic. Next, we homogenise the resulting images to correct the artefacts and the empty pixels to be comparable to images available in common training libraries. These images are then used as input to the Mask R-CNN neural network, designed for 2D instance segmentation. Finally, the obtained predictions are reprojected to the point cloud to obtain the segmentation results. We link the results to a context-aware neural network to augment the semantics. Several tests were performed on different datasets to test the adequacy of the method and its potential for generalisation. The developed algorithm uses only the attributes X, Y, Z, and a projection centre (virtual camera) position as inputs.https://www.mdpi.com/2072-4292/13/18/36473D point cloudinstance segmentation3D projectionpanoramic imagedeep learningI point cloud semantics |
| spellingShingle | Ghizlane Karara Rafika Hajji Florent Poux 3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques Remote Sensing 3D point cloud instance segmentation 3D projection panoramic image deep learning I point cloud semantics |
| title | 3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques |
| title_full | 3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques |
| title_fullStr | 3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques |
| title_full_unstemmed | 3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques |
| title_short | 3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques |
| title_sort | 3d point cloud semantic augmentation instance segmentation of 360° panoramas by deep learning techniques |
| topic | 3D point cloud instance segmentation 3D projection panoramic image deep learning I point cloud semantics |
| url | https://www.mdpi.com/2072-4292/13/18/3647 |
| work_keys_str_mv | AT ghizlanekarara 3dpointcloudsemanticaugmentationinstancesegmentationof360panoramasbydeeplearningtechniques AT rafikahajji 3dpointcloudsemanticaugmentationinstancesegmentationof360panoramasbydeeplearningtechniques AT florentpoux 3dpointcloudsemanticaugmentationinstancesegmentationof360panoramasbydeeplearningtechniques |