Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface Vehicles
Water segmentation is essential for the autonomous driving system of unmanned surface vehicles (USVs), which provides reliable navigation for making safety decisions. However, existing methods have only used monocular images as input, which often suffer from the changes in illumination and weather....
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Journal of Marine Science and Engineering |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2077-1312/10/6/744 |
| _version_ | 1827659520139067392 |
|---|---|
| author | Jiantao Gao Jingting Zhang Chang Liu Xiaomao Li Yan Peng |
| author_facet | Jiantao Gao Jingting Zhang Chang Liu Xiaomao Li Yan Peng |
| author_sort | Jiantao Gao |
| collection | DOAJ |
| description | Water segmentation is essential for the autonomous driving system of unmanned surface vehicles (USVs), which provides reliable navigation for making safety decisions. However, existing methods have only used monocular images as input, which often suffer from the changes in illumination and weather. Compared with monocular images, LiDAR point clouds can be collected independently of ambient light and provide sufficient 3D information but lack the color and texture that images own. Thus, in this paper, we propose a novel camera-LiDAR cross-modality fusion water segmentation method, which combines the data characteristics of the 2D image and 3D LiDAR point cloud in water segmentation for the first time. Specifically, the 3D point clouds are first supplemented with 2D color and texture information from the images and then distinguished into water surface points and non-water points by the early 3D cross-modality segmentation module. Subsequently, the 3D segmentation results and features are fed into the late 2D cross-modality segmentation module to perform 2D water segmentation. Finally, the 2D and 3D water segmentation results are fused for the refinement by an uncertainty-aware cross-modality fusion module. We further collect, annotate and present a novel Cross-modality Water Segmentation (CMWS) dataset to validate our proposed method. To the best of our knowledge, this is the first water segmentation dataset for USVs in inland waterways consisting of images and corresponding point clouds. Extensive experiments on the CMWS dataset demonstrate that our proposed method can significantly improve image-only-based methods, achieving improvements in accuracy and MaxF of approximately 2% for all the image-only-based methods. |
| first_indexed | 2024-03-09T23:23:35Z |
| format | Article |
| id | doaj.art-98fddcd662134b659a5a470521c73dd3 |
| institution | Directory Open Access Journal |
| issn | 2077-1312 |
| language | English |
| last_indexed | 2024-03-09T23:23:35Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Journal of Marine Science and Engineering |
| spelling | doaj.art-98fddcd662134b659a5a470521c73dd32023-11-23T17:22:12ZengMDPI AGJournal of Marine Science and Engineering2077-13122022-05-0110674410.3390/jmse10060744Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface VehiclesJiantao Gao0Jingting Zhang1Chang Liu2Xiaomao Li3Yan Peng4Research Institute of USV Engineering, School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, ChinaResearch Institute of USV Engineering, School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, ChinaResearch Institute of USV Engineering, School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, ChinaResearch Institute of USV Engineering, School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, ChinaResearch Institute of USV Engineering, School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, ChinaWater segmentation is essential for the autonomous driving system of unmanned surface vehicles (USVs), which provides reliable navigation for making safety decisions. However, existing methods have only used monocular images as input, which often suffer from the changes in illumination and weather. Compared with monocular images, LiDAR point clouds can be collected independently of ambient light and provide sufficient 3D information but lack the color and texture that images own. Thus, in this paper, we propose a novel camera-LiDAR cross-modality fusion water segmentation method, which combines the data characteristics of the 2D image and 3D LiDAR point cloud in water segmentation for the first time. Specifically, the 3D point clouds are first supplemented with 2D color and texture information from the images and then distinguished into water surface points and non-water points by the early 3D cross-modality segmentation module. Subsequently, the 3D segmentation results and features are fed into the late 2D cross-modality segmentation module to perform 2D water segmentation. Finally, the 2D and 3D water segmentation results are fused for the refinement by an uncertainty-aware cross-modality fusion module. We further collect, annotate and present a novel Cross-modality Water Segmentation (CMWS) dataset to validate our proposed method. To the best of our knowledge, this is the first water segmentation dataset for USVs in inland waterways consisting of images and corresponding point clouds. Extensive experiments on the CMWS dataset demonstrate that our proposed method can significantly improve image-only-based methods, achieving improvements in accuracy and MaxF of approximately 2% for all the image-only-based methods.https://www.mdpi.com/2077-1312/10/6/744water segmentationsemantic segmentationimage segmentationLiDAR point clouddeep learningunmanned surface vessel |
| spellingShingle | Jiantao Gao Jingting Zhang Chang Liu Xiaomao Li Yan Peng Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface Vehicles Journal of Marine Science and Engineering water segmentation semantic segmentation image segmentation LiDAR point cloud deep learning unmanned surface vessel |
| title | Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface Vehicles |
| title_full | Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface Vehicles |
| title_fullStr | Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface Vehicles |
| title_full_unstemmed | Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface Vehicles |
| title_short | Camera-LiDAR Cross-Modality Fusion Water Segmentation for Unmanned Surface Vehicles |
| title_sort | camera lidar cross modality fusion water segmentation for unmanned surface vehicles |
| topic | water segmentation semantic segmentation image segmentation LiDAR point cloud deep learning unmanned surface vessel |
| url | https://www.mdpi.com/2077-1312/10/6/744 |
| work_keys_str_mv | AT jiantaogao cameralidarcrossmodalityfusionwatersegmentationforunmannedsurfacevehicles AT jingtingzhang cameralidarcrossmodalityfusionwatersegmentationforunmannedsurfacevehicles AT changliu cameralidarcrossmodalityfusionwatersegmentationforunmannedsurfacevehicles AT xiaomaoli cameralidarcrossmodalityfusionwatersegmentationforunmannedsurfacevehicles AT yanpeng cameralidarcrossmodalityfusionwatersegmentationforunmannedsurfacevehicles |