Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF
The semantic segmentation of remote sensing images is a significant research direction in digital image processing. The complex background environment, irregular size and shape of objects, and similar appearance of different categories of remote sensing images have brought great challenges to remote...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/5/1229 |
| _version_ | 1827752308749893632 |
|---|---|
| author | Xiang Cheng Hong Lei |
| author_facet | Xiang Cheng Hong Lei |
| author_sort | Xiang Cheng |
| collection | DOAJ |
| description | The semantic segmentation of remote sensing images is a significant research direction in digital image processing. The complex background environment, irregular size and shape of objects, and similar appearance of different categories of remote sensing images have brought great challenges to remote sensing image segmentation tasks. Traditional convolutional-neural-network-based models often ignore spatial information in the feature extraction stage and pay less attention to global context information. However, spatial context information is important in complex remote sensing images, which means that the segmentation effect of traditional models needs to be improved. In addition, neural networks with a superior segmentation performance often suffer from the problem of high computational resource consumption. To address the above issues, this paper proposes a combination model of a modified multiscale deformable convolutional neural network (mmsDCNN) and dense conditional random field (DenseCRF). Firstly, we designed a lightweight multiscale deformable convolutional network (mmsDCNN) with a large receptive field to generate a preliminary prediction probability map at each pixel. The output of the mmsDCNN model is a coarse segmentation result map, which has the same size as the input image. In addition, the preliminary segmentation result map contains rich multiscale features. Then, the multi-level DenseCRF model based on the superpixel level and the pixel level is proposed, which can make full use of the context information of the image at different levels and further optimize the rough segmentation result of mmsDCNN. To be specific, we converted the pixel-level preliminary probability map into a superpixel-level predicted probability map according to the simple linear iterative clustering (SILC) algorithm and defined the potential function of the DenseCRF model based on this. Furthermore, we added the pixel-level potential function constraint term to the superpixel-based Gaussian potential function to obtain a combined Gaussian potential function, which enabled our model to consider the features of various scales and prevent poor superpixel segmentation results from affecting the final result. To restore the contour of the object more clearly, we utilized the Sketch token edge detection algorithm to extract the edge contour features of the image and fused them into the potential function of the DenseCRF model. Finally, extensive experiments on the Potsdam and Vaihingen datasets demonstrated that the proposed model exhibited significant advantages compared to the current state-of-the-art models. |
| first_indexed | 2024-03-11T07:12:19Z |
| format | Article |
| id | doaj.art-701dee6950664d14adfef42bd9588fad |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-11T07:12:19Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-701dee6950664d14adfef42bd9588fad2023-11-17T08:30:11ZengMDPI AGRemote Sensing2072-42922023-02-01155122910.3390/rs15051229Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRFXiang Cheng0Hong Lei1Department of Space Microwave Remote Sensing System, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, ChinaDepartment of Space Microwave Remote Sensing System, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, ChinaThe semantic segmentation of remote sensing images is a significant research direction in digital image processing. The complex background environment, irregular size and shape of objects, and similar appearance of different categories of remote sensing images have brought great challenges to remote sensing image segmentation tasks. Traditional convolutional-neural-network-based models often ignore spatial information in the feature extraction stage and pay less attention to global context information. However, spatial context information is important in complex remote sensing images, which means that the segmentation effect of traditional models needs to be improved. In addition, neural networks with a superior segmentation performance often suffer from the problem of high computational resource consumption. To address the above issues, this paper proposes a combination model of a modified multiscale deformable convolutional neural network (mmsDCNN) and dense conditional random field (DenseCRF). Firstly, we designed a lightweight multiscale deformable convolutional network (mmsDCNN) with a large receptive field to generate a preliminary prediction probability map at each pixel. The output of the mmsDCNN model is a coarse segmentation result map, which has the same size as the input image. In addition, the preliminary segmentation result map contains rich multiscale features. Then, the multi-level DenseCRF model based on the superpixel level and the pixel level is proposed, which can make full use of the context information of the image at different levels and further optimize the rough segmentation result of mmsDCNN. To be specific, we converted the pixel-level preliminary probability map into a superpixel-level predicted probability map according to the simple linear iterative clustering (SILC) algorithm and defined the potential function of the DenseCRF model based on this. Furthermore, we added the pixel-level potential function constraint term to the superpixel-based Gaussian potential function to obtain a combined Gaussian potential function, which enabled our model to consider the features of various scales and prevent poor superpixel segmentation results from affecting the final result. To restore the contour of the object more clearly, we utilized the Sketch token edge detection algorithm to extract the edge contour features of the image and fused them into the potential function of the DenseCRF model. Finally, extensive experiments on the Potsdam and Vaihingen datasets demonstrated that the proposed model exhibited significant advantages compared to the current state-of-the-art models.https://www.mdpi.com/2072-4292/15/5/1229semantic segmentation of remote sensing imagerydeep learningconvolutional neural network (CNN)conditional random field (CRF) |
| spellingShingle | Xiang Cheng Hong Lei Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF Remote Sensing semantic segmentation of remote sensing imagery deep learning convolutional neural network (CNN) conditional random field (CRF) |
| title | Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF |
| title_full | Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF |
| title_fullStr | Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF |
| title_full_unstemmed | Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF |
| title_short | Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF |
| title_sort | semantic segmentation of remote sensing imagery based on multiscale deformable cnn and densecrf |
| topic | semantic segmentation of remote sensing imagery deep learning convolutional neural network (CNN) conditional random field (CRF) |
| url | https://www.mdpi.com/2072-4292/15/5/1229 |
| work_keys_str_mv | AT xiangcheng semanticsegmentationofremotesensingimagerybasedonmultiscaledeformablecnnanddensecrf AT honglei semanticsegmentationofremotesensingimagerybasedonmultiscaledeformablecnnanddensecrf |