A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATION
In recent years, the developments for Fully Convolutional Networks (FCN) have led to great improvements for semantic segmentation in various applications including fused remote sensing data. There is, however, a lack of an in-depth study inside FCN models which would lead to an understanding of the...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Copernicus Publications
2017-05-01
|
| Series: | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Online Access: | http://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-1-W1/513/2017/isprs-archives-XLII-1-W1-513-2017.pdf |
| _version_ | 1818333873395728384 |
|---|---|
| author | W. Zhang W. Zhang H. Huang M. Schmitz X. Sun H. Wang H. Mayer |
| author_facet | W. Zhang W. Zhang H. Huang M. Schmitz X. Sun H. Wang H. Mayer |
| author_sort | W. Zhang |
| collection | DOAJ |
| description | In recent years, the developments for Fully Convolutional Networks (FCN) have led to great improvements for semantic segmentation in various applications including fused remote sensing data. There is, however, a lack of an in-depth study inside FCN models which would lead to an understanding of the contribution of individual layers to specific classes and their sensitivity to different types of input data. In this paper, we address this problem and propose a fusion model incorporating infrared imagery and Digital Surface Models (DSM) for semantic segmentation. The goal is to utilize heterogeneous data more accurately and effectively in a single model instead of to assemble multiple models. First, the contribution and sensitivity of layers concerning the given classes are quantified by means of their recall in FCN. The contribution of different modalities on the pixel-wise prediction is then analyzed based on visualization. Finally, an optimized scheme for the fusion of layers with color and elevation information into a single FCN model is derived based on the analysis. Experiments are performed on the ISPRS Vaihingen 2D Semantic Labeling dataset. Comprehensive evaluations demonstrate the potential of the proposed approach. |
| first_indexed | 2024-12-13T13:58:33Z |
| format | Article |
| id | doaj.art-c4c2cfc2e0614865a4d2d8370db48db3 |
| institution | Directory Open Access Journal |
| issn | 1682-1750 2194-9034 |
| language | English |
| last_indexed | 2024-12-13T13:58:33Z |
| publishDate | 2017-05-01 |
| publisher | Copernicus Publications |
| record_format | Article |
| series | The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| spelling | doaj.art-c4c2cfc2e0614865a4d2d8370db48db32022-12-21T23:42:50ZengCopernicus PublicationsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences1682-17502194-90342017-05-01XLII-1-W151351710.5194/isprs-archives-XLII-1-W1-513-2017A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATIONW. Zhang0W. Zhang1H. Huang2M. Schmitz3X. Sun4H. Wang5H. Mayer6Key Laboratory of Spatial Information Processing and Application System Technology, Institute of Electronics, Chinese Academy of Sciences, Beijing, 100190, ChinaUniversity of Chinese Academy of Sciences, Beijing, 100190, ChinaInstitute for Applied Computer Science, Bundeswehr University Munich, Werner-Heisenberg-Weg 39, D-85577 Neubiberg, GermanyInstitute for Applied Computer Science, Bundeswehr University Munich, Werner-Heisenberg-Weg 39, D-85577 Neubiberg, GermanyKey Laboratory of Spatial Information Processing and Application System Technology, Institute of Electronics, Chinese Academy of Sciences, Beijing, 100190, ChinaKey Laboratory of Spatial Information Processing and Application System Technology, Institute of Electronics, Chinese Academy of Sciences, Beijing, 100190, ChinaInstitute for Applied Computer Science, Bundeswehr University Munich, Werner-Heisenberg-Weg 39, D-85577 Neubiberg, GermanyIn recent years, the developments for Fully Convolutional Networks (FCN) have led to great improvements for semantic segmentation in various applications including fused remote sensing data. There is, however, a lack of an in-depth study inside FCN models which would lead to an understanding of the contribution of individual layers to specific classes and their sensitivity to different types of input data. In this paper, we address this problem and propose a fusion model incorporating infrared imagery and Digital Surface Models (DSM) for semantic segmentation. The goal is to utilize heterogeneous data more accurately and effectively in a single model instead of to assemble multiple models. First, the contribution and sensitivity of layers concerning the given classes are quantified by means of their recall in FCN. The contribution of different modalities on the pixel-wise prediction is then analyzed based on visualization. Finally, an optimized scheme for the fusion of layers with color and elevation information into a single FCN model is derived based on the analysis. Experiments are performed on the ISPRS Vaihingen 2D Semantic Labeling dataset. Comprehensive evaluations demonstrate the potential of the proposed approach.http://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-1-W1/513/2017/isprs-archives-XLII-1-W1-513-2017.pdf |
| spellingShingle | W. Zhang W. Zhang H. Huang M. Schmitz X. Sun H. Wang H. Mayer A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATION The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| title | A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATION |
| title_full | A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATION |
| title_fullStr | A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATION |
| title_full_unstemmed | A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATION |
| title_short | A MULTI-RESOLUTION FUSION MODEL INCORPORATING COLOR AND ELEVATION FOR SEMANTIC SEGMENTATION |
| title_sort | multi resolution fusion model incorporating color and elevation for semantic segmentation |
| url | http://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-1-W1/513/2017/isprs-archives-XLII-1-W1-513-2017.pdf |
| work_keys_str_mv | AT wzhang amultiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT wzhang amultiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT hhuang amultiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT mschmitz amultiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT xsun amultiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT hwang amultiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT hmayer amultiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT wzhang multiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT wzhang multiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT hhuang multiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT mschmitz multiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT xsun multiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT hwang multiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation AT hmayer multiresolutionfusionmodelincorporatingcolorandelevationforsemanticsegmentation |