SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image Fusion
Infrared and visible image fusion is a solution that generates an information-rich individual image with different modal information by fusing images obtained from various sensors. Salient detection can better emphasize the targets of concern. We propose a residual Swin Transformer fusion network ba...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/18/4467 |
| _version_ | 1797577252683120640 |
|---|---|
| author | Shengshi Li Guanjun Wang Hui Zhang Yonghua Zou |
| author_facet | Shengshi Li Guanjun Wang Hui Zhang Yonghua Zou |
| author_sort | Shengshi Li |
| collection | DOAJ |
| description | Infrared and visible image fusion is a solution that generates an information-rich individual image with different modal information by fusing images obtained from various sensors. Salient detection can better emphasize the targets of concern. We propose a residual Swin Transformer fusion network based on saliency detection, termed SDRSwin, aiming to highlight the salient thermal targets in the infrared image while maintaining the texture details in the visible image. The SDRSwin network is trained with a two-stage training approach. In the first stage, we train an encoder–decoder network based on residual Swin Transformers to achieve powerful feature extraction and reconstruction capabilities. In the second stage, we develop a novel salient loss function to guide the network to fuse the salient targets in the infrared image and the background detail regions in the visible image. The extensive results indicate that our method has abundant texture details with clear bright infrared targets and achieves a better performance than the twenty-one state-of-the-art methods in both subjective and objective evaluation. |
| first_indexed | 2024-03-10T22:05:35Z |
| format | Article |
| id | doaj.art-9ae9bdeae9854ab7b194b36945cc5040 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T22:05:35Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-9ae9bdeae9854ab7b194b36945cc50402023-11-19T12:48:06ZengMDPI AGRemote Sensing2072-42922023-09-011518446710.3390/rs15184467SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image FusionShengshi Li0Guanjun Wang1Hui Zhang2Yonghua Zou3School of Information and Communication Engineering, Hainan University, Haikou 570228, ChinaSchool of Information and Communication Engineering, Hainan University, Haikou 570228, ChinaKey Laboratory of Genetics and Germplasm Innovation of Tropical Special Forest Trees and Ornamental Plants (Hainan University), Ministry of Education, School of Forestry, Hainan University, Haikou 570228, ChinaSchool of Information and Communication Engineering, Hainan University, Haikou 570228, ChinaInfrared and visible image fusion is a solution that generates an information-rich individual image with different modal information by fusing images obtained from various sensors. Salient detection can better emphasize the targets of concern. We propose a residual Swin Transformer fusion network based on saliency detection, termed SDRSwin, aiming to highlight the salient thermal targets in the infrared image while maintaining the texture details in the visible image. The SDRSwin network is trained with a two-stage training approach. In the first stage, we train an encoder–decoder network based on residual Swin Transformers to achieve powerful feature extraction and reconstruction capabilities. In the second stage, we develop a novel salient loss function to guide the network to fuse the salient targets in the infrared image and the background detail regions in the visible image. The extensive results indicate that our method has abundant texture details with clear bright infrared targets and achieves a better performance than the twenty-one state-of-the-art methods in both subjective and objective evaluation.https://www.mdpi.com/2072-4292/15/18/4467image fusionsaliency detectionresidual Swin Transformerinfrared imageHainan gibbon |
| spellingShingle | Shengshi Li Guanjun Wang Hui Zhang Yonghua Zou SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image Fusion Remote Sensing image fusion saliency detection residual Swin Transformer infrared image Hainan gibbon |
| title | SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image Fusion |
| title_full | SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image Fusion |
| title_fullStr | SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image Fusion |
| title_full_unstemmed | SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image Fusion |
| title_short | SDRSwin: A Residual Swin Transformer Network with Saliency Detection for Infrared and Visible Image Fusion |
| title_sort | sdrswin a residual swin transformer network with saliency detection for infrared and visible image fusion |
| topic | image fusion saliency detection residual Swin Transformer infrared image Hainan gibbon |
| url | https://www.mdpi.com/2072-4292/15/18/4467 |
| work_keys_str_mv | AT shengshili sdrswinaresidualswintransformernetworkwithsaliencydetectionforinfraredandvisibleimagefusion AT guanjunwang sdrswinaresidualswintransformernetworkwithsaliencydetectionforinfraredandvisibleimagefusion AT huizhang sdrswinaresidualswintransformernetworkwithsaliencydetectionforinfraredandvisibleimagefusion AT yonghuazou sdrswinaresidualswintransformernetworkwithsaliencydetectionforinfraredandvisibleimagefusion |