Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception
Infrared-visible fusion has great potential in night-vision enhancement for intelligent vehicles. The fusion performance depends on fusion rules that balance target saliency and visual perception. However, most existing methods do not have explicit and effective rules, which leads to the poor contra...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-09-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/24/10/1327 |
| _version_ | 1797473505013399552 |
|---|---|
| author | Xiaoyu Chen Zhijie Teng Yingqi Liu Jun Lu Lianfa Bai Jing Han |
| author_facet | Xiaoyu Chen Zhijie Teng Yingqi Liu Jun Lu Lianfa Bai Jing Han |
| author_sort | Xiaoyu Chen |
| collection | DOAJ |
| description | Infrared-visible fusion has great potential in night-vision enhancement for intelligent vehicles. The fusion performance depends on fusion rules that balance target saliency and visual perception. However, most existing methods do not have explicit and effective rules, which leads to the poor contrast and saliency of the target. In this paper, we propose the SGVPGAN, an adversarial framework for high-quality infrared-visible image fusion, which consists of an infrared-visible image fusion network based on Adversarial Semantic Guidance (ASG) and Adversarial Visual Perception (AVP) modules. Specifically, the ASG module transfers the semantics of the target and background to the fusion process for target highlighting. The AVP module analyzes the visual features from the global structure and local details of the visible and fusion images and then guides the fusion network to adaptively generate a weight map of signal completion so that the resulting fusion images possess a natural and visible appearance. We construct a joint distribution function between the fusion images and the corresponding semantics and use the discriminator to improve the fusion performance in terms of natural appearance and target saliency. Experimental results demonstrate that our proposed ASG and AVP modules can effectively guide the image-fusion process by selectively preserving the details in visible images and the salient information of targets in infrared images. The SGVPGAN exhibits significant improvements over other fusion methods. |
| first_indexed | 2024-03-09T20:15:26Z |
| format | Article |
| id | doaj.art-5a85316709dd4894a35ff3c2c83eadcb |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-09T20:15:26Z |
| publishDate | 2022-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-5a85316709dd4894a35ff3c2c83eadcb2023-11-24T00:01:48ZengMDPI AGEntropy1099-43002022-09-012410132710.3390/e24101327Infrared-Visible Image Fusion Based on Semantic Guidance and Visual PerceptionXiaoyu Chen0Zhijie Teng1Yingqi Liu2Jun Lu3Lianfa Bai4Jing Han5Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, ChinaJiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, ChinaJiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, ChinaJiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, ChinaJiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, ChinaJiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, ChinaInfrared-visible fusion has great potential in night-vision enhancement for intelligent vehicles. The fusion performance depends on fusion rules that balance target saliency and visual perception. However, most existing methods do not have explicit and effective rules, which leads to the poor contrast and saliency of the target. In this paper, we propose the SGVPGAN, an adversarial framework for high-quality infrared-visible image fusion, which consists of an infrared-visible image fusion network based on Adversarial Semantic Guidance (ASG) and Adversarial Visual Perception (AVP) modules. Specifically, the ASG module transfers the semantics of the target and background to the fusion process for target highlighting. The AVP module analyzes the visual features from the global structure and local details of the visible and fusion images and then guides the fusion network to adaptively generate a weight map of signal completion so that the resulting fusion images possess a natural and visible appearance. We construct a joint distribution function between the fusion images and the corresponding semantics and use the discriminator to improve the fusion performance in terms of natural appearance and target saliency. Experimental results demonstrate that our proposed ASG and AVP modules can effectively guide the image-fusion process by selectively preserving the details in visible images and the salient information of targets in infrared images. The SGVPGAN exhibits significant improvements over other fusion methods.https://www.mdpi.com/1099-4300/24/10/1327infrared-visible image fusionsemantic guidancevisual perceptionintelligent vehicles |
| spellingShingle | Xiaoyu Chen Zhijie Teng Yingqi Liu Jun Lu Lianfa Bai Jing Han Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception Entropy infrared-visible image fusion semantic guidance visual perception intelligent vehicles |
| title | Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception |
| title_full | Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception |
| title_fullStr | Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception |
| title_full_unstemmed | Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception |
| title_short | Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception |
| title_sort | infrared visible image fusion based on semantic guidance and visual perception |
| topic | infrared-visible image fusion semantic guidance visual perception intelligent vehicles |
| url | https://www.mdpi.com/1099-4300/24/10/1327 |
| work_keys_str_mv | AT xiaoyuchen infraredvisibleimagefusionbasedonsemanticguidanceandvisualperception AT zhijieteng infraredvisibleimagefusionbasedonsemanticguidanceandvisualperception AT yingqiliu infraredvisibleimagefusionbasedonsemanticguidanceandvisualperception AT junlu infraredvisibleimagefusionbasedonsemanticguidanceandvisualperception AT lianfabai infraredvisibleimagefusionbasedonsemanticguidanceandvisualperception AT jinghan infraredvisibleimagefusionbasedonsemanticguidanceandvisualperception |