General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks
In this paper, we propose a unified and flexible framework for general image fusion tasks, including multi-exposure image fusion, multi-focus image fusion, infrared/visible image fusion, and multi-modality medical image fusion. Unlike other deep learning-based image fusion methods applied to a fixed...
| Principais autores: | , , , |
|---|---|
| Formato: | Artigo |
| Idioma: | English |
| Publicado em: |
MDPI AG
2022-03-01
|
| coleção: | Sensors |
| Assuntos: | |
| Acesso em linha: | https://www.mdpi.com/1424-8220/22/7/2457 |
| _version_ | 1827622544174219264 |
|---|---|
| author | Yifan Xiao Zhixin Guo Peter Veelaert Wilfried Philips |
| author_facet | Yifan Xiao Zhixin Guo Peter Veelaert Wilfried Philips |
| author_sort | Yifan Xiao |
| collection | DOAJ |
| description | In this paper, we propose a unified and flexible framework for general image fusion tasks, including multi-exposure image fusion, multi-focus image fusion, infrared/visible image fusion, and multi-modality medical image fusion. Unlike other deep learning-based image fusion methods applied to a fixed number of input sources (normally two inputs), the proposed framework can simultaneously handle an arbitrary number of inputs. Specifically, we use the symmetrical function (e.g., Max-pooling) to extract the most significant features from all the input images, which are then fused with the respective features from each input source. This symmetry function enables permutation-invariance of the network, which means the network can successfully extract and fuse the saliency features of each image without needing to remember the input order of the inputs. The property of permutation-invariance also brings convenience for the network during inference with unfixed inputs. To handle multiple image fusion tasks with one unified framework, we adopt continual learning based on Elastic Weight Consolidation (EWC) for different fusion tasks. Subjective and objective experiments on several public datasets demonstrate that the proposed method outperforms state-of-the-art methods on multiple image fusion tasks. |
| first_indexed | 2024-03-09T11:27:22Z |
| format | Article |
| id | doaj.art-3ad31f52647e4899a9f8a8185c2b7e47 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T11:27:22Z |
| publishDate | 2022-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-3ad31f52647e4899a9f8a8185c2b7e472023-11-30T23:59:09ZengMDPI AGSensors1424-82202022-03-01227245710.3390/s22072457General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural NetworksYifan Xiao0Zhixin Guo1Peter Veelaert2Wilfried Philips3Department of Telecommunications and Information Processing, IPI-IMEC, Ghent University, 9000 Ghent, BelgiumDepartment of Telecommunications and Information Processing, IPI-IMEC, Ghent University, 9000 Ghent, BelgiumDepartment of Telecommunications and Information Processing, IPI-IMEC, Ghent University, 9000 Ghent, BelgiumDepartment of Telecommunications and Information Processing, IPI-IMEC, Ghent University, 9000 Ghent, BelgiumIn this paper, we propose a unified and flexible framework for general image fusion tasks, including multi-exposure image fusion, multi-focus image fusion, infrared/visible image fusion, and multi-modality medical image fusion. Unlike other deep learning-based image fusion methods applied to a fixed number of input sources (normally two inputs), the proposed framework can simultaneously handle an arbitrary number of inputs. Specifically, we use the symmetrical function (e.g., Max-pooling) to extract the most significant features from all the input images, which are then fused with the respective features from each input source. This symmetry function enables permutation-invariance of the network, which means the network can successfully extract and fuse the saliency features of each image without needing to remember the input order of the inputs. The property of permutation-invariance also brings convenience for the network during inference with unfixed inputs. To handle multiple image fusion tasks with one unified framework, we adopt continual learning based on Elastic Weight Consolidation (EWC) for different fusion tasks. Subjective and objective experiments on several public datasets demonstrate that the proposed method outperforms state-of-the-art methods on multiple image fusion tasks.https://www.mdpi.com/1424-8220/22/7/2457image fusionmultiple inputspermutation-invariant networkcontinual learning |
| spellingShingle | Yifan Xiao Zhixin Guo Peter Veelaert Wilfried Philips General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks Sensors image fusion multiple inputs permutation-invariant network continual learning |
| title | General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks |
| title_full | General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks |
| title_fullStr | General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks |
| title_full_unstemmed | General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks |
| title_short | General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks |
| title_sort | general image fusion for an arbitrary number of inputs using convolutional neural networks |
| topic | image fusion multiple inputs permutation-invariant network continual learning |
| url | https://www.mdpi.com/1424-8220/22/7/2457 |
| work_keys_str_mv | AT yifanxiao generalimagefusionforanarbitrarynumberofinputsusingconvolutionalneuralnetworks AT zhixinguo generalimagefusionforanarbitrarynumberofinputsusingconvolutionalneuralnetworks AT peterveelaert generalimagefusionforanarbitrarynumberofinputsusingconvolutionalneuralnetworks AT wilfriedphilips generalimagefusionforanarbitrarynumberofinputsusingconvolutionalneuralnetworks |