Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network
Aiming at the current multimodal medical image fusion methods that cannot fully characterize the complex textures and edge information of the lesion in the fused image, a method based on Gabor representation of multi-CNN combination and fuzzy neural network is proposed. This method first filters the...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9416689/ |
| _version_ | 1798001979460419584 |
|---|---|
| author | Lifang Wang Jin Zhang Yang Liu Jia Mi Jiong Zhang |
| author_facet | Lifang Wang Jin Zhang Yang Liu Jia Mi Jiong Zhang |
| author_sort | Lifang Wang |
| collection | DOAJ |
| description | Aiming at the current multimodal medical image fusion methods that cannot fully characterize the complex textures and edge information of the lesion in the fused image, a method based on Gabor representation of multi-CNN combination and fuzzy neural network is proposed. This method first filters the CT and MR image sets through a set of Gabor filter banks with different proportions and directions to obtain different Gabor representations pairs of CT and MR, each pair of different Gabor representations is used to train the corresponding CNN to generate a G- CNN and multiple G- CNN form a G- CNN group, namely G- CNNs; then when fusing CT and MR images, CT and MR are represented by Gabors to get Gabor representation pairs firstly, each Gabor representation pair is put into the corresponding trained G- CNN for preliminary fusion, then use the fuzzy neural network to fuse multiple outputs of the G- CNNs to obtain the final fused image. Compared with the nine recent state-of-the-art multimodal fusion methods, the average mutual information of the three groups of experiments has increased by 13%, 10.3%, and 10% respectively; the average spatial frequency has increased by 10.3%, 20%, and 10.7%; the average standard deviation has increased respectively 12.4%, 10.8%, 14.4%; the average edge retention information increased by 33.5%, 22%, and 43%. The experimental results show that the proposed fusion method is significantly better than the other comparative fusion methods in objective evaluation and visual quality. It has the best performance on the four indicators and can better integrate the rich texture features and the clear edge information of the source images into the final fused image, which improves the quality of multimodal medical image fusion, and effectively assists doctors in disease diagnosis. |
| first_indexed | 2024-04-11T11:44:48Z |
| format | Article |
| id | doaj.art-c56b7a66d70d45ce82060e474d890372 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-11T11:44:48Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-c56b7a66d70d45ce82060e474d8903722022-12-22T04:25:40ZengIEEEIEEE Access2169-35362021-01-019676346764710.1109/ACCESS.2021.30759539416689Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural NetworkLifang Wang0Jin Zhang1https://orcid.org/0000-0003-2732-7874Yang Liu2Jia Mi3Jiong Zhang4College of Big Data, North University of China, Shanxi, ChinaCollege of Big Data, North University of China, Shanxi, ChinaCollege of Big Data, North University of China, Shanxi, ChinaCollege of Big Data, North University of China, Shanxi, ChinaCollege of Big Data, North University of China, Shanxi, ChinaAiming at the current multimodal medical image fusion methods that cannot fully characterize the complex textures and edge information of the lesion in the fused image, a method based on Gabor representation of multi-CNN combination and fuzzy neural network is proposed. This method first filters the CT and MR image sets through a set of Gabor filter banks with different proportions and directions to obtain different Gabor representations pairs of CT and MR, each pair of different Gabor representations is used to train the corresponding CNN to generate a G- CNN and multiple G- CNN form a G- CNN group, namely G- CNNs; then when fusing CT and MR images, CT and MR are represented by Gabors to get Gabor representation pairs firstly, each Gabor representation pair is put into the corresponding trained G- CNN for preliminary fusion, then use the fuzzy neural network to fuse multiple outputs of the G- CNNs to obtain the final fused image. Compared with the nine recent state-of-the-art multimodal fusion methods, the average mutual information of the three groups of experiments has increased by 13%, 10.3%, and 10% respectively; the average spatial frequency has increased by 10.3%, 20%, and 10.7%; the average standard deviation has increased respectively 12.4%, 10.8%, 14.4%; the average edge retention information increased by 33.5%, 22%, and 43%. The experimental results show that the proposed fusion method is significantly better than the other comparative fusion methods in objective evaluation and visual quality. It has the best performance on the four indicators and can better integrate the rich texture features and the clear edge information of the source images into the final fused image, which improves the quality of multimodal medical image fusion, and effectively assists doctors in disease diagnosis.https://ieeexplore.ieee.org/document/9416689/Medical image fusionG-CNNsGabor representationconvolutional neural networkfuzzy neural network |
| spellingShingle | Lifang Wang Jin Zhang Yang Liu Jia Mi Jiong Zhang Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network IEEE Access Medical image fusion G-CNNs Gabor representation convolutional neural network fuzzy neural network |
| title | Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network |
| title_full | Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network |
| title_fullStr | Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network |
| title_full_unstemmed | Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network |
| title_short | Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network |
| title_sort | multimodal medical image fusion based on gabor representation combination of multi cnn and fuzzy neural network |
| topic | Medical image fusion G-CNNs Gabor representation convolutional neural network fuzzy neural network |
| url | https://ieeexplore.ieee.org/document/9416689/ |
| work_keys_str_mv | AT lifangwang multimodalmedicalimagefusionbasedongaborrepresentationcombinationofmulticnnandfuzzyneuralnetwork AT jinzhang multimodalmedicalimagefusionbasedongaborrepresentationcombinationofmulticnnandfuzzyneuralnetwork AT yangliu multimodalmedicalimagefusionbasedongaborrepresentationcombinationofmulticnnandfuzzyneuralnetwork AT jiami multimodalmedicalimagefusionbasedongaborrepresentationcombinationofmulticnnandfuzzyneuralnetwork AT jiongzhang multimodalmedicalimagefusionbasedongaborrepresentationcombinationofmulticnnandfuzzyneuralnetwork |