Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture
In this study, we introduce the U-Swin fusion model, an effective and efficient transformer-based architecture designed for the fusion of multi-focus microscope images. We utilized the Swin-Transformer with shifted window and path merging as the encoder for extracted hierarchical context features. A...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/23/12798 |
| _version_ | 1827592442963034112 |
|---|---|
| author | Han Hank Xia Hao Gao Hang Shao Kun Gao Wei Liu |
| author_facet | Han Hank Xia Hao Gao Hang Shao Kun Gao Wei Liu |
| author_sort | Han Hank Xia |
| collection | DOAJ |
| description | In this study, we introduce the U-Swin fusion model, an effective and efficient transformer-based architecture designed for the fusion of multi-focus microscope images. We utilized the Swin-Transformer with shifted window and path merging as the encoder for extracted hierarchical context features. Additionally, a Swin-Transformer-based decoder with patch expansion was designed to perform the un-sampling operation, generating the fully focused image. To enhance the performance of the feature decoder, the skip connections were applied to concatenate the hierarchical features from the encoder with the decoder up-sample features, like U-net. To facilitate comprehensive model training, we created a substantial dataset of multi-focus images, primarily derived from texture datasets. Our modulators demonstrated superior capability for multi-focus image fusion to achieve comparable or even better fusion images than the existing state-of-the-art image fusion algorithms and demonstrated adequate generalization ability for multi-focus microscope image fusion. Remarkably, for multi-focus microscope image fusion, the pure transformer-based U-Swin fusion model incorporating channel mix fusion rules delivers optimal performance compared with most existing end-to-end fusion models. |
| first_indexed | 2024-03-09T01:54:40Z |
| format | Article |
| id | doaj.art-3c3d07149f824cd3ab9ac7713c01657a |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T01:54:40Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-3c3d07149f824cd3ab9ac7713c01657a2023-12-08T15:11:45ZengMDPI AGApplied Sciences2076-34172023-11-0113231279810.3390/app132312798Multi-Focus Microscopy Image Fusion Based on Swin Transformer ArchitectureHan Hank Xia0Hao Gao1Hang Shao2Kun Gao3Wei Liu4School of Electronic Science and Engineering, Nanjing University, Nanjing 210093, ChinaInstitute of Advanced Technology, Nanjing University of Posts and Telecommunications, Nanjing 210023, ChinaYangtze Delta Region Institute of Tsinghua University, Jiaxing 314006, ChinaYangtze Delta Region Institute of Tsinghua University, Jiaxing 314006, ChinaYangtze Delta Region Institute of Tsinghua University, Jiaxing 314006, ChinaIn this study, we introduce the U-Swin fusion model, an effective and efficient transformer-based architecture designed for the fusion of multi-focus microscope images. We utilized the Swin-Transformer with shifted window and path merging as the encoder for extracted hierarchical context features. Additionally, a Swin-Transformer-based decoder with patch expansion was designed to perform the un-sampling operation, generating the fully focused image. To enhance the performance of the feature decoder, the skip connections were applied to concatenate the hierarchical features from the encoder with the decoder up-sample features, like U-net. To facilitate comprehensive model training, we created a substantial dataset of multi-focus images, primarily derived from texture datasets. Our modulators demonstrated superior capability for multi-focus image fusion to achieve comparable or even better fusion images than the existing state-of-the-art image fusion algorithms and demonstrated adequate generalization ability for multi-focus microscope image fusion. Remarkably, for multi-focus microscope image fusion, the pure transformer-based U-Swin fusion model incorporating channel mix fusion rules delivers optimal performance compared with most existing end-to-end fusion models.https://www.mdpi.com/2076-3417/13/23/12798multi-focus fusionmicroscope imagesSwin Transformer |
| spellingShingle | Han Hank Xia Hao Gao Hang Shao Kun Gao Wei Liu Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture Applied Sciences multi-focus fusion microscope images Swin Transformer |
| title | Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture |
| title_full | Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture |
| title_fullStr | Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture |
| title_full_unstemmed | Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture |
| title_short | Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture |
| title_sort | multi focus microscopy image fusion based on swin transformer architecture |
| topic | multi-focus fusion microscope images Swin Transformer |
| url | https://www.mdpi.com/2076-3417/13/23/12798 |
| work_keys_str_mv | AT hanhankxia multifocusmicroscopyimagefusionbasedonswintransformerarchitecture AT haogao multifocusmicroscopyimagefusionbasedonswintransformerarchitecture AT hangshao multifocusmicroscopyimagefusionbasedonswintransformerarchitecture AT kungao multifocusmicroscopyimagefusionbasedonswintransformerarchitecture AT weiliu multifocusmicroscopyimagefusionbasedonswintransformerarchitecture |