Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion Network
High Dynamic Range (HDR) images are widely used in automotive, aerospace, AI, and other fields but are limited by the maximum dynamic range of a single data acquisition using CMOS image sensors. High dynamic range images are usually synthesized through multiple exposure techniques and image processi...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Symmetry |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-8994/15/7/1463 |
| _version_ | 1797587418883293184 |
|---|---|
| author | Bo Song Rui Gao Yong Wang Qi Yu |
| author_facet | Bo Song Rui Gao Yong Wang Qi Yu |
| author_sort | Bo Song |
| collection | DOAJ |
| description | High Dynamic Range (HDR) images are widely used in automotive, aerospace, AI, and other fields but are limited by the maximum dynamic range of a single data acquisition using CMOS image sensors. High dynamic range images are usually synthesized through multiple exposure techniques and image processing techniques. One of the most challenging task in multiframe Low Dynamic Range (LDR) images fusion for HDR is to eliminate ghosting artifacts caused by motion. In traditional algorithms, optical flow is generally used to align dynamic scenes before image fusion, which can achieve good results in cases of small-scale motion scenes but causes obvious ghosting artifacts when motion magnitude is large. Recently, attention mechanisms have been introduced during the alignment stage to enhance the network’s ability to remove ghosts. However, significant ghosting artifacts still occur in some scenarios with large-scale motion or oversaturated areas. We proposea novel Distilled Feature TransformerBlock (DFTB) structure to distill and re-extract information from deep image features obtained after U-Net downsampling, achieving ghost removal at the semantic level for HDR fusion. We introduce a Feature Distillation Transformer Block (FDTB), based on the Swin-Transformer and RFDB structure. FDTB uses multiple distillation connections to learn more discriminative feature representations. For the multiexposure moving scene image fusion HDR ghost removal task, in the previous method, the use of deep learning to remove the ghost effect in the composite image has been perfect, and it is almost difficult to observe the ghost residue of moving objects in the composite HDR image. The method in this paper focuses more on how to save the details of LDR image more completely after removing the ghost to synthesize high-quality HDR image. After using the proposed FDTB, the edge texture details of the synthesized HDR image are saved more perfectly, which shows that FDTB has a better effect in saving the details of image fusion. Futhermore, we propose a new depth framework based on DFTB for fusing and removing ghosts from deep image features, called TransU-Fusion. First of all, we use the encoder in U-Net to extract image features of different exposures and map them to different dimensional feature spaces. By utilizing the symmetry of the U-Net structure, we can ultimately output these feature images as original size HDR images. Then, we further fuse high-dimensional space features using Dilated Residual Dense Block (DRDB) to expand the receptive field, which is beneficial for repairing over-saturated regions. We use the transformer in DFTB to perform low-pass filtering on low-dimensional space features and interact with global information to remove ghosts. Finally, the processed features are merged and output as an HDR image without ghosting artifacts through the decoder. After testing on datasets and comparing with benchmark and state-of-the-art models, the results demonstrate our model’s excellent information fusion ability and stronger ghost removal capability. |
| first_indexed | 2024-03-11T00:36:43Z |
| format | Article |
| id | doaj.art-f596fa47cc51446295e43150e03701a8 |
| institution | Directory Open Access Journal |
| issn | 2073-8994 |
| language | English |
| last_indexed | 2024-03-11T00:36:43Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Symmetry |
| spelling | doaj.art-f596fa47cc51446295e43150e03701a82023-11-18T21:35:23ZengMDPI AGSymmetry2073-89942023-07-01157146310.3390/sym15071463Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion NetworkBo Song0Rui Gao1Yong Wang2Qi Yu3State Key Laboratory of Electronic Thin Films and Integrated Devices, School of Integrated Circult Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, ChinaState Key Laboratory of Electronic Thin Films and Integrated Devices, School of Integrated Circult Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, ChinaChengdu Image Design Technology Co., Ltd., 171# Hele 2nd Street, Chengdu 610213, ChinaState Key Laboratory of Electronic Thin Films and Integrated Devices, School of Integrated Circult Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, ChinaHigh Dynamic Range (HDR) images are widely used in automotive, aerospace, AI, and other fields but are limited by the maximum dynamic range of a single data acquisition using CMOS image sensors. High dynamic range images are usually synthesized through multiple exposure techniques and image processing techniques. One of the most challenging task in multiframe Low Dynamic Range (LDR) images fusion for HDR is to eliminate ghosting artifacts caused by motion. In traditional algorithms, optical flow is generally used to align dynamic scenes before image fusion, which can achieve good results in cases of small-scale motion scenes but causes obvious ghosting artifacts when motion magnitude is large. Recently, attention mechanisms have been introduced during the alignment stage to enhance the network’s ability to remove ghosts. However, significant ghosting artifacts still occur in some scenarios with large-scale motion or oversaturated areas. We proposea novel Distilled Feature TransformerBlock (DFTB) structure to distill and re-extract information from deep image features obtained after U-Net downsampling, achieving ghost removal at the semantic level for HDR fusion. We introduce a Feature Distillation Transformer Block (FDTB), based on the Swin-Transformer and RFDB structure. FDTB uses multiple distillation connections to learn more discriminative feature representations. For the multiexposure moving scene image fusion HDR ghost removal task, in the previous method, the use of deep learning to remove the ghost effect in the composite image has been perfect, and it is almost difficult to observe the ghost residue of moving objects in the composite HDR image. The method in this paper focuses more on how to save the details of LDR image more completely after removing the ghost to synthesize high-quality HDR image. After using the proposed FDTB, the edge texture details of the synthesized HDR image are saved more perfectly, which shows that FDTB has a better effect in saving the details of image fusion. Futhermore, we propose a new depth framework based on DFTB for fusing and removing ghosts from deep image features, called TransU-Fusion. First of all, we use the encoder in U-Net to extract image features of different exposures and map them to different dimensional feature spaces. By utilizing the symmetry of the U-Net structure, we can ultimately output these feature images as original size HDR images. Then, we further fuse high-dimensional space features using Dilated Residual Dense Block (DRDB) to expand the receptive field, which is beneficial for repairing over-saturated regions. We use the transformer in DFTB to perform low-pass filtering on low-dimensional space features and interact with global information to remove ghosts. Finally, the processed features are merged and output as an HDR image without ghosting artifacts through the decoder. After testing on datasets and comparing with benchmark and state-of-the-art models, the results demonstrate our model’s excellent information fusion ability and stronger ghost removal capability.https://www.mdpi.com/2073-8994/15/7/1463HDR fusionDFTBDRDBU-Netghosting artifact |
| spellingShingle | Bo Song Rui Gao Yong Wang Qi Yu Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion Network Symmetry HDR fusion DFTB DRDB U-Net ghosting artifact |
| title | Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion Network |
| title_full | Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion Network |
| title_fullStr | Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion Network |
| title_full_unstemmed | Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion Network |
| title_short | Enhanced LDR Detail Rendering for HDR Fusion by TransU-Fusion Network |
| title_sort | enhanced ldr detail rendering for hdr fusion by transu fusion network |
| topic | HDR fusion DFTB DRDB U-Net ghosting artifact |
| url | https://www.mdpi.com/2073-8994/15/7/1463 |
| work_keys_str_mv | AT bosong enhancedldrdetailrenderingforhdrfusionbytransufusionnetwork AT ruigao enhancedldrdetailrenderingforhdrfusionbytransufusionnetwork AT yongwang enhancedldrdetailrenderingforhdrfusionbytransufusionnetwork AT qiyu enhancedldrdetailrenderingforhdrfusionbytransufusionnetwork |