Illumination Robust Semantic Segmentation Based on Cross-Dimensional Multispectral Edge Fusion in Dynamic Traffic Scenes - Peizhou Ni; Weiming Hu; Jinchao Hu; Xiong Chen; Yan Liu (IEEE Access, 2024)
Semantic segmentation in dynamic urban traffic scenes holds paramount importance for enhancing the safety and efficiency of intelligent vehicles. However, the performance of this technique can be hindered in challenging lighting conditions and intricate backgrounds, even with the aid of geometric information derived from depth-images or LiDAR point clouds alongside single-modal RGB images. To overcome this limitation, it is a feasible method to introduce thermal data alongside visible images, which could boost the accuracy and robustness of semantic segmentation in dynamic urban traffic scenes with varying illumination conditions. Considering that, we propose a novel cross-dimensional multispectral edge fusion network (CMEFNet) specifically designed for RGB-Thermal (RGB-T) semantic segmentation in dynamic traffic scenes. It incorporates the dual encoders regrouping low-level and high-level RGB-T features, the hierarchical attentional fusion module to refine multiscale multimodal features, a skip edge guidance structure for information supplement and further promoting accuracy, and a deep supervision mechanism for fine-tuning. Experimental results demonstrate that our method can significantly improves segmentation accuracy and robustness (improving 0.9-18.3% in mIoU), particularly in detecting dynamic traffic participants such as vehicles and pedestrians (improving 0.4-24.6%). Furthermore, it displays commendable accuracy and robustness in both daytime and nighttime scenarios, surpassing the performance of state-of-the-art networks.