Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception - Natan, Oskar; Miura, Jun (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
In this research, we present a novel deep multi-task learning model to handle the perception stage of an autonomous driving system. The model leverages the fusion of RGB and dynamic vision sensor (DVS) images to perform semantic segmentation and depth estimation in four different perspectives of view simultaneously. As for the experiment, CARLA simulator is used to generate thousands of simulation data for training, validation, and testing processes. A dynamically changing environment with various weather conditions, daytime, maps, and non-player characters (NPC) is also considered to simulate a more realistic condition with expecting a better generalization of the model. An ablation study is conducted by modifying the network architecture to evaluate the influence of the sensor fusion technique. Based on the test result on 2 different datasets, the model that leverages feature maps sharing from RGB and DVS encoders is performing better. Furthermore, we show that our model can inference faster and have a comparable performance against another recent model. Official implementation code is shared at https://github.com/oskarnatan/RGBDVS-fusion.