Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception
In this research, we present a novel deep multi-task learning model to handle the perception stage of an autonomous driving system. The model leverages the fusion of RGB and dynamic vision sensor (DVS) images to perform semantic segmentation and depth estimation in four different perspectives of vie...
| Main Authors: | , |
|---|---|
| Format: | Other |
| Published: |
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
2022
|
| Subjects: |
| _version_ | 1797037815113973760 |
|---|---|
| author | Natan, Oskar Miura, Jun |
| author_facet | Natan, Oskar Miura, Jun |
| author_sort | Natan, Oskar |
| collection | UGM |
| description | In this research, we present a novel deep multi-task learning model to handle the perception stage of an autonomous driving system. The model leverages the fusion of RGB and dynamic vision sensor (DVS) images to perform semantic segmentation and depth estimation in four different perspectives of view simultaneously. As for the experiment, CARLA simulator is used to generate thousands of simulation data for training, validation, and testing processes. A dynamically changing environment with various weather conditions, daytime, maps, and non-player characters (NPC) is also considered to simulate a more realistic condition with expecting a better generalization of the model. An ablation study is conducted by modifying the network architecture to evaluate the influence of the sensor fusion technique. Based on the test result on 2 different datasets, the model that leverages feature maps sharing from RGB and DVS encoders is performing better. Furthermore, we show that our model can inference faster and have a comparable performance against another recent model. Official implementation code is shared at https://github.com/oskarnatan/RGBDVS-fusion. |
| first_indexed | 2024-03-14T00:09:43Z |
| format | Other |
| id | oai:generic.eprints.org:284221 |
| institution | Universiti Gadjah Mada |
| last_indexed | 2024-03-14T00:09:43Z |
| publishDate | 2022 |
| publisher | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
| record_format | dspace |
| spelling | oai:generic.eprints.org:2842212023-12-05T07:36:23Z https://repository.ugm.ac.id/284221/ Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception Natan, Oskar Miura, Jun Artificial Intelligence and Image Processing not elsewhere classified In this research, we present a novel deep multi-task learning model to handle the perception stage of an autonomous driving system. The model leverages the fusion of RGB and dynamic vision sensor (DVS) images to perform semantic segmentation and depth estimation in four different perspectives of view simultaneously. As for the experiment, CARLA simulator is used to generate thousands of simulation data for training, validation, and testing processes. A dynamically changing environment with various weather conditions, daytime, maps, and non-player characters (NPC) is also considered to simulate a more realistic condition with expecting a better generalization of the model. An ablation study is conducted by modifying the network architecture to evaluate the influence of the sensor fusion technique. Based on the test result on 2 different datasets, the model that leverages feature maps sharing from RGB and DVS encoders is performing better. Furthermore, we show that our model can inference faster and have a comparable performance against another recent model. Official implementation code is shared at https://github.com/oskarnatan/RGBDVS-fusion. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2022 Other NonPeerReviewed Natan, Oskar and Miura, Jun (2022) Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). https://dl.acm.org/doi/10.1007/978-3-031-02375-0_26 10.1007/978-3-031-02375-0_26 |
| spellingShingle | Artificial Intelligence and Image Processing not elsewhere classified Natan, Oskar Miura, Jun Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception |
| title | Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception |
| title_full | Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception |
| title_fullStr | Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception |
| title_full_unstemmed | Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception |
| title_short | Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception |
| title_sort | semantic segmentation and depth estimation with rgb and dvs sensor fusion for multi view driving perception |
| topic | Artificial Intelligence and Image Processing not elsewhere classified |
| work_keys_str_mv | AT natanoskar semanticsegmentationanddepthestimationwithrgbanddvssensorfusionformultiviewdrivingperception AT miurajun semanticsegmentationanddepthestimationwithrgbanddvssensorfusionformultiviewdrivingperception |