SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion
Depth completion involves recovering a dense depth map from a sparse map and an RGB image. Recent approaches focus on utilizing color images as guidance images to recover depth at invalid pixels. However, color images alone are not enough to provide the necessary semantic understanding of the scene....
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9918022/ |
| _version_ | 1828096951725326336 |
|---|---|
| author | Danish Nazir Alain Pagani Marcus Liwicki Didier Stricker Muhammad Zeshan Afzal |
| author_facet | Danish Nazir Alain Pagani Marcus Liwicki Didier Stricker Muhammad Zeshan Afzal |
| author_sort | Danish Nazir |
| collection | DOAJ |
| description | Depth completion involves recovering a dense depth map from a sparse map and an RGB image. Recent approaches focus on utilizing color images as guidance images to recover depth at invalid pixels. However, color images alone are not enough to provide the necessary semantic understanding of the scene. Consequently, the depth completion task suffers from sudden illumination changes in RGB images (e.g., shadows). In this paper, we propose a novel three-branch backbone comprising color-guided, semantic-guided, and depth-guided branches. Specifically, the color-guided branch takes a sparse depth map and RGB image as an input and generates color depth which includes color cues (e.g., object boundaries) of the scene. The predicted dense depth map of color-guided branch along-with semantic image and sparse depth map is passed as input to semantic-guided branch for estimating semantic depth. The depth-guided branch takes sparse, color, and semantic depths to generate the dense depth map. The color depth, semantic depth, and guided depth are adaptively fused to produce the output of our proposed three-branch backbone. In addition, we also propose to apply semantic-aware multi-modal attention-based fusion block (SAMMAFB) to fuse features between all three branches. We further use CSPN++ with Atrous convolutions to refine the dense depth map produced by our three-branch backbone. Extensive experiments show that our model achieves state-of-the-art performance in the KITTI depth completion benchmark at the time of submission. |
| first_indexed | 2024-04-11T07:38:25Z |
| format | Article |
| id | doaj.art-4cdc388f284b4efb88c90b9fb2cd5870 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-11T07:38:25Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-4cdc388f284b4efb88c90b9fb2cd58702022-12-22T04:36:39ZengIEEEIEEE Access2169-35362022-01-011012078112079110.1109/ACCESS.2022.32143169918022SemAttNet: Toward Attention-Based Semantic Aware Guided Depth CompletionDanish Nazir0https://orcid.org/0000-0001-6364-8427Alain Pagani1Marcus Liwicki2https://orcid.org/0000-0003-4029-6574Didier Stricker3Muhammad Zeshan Afzal4https://orcid.org/0000-0002-0536-6867Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, GermanyDeutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, GermanyDepartment of Computer Science, Luleå University of Technology, Luleå, SwedenDeutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, GermanyDeutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, GermanyDepth completion involves recovering a dense depth map from a sparse map and an RGB image. Recent approaches focus on utilizing color images as guidance images to recover depth at invalid pixels. However, color images alone are not enough to provide the necessary semantic understanding of the scene. Consequently, the depth completion task suffers from sudden illumination changes in RGB images (e.g., shadows). In this paper, we propose a novel three-branch backbone comprising color-guided, semantic-guided, and depth-guided branches. Specifically, the color-guided branch takes a sparse depth map and RGB image as an input and generates color depth which includes color cues (e.g., object boundaries) of the scene. The predicted dense depth map of color-guided branch along-with semantic image and sparse depth map is passed as input to semantic-guided branch for estimating semantic depth. The depth-guided branch takes sparse, color, and semantic depths to generate the dense depth map. The color depth, semantic depth, and guided depth are adaptively fused to produce the output of our proposed three-branch backbone. In addition, we also propose to apply semantic-aware multi-modal attention-based fusion block (SAMMAFB) to fuse features between all three branches. We further use CSPN++ with Atrous convolutions to refine the dense depth map produced by our three-branch backbone. Extensive experiments show that our model achieves state-of-the-art performance in the KITTI depth completion benchmark at the time of submission.https://ieeexplore.ieee.org/document/9918022/State-of-the-art depth completion approach on KITTI depth completion benchmarkattention-based fusion for depth completionsemantic-guided depth completion |
| spellingShingle | Danish Nazir Alain Pagani Marcus Liwicki Didier Stricker Muhammad Zeshan Afzal SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion IEEE Access State-of-the-art depth completion approach on KITTI depth completion benchmark attention-based fusion for depth completion semantic-guided depth completion |
| title | SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion |
| title_full | SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion |
| title_fullStr | SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion |
| title_full_unstemmed | SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion |
| title_short | SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion |
| title_sort | semattnet toward attention based semantic aware guided depth completion |
| topic | State-of-the-art depth completion approach on KITTI depth completion benchmark attention-based fusion for depth completion semantic-guided depth completion |
| url | https://ieeexplore.ieee.org/document/9918022/ |
| work_keys_str_mv | AT danishnazir semattnettowardattentionbasedsemanticawareguideddepthcompletion AT alainpagani semattnettowardattentionbasedsemanticawareguideddepthcompletion AT marcusliwicki semattnettowardattentionbasedsemanticawareguideddepthcompletion AT didierstricker semattnettowardattentionbasedsemanticawareguideddepthcompletion AT muhammadzeshanafzal semattnettowardattentionbasedsemanticawareguideddepthcompletion |