Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary
As the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, AR/VR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress h...
Main Authors: | , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2020-01-01
|
Series: | Sensors |
Subjects: | |
Online Access: | https://www.mdpi.com/1424-8220/20/3/635 |
_version_ | 1798039538884411392 |
---|---|
author | Nan Zou Zhiyu Xiang Yiman Chen Shuya Chen Chengyu Qiao |
author_facet | Nan Zou Zhiyu Xiang Yiman Chen Shuya Chen Chengyu Qiao |
author_sort | Nan Zou |
collection | DOAJ |
description | As the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, AR/VR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress has been made in both tasks through deep learning technologies, few works have been done on building a joint model by deeply exploring the inner relationship of the above tasks. In this paper, semantic segmentation and depth completion are jointly considered under a multi-task learning framework. By sharing a common encoder part and introducing boundary features as inner constraints in the decoder part, the two tasks can properly share the required information from each other. An extra boundary detection sub-task is responsible for providing the boundary features and constructing cross-task joint loss functions for network training. The entire network is implemented end-to-end and evaluated with both RGB and sparse depth input. Experiments conducted on synthesized and real scene datasets show that our proposed multi-task CNN model can effectively improve the performance of every single task. |
first_indexed | 2024-04-11T21:55:16Z |
format | Article |
id | doaj.art-da97cc8ba0324d18848d0e3b7132bdb0 |
institution | Directory Open Access Journal |
issn | 1424-8220 |
language | English |
last_indexed | 2024-04-11T21:55:16Z |
publishDate | 2020-01-01 |
publisher | MDPI AG |
record_format | Article |
series | Sensors |
spelling | doaj.art-da97cc8ba0324d18848d0e3b7132bdb02022-12-22T04:01:07ZengMDPI AGSensors1424-82202020-01-0120363510.3390/s20030635s20030635Simultaneous Semantic Segmentation and Depth Completion with Constraint of BoundaryNan Zou0Zhiyu Xiang1Yiman Chen2Shuya Chen3Chengyu Qiao4College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaZhejiang Provincial Key Laboratory of Information Processing, Communication and Networking, Zhejiang University, Hangzhou 310000, ChinaCollege of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaCollege of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaCollege of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaAs the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, AR/VR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress has been made in both tasks through deep learning technologies, few works have been done on building a joint model by deeply exploring the inner relationship of the above tasks. In this paper, semantic segmentation and depth completion are jointly considered under a multi-task learning framework. By sharing a common encoder part and introducing boundary features as inner constraints in the decoder part, the two tasks can properly share the required information from each other. An extra boundary detection sub-task is responsible for providing the boundary features and constructing cross-task joint loss functions for network training. The entire network is implemented end-to-end and evaluated with both RGB and sparse depth input. Experiments conducted on synthesized and real scene datasets show that our proposed multi-task CNN model can effectively improve the performance of every single task.https://www.mdpi.com/1424-8220/20/3/635cnnsemantic segmentationdepth completionmulti-task learning |
spellingShingle | Nan Zou Zhiyu Xiang Yiman Chen Shuya Chen Chengyu Qiao Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary Sensors cnn semantic segmentation depth completion multi-task learning |
title | Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary |
title_full | Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary |
title_fullStr | Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary |
title_full_unstemmed | Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary |
title_short | Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary |
title_sort | simultaneous semantic segmentation and depth completion with constraint of boundary |
topic | cnn semantic segmentation depth completion multi-task learning |
url | https://www.mdpi.com/1424-8220/20/3/635 |
work_keys_str_mv | AT nanzou simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary AT zhiyuxiang simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary AT yimanchen simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary AT shuyachen simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary AT chengyuqiao simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary |