Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary

As the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, AR/VR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress h...

Full description

Bibliographic Details
Main Authors: Nan Zou, Zhiyu Xiang, Yiman Chen, Shuya Chen, Chengyu Qiao
Format: Article
Language:English
Published: MDPI AG 2020-01-01
Series:Sensors
Subjects:
Online Access:https://www.mdpi.com/1424-8220/20/3/635
_version_ 1798039538884411392
author Nan Zou
Zhiyu Xiang
Yiman Chen
Shuya Chen
Chengyu Qiao
author_facet Nan Zou
Zhiyu Xiang
Yiman Chen
Shuya Chen
Chengyu Qiao
author_sort Nan Zou
collection DOAJ
description As the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, AR/VR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress has been made in both tasks through deep learning technologies, few works have been done on building a joint model by deeply exploring the inner relationship of the above tasks. In this paper, semantic segmentation and depth completion are jointly considered under a multi-task learning framework. By sharing a common encoder part and introducing boundary features as inner constraints in the decoder part, the two tasks can properly share the required information from each other. An extra boundary detection sub-task is responsible for providing the boundary features and constructing cross-task joint loss functions for network training. The entire network is implemented end-to-end and evaluated with both RGB and sparse depth input. Experiments conducted on synthesized and real scene datasets show that our proposed multi-task CNN model can effectively improve the performance of every single task.
first_indexed 2024-04-11T21:55:16Z
format Article
id doaj.art-da97cc8ba0324d18848d0e3b7132bdb0
institution Directory Open Access Journal
issn 1424-8220
language English
last_indexed 2024-04-11T21:55:16Z
publishDate 2020-01-01
publisher MDPI AG
record_format Article
series Sensors
spelling doaj.art-da97cc8ba0324d18848d0e3b7132bdb02022-12-22T04:01:07ZengMDPI AGSensors1424-82202020-01-0120363510.3390/s20030635s20030635Simultaneous Semantic Segmentation and Depth Completion with Constraint of BoundaryNan Zou0Zhiyu Xiang1Yiman Chen2Shuya Chen3Chengyu Qiao4College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaZhejiang Provincial Key Laboratory of Information Processing, Communication and Networking, Zhejiang University, Hangzhou 310000, ChinaCollege of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaCollege of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaCollege of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, ChinaAs the core task of scene understanding, semantic segmentation and depth completion play a vital role in lots of applications such as robot navigation, AR/VR and autonomous driving. They are responsible for parsing scenes from the angle of semantics and geometry, respectively. While great progress has been made in both tasks through deep learning technologies, few works have been done on building a joint model by deeply exploring the inner relationship of the above tasks. In this paper, semantic segmentation and depth completion are jointly considered under a multi-task learning framework. By sharing a common encoder part and introducing boundary features as inner constraints in the decoder part, the two tasks can properly share the required information from each other. An extra boundary detection sub-task is responsible for providing the boundary features and constructing cross-task joint loss functions for network training. The entire network is implemented end-to-end and evaluated with both RGB and sparse depth input. Experiments conducted on synthesized and real scene datasets show that our proposed multi-task CNN model can effectively improve the performance of every single task.https://www.mdpi.com/1424-8220/20/3/635cnnsemantic segmentationdepth completionmulti-task learning
spellingShingle Nan Zou
Zhiyu Xiang
Yiman Chen
Shuya Chen
Chengyu Qiao
Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary
Sensors
cnn
semantic segmentation
depth completion
multi-task learning
title Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary
title_full Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary
title_fullStr Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary
title_full_unstemmed Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary
title_short Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary
title_sort simultaneous semantic segmentation and depth completion with constraint of boundary
topic cnn
semantic segmentation
depth completion
multi-task learning
url https://www.mdpi.com/1424-8220/20/3/635
work_keys_str_mv AT nanzou simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary
AT zhiyuxiang simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary
AT yimanchen simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary
AT shuyachen simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary
AT chengyuqiao simultaneoussemanticsegmentationanddepthcompletionwithconstraintofboundary