Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera
© 2019 IEEE. Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020
|
| Online Access: | https://hdl.handle.net/1721.1/126545 |
| _version_ | 1826202847775031296 |
|---|---|
| author | Ma, Fangchang Venturelli Cavalheiro, Guilherme. Karaman, Sertac |
| author2 | Massachusetts Institute of Technology. Department of Aeronautics and Astronautics |
| author_facet | Massachusetts Institute of Technology. Department of Aeronautics and Astronautics Ma, Fangchang Venturelli Cavalheiro, Guilherme. Karaman, Sertac |
| author_sort | Ma, Fangchang |
| collection | MIT |
| description | © 2019 IEEE. Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels for training. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) input to dense depth prediction. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that the self-supervised framework outperforms a number of existing solutions trained with semi-dense annotations. Furthermore, when trained with semi-dense annotations, our network attains state-of-the-art accuracy and is the winning approach on the KITTI depth completion benchmark² at the time of submission. Furthermore, the self-supervised framework outperforms a number of existing solutions trained with semi-dense annotations. |
| first_indexed | 2024-09-23T12:21:52Z |
| format | Article |
| id | mit-1721.1/126545 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T12:21:52Z |
| publishDate | 2020 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1265452022-09-28T07:52:47Z Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera Ma, Fangchang Venturelli Cavalheiro, Guilherme. Karaman, Sertac Massachusetts Institute of Technology. Department of Aeronautics and Astronautics © 2019 IEEE. Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels for training. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) input to dense depth prediction. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that the self-supervised framework outperforms a number of existing solutions trained with semi-dense annotations. Furthermore, when trained with semi-dense annotations, our network attains state-of-the-art accuracy and is the winning approach on the KITTI depth completion benchmark² at the time of submission. Furthermore, the self-supervised framework outperforms a number of existing solutions trained with semi-dense annotations. United States. Office of Naval Research (Grant N00014-17-1-2670) 2020-08-12T17:05:50Z 2020-08-12T17:05:50Z 2019-05 2019-10-29T16:03:10Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/126545 Ma, Fangchang, Guilherme Venturelli Cavalheiro and Sertac Karaman. “Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera.” Paper presented at the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 20-24 May 2019, IEEE © 2019 The Author(s) en 10.1109/ICRA.2019.8793637 2019 International Conference on Robotics and Automation (ICRA) Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf IEEE arXiv |
| spellingShingle | Ma, Fangchang Venturelli Cavalheiro, Guilherme. Karaman, Sertac Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera |
| title | Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera |
| title_full | Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera |
| title_fullStr | Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera |
| title_full_unstemmed | Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera |
| title_short | Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera |
| title_sort | self supervised sparse to dense self supervised depth completion from lidar and monocular camera |
| url | https://hdl.handle.net/1721.1/126545 |
| work_keys_str_mv | AT mafangchang selfsupervisedsparsetodenseselfsuperviseddepthcompletionfromlidarandmonocularcamera AT venturellicavalheiroguilherme selfsupervisedsparsetodenseselfsuperviseddepthcompletionfromlidarandmonocularcamera AT karamansertac selfsupervisedsparsetodenseselfsuperviseddepthcompletionfromlidarandmonocularcamera |