Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion
The recently proposed spacecraft three-dimensional (3D) structure recovery method based on optical images and LIDAR has enhanced the working distance of a spacecraft’s 3D perception system. However, the existing methods ignore the richness of temporal features and fail to capture the temporal cohere...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/19/4786 |
| _version_ | 1827722303697321984 |
|---|---|
| author | Xiang Liu Hongyuan Wang Xinlong Chen Weichun Chen Zhengyou Xie |
| author_facet | Xiang Liu Hongyuan Wang Xinlong Chen Weichun Chen Zhengyou Xie |
| author_sort | Xiang Liu |
| collection | DOAJ |
| description | The recently proposed spacecraft three-dimensional (3D) structure recovery method based on optical images and LIDAR has enhanced the working distance of a spacecraft’s 3D perception system. However, the existing methods ignore the richness of temporal features and fail to capture the temporal coherence of consecutive frames. This paper proposes a sequential spacecraft depth completion network (S2DCNet) for generating accurate and temporally consistent depth prediction results, and it can fully exploit temporal–spatial coherence in sequential frames. Specifically, two parallel convolution neural network (CNN) branches were first adopted to extract the features latent in different inputs. The gray image features and the depth features were hierarchically encapsulated into unified feature representations through fusion modules. In the decoding stage, the convolutional long short-term memory (ConvLSTM) networks were embedded with the multi-scale scheme to capture the feature spatial–temporal distribution variation, which could reflect the past state and generate more accurate and temporally consistent depth maps. In addition, a large-scale dataset was constructed, and the experiments revealed the outstanding performance of the proposed S2DCNet, achieving a mean absolute error of 0.192 m within the region of interest. |
| first_indexed | 2024-03-10T21:36:10Z |
| format | Article |
| id | doaj.art-cba0e9e1b5ff4315a7367388932985d0 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T21:36:10Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-cba0e9e1b5ff4315a7367388932985d02023-11-19T14:59:56ZengMDPI AGRemote Sensing2072-42922023-09-011519478610.3390/rs15194786Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth CompletionXiang Liu0Hongyuan Wang1Xinlong Chen2Weichun Chen3Zhengyou Xie4School of Astronautics, Harbin Institute of Technology, Harbin 150001, ChinaSchool of Astronautics, Harbin Institute of Technology, Harbin 150001, ChinaQian Xuesen Laboratory of Space Technology, China Academy of Space Technology, Beijing 100080, ChinaQian Xuesen Laboratory of Space Technology, China Academy of Space Technology, Beijing 100080, ChinaQian Xuesen Laboratory of Space Technology, China Academy of Space Technology, Beijing 100080, ChinaThe recently proposed spacecraft three-dimensional (3D) structure recovery method based on optical images and LIDAR has enhanced the working distance of a spacecraft’s 3D perception system. However, the existing methods ignore the richness of temporal features and fail to capture the temporal coherence of consecutive frames. This paper proposes a sequential spacecraft depth completion network (S2DCNet) for generating accurate and temporally consistent depth prediction results, and it can fully exploit temporal–spatial coherence in sequential frames. Specifically, two parallel convolution neural network (CNN) branches were first adopted to extract the features latent in different inputs. The gray image features and the depth features were hierarchically encapsulated into unified feature representations through fusion modules. In the decoding stage, the convolutional long short-term memory (ConvLSTM) networks were embedded with the multi-scale scheme to capture the feature spatial–temporal distribution variation, which could reflect the past state and generate more accurate and temporally consistent depth maps. In addition, a large-scale dataset was constructed, and the experiments revealed the outstanding performance of the proposed S2DCNet, achieving a mean absolute error of 0.192 m within the region of interest.https://www.mdpi.com/2072-4292/15/19/4786depth completionsequential depth completionmulti-modal fusionConvLSTMsatellite datasetdeep learning |
| spellingShingle | Xiang Liu Hongyuan Wang Xinlong Chen Weichun Chen Zhengyou Xie Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion Remote Sensing depth completion sequential depth completion multi-modal fusion ConvLSTM satellite dataset deep learning |
| title | Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion |
| title_full | Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion |
| title_fullStr | Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion |
| title_full_unstemmed | Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion |
| title_short | Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion |
| title_sort | exploiting temporal spatial feature correlations for sequential spacecraft depth completion |
| topic | depth completion sequential depth completion multi-modal fusion ConvLSTM satellite dataset deep learning |
| url | https://www.mdpi.com/2072-4292/15/19/4786 |
| work_keys_str_mv | AT xiangliu exploitingtemporalspatialfeaturecorrelationsforsequentialspacecraftdepthcompletion AT hongyuanwang exploitingtemporalspatialfeaturecorrelationsforsequentialspacecraftdepthcompletion AT xinlongchen exploitingtemporalspatialfeaturecorrelationsforsequentialspacecraftdepthcompletion AT weichunchen exploitingtemporalspatialfeaturecorrelationsforsequentialspacecraftdepthcompletion AT zhengyouxie exploitingtemporalspatialfeaturecorrelationsforsequentialspacecraftdepthcompletion |