Learning Dense Consistent Features for Aerial-to-Ground Structure-From-Motion
The integration of aerial and ground images is known to be effective for enhancing the quality of 3-D reconstruction in complex urban scenarios. However, directly applying the structure-from-motion (SfM) technique for unified 3-D reconstruction with aerial and ground images is particularly difficult...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10131998/ |
| _version_ | 1797804907889164288 |
|---|---|
| author | Hongjie Li Aonan Liu Xiao Xie Han Guo Hanjiang Xiong Xianwei Zheng |
| author_facet | Hongjie Li Aonan Liu Xiao Xie Han Guo Hanjiang Xiong Xianwei Zheng |
| author_sort | Hongjie Li |
| collection | DOAJ |
| description | The integration of aerial and ground images is known to be effective for enhancing the quality of 3-D reconstruction in complex urban scenarios. However, directly applying the structure-from-motion (SfM) technique for unified 3-D reconstruction with aerial and ground images is particularly difficult, due to the large differences in viewpoint, scale, and appearance between those two types of images. Previous studies mainly rely on viewpoint rectification or view rendering/synthesis to improve the feature matching quality for aligning the aerial and ground models. Nevertheless, these approaches still fail to address the inherent information differences between aerial and ground images. In this article, we propose a learning-based matching framework for direct SfM with ground and aerial images. The key idea of our method is to learn the pixel-wise consistent features between aerial and ground images to handle the large heterogeneity of these two types of images. Specifically, we deploy a learning-based matching framework to robustly correspond the aerial and ground images. With the high-quality feature matching, learned feature maps are used for refining keypoint locations and fusing featuremetric error into bundle adjustment with the consideration of geometric error, both of which can further improve the accuracy and completeness of the recovered 3-D scene. Extensive experiments conducted on six datasets demonstrate that the proposed method can reconstruct high-fidelity 3-D models with direct aerial-to-ground SfM, which cannot be achieved by existing methods. In addition, our method also shows outstanding performance in subtasks of feature matching and point cloud recovery. |
| first_indexed | 2024-03-13T05:44:07Z |
| format | Article |
| id | doaj.art-c0f6d23f247c493c8e47787d155a4a66 |
| institution | Directory Open Access Journal |
| issn | 2151-1535 |
| language | English |
| last_indexed | 2024-03-13T05:44:07Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj.art-c0f6d23f247c493c8e47787d155a4a662023-06-13T23:00:20ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing2151-15352023-01-01165089510210.1109/JSTARS.2023.327919910131998Learning Dense Consistent Features for Aerial-to-Ground Structure-From-MotionHongjie Li0https://orcid.org/0009-0005-0259-7984Aonan Liu1Xiao Xie2https://orcid.org/0000-0002-2598-0047Han Guo3https://orcid.org/0009-0008-1006-4928Hanjiang Xiong4https://orcid.org/0000-0003-3949-6368Xianwei Zheng5https://orcid.org/0000-0001-9783-3030Key Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources, Shenzhen and the State Key Laboratory LIESMARS, Wuhan University, Wuhan, ChinaState Key Laboratory LIESMARS, Wuhan University, Wuhan, ChinaKey Laboratory for Environmental Computation and Sustainability of Liaoning Province, Institute of Applied Ecology, Chinese Academy of Sciences, Shenyang, ChinaKey Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources, Shenzhen, ChinaState Key Laboratory LIESMARS, Wuhan University, Wuhan, ChinaKey Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources, Shenzhen and the State Key Laboratory LIESMARS, Wuhan University, Wuhan, ChinaThe integration of aerial and ground images is known to be effective for enhancing the quality of 3-D reconstruction in complex urban scenarios. However, directly applying the structure-from-motion (SfM) technique for unified 3-D reconstruction with aerial and ground images is particularly difficult, due to the large differences in viewpoint, scale, and appearance between those two types of images. Previous studies mainly rely on viewpoint rectification or view rendering/synthesis to improve the feature matching quality for aligning the aerial and ground models. Nevertheless, these approaches still fail to address the inherent information differences between aerial and ground images. In this article, we propose a learning-based matching framework for direct SfM with ground and aerial images. The key idea of our method is to learn the pixel-wise consistent features between aerial and ground images to handle the large heterogeneity of these two types of images. Specifically, we deploy a learning-based matching framework to robustly correspond the aerial and ground images. With the high-quality feature matching, learned feature maps are used for refining keypoint locations and fusing featuremetric error into bundle adjustment with the consideration of geometric error, both of which can further improve the accuracy and completeness of the recovered 3-D scene. Extensive experiments conducted on six datasets demonstrate that the proposed method can reconstruct high-fidelity 3-D models with direct aerial-to-ground SfM, which cannot be achieved by existing methods. In addition, our method also shows outstanding performance in subtasks of feature matching and point cloud recovery.https://ieeexplore.ieee.org/document/10131998/Aerial-ground integrationdense consistent featuresfeature map-based bundle adjustment (BA)keypoint location refinement (LR)structure-from-motion (SfM) |
| spellingShingle | Hongjie Li Aonan Liu Xiao Xie Han Guo Hanjiang Xiong Xianwei Zheng Learning Dense Consistent Features for Aerial-to-Ground Structure-From-Motion IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Aerial-ground integration dense consistent features feature map-based bundle adjustment (BA) keypoint location refinement (LR) structure-from-motion (SfM) |
| title | Learning Dense Consistent Features for Aerial-to-Ground Structure-From-Motion |
| title_full | Learning Dense Consistent Features for Aerial-to-Ground Structure-From-Motion |
| title_fullStr | Learning Dense Consistent Features for Aerial-to-Ground Structure-From-Motion |
| title_full_unstemmed | Learning Dense Consistent Features for Aerial-to-Ground Structure-From-Motion |
| title_short | Learning Dense Consistent Features for Aerial-to-Ground Structure-From-Motion |
| title_sort | learning dense consistent features for aerial to ground structure from motion |
| topic | Aerial-ground integration dense consistent features feature map-based bundle adjustment (BA) keypoint location refinement (LR) structure-from-motion (SfM) |
| url | https://ieeexplore.ieee.org/document/10131998/ |
| work_keys_str_mv | AT hongjieli learningdenseconsistentfeaturesforaerialtogroundstructurefrommotion AT aonanliu learningdenseconsistentfeaturesforaerialtogroundstructurefrommotion AT xiaoxie learningdenseconsistentfeaturesforaerialtogroundstructurefrommotion AT hanguo learningdenseconsistentfeaturesforaerialtogroundstructurefrommotion AT hanjiangxiong learningdenseconsistentfeaturesforaerialtogroundstructurefrommotion AT xianweizheng learningdenseconsistentfeaturesforaerialtogroundstructurefrommotion |