FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor Motion
Real-time 3D scene reconstruction has attracted a great amount of attention in the fields of augmented reality, virtual reality and robotics. Previous works usually assumed slow sensor motions to avoid large interframe differences and strong image blur, but this limits the applicability of the techn...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-07-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/14/15/3551 |
| _version_ | 1797412497030905856 |
|---|---|
| author | Zunjie Zhu Zhefeng Xu Ruolin Chen Tingyu Wang Can Wang Chenggang Yan Feng Xu |
| author_facet | Zunjie Zhu Zhefeng Xu Ruolin Chen Tingyu Wang Can Wang Chenggang Yan Feng Xu |
| author_sort | Zunjie Zhu |
| collection | DOAJ |
| description | Real-time 3D scene reconstruction has attracted a great amount of attention in the fields of augmented reality, virtual reality and robotics. Previous works usually assumed slow sensor motions to avoid large interframe differences and strong image blur, but this limits the applicability of the techniques in real cases. In this study, we propose an end-to-end 3D reconstruction system that combines color, depth and inertial measurements to achieve a robust reconstruction with fast sensor motions. We involved an extended Kalman filter (EKF) to fuse RGB-D-IMU data and jointly optimize feature correspondences, camera poses and scene geometry by using an iterative method. A novel geometry-aware patch deformation technique is proposed to adapt the changes in patch features in the image domain, leading to highly accurate feature tracking with fast sensor motions. In addition, we maintained the global consistency of the reconstructed model by achieving loop closure with submap-based depth image encoding and 3D map deformation. The experiments revealed that our patch deformation method improves the accuracy of feature tracking, that our improved loop detection method is more efficient than the original method and that our system possesses superior 3D reconstruction results compared with the state-of-the-art solutions in handling fast camera motions. |
| first_indexed | 2024-03-09T05:03:27Z |
| format | Article |
| id | doaj.art-95b9e269bd1a4f1393faa58fe201c7c2 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-09T05:03:27Z |
| publishDate | 2022-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-95b9e269bd1a4f1393faa58fe201c7c22023-12-03T12:57:43ZengMDPI AGRemote Sensing2072-42922022-07-011415355110.3390/rs14153551FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor MotionZunjie Zhu0Zhefeng Xu1Ruolin Chen2Tingyu Wang3Can Wang4Chenggang Yan5Feng Xu6School of Communication Engineering, Hangzhou Dianzi University, Hangzhou 310018, ChinaLishui Institute of Hangzhou Dianzi University, Lishui 323010, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaLinx Robot Company, Hangzhou 311100, ChinaSchool of Automation, Hangzhou Dianzi University, Hangzhou 310018, ChinaBNRist and School of Software, Tsinghua University, Beijing 100084, ChinaReal-time 3D scene reconstruction has attracted a great amount of attention in the fields of augmented reality, virtual reality and robotics. Previous works usually assumed slow sensor motions to avoid large interframe differences and strong image blur, but this limits the applicability of the techniques in real cases. In this study, we propose an end-to-end 3D reconstruction system that combines color, depth and inertial measurements to achieve a robust reconstruction with fast sensor motions. We involved an extended Kalman filter (EKF) to fuse RGB-D-IMU data and jointly optimize feature correspondences, camera poses and scene geometry by using an iterative method. A novel geometry-aware patch deformation technique is proposed to adapt the changes in patch features in the image domain, leading to highly accurate feature tracking with fast sensor motions. In addition, we maintained the global consistency of the reconstructed model by achieving loop closure with submap-based depth image encoding and 3D map deformation. The experiments revealed that our patch deformation method improves the accuracy of feature tracking, that our improved loop detection method is more efficient than the original method and that our system possesses superior 3D reconstruction results compared with the state-of-the-art solutions in handling fast camera motions.https://www.mdpi.com/2072-4292/14/15/35513D reconstructionfast motionEKFIMUloop closure |
| spellingShingle | Zunjie Zhu Zhefeng Xu Ruolin Chen Tingyu Wang Can Wang Chenggang Yan Feng Xu FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor Motion Remote Sensing 3D reconstruction fast motion EKF IMU loop closure |
| title | FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor Motion |
| title_full | FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor Motion |
| title_fullStr | FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor Motion |
| title_full_unstemmed | FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor Motion |
| title_short | FastFusion: Real-Time Indoor Scene Reconstruction with Fast Sensor Motion |
| title_sort | fastfusion real time indoor scene reconstruction with fast sensor motion |
| topic | 3D reconstruction fast motion EKF IMU loop closure |
| url | https://www.mdpi.com/2072-4292/14/15/3551 |
| work_keys_str_mv | AT zunjiezhu fastfusionrealtimeindoorscenereconstructionwithfastsensormotion AT zhefengxu fastfusionrealtimeindoorscenereconstructionwithfastsensormotion AT ruolinchen fastfusionrealtimeindoorscenereconstructionwithfastsensormotion AT tingyuwang fastfusionrealtimeindoorscenereconstructionwithfastsensormotion AT canwang fastfusionrealtimeindoorscenereconstructionwithfastsensormotion AT chenggangyan fastfusionrealtimeindoorscenereconstructionwithfastsensormotion AT fengxu fastfusionrealtimeindoorscenereconstructionwithfastsensormotion |