Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas
Conventional multi-view stereo (MVS) approaches based on photo-consistency measures are generally robust, yet often fail in calculating valid depth pixel estimates in low textured areas of the scene. In this study, a novel approach is proposed to tackle this challenge by leveraging semantic priors i...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-03-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/13/6/1053 |
| _version_ | 1797541802816831488 |
|---|---|
| author | Elisavet Konstantina Stathopoulou Roberto Battisti Dan Cernea Fabio Remondino Andreas Georgopoulos |
| author_facet | Elisavet Konstantina Stathopoulou Roberto Battisti Dan Cernea Fabio Remondino Andreas Georgopoulos |
| author_sort | Elisavet Konstantina Stathopoulou |
| collection | DOAJ |
| description | Conventional multi-view stereo (MVS) approaches based on photo-consistency measures are generally robust, yet often fail in calculating valid depth pixel estimates in low textured areas of the scene. In this study, a novel approach is proposed to tackle this challenge by leveraging semantic priors into a PatchMatch-based MVS in order to increase confidence and support depth and normal map estimation. Semantic class labels on image pixels are used to impose class-specific geometric constraints during multiview stereo, optimising the depth estimation on weakly supported, textureless areas, commonly present in urban scenarios of building facades, indoor scenes, or aerial datasets. Detecting dominant shapes, e.g., planes, with RANSAC, an adjusted cost function is introduced that combines and weighs both photometric and semantic scores propagating, thus, more accurate depth estimates. Being adaptive, it fills in apparent information gaps and smoothing local roughness in problematic regions while at the same time preserves important details. Experiments on benchmark and custom datasets demonstrate the effectiveness of the presented approach. |
| first_indexed | 2024-03-10T13:21:53Z |
| format | Article |
| id | doaj.art-064071eda2734ce99ec12d32cb15bc3a |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T13:21:53Z |
| publishDate | 2021-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-064071eda2734ce99ec12d32cb15bc3a2023-11-21T09:57:05ZengMDPI AGRemote Sensing2072-42922021-03-01136105310.3390/rs13061053Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless AreasElisavet Konstantina Stathopoulou0Roberto Battisti1Dan Cernea2Fabio Remondino3Andreas Georgopoulos43D Optical Metrology (3DOM) Unit, Bruno Kessler Foundation (FBK), 38123 Trento, Italy3D Optical Metrology (3DOM) Unit, Bruno Kessler Foundation (FBK), 38123 Trento, ItalySeaCave, 010747 Bucharest, Romania3D Optical Metrology (3DOM) Unit, Bruno Kessler Foundation (FBK), 38123 Trento, ItalyLaboratory of Photogrammetry, School of Rural and Surveying Engineering, National Technical University of Athens, 15780 Athens, GreeceConventional multi-view stereo (MVS) approaches based on photo-consistency measures are generally robust, yet often fail in calculating valid depth pixel estimates in low textured areas of the scene. In this study, a novel approach is proposed to tackle this challenge by leveraging semantic priors into a PatchMatch-based MVS in order to increase confidence and support depth and normal map estimation. Semantic class labels on image pixels are used to impose class-specific geometric constraints during multiview stereo, optimising the depth estimation on weakly supported, textureless areas, commonly present in urban scenarios of building facades, indoor scenes, or aerial datasets. Detecting dominant shapes, e.g., planes, with RANSAC, an adjusted cost function is introduced that combines and weighs both photometric and semantic scores propagating, thus, more accurate depth estimates. Being adaptive, it fills in apparent information gaps and smoothing local roughness in problematic regions while at the same time preserves important details. Experiments on benchmark and custom datasets demonstrate the effectiveness of the presented approach.https://www.mdpi.com/2072-4292/13/6/1053multi view stereo (MVS)PatchMatchdepth estimationdense point cloud3D reconstructionsemantic segmentation |
| spellingShingle | Elisavet Konstantina Stathopoulou Roberto Battisti Dan Cernea Fabio Remondino Andreas Georgopoulos Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas Remote Sensing multi view stereo (MVS) PatchMatch depth estimation dense point cloud 3D reconstruction semantic segmentation |
| title | Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas |
| title_full | Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas |
| title_fullStr | Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas |
| title_full_unstemmed | Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas |
| title_short | Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas |
| title_sort | semantically derived geometric constraints for mvs reconstruction of textureless areas |
| topic | multi view stereo (MVS) PatchMatch depth estimation dense point cloud 3D reconstruction semantic segmentation |
| url | https://www.mdpi.com/2072-4292/13/6/1053 |
| work_keys_str_mv | AT elisavetkonstantinastathopoulou semanticallyderivedgeometricconstraintsformvsreconstructionoftexturelessareas AT robertobattisti semanticallyderivedgeometricconstraintsformvsreconstructionoftexturelessareas AT dancernea semanticallyderivedgeometricconstraintsformvsreconstructionoftexturelessareas AT fabioremondino semanticallyderivedgeometricconstraintsformvsreconstructionoftexturelessareas AT andreasgeorgopoulos semanticallyderivedgeometricconstraintsformvsreconstructionoftexturelessareas |