Unified shape and appearance reconstruction with joint camera parameter refinement
In this paper, we present an inverse rendering method for the simple reconstruction of shape and appearance of real-world objects from only roughly calibrated RGB images captured under collocated point light illumination. To this end, we gradually reconstruct the lower-frequency geometry information...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-10-01
|
| Series: | Graphical Models |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1524070323000231 |
| _version_ | 1797672102017368064 |
|---|---|
| author | Julian Kaltheuner Patrick Stotko Reinhard Klein |
| author_facet | Julian Kaltheuner Patrick Stotko Reinhard Klein |
| author_sort | Julian Kaltheuner |
| collection | DOAJ |
| description | In this paper, we present an inverse rendering method for the simple reconstruction of shape and appearance of real-world objects from only roughly calibrated RGB images captured under collocated point light illumination. To this end, we gradually reconstruct the lower-frequency geometry information using automatically generated occupancy mask images based on a visual hull initialization of the mesh, to infer the object topology, and a smoothness-preconditioned optimization. By combining this geometry estimation with learning-based SVBRDF parameter inference as well as intrinsic and extrinsic camera parameter refinement in a joint and unified formulation, our novel method is able to reconstruct shape and an isotropic SVBRDF from fewer input images than previous methods. Unlike in other works, we also estimate normal maps as part of the SVBRDF to capture and represent higher-frequency geometric details in a compact way. Furthermore, by regularizing the appearance estimation with a GAN-based SVBRDF generator, we are able to meaningfully limit the solution space. In summary, this leads to a robust automatic reconstruction algorithm for shape and appearance. We evaluated our algorithm on synthetic as well as on real-world data and demonstrate that our method is able to reconstruct complex objects with high-fidelity reflection properties in a robust way, also in the presence of imperfect camera parameter data. |
| first_indexed | 2024-03-11T21:25:11Z |
| format | Article |
| id | doaj.art-bcff26e63559495dbe45af6b99986c7e |
| institution | Directory Open Access Journal |
| issn | 1524-0703 |
| language | English |
| last_indexed | 2024-03-11T21:25:11Z |
| publishDate | 2023-10-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Graphical Models |
| spelling | doaj.art-bcff26e63559495dbe45af6b99986c7e2023-09-28T05:25:07ZengElsevierGraphical Models1524-07032023-10-01129101193Unified shape and appearance reconstruction with joint camera parameter refinementJulian Kaltheuner0Patrick Stotko1Reinhard Klein2Corresponding author.; University of Bonn, Friedrich-Hirzebruch-Allee 8, Bonn, 53115, GermanyUniversity of Bonn, Friedrich-Hirzebruch-Allee 8, Bonn, 53115, GermanyUniversity of Bonn, Friedrich-Hirzebruch-Allee 8, Bonn, 53115, GermanyIn this paper, we present an inverse rendering method for the simple reconstruction of shape and appearance of real-world objects from only roughly calibrated RGB images captured under collocated point light illumination. To this end, we gradually reconstruct the lower-frequency geometry information using automatically generated occupancy mask images based on a visual hull initialization of the mesh, to infer the object topology, and a smoothness-preconditioned optimization. By combining this geometry estimation with learning-based SVBRDF parameter inference as well as intrinsic and extrinsic camera parameter refinement in a joint and unified formulation, our novel method is able to reconstruct shape and an isotropic SVBRDF from fewer input images than previous methods. Unlike in other works, we also estimate normal maps as part of the SVBRDF to capture and represent higher-frequency geometric details in a compact way. Furthermore, by regularizing the appearance estimation with a GAN-based SVBRDF generator, we are able to meaningfully limit the solution space. In summary, this leads to a robust automatic reconstruction algorithm for shape and appearance. We evaluated our algorithm on synthetic as well as on real-world data and demonstrate that our method is able to reconstruct complex objects with high-fidelity reflection properties in a robust way, also in the presence of imperfect camera parameter data.http://www.sciencedirect.com/science/article/pii/S1524070323000231Inverse renderingReal-world objectsMesh reconstructionSVBRDFAppearance estimationCamera parameter refinement |
| spellingShingle | Julian Kaltheuner Patrick Stotko Reinhard Klein Unified shape and appearance reconstruction with joint camera parameter refinement Graphical Models Inverse rendering Real-world objects Mesh reconstruction SVBRDF Appearance estimation Camera parameter refinement |
| title | Unified shape and appearance reconstruction with joint camera parameter refinement |
| title_full | Unified shape and appearance reconstruction with joint camera parameter refinement |
| title_fullStr | Unified shape and appearance reconstruction with joint camera parameter refinement |
| title_full_unstemmed | Unified shape and appearance reconstruction with joint camera parameter refinement |
| title_short | Unified shape and appearance reconstruction with joint camera parameter refinement |
| title_sort | unified shape and appearance reconstruction with joint camera parameter refinement |
| topic | Inverse rendering Real-world objects Mesh reconstruction SVBRDF Appearance estimation Camera parameter refinement |
| url | http://www.sciencedirect.com/science/article/pii/S1524070323000231 |
| work_keys_str_mv | AT juliankaltheuner unifiedshapeandappearancereconstructionwithjointcameraparameterrefinement AT patrickstotko unifiedshapeandappearancereconstructionwithjointcameraparameterrefinement AT reinhardklein unifiedshapeandappearancereconstructionwithjointcameraparameterrefinement |