Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings
In this paper, we address the challenge of estimating the 6DoF pose of objects in 2D equirectangular images. This solution allows the transition to the objects’ 3D model from their current pose. In particular, it finds application in the educational use of 360° videos, where it enhances the learning...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/11/5309 |
| _version_ | 1797494313664380928 |
|---|---|
| author | Matteo Zanetti Alessandro Luchetti Sharad Maheshwari Denis Kalkofen Manuel Labrador Ortega Mariolino De Cecco |
| author_facet | Matteo Zanetti Alessandro Luchetti Sharad Maheshwari Denis Kalkofen Manuel Labrador Ortega Mariolino De Cecco |
| author_sort | Matteo Zanetti |
| collection | DOAJ |
| description | In this paper, we address the challenge of estimating the 6DoF pose of objects in 2D equirectangular images. This solution allows the transition to the objects’ 3D model from their current pose. In particular, it finds application in the educational use of 360° videos, where it enhances the learning experience of students by making it more engaging and immersive due to the possible interaction with 3D virtual models. We developed a general approach usable for any object and shape. The only requirement is to have an accurate CAD model, even without textures of the item, whose pose must be estimated. The developed pipeline has two main steps: vehicle segmentation from the image background and estimation of the vehicle pose. To accomplish the first task, we used deep learning methods, while for the second, we developed a 360° camera simulator in Unity to generate synthetic equirectangular images used for comparison. We conducted our tests using a miniature truck model whose CAD was at our disposal. The developed algorithm was tested using a metrological analysis applied to real data. The results showed a mean difference of 1.5° with a standard deviation of 1° from the ground truth data for rotations, and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1.4</mn></mrow></semantics></math></inline-formula> cm with a standard deviation of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1.5</mn></mrow></semantics></math></inline-formula> cm for translations over a research range of ±20° and ±20 cm, respectively. |
| first_indexed | 2024-03-10T01:32:29Z |
| format | Article |
| id | doaj.art-4dab5381c5244acf8fd997c475fa9620 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T01:32:29Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-4dab5381c5244acf8fd997c475fa96202023-11-23T13:39:29ZengMDPI AGApplied Sciences2076-34172022-05-011211530910.3390/app12115309Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational SettingsMatteo Zanetti0Alessandro Luchetti1Sharad Maheshwari2Denis Kalkofen3Manuel Labrador Ortega4Mariolino De Cecco5Department of Industrial Engineering, University of Trento, Via Sommarive 9, 38123 Trento, ItalyDepartment of Industrial Engineering, University of Trento, Via Sommarive 9, 38123 Trento, ItalyDepartment of Industrial Engineering, University of Trento, Via Sommarive 9, 38123 Trento, ItalyInstitute of Computer Graphics and Vision, Graz University of Technology, Rechbauerstraße 12, 8010 Graz, AustriaResources Innovation Center Leoben, Montanuniversität Leoben, Franz Josef Strasse 18, 8700 Leoben, AustriaDepartment of Industrial Engineering, University of Trento, Via Sommarive 9, 38123 Trento, ItalyIn this paper, we address the challenge of estimating the 6DoF pose of objects in 2D equirectangular images. This solution allows the transition to the objects’ 3D model from their current pose. In particular, it finds application in the educational use of 360° videos, where it enhances the learning experience of students by making it more engaging and immersive due to the possible interaction with 3D virtual models. We developed a general approach usable for any object and shape. The only requirement is to have an accurate CAD model, even without textures of the item, whose pose must be estimated. The developed pipeline has two main steps: vehicle segmentation from the image background and estimation of the vehicle pose. To accomplish the first task, we used deep learning methods, while for the second, we developed a 360° camera simulator in Unity to generate synthetic equirectangular images used for comparison. We conducted our tests using a miniature truck model whose CAD was at our disposal. The developed algorithm was tested using a metrological analysis applied to real data. The results showed a mean difference of 1.5° with a standard deviation of 1° from the ground truth data for rotations, and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1.4</mn></mrow></semantics></math></inline-formula> cm with a standard deviation of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1.5</mn></mrow></semantics></math></inline-formula> cm for translations over a research range of ±20° and ±20 cm, respectively.https://www.mdpi.com/2076-3417/12/11/5309image processing6DoF pose estimationmixed realityhuman empowermenteducational setting |
| spellingShingle | Matteo Zanetti Alessandro Luchetti Sharad Maheshwari Denis Kalkofen Manuel Labrador Ortega Mariolino De Cecco Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings Applied Sciences image processing 6DoF pose estimation mixed reality human empowerment educational setting |
| title | Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings |
| title_full | Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings |
| title_fullStr | Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings |
| title_full_unstemmed | Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings |
| title_short | Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings |
| title_sort | object pose detection to enable 3d interaction from 2d equirectangular images in mixed reality educational settings |
| topic | image processing 6DoF pose estimation mixed reality human empowerment educational setting |
| url | https://www.mdpi.com/2076-3417/12/11/5309 |
| work_keys_str_mv | AT matteozanetti objectposedetectiontoenable3dinteractionfrom2dequirectangularimagesinmixedrealityeducationalsettings AT alessandroluchetti objectposedetectiontoenable3dinteractionfrom2dequirectangularimagesinmixedrealityeducationalsettings AT sharadmaheshwari objectposedetectiontoenable3dinteractionfrom2dequirectangularimagesinmixedrealityeducationalsettings AT deniskalkofen objectposedetectiontoenable3dinteractionfrom2dequirectangularimagesinmixedrealityeducationalsettings AT manuellabradorortega objectposedetectiontoenable3dinteractionfrom2dequirectangularimagesinmixedrealityeducationalsettings AT mariolinodececco objectposedetectiontoenable3dinteractionfrom2dequirectangularimagesinmixedrealityeducationalsettings |