SemanticPaint: interactive segmentation and learning of 3D worlds
We present a real-time, interactive system for the geometric reconstruction, object-class segmentation and learning of 3D scenes [Valentin et al. 2015]. Using our system, a user can walk into a room wearing a depth camera and a virtual reality headset, and both densely reconstruct the 3D scene [Newc...
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Association for Computing Machinery
2015
|
| _version_ | 1826313900219432960 |
|---|---|
| author | Golodetz, S Sapienza, M Valentin, JPC Vineet, V Cheng, M-M Prisacariu, VA Kähler, O Ren, CY Arnab, A Hicks, SL Murray, DW Izadi, S Torr, PHS |
| author_facet | Golodetz, S Sapienza, M Valentin, JPC Vineet, V Cheng, M-M Prisacariu, VA Kähler, O Ren, CY Arnab, A Hicks, SL Murray, DW Izadi, S Torr, PHS |
| author_sort | Golodetz, S |
| collection | OXFORD |
| description | We present a real-time, interactive system for the geometric reconstruction, object-class segmentation and learning of 3D scenes [Valentin et al. 2015]. Using our system, a user can walk into a room wearing a depth camera and a virtual reality headset, and both densely reconstruct the 3D scene [Newcombe et al. 2011; Nießner et al. 2013; Prisacariu et al. 2014]) and interactively segment the environment into object classes such as 'chair', 'floor' and 'table'. The user interacts physically with the real-world scene, touching objects and using voice commands to assign them appropriate labels. These user-generated labels are leveraged by an online random forest-based machine learning algorithm, which is used to predict labels for previously unseen parts of the scene. The predicted labels, together with those provided directly by the user, are incorporated into a dense 3D conditional random field model, over which we perform mean-field inference to filter out label inconsistencies. The entire pipeline runs in real time, and the user stays 'in the loop' throughout the process, receiving immediate feedback about the progress of the labelling and interacting with the scene as necessary to refine the predicted segmentation. |
| first_indexed | 2024-09-25T04:23:39Z |
| format | Conference item |
| id | oxford-uuid:7a1e9677-2c5e-40a4-a459-ce2d90760954 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-09-25T04:23:39Z |
| publishDate | 2015 |
| publisher | Association for Computing Machinery |
| record_format | dspace |
| spelling | oxford-uuid:7a1e9677-2c5e-40a4-a459-ce2d907609542024-08-21T15:51:34ZSemanticPaint: interactive segmentation and learning of 3D worldsConference itemhttp://purl.org/coar/resource_type/c_c94fuuid:7a1e9677-2c5e-40a4-a459-ce2d90760954EnglishSymplectic ElementsAssociation for Computing Machinery2015Golodetz, SSapienza, MValentin, JPCVineet, VCheng, M-MPrisacariu, VAKähler, ORen, CYArnab, AHicks, SLMurray, DWIzadi, STorr, PHSWe present a real-time, interactive system for the geometric reconstruction, object-class segmentation and learning of 3D scenes [Valentin et al. 2015]. Using our system, a user can walk into a room wearing a depth camera and a virtual reality headset, and both densely reconstruct the 3D scene [Newcombe et al. 2011; Nießner et al. 2013; Prisacariu et al. 2014]) and interactively segment the environment into object classes such as 'chair', 'floor' and 'table'. The user interacts physically with the real-world scene, touching objects and using voice commands to assign them appropriate labels. These user-generated labels are leveraged by an online random forest-based machine learning algorithm, which is used to predict labels for previously unseen parts of the scene. The predicted labels, together with those provided directly by the user, are incorporated into a dense 3D conditional random field model, over which we perform mean-field inference to filter out label inconsistencies. The entire pipeline runs in real time, and the user stays 'in the loop' throughout the process, receiving immediate feedback about the progress of the labelling and interacting with the scene as necessary to refine the predicted segmentation. |
| spellingShingle | Golodetz, S Sapienza, M Valentin, JPC Vineet, V Cheng, M-M Prisacariu, VA Kähler, O Ren, CY Arnab, A Hicks, SL Murray, DW Izadi, S Torr, PHS SemanticPaint: interactive segmentation and learning of 3D worlds |
| title | SemanticPaint: interactive segmentation and learning of 3D worlds |
| title_full | SemanticPaint: interactive segmentation and learning of 3D worlds |
| title_fullStr | SemanticPaint: interactive segmentation and learning of 3D worlds |
| title_full_unstemmed | SemanticPaint: interactive segmentation and learning of 3D worlds |
| title_short | SemanticPaint: interactive segmentation and learning of 3D worlds |
| title_sort | semanticpaint interactive segmentation and learning of 3d worlds |
| work_keys_str_mv | AT golodetzs semanticpaintinteractivesegmentationandlearningof3dworlds AT sapienzam semanticpaintinteractivesegmentationandlearningof3dworlds AT valentinjpc semanticpaintinteractivesegmentationandlearningof3dworlds AT vineetv semanticpaintinteractivesegmentationandlearningof3dworlds AT chengmm semanticpaintinteractivesegmentationandlearningof3dworlds AT prisacariuva semanticpaintinteractivesegmentationandlearningof3dworlds AT kahlero semanticpaintinteractivesegmentationandlearningof3dworlds AT rency semanticpaintinteractivesegmentationandlearningof3dworlds AT arnaba semanticpaintinteractivesegmentationandlearningof3dworlds AT hickssl semanticpaintinteractivesegmentationandlearningof3dworlds AT murraydw semanticpaintinteractivesegmentationandlearningof3dworlds AT izadis semanticpaintinteractivesegmentationandlearningof3dworlds AT torrphs semanticpaintinteractivesegmentationandlearningof3dworlds |