ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition
3D decomposition/segmentation still remains a challenge as large-scale 3D annotated data is not readily available. Contemporary approaches typically leverage 2D machine-generated segments, integrating them for 3D consistency. While the majority of these methods are based on NeRFs, they face a pot...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Conference Paper |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/180249 http://arxiv.org/abs/2403.14619v1 |
| _version_ | 1824455120114417664 |
|---|---|
| author | Wu, Tianhao Zheng, Chuanxia Cham, Tat-Jen Wu, Qianyi |
| author2 | College of Computing and Data Science |
| author_facet | College of Computing and Data Science Wu, Tianhao Zheng, Chuanxia Cham, Tat-Jen Wu, Qianyi |
| author_sort | Wu, Tianhao |
| collection | NTU |
| description | 3D decomposition/segmentation still remains a challenge as large-scale 3D
annotated data is not readily available. Contemporary approaches typically
leverage 2D machine-generated segments, integrating them for 3D consistency.
While the majority of these methods are based on NeRFs, they face a potential
weakness that the instance/semantic embedding features derive from independent
MLPs, thus preventing the segmentation network from learning the geometric
details of the objects directly through radiance and density. In this paper, we
propose ClusteringSDF, a novel approach to achieve both segmentation and
reconstruction in 3D via the neural implicit surface representation,
specifically Signal Distance Function (SDF), where the segmentation rendering
is directly integrated with the volume rendering of neural implicit surfaces.
Although based on ObjectSDF++, ClusteringSDF no longer requires the
ground-truth segments for supervision while maintaining the capability of
reconstructing individual object surfaces, but purely with the noisy and
inconsistent labels from pre-trained models.As the core of ClusteringSDF, we
introduce a high-efficient clustering mechanism for lifting the 2D labels to 3D
and the experimental results on the challenging scenes from ScanNet and Replica
datasets show that ClusteringSDF can achieve competitive performance compared
against the state-of-the-art with significantly reduced training time. |
| first_indexed | 2025-02-19T03:33:09Z |
| format | Conference Paper |
| id | ntu-10356/180249 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:33:09Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1802492024-10-01T06:03:28Z ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition Wu, Tianhao Zheng, Chuanxia Cham, Tat-Jen Wu, Qianyi College of Computing and Data Science 2024 European Conference on Computer Vision (ECCV) S-Lab Computer and Information Science 3D segmentation Neural implicit surface representation 3D decomposition/segmentation still remains a challenge as large-scale 3D annotated data is not readily available. Contemporary approaches typically leverage 2D machine-generated segments, integrating them for 3D consistency. While the majority of these methods are based on NeRFs, they face a potential weakness that the instance/semantic embedding features derive from independent MLPs, thus preventing the segmentation network from learning the geometric details of the objects directly through radiance and density. In this paper, we propose ClusteringSDF, a novel approach to achieve both segmentation and reconstruction in 3D via the neural implicit surface representation, specifically Signal Distance Function (SDF), where the segmentation rendering is directly integrated with the volume rendering of neural implicit surfaces. Although based on ObjectSDF++, ClusteringSDF no longer requires the ground-truth segments for supervision while maintaining the capability of reconstructing individual object surfaces, but purely with the noisy and inconsistent labels from pre-trained models.As the core of ClusteringSDF, we introduce a high-efficient clustering mechanism for lifting the 2D labels to 3D and the experimental results on the challenging scenes from ScanNet and Replica datasets show that ClusteringSDF can achieve competitive performance compared against the state-of-the-art with significantly reduced training time. Submitted/Accepted version 2024-09-26T02:28:52Z 2024-09-26T02:28:52Z 2024 Conference Paper Wu, T., Zheng, C., Cham, T. & Wu, Q. (2024). ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition. 2024 European Conference on Computer Vision (ECCV). https://dx.doi.org/10.48550/arXiv.2403.14619 https://hdl.handle.net/10356/180249 10.48550/arXiv.2403.14619 http://arxiv.org/abs/2403.14619v1 en 10.21979/N9/RJUHMC © 2024 ECCV. All rights reserved. This article may be downloaded for personal use only. Any other use requires prior permission of the copyright holder. application/pdf |
| spellingShingle | Computer and Information Science 3D segmentation Neural implicit surface representation Wu, Tianhao Zheng, Chuanxia Cham, Tat-Jen Wu, Qianyi ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition |
| title | ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition |
| title_full | ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition |
| title_fullStr | ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition |
| title_full_unstemmed | ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition |
| title_short | ClusteringSDF: self-organized neural implicit surfaces for 3D decomposition |
| title_sort | clusteringsdf self organized neural implicit surfaces for 3d decomposition |
| topic | Computer and Information Science 3D segmentation Neural implicit surface representation |
| url | https://hdl.handle.net/10356/180249 http://arxiv.org/abs/2403.14619v1 |
| work_keys_str_mv | AT wutianhao clusteringsdfselforganizedneuralimplicitsurfacesfor3ddecomposition AT zhengchuanxia clusteringsdfselforganizedneuralimplicitsurfacesfor3ddecomposition AT chamtatjen clusteringsdfselforganizedneuralimplicitsurfacesfor3ddecomposition AT wuqianyi clusteringsdfselforganizedneuralimplicitsurfacesfor3ddecomposition |