Neural feature fusion fields: 3D distillation of self-supervised 2D image representations
We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a tea...
| Main Authors: | , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
IEEE
2023
|
| _version_ | 1797109045980561408 |
|---|---|
| author | Tschernezki, V Laina, I Larlus, D Vedaldi, A |
| author_facet | Tschernezki, V Laina, I Larlus, D Vedaldi, A |
| author_sort | Tschernezki, V |
| collection | OXFORD |
| description | We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space. The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery. As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes. We show that our method not only enables semantic understanding in the context of scene-specific neural fields without the use of manual labels, but also consistently improves over the self-supervised 2D baselines. This is demonstrated by considering various tasks, such as 2D object retrieval, 3D segmentation, and scene editing, in diverse sequences, including long egocentric videos in the EPIC-KITCHENS benchmark. Project page: https://www.robots.ox.ac.uk/-vadim/n3f/ |
| first_indexed | 2024-03-07T07:36:34Z |
| format | Conference item |
| id | oxford-uuid:809a6891-52af-4257-8bc2-66f1419ff9eb |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:36:34Z |
| publishDate | 2023 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:809a6891-52af-4257-8bc2-66f1419ff9eb2023-03-16T10:24:10ZNeural feature fusion fields: 3D distillation of self-supervised 2D image representationsConference itemhttp://purl.org/coar/resource_type/c_5794uuid:809a6891-52af-4257-8bc2-66f1419ff9ebEnglishSymplectic ElementsIEEE2023Tschernezki, VLaina, ILarlus, DVedaldi, AWe present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space. The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery. As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes. We show that our method not only enables semantic understanding in the context of scene-specific neural fields without the use of manual labels, but also consistently improves over the self-supervised 2D baselines. This is demonstrated by considering various tasks, such as 2D object retrieval, 3D segmentation, and scene editing, in diverse sequences, including long egocentric videos in the EPIC-KITCHENS benchmark. Project page: https://www.robots.ox.ac.uk/-vadim/n3f/ |
| spellingShingle | Tschernezki, V Laina, I Larlus, D Vedaldi, A Neural feature fusion fields: 3D distillation of self-supervised 2D image representations |
| title | Neural feature fusion fields: 3D distillation of self-supervised 2D image representations |
| title_full | Neural feature fusion fields: 3D distillation of self-supervised 2D image representations |
| title_fullStr | Neural feature fusion fields: 3D distillation of self-supervised 2D image representations |
| title_full_unstemmed | Neural feature fusion fields: 3D distillation of self-supervised 2D image representations |
| title_short | Neural feature fusion fields: 3D distillation of self-supervised 2D image representations |
| title_sort | neural feature fusion fields 3d distillation of self supervised 2d image representations |
| work_keys_str_mv | AT tschernezkiv neuralfeaturefusionfields3ddistillationofselfsupervised2dimagerepresentations AT lainai neuralfeaturefusionfields3ddistillationofselfsupervised2dimagerepresentations AT larlusd neuralfeaturefusionfields3ddistillationofselfsupervised2dimagerepresentations AT vedaldia neuralfeaturefusionfields3ddistillationofselfsupervised2dimagerepresentations |