NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields
2022 International Conference on Robotics and Automation (ICRA) 23-27 May 2022
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
IEEE
2024
|
| Online Access: | https://hdl.handle.net/1721.1/153644 |
| _version_ | 1826202072521900032 |
|---|---|
| author | Yen-Chen, Lin Florence, Pete Barron, Jonathan T. Lin, Tsung-Yi Rodriguez, Alberto Isola, Phillip |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Yen-Chen, Lin Florence, Pete Barron, Jonathan T. Lin, Tsung-Yi Rodriguez, Alberto Isola, Phillip |
| author_sort | Yen-Chen, Lin |
| collection | MIT |
| description | 2022 International Conference on Robotics and Automation (ICRA) 23-27 May 2022 |
| first_indexed | 2024-09-23T12:01:27Z |
| format | Article |
| id | mit-1721.1/153644 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T12:01:27Z |
| publishDate | 2024 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1536442024-09-20T19:21:34Z NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields Yen-Chen, Lin Florence, Pete Barron, Jonathan T. Lin, Tsung-Yi Rodriguez, Alberto Isola, Phillip Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Department of Mechanical Engineering Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory 2022 International Conference on Robotics and Automation (ICRA) 23-27 May 2022 Thin, reflective objects such as forks and whisks are common in our daily lives, but they are particularly chal-lenging for robot perception because it is hard to reconstruct them using commodity RGB-D cameras or multi-view stereo techniques. While traditional pipelines struggle with objects like these, Neural Radiance Fields (NeRFs) have recently been shown to be remarkably effective for performing view synthesis on objects with thin structures or reflective materials. In this paper we explore the use of NeRF as a new source of supervision for robust robot vision systems. In particular, we demonstrate that a NeRF representation of a scene can be used to train dense object descriptors. We use an optimized NeRF to extract dense correspondences between multiple views of an object, and then use these correspondences as training data for learning a view-invariant representation of the object. NeRF's usage of a density field allows us to reformulate the correspondence problem with a novel distribution-of-depths formulation, as opposed to the conventional approach of using a depth map. Dense correspondence models supervised with our method significantly outperform off-the-shelf learned descriptors by 106% (PCK@3px metric, more than doubling performance) and outperform our baseline supervised with multi-view stereo by 29%. Furthermore, we demonstrate the learned dense descriptors enable robots to perform accurate 6-degree of freedom (6-DoF) pick and place of thin and reflective objects. 2024-03-08T16:58:27Z 2024-03-08T16:58:27Z 2022-05-23 Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/153644 Yen-Chen, Lin, Florence, Pete, Barron, Jonathan T., Lin, Tsung-Yi, Rodriguez, Alberto et al. 2022. "NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields." en_US 10.1109/icra46639.2022.9812291 Creative Commons Attribution-Noncommercial-ShareAlike Attribution-NonCommercial-ShareAlike 4.0 International http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf IEEE IEEE |
| spellingShingle | Yen-Chen, Lin Florence, Pete Barron, Jonathan T. Lin, Tsung-Yi Rodriguez, Alberto Isola, Phillip NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields |
| title | NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields |
| title_full | NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields |
| title_fullStr | NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields |
| title_full_unstemmed | NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields |
| title_short | NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields |
| title_sort | nerf supervision learning dense object descriptors from neural radiance fields |
| url | https://hdl.handle.net/1721.1/153644 |
| work_keys_str_mv | AT yenchenlin nerfsupervisionlearningdenseobjectdescriptorsfromneuralradiancefields AT florencepete nerfsupervisionlearningdenseobjectdescriptorsfromneuralradiancefields AT barronjonathant nerfsupervisionlearningdenseobjectdescriptorsfromneuralradiancefields AT lintsungyi nerfsupervisionlearningdenseobjectdescriptorsfromneuralradiancefields AT rodriguezalberto nerfsupervisionlearningdenseobjectdescriptorsfromneuralradiancefields AT isolaphillip nerfsupervisionlearningdenseobjectdescriptorsfromneuralradiancefields |