Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes
Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement fo...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/129426 |
| _version_ | 1826192853456388096 |
|---|---|
| author | Marion, James Patrick Florence, Peter Raymond Manuelli, Lucas Tedrake, Russell L |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Marion, James Patrick Florence, Peter Raymond Manuelli, Lucas Tedrake, Russell L |
| author_sort | Marion, James Patrick |
| collection | MIT |
| description | Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture. Our dataset and annotation pipeline are available at labelfusion.csail.mit.edu. |
| first_indexed | 2024-09-23T09:30:12Z |
| format | Article |
| id | mit-1721.1/129426 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T09:30:12Z |
| publishDate | 2021 |
| publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| record_format | dspace |
| spelling | mit-1721.1/1294262022-09-30T14:52:05Z Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes Marion, James Patrick Florence, Peter Raymond Manuelli, Lucas Tedrake, Russell L Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture. Our dataset and annotation pipeline are available at labelfusion.csail.mit.edu. 2021-01-14T20:45:25Z 2021-01-14T20:45:25Z 2018-09 2018-05 2019-07-16T12:01:18Z Article http://purl.org/eprint/type/ConferencePaper 9781538630815 2577-087X https://hdl.handle.net/1721.1/129426 Marion, Pat et al. "Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes." 2018 IEEE International Conference on Robotics and Automation, May 2018, Brisbane, Australia, Institute of Electrical and Electronics Engineers, September 2018. © 2018 IEEE en http://dx.doi.org/10.1109/icra.2018.8460950 2018 IEEE International Conference on Robotics and Automation Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute of Electrical and Electronics Engineers (IEEE) arXiv |
| spellingShingle | Marion, James Patrick Florence, Peter Raymond Manuelli, Lucas Tedrake, Russell L Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes |
| title | Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes |
| title_full | Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes |
| title_fullStr | Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes |
| title_full_unstemmed | Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes |
| title_short | Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes |
| title_sort | label fusion a pipeline for generating ground truth labels for real rgbd data of cluttered scenes |
| url | https://hdl.handle.net/1721.1/129426 |
| work_keys_str_mv | AT marionjamespatrick labelfusionapipelineforgeneratinggroundtruthlabelsforrealrgbddataofclutteredscenes AT florencepeterraymond labelfusionapipelineforgeneratinggroundtruthlabelsforrealrgbddataofclutteredscenes AT manuellilucas labelfusionapipelineforgeneratinggroundtruthlabelsforrealrgbddataofclutteredscenes AT tedrakerusselll labelfusionapipelineforgeneratinggroundtruthlabelsforrealrgbddataofclutteredscenes |