MarrNet: 3D shape reconstruction via 2.5D sketches
3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data wit...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Neural Information Processing Systems Foundation, Inc.
2020
|
| Online Access: | https://hdl.handle.net/1721.1/124813 |
| _version_ | 1826217403843870720 |
|---|---|
| author | Wu, Jiajun Wang, Yifan Xue, Tianfan Sun, Xingyuan Freeman, William T. Tenenbaum, Joshua B. |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Wu, Jiajun Wang, Yifan Xue, Tianfan Sun, Xingyuan Freeman, William T. Tenenbaum, Joshua B. |
| author_sort | Wu, Jiajun |
| collection | MIT |
| description | 3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction. ©2017 Presented as a poster session at the 31st Conference on Neural Information Processing Systems (NeurIPS 2017), December 4-9, 2017, Long Beach, California |
| first_indexed | 2024-09-23T17:03:06Z |
| format | Article |
| id | mit-1721.1/124813 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T17:03:06Z |
| publishDate | 2020 |
| publisher | Neural Information Processing Systems Foundation, Inc. |
| record_format | dspace |
| spelling | mit-1721.1/1248132022-10-03T10:05:03Z MarrNet: 3D shape reconstruction via 2.5D sketches Wu, Jiajun Wang, Yifan Xue, Tianfan Sun, Xingyuan Freeman, William T. Tenenbaum, Joshua B. Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory 3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction. ©2017 Presented as a poster session at the 31st Conference on Neural Information Processing Systems (NeurIPS 2017), December 4-9, 2017, Long Beach, California 2020-04-22T19:21:17Z 2020-04-22T19:21:17Z 2017 2019-05-28T12:52:50Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/124813 Wu, Jiajun, et al., "MarrNet: 3D shape reconstruction via 2.5D sketches." In Guyon, I., et al., eds., Advances in Neural Information Processing Systems 30 (San Diego, California: Neural Information Processing Systems Foundation, Inc., 2017) url https://papers.nips.cc/paper/6657-marrnet-3d-shape-reconstruction-via-25d-sketches ©2017 Author(s) en https://papers.nips.cc/paper/6657-marrnet-3d-shape-reconstruction-via-25d-sketches Advances in Neural Information Processing Systems Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Neural Information Processing Systems Foundation, Inc. Neural Information Processing Systems (NIPS) |
| spellingShingle | Wu, Jiajun Wang, Yifan Xue, Tianfan Sun, Xingyuan Freeman, William T. Tenenbaum, Joshua B. MarrNet: 3D shape reconstruction via 2.5D sketches |
| title | MarrNet: 3D shape reconstruction via 2.5D sketches |
| title_full | MarrNet: 3D shape reconstruction via 2.5D sketches |
| title_fullStr | MarrNet: 3D shape reconstruction via 2.5D sketches |
| title_full_unstemmed | MarrNet: 3D shape reconstruction via 2.5D sketches |
| title_short | MarrNet: 3D shape reconstruction via 2.5D sketches |
| title_sort | marrnet 3d shape reconstruction via 2 5d sketches |
| url | https://hdl.handle.net/1721.1/124813 |
| work_keys_str_mv | AT wujiajun marrnet3dshapereconstructionvia25dsketches AT wangyifan marrnet3dshapereconstructionvia25dsketches AT xuetianfan marrnet3dshapereconstructionvia25dsketches AT sunxingyuan marrnet3dshapereconstructionvia25dsketches AT freemanwilliamt marrnet3dshapereconstructionvia25dsketches AT tenenbaumjoshuab marrnet3dshapereconstructionvia25dsketches |