Building 3D Generative Models from Minimal Data
Abstract We propose a method for constructing generative models of 3D objects from a single 3D mesh and improving them through unsupervised low-shot learning from 2D images. Our method produces a 3D morphable model that represents shape and albedo in terms of Gaussian processes. Whereas...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer US
2023
|
| Online Access: | https://hdl.handle.net/1721.1/152192 |
| _version_ | 1826193756546662400 |
|---|---|
| author | Sutherland, Skylar Egger, Bernhard Tenenbaum, Joshua |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Sutherland, Skylar Egger, Bernhard Tenenbaum, Joshua |
| author_sort | Sutherland, Skylar |
| collection | MIT |
| description | Abstract
We propose a method for constructing generative models of 3D objects from a single 3D mesh and improving them through unsupervised low-shot learning from 2D images. Our method produces a 3D morphable model that represents shape and albedo in terms of Gaussian processes. Whereas previous approaches have typically built 3D morphable models from multiple high-quality 3D scans through principal component analysis, we build 3D morphable models from a single scan or template. As we demonstrate in the face domain, these models can be used to infer 3D reconstructions from 2D data (inverse graphics) or 3D data (registration). Specifically, we show that our approach can be used to perform face recognition using only a single 3D template (one scan total, not one per person). We extend our model to a preliminary unsupervised learning framework that enables the learning of the distribution of 3D faces using one 3D template and a small number of 2D images. Our approach is motivated as a potential model for the origins of face perception in human infants, who appear to start with an innate face template and subsequently develop a flexible system for perceiving the 3D structure of any novel face from experience with only 2D images of a relatively small number of familiar faces. |
| first_indexed | 2024-09-23T09:44:25Z |
| format | Article |
| id | mit-1721.1/152192 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T09:44:25Z |
| publishDate | 2023 |
| publisher | Springer US |
| record_format | dspace |
| spelling | mit-1721.1/1521922024-01-19T21:02:50Z Building 3D Generative Models from Minimal Data Sutherland, Skylar Egger, Bernhard Tenenbaum, Joshua Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Abstract We propose a method for constructing generative models of 3D objects from a single 3D mesh and improving them through unsupervised low-shot learning from 2D images. Our method produces a 3D morphable model that represents shape and albedo in terms of Gaussian processes. Whereas previous approaches have typically built 3D morphable models from multiple high-quality 3D scans through principal component analysis, we build 3D morphable models from a single scan or template. As we demonstrate in the face domain, these models can be used to infer 3D reconstructions from 2D data (inverse graphics) or 3D data (registration). Specifically, we show that our approach can be used to perform face recognition using only a single 3D template (one scan total, not one per person). We extend our model to a preliminary unsupervised learning framework that enables the learning of the distribution of 3D faces using one 3D template and a small number of 2D images. Our approach is motivated as a potential model for the origins of face perception in human infants, who appear to start with an innate face template and subsequently develop a flexible system for perceiving the 3D structure of any novel face from experience with only 2D images of a relatively small number of familiar faces. 2023-09-21T19:43:21Z 2023-09-21T19:43:21Z 2023-09-13 2023-09-17T03:10:16Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/152192 Sutherland, Skylar, Egger, Bernhard and Tenenbaum, Joshua. 2023. "Building 3D Generative Models from Minimal Data." PUBLISHER_CC en https://doi.org/10.1007/s11263-023-01870-2 Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ The Author(s) application/pdf Springer US Springer US |
| spellingShingle | Sutherland, Skylar Egger, Bernhard Tenenbaum, Joshua Building 3D Generative Models from Minimal Data |
| title | Building 3D Generative Models from Minimal Data |
| title_full | Building 3D Generative Models from Minimal Data |
| title_fullStr | Building 3D Generative Models from Minimal Data |
| title_full_unstemmed | Building 3D Generative Models from Minimal Data |
| title_short | Building 3D Generative Models from Minimal Data |
| title_sort | building 3d generative models from minimal data |
| url | https://hdl.handle.net/1721.1/152192 |
| work_keys_str_mv | AT sutherlandskylar building3dgenerativemodelsfromminimaldata AT eggerbernhard building3dgenerativemodelsfromminimaldata AT tenenbaumjoshua building3dgenerativemodelsfromminimaldata |