SceneCrafter - 3D scene generation and stylization
3D landscape assets are widely used in modern games and virtual reality worlds. Often, a 3D landscape is unbounded in nature. To design such a vast landscape manually requires much time and effort, thus it is favourable to develop a generative approach that can synthesize a realistic 3D world fro...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Final Year Project (FYP) |
| Language: | English |
| Published: |
Nanyang Technological University
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/176074 |
| _version_ | 1811697087650201600 |
|---|---|
| author | Yew Fu Yen |
| author2 | Liu Ziwei |
| author_facet | Liu Ziwei Yew Fu Yen |
| author_sort | Yew Fu Yen |
| collection | NTU |
| description | 3D landscape assets are widely used in modern games and virtual reality worlds.
Often, a 3D landscape is unbounded in nature. To design such a vast landscape
manually requires much time and effort, thus it is favourable to develop a generative
approach that can synthesize a realistic 3D world from noise inputs. To tackle this issue,
SceneDreamer was proposed as a generative model trained from in-the-wild images
for the task of unbounded landscape generation. Though SceneDreamer is robust in its
ability to render photorealistic views of 3D scenes, it does not offer much
controllability in its ability to stylize rendering outputs and lightings. As an
improvement to the original SceneDreamer model, I propose SceneCrafter, a
finetuning approach based on Score Distillation Sampling (SDS), which allows the
model to conditionally stylize 3D generated scenes according to user text prompt
inputs. SceneCrafter utilizes a pretrained Stable Diffusion model to guide the
finetuning of SceneDreamer. Different parameter groups in SceneDreamer are
finetuned at appropriately set learning rates to maximize stylization effects while
retaining 3D consistency. The proposed loss formulation also allows for flexible
adjustment in the degree of stylization. |
| first_indexed | 2024-10-01T07:49:41Z |
| format | Final Year Project (FYP) |
| id | ntu-10356/176074 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:49:41Z |
| publishDate | 2024 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1760742024-05-17T15:38:12Z SceneCrafter - 3D scene generation and stylization Yew Fu Yen Liu Ziwei School of Computer Science and Engineering ziwei.liu@ntu.edu.sg Computer and Information Science 3D stylization Score distillation sampling Neural rendering 3D generative model Unbounded scene generation 3D landscape assets are widely used in modern games and virtual reality worlds. Often, a 3D landscape is unbounded in nature. To design such a vast landscape manually requires much time and effort, thus it is favourable to develop a generative approach that can synthesize a realistic 3D world from noise inputs. To tackle this issue, SceneDreamer was proposed as a generative model trained from in-the-wild images for the task of unbounded landscape generation. Though SceneDreamer is robust in its ability to render photorealistic views of 3D scenes, it does not offer much controllability in its ability to stylize rendering outputs and lightings. As an improvement to the original SceneDreamer model, I propose SceneCrafter, a finetuning approach based on Score Distillation Sampling (SDS), which allows the model to conditionally stylize 3D generated scenes according to user text prompt inputs. SceneCrafter utilizes a pretrained Stable Diffusion model to guide the finetuning of SceneDreamer. Different parameter groups in SceneDreamer are finetuned at appropriately set learning rates to maximize stylization effects while retaining 3D consistency. The proposed loss formulation also allows for flexible adjustment in the degree of stylization. Bachelor's degree 2024-05-13T10:46:50Z 2024-05-13T10:46:50Z 2024 Final Year Project (FYP) Yew Fu Yen (2024). SceneCrafter - 3D scene generation and stylization. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/176074 https://hdl.handle.net/10356/176074 en application/pdf Nanyang Technological University |
| spellingShingle | Computer and Information Science 3D stylization Score distillation sampling Neural rendering 3D generative model Unbounded scene generation Yew Fu Yen SceneCrafter - 3D scene generation and stylization |
| title | SceneCrafter - 3D scene generation and stylization |
| title_full | SceneCrafter - 3D scene generation and stylization |
| title_fullStr | SceneCrafter - 3D scene generation and stylization |
| title_full_unstemmed | SceneCrafter - 3D scene generation and stylization |
| title_short | SceneCrafter - 3D scene generation and stylization |
| title_sort | scenecrafter 3d scene generation and stylization |
| topic | Computer and Information Science 3D stylization Score distillation sampling Neural rendering 3D generative model Unbounded scene generation |
| url | https://hdl.handle.net/10356/176074 |
| work_keys_str_mv | AT yewfuyen scenecrafter3dscenegenerationandstylization |