| Summary: | 3D landscape assets are widely used in modern games and virtual reality worlds.
Often, a 3D landscape is unbounded in nature. To design such a vast landscape
manually requires much time and effort, thus it is favourable to develop a generative
approach that can synthesize a realistic 3D world from noise inputs. To tackle this issue,
SceneDreamer was proposed as a generative model trained from in-the-wild images
for the task of unbounded landscape generation. Though SceneDreamer is robust in its
ability to render photorealistic views of 3D scenes, it does not offer much
controllability in its ability to stylize rendering outputs and lightings. As an
improvement to the original SceneDreamer model, I propose SceneCrafter, a
finetuning approach based on Score Distillation Sampling (SDS), which allows the
model to conditionally stylize 3D generated scenes according to user text prompt
inputs. SceneCrafter utilizes a pretrained Stable Diffusion model to guide the
finetuning of SceneDreamer. Different parameter groups in SceneDreamer are
finetuned at appropriately set learning rates to maximize stylization effects while
retaining 3D consistency. The proposed loss formulation also allows for flexible
adjustment in the degree of stylization.
|
|---|