Generative 3D scene modeling: algorithm and applications

Shen, Yuan

Permalink

https://hdl.handle.net/2142/132482 Copy

Description

Title

Generative 3D scene modeling: algorithm and applications

Author(s)

Shen, Yuan

Issue Date

2025-11-11

Director of Research (if dissertation) or Advisor (if thesis)

Wang, Shenlong

Doctoral Committee Chair(s)

Wang, Shenlong

Committee Member(s)

• Hoiem, Derek
• Forsyth, David
• Ceylan, Duygu

Department of Study

Siebel School Comp & Data Sci

Discipline

Computer Science

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• 3d scene generation
• 3d scene editing

Abstract

The ability to generate realistic and functional 3D scenes at scale is becoming increasingly important across domains such as agriculture, autonomous driving, immersive gaming, and robot learning. However, manually constructing such environments is prohibitively expensive and often introduces domain gaps from real-world distributions. Generative modeling of 3D scenes offers a scalable alternative, with the promise of producing vast quantities of coherent, diverse, and controllable environments directly from data. This thesis aims to advance generative 3D scene modeling with a particular emphasis on large, outdoor, and multi-modal environments. We define two desiderata for an ideal generative scene model: the ability to imagine worlds beyond the observed extent (scene generation), and the ability to reinterpret observed environments with alternative variations (scene editing). Realizing these goals is far from trivial, raising three key research challenges: (1) how to enable large-scale scene generation with limited 3D data; (2) how to move beyond single-modality generation to handle diverse and integrated scene modalities; and (3) how to ensure that generative scene models bring tangible benefits to downstream applications. Our approach to these challenges is developed along two complementary dimensions: algorithms and applications. Algorithmically, the guiding philosophy is to leverage powerful 2D generative priors to overcome the scarcity of high-quality 3D data. This leads to two chapters: (1) SuperGaussian, which adapts pre-trained video models for 3D scene editing, showing that video priors can be repurposed for high-fidelity 3D super-resolution; and (2) SGAM, a progressive framework that combines generative sensor modeling with reconstruction to create globally consistent, large-scale virtual worlds from RGB-D sequences. On the application side, we demonstrate how generative scene modeling can directly support real-world use cases. In particular, Sim-on-Wheels provides a safe and realistic framework for testing autonomous driving by augmenting real-world driving scenes with virtual tra"c scenarios, while MMCityGen showcases how multi-modal generation can power urban digital twins for planning and analysis. Together, these contributions establish a principled approach to repurposing 2D priors for 3D scene modeling, extend generation into the multi-modal domain, and demonstrate concrete uses in autonomy and urban planning.

Graduation Semester

2025-12

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/132482

Copyright and License Information

Copyright 2025 Yuan Shen

Owning Collections