Adaptation of Generative Models for Image Manipulation and Recontextualization

Shah, Viraj

Permalink

https://hdl.handle.net/2142/125619 Copy

Description

Title

Adaptation of Generative Models for Image Manipulation and Recontextualization

Author(s)

Shah, Viraj

Issue Date

2024-07-12

Director of Research (if dissertation) or Advisor (if thesis)

Lazebnik, Svetlana

Doctoral Committee Chair(s)

Lazebnik, Svetlana

Committee Member(s)

• Forsyth, David
• Schwing, Alexander
• Hasegawa-Johnson, Mark

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Generative Models
• Generative Adversarial Networks
• Diffusion Models
• Image Editing
• Image Stylization
• Intrinsic Image Decomposition
• Recontextualization

Abstract

In recent works, generative models such as generative adversarial networks (GANs) and diffusion models have demonstrated a remarkable ability to mimic the image data distributions and to synthesize highly photo-realistic images representing diverse sets of concepts. Such models capture rich semantic information of the image data, and can potentially be used as a prior in solving image manipulation and recontextualization problems. In this work, we aim to study various challenges in employing generative models as priors in solving image attribute editing, intrinsic image decomposition, and one-shot image stylization. First, we discuss the challenge of inverting a pre-trained GAN, a crucial step in exploiting the rich GAN image priors, and how we can achieve a near-perfect GAN Inversion for accurate image reconstruction and attribute editing. We extend the framework of GAN Inversion to multiple GANs that allow for jointly leveraging multiple GAN priors for the successful decomposition of an image into its intrinsic components such as albedo, shading, and specular. Further, we propose a GAN-based one-shot stylization method that can stylize an input image into multiple styles at once while using only one example of each reference style. Since the GAN-based stylization approaches are typically limited to specific subject domains, we also propose a diffusion model-based one-shot stylization approach, ZipLoRA, that allows for generating any subject in any style along with text-driven recontextualization capabilities.

Graduation Semester

2024-08

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/125619

Copyright and License Information

Copyright 2024 Viraj Shah

Owning Collections