Studies of lighting for dense prediction and generation in computer vision
Soole, James
Loading…
Permalink
https://hdl.handle.net/2142/124287
Description
Title
Studies of lighting for dense prediction and generation in computer vision
Author(s)
Soole, James
Issue Date
2024-04-25
Director of Research (if dissertation) or Advisor (if thesis)
Forsyth, David A
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
Computer Vision
Artificial Intelligence
Deep Learning
Generative Ai
Intrinsics
Dense Prediction
Language
eng
Abstract
Modern dense prediction models do exceedingly well on benchmarks for the standard computer vision tasks of depth and normal estimation, object detection, and semantic segmentation. However, we show that the lighting in a scene has a significant effect on predictions, often producing inconsistent results for relit versions of the exact same scene. For surface normal prediction, we demonstrate that fine-tuning to enforce consistency under various lightings can mitigate this problem without sacrificing base accuracy of the pretrained model. Yet, such fine-tuning requires a dataset of relit scenes, which exist in limited quantity and are burdensome to produce. We therefore explore existing generative methods to create a synthetic relighting dataset, and propose our new method StyLitGAN. Based on StyleGAN architecture and the use of latent stylecode directions, StyLitGAN can realistically relight complex scenes without the need for labeled data. Fine-tuning a dense predictor with StyLitGAN images results in improvements comparable with that obtained by fine-tuning with true multi-illuminant images. We continue to investigate the effect of stylecode directions across different scenes, and show their use in producing desired results from reference images. We explore stylecode applications in explicitly-controllable scene lighting and uncover hints at their internal representation.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
