Hardware acceleration of neural graphics

Mubarik, Muhammad Husnain

Permalink

https://hdl.handle.net/2142/125559 Copy

Description

Title

Hardware acceleration of neural graphics

Author(s)

Mubarik, Muhammad Husnain

Issue Date

2024-07-12

Director of Research (if dissertation) or Advisor (if thesis)

Kumar, Rakesh

Doctoral Committee Chair(s)

Kumar, Rakesh

Committee Member(s)

• Chen, Deming
• Gupta, Saurabh
• Iyer, Ravi
• Lin, Yingyan

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Neural Radiance Fields
• Latent Diffusion Models
• Neural Graphics
• Neural Representations
• Diffusion Models
• Virtual Reality
• Augmented Reality
• Mixed Reality

Abstract

Neural graphics is a rapidly evolving domain which uses neural networks to replace entire or parts of rendering pipeline. There are two broad classes of neural graphics: 1) Non-generative neural graphics where conventional DNNs are used for rendering. 2) Generative neural graphics where generative models such as latent diffusion models are employed. This dissertation studies the performance characteristic of both non-generative as well as generative neural graphics pipelines on modern hardware platforms, and asks the question: Does neural graphics need hardware support? The research is segmented into three primary areas: non-generative neural graphics via Neural Representations (NRs), generative neural graphics via Latent Diffusion Models (LDMs), and performance enhancements through integrating neural graphics with relatively more mature neural super-resolution techniques. Firstly, we study non-generative neural graphics by focusing on Neural Representations (NRs) which form the backbone of various applications like Neural Radiance and Density Fields (NeRF), Neural Signed Distance Functions (NSDF), Neural Volume Rendering (NVR), and Gigapixel Image Approximation (GIA). We identify significant performance gaps when rendering 4K resolution frames at 60 FPS on current GPUs, with a shortfall ranging from 1.51X to 55.50X. To bridge this gap, we propose the Neural Graphics Processing Cluster (NGPC), an architecture that accelerates input encoding and multi-layer perceptron kernels, achieving up to 58.36× improvement in application-level performance. Secondly, in the realm of generative neural graphics, we explore Latent Diffusion Models (LDMs) which have surpassed all the other generative processes in high-fidelity image generation domain. The computational demands of LDMs are immense, particularly at high resolutions. We performed in-depth breakdown analysis of the primitive compute kernels of the latent diffusion models and used our kernel-level analysis to design a scalable architecture – Latent Diffusion Model Processing Unit (LDPU). Our estimates show that LDPU increasing the energy efficiency by up to 572× compared to traditional GPU baselines. Lastly, the dissertation combines neural graphics with neural super-resolution techniques to further enhance the output quality and computational efficiency of neural rendering processes. By employing these techniques, we can render at lower resolutions and upscale the outputs to achieve desired higher resolutions without the computational cost typically associated with high-resolution rendering. This comprehensive study not only underscores the necessity of specialized hardware for neural graphics but also sets a precedent for future innovations in this domain, potentially revolutionizing applications in virtual reality (VR), augmented reality (AR), mixed reality (XR), and beyond.

Graduation Semester

2024-08

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/125559

Copyright and License Information

Copyright 2024 Muhammad Husnain Mubarik

Owning Collections