2026-09-24T00:30:03Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/127369 2026-02-03 col_2142_5131 col_2142_8888 com_2142_5130 com_2142_8887 com_2142_234

Audiovisual processing for generation and enhancement Fan, Xulin 2024-11-27 Hasegawa-Johnson, Mark Allan Multimodal Signal Processing Speech Processing Speech Enhancement Signal Processing eng application/pdf en text Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-12-01 The student, Xulin Fan, accepted the attached license on 2024-11-26 at 14:26. The student, Xulin Fan, submitted this Thesis for approval on 2024-11-26 at 14:39. This Thesis was approved for publication on 2024-11-27 at 09:26. DSpace SAF Submission Ingestion Package generated from Vireo submission #21403 on 2025-03-28 at 14:43:27 With advances in deep neural networks and the increasing availability of computational power, machine learning researchers working on unimodal tasks, such as text, audio, or vision, have begun to explore methods that address multimodal problems, which involve inputs from multiple modalities. Each modality carries distinct types of information represented in different formats. For example, audiovisual tasks typically involve an audio-visual aligned video, where the video channel is essentially a sequence of images represented as a 4D tensor (number of frames, RGB channels, height, width), while the audio channel is a 1D signal sampled at a much higher rate. Popular multimodal neural architectures generally consist of three stages: modality-specific encoders, modality fusion, and one or more task-specific decoders. To handle inputs of varying formats, a common design choice is to employ modality-specific encoders, which project each modality into a learnable embedding space. This embedding space is structured to facilitate cross-modal similarity and ease the subsequent fusion process. The modality fusion step, however, is tailored to the requirements of the specific downstream task. For instance, an audiovisual task that outputs an audio signal, such as audiovisual speech enhancement, may require high temporal resolution, whereas tasks that produce textual outputs, such as audiovisual automatic speech recognition, may prioritize different fusion strategies. In this thesis, we explore various design choices for two audiovisual tasks: audio-driven talking head synthesis and audiovisual target speaker extraction. Through extensive experimentation, we identify key considerations for adapting transformer-based and diffusion-based methods to multimodal scenarios. 2024-12 Thesis https://hdl.handle.net/2142/127369 Copyright 2024 Xulin Fan Electrical & Computer Eng Electrical & Computer Engr University of Illinois at Urbana-Champaign M.S. Thesis