Files in this item



application/pdf3086036.pdf (5MB)Restricted to U of Illinois
(no description provided)PDF


Title:Multimodal Fusion With Applications to Audio -Visual Speech Recognition
Author(s):Chu, Stephen Mingyu
Doctoral Committee Chair(s):Huang, Thomas S.
Department / Program:Electrical Engineering
Discipline:Electrical Engineering
Degree Granting Institution:University of Illinois at Urbana-Champaign
Subject(s):Computer Science
Abstract:Differences in the characteristics of the intermodal couplings in audio-visual speech recognition and in multichannel biometrics defy a universal fusion method for both applications. For audio-visual speech modeling, we propose a novel sensory fusion method based on the coupled hidden Markov models (CHMMs). The CHMM framework allows the fusion of two temporally coupled information sources to take place as an integral part of the statistical modeling process. An important advantage of the CHMM-based fusion method lies in its ability to model asynchronies between the audio and visual channels. We describe two approaches to carry out inference and learning in CHMMs. The first is an exact algorithm derived by extending the forward-backward procedure used in hidden Markov model (HMM) inference. The second method relies on the model transformation strategy that maps the state space of a CHMM onto the state space of a classic HMM, and therefore facilitates the development of sophisticated audio-visual speech recognition systems using existing infrastructures. For multichannel biometrics, we introduce a general formulation based on the late integration paradigm and address the environmental robustness issue through multichannel fusion. Based on this formulation, two effective approaches to carry out environment-adaptive decision fusion are developed: the environmental confidence weighting method and the optimal channel weighting method.
Issue Date:2003
Description:87 p.
Thesis (Ph.D.)--University of Illinois at Urbana-Champaign, 2003.
Other Identifier(s):(MiAaPQ)AAI3086036
Date Available in IDEALS:2015-09-25
Date Deposited:2003

This item appears in the following Collection(s)

Item Statistics