IDEALS Home University of Illinois at Urbana-Champaign logo The Alma Mater The Main Quad

Non-speech Acoustic Event Detection Using Multimodal Information

Show full item record

Bookmark or cite this item:

Files in this item

File Description Format
PDF Huang_Po-Sen.pdf (2MB) (no description provided) PDF
Title: Non-speech Acoustic Event Detection Using Multimodal Information
Author(s): Huang, Po-Sen
Advisor(s): Hasegawa-Johnson, Mark A.
Department / Program: Electrical & Computer Eng
Discipline: Electrical & Computer Engr
Degree Granting Institution: University of Illinois at Urbana-Champaign
Degree: M.S.
Genre: Masters
Subject(s): Acoustic Event Detection Optical Flow Hidden Markov Models Multistream Hidden Markov Models Coupled Hidden Markov Models Gaussian Mixture Models Support Vector Machines Sensor Fusion Footstep Detection Person Detection.
Abstract: Non-speech acoustic event detection (AED) aims to recognize events that are relevant to human activities associated with audio information. Much previous research has been focused on restricted highlight events, and highly relied on ad-hoc detectors for these events. This thesis focuses on using multimodal data in order to make non-speech acoustic event detection and classification tasks more robust, requiring no expensive annotation. To be specific, the thesis emphasizes designing suitable feature representations for different modalities and fusing the information properly. Two cases are studied in this thesis: (1) Acoustic event detection in a meeting room scenario using single-microphone audio cues and single-camera visual cues. Non-speech event cues often exist in both audio and vision, but not necessarily in a synchronized fashion. We jointly model audio and visual cues in order to improve event detection using multistream HMMs and coupled HMMs (CHMM). Spatial pyramid histograms based on the optical flow are proposed as a generalizable visual representation that does not require training on labeled video data. In a multimedia meeting room non-speech event detection task, the proposed methods outperform previously reported systems leveraging ad-hoc visual object detectors and sound localization information obtained from multiple microphones. (2) Multimodal feature representation for person detection at border crossings. Based on phenomenology of the differences between humans and four-legged animals, we propose using enhanced autocorrelation pattern for feature extraction for seismic sensors, and an exemplar selection framework for acoustic sensors. We also propose using temporal pattens from ultrasonic sensors. We perform decision and feature fusion to combine the information from all three modalities. From experimental results, we show that our proposed methods improve the robustness of the system.
Issue Date: 2012-02-06
Genre: thesis
Rights Information: Copyright 2011 Po-Sen Huang
Date Available in IDEALS: 2012-02-06
Date Deposited: 2011-12

This item appears in the following Collection(s)

Show full item record

Item Statistics

  • Total Downloads: 208
  • Downloads this Month: 4
  • Downloads Today: 0


My Account


Access Key