A survey of IMU based cross-modal transfer learning in human activity recognition

Kamboj, Abhi

Permalink

https://hdl.handle.net/2142/124581 Copy

Description

Title

A survey of IMU based cross-modal transfer learning in human activity recognition

Author(s)

Kamboj, Abhi

Issue Date

2024-04-30

Director of Research (if dissertation) or Advisor (if thesis)

Do, Minh

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Inertial Measurement Units
• Human Action Recognition
• Transfer Learning
• Cross-modal Learning
• Multimodal Learning
• Sensor Fusion

Language

eng

Abstract

Despite living in a multi-sensory world, most AI models are limited to textual and visual understanding of human motion and behavior. Inertial measurement units (IMUs) provide a salient signal to understand human motion; however, in practice, they have been understudied due to numerous difficulties, including the uniterpretability and lack of data. In fact, full situational awareness of human motion could best be understood through a combination of sensors. In this survey, we investigate how knowledge can be transferred and utilized amongst modalities for Human Activity or Action Recognition (HAR), i.e. cross-modality transfer learning. We motivate the importance and potential of IMU data and its applicability in cross-modality learning as well as the importance of studying the HAR problem. We categorize HAR related tasks by time and abstractness and then compare various types of multimodal HAR datasets. We also distinguish and expound on many related but inconsistently used terms in the literature, such as transfer learning, domain adaptation, representation learning, sensor fusion, and multimodal learning, and describe how cross-modal learning fits with all these concepts. Then, we review the literature on IMU-based cross-modal transfer for HAR. The two main approaches for cross-modal transfer are instance-based transfer, where instances of one modality are mapped to another (e.g. knowledge is transferred in the input space), or feature-based transfer, where the model relates the modalities in an intermediate latent space (e.g. knowledge is transferred in the feature space). Finally, we discuss future research directions and applications in cross-modal HAR.

Graduation Semester

2024-05

Type of Resource

Text

Handle URL

https://hdl.handle.net/2142/124581

Copyright and License Information

Copyright 2024 Abhi Kamboj

Owning Collections