Towards micro foundation models for robust multimodal IoT sensing

Kimura, Tomoyoshi

Permalink

https://hdl.handle.net/2142/127289 Copy

Description

Title

Towards micro foundation models for robust multimodal IoT sensing

Author(s)

Kimura, Tomoyoshi

Issue Date

2024-12-10

Director of Research (if dissertation) or Advisor (if thesis)

Abdelzaher, Tarek F

Department of Study

Siebel School Comp & Data Sci

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Foundation models
• Internet of Things
• Self-Supervised Learning

Abstract

The thesis argues and advocates for the feasibility and utility of micro foundation models (µFMs), a key direction for future smart IoT/CPS systems that exploits advances in self-supervised pre-training to support robust intelligent inference tasks. We demonstrate key beneficial properties such as latent representation independence from the downstream task, robustness to domain shifts, ability to learn from unlabeled data, and enhanced structural resiliency of edge systems. Importantly, we demonstrate the emergence of these properties after pre-training with only moderate amounts of unlabeled data, earning the name µFMs. To make the argument, evaluate model efficacy, and surface some of the underlying challenges, this thesis describes a vibration-based µFM, called VibroFM, pre-trained with moderate amounts of unlabeled acoustic and seismic sensing data, to support target classification and tracking applications. VibroFM is pre-trained in an environment-agnostic fashion using unlabeled sensor data. It can then be fine-tuned to a given deployment using a small amount of in-situ labeled data. The paper shows that VibroFM (i) improves the robustness of several downstream tasks, (ii) efficiently adapts to different environmental conditions (using only small amounts of fine-tuning), (iii) allows few-shot generalization to unseen targets, and (iv) generalizable to multiple downstream tasks each at a minimal labeling and system cost. We further show that VibroFM can execute in real-time on embedded sensor nodes. We compare the robustness and performance of VibroFM to conventional supervised deep neural networks, showing the advantages of the former. Combined with the feasibility of executing µFMs in resource-limited settings and the sufficiency of only moderate amounts of data for their pre-training, we conclude the importance of micro foundation models as a promising research direction for the IoT/CPS community.

Graduation Semester

2024-12

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/127289

Copyright and License Information

Copyright 2024 Tomoyoshi Kimura

Owning Collections