Structured model learning for adaptive robot generalists

Chen, Haonan

Permalink

https://hdl.handle.net/2142/132575 Copy

Description

Title

Structured model learning for adaptive robot generalists

Author(s)

Chen, Haonan

Issue Date

2025-12-05

Director of Research (if dissertation) or Advisor (if thesis)

Driggs-Campbell, Katherine

Doctoral Committee Chair(s)

Driggs-Campbell, Katherine

Committee Member(s)

• Li, Yunzhu
• Amato, Nancy
• Schwing, Alexander

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Robotics
• Machine Learning
• Artificial Intelligence

Abstract

As robots move from factories into homes, hospitals, and warehouses, the grand challenge is to create adaptive robot generalists: systems that can learn diverse manipulation skills, transfer knowledge across tasks and embodiments, and operate robustly under varying sensing conditions. However, robotic manipulation in such unstructured environments requires integrating diverse sensory modalities, long-horizon reasoning, and learning from heterogeneous data sources. Traditional monolithic learning approaches remain brittle, lack modularity, and struggle to generalize across tasks, embodiments, and sensing configurations. This dissertation presents a unified framework based on structured model learning that decomposes the robot learning problem into modular, reusable components. This approach yields policies that generalize across tasks, adapt to new contexts, and operate robustly under diverse conditions. The framework is developed progressively across manipulation challenges of increasing complexity. It begins by integrating learned policies with analytical models for hybrid control in traffic simulation and robotic manipulation. This foundation extends to hierarchical decomposition, where tasks are separated into high-level goals and low-level skills learned from direct physical human guidance. For contact-rich scenarios involving multiple objects, learned dynamics models combined with behavior primitives enable reasoning about object interactions. The framework further separates state prediction (using diffusion models) from action generation (using inverse dynamics models), enabling coordinated bimanual manipulation of deformable objects. Beyond structural decomposition within individual tasks, the framework enables generalization across embodiments and sensing modalities. Tool-centric representations bridge embodiment differences, enabling robots to learn manipulation skills from human videos. For multimodal sensor fusion, the framework uses separate policy experts for vision and touch, coordinated through learned routing that dynamically balances their contributions based on task context. Validated on tasks including contact-rich stowing, occluded object retrieval, in-hand reorientation, and coordinated bimanual manipulation, the results demonstrate that structured model learning provides a scalable foundation for adaptive robot generalists capable of operating effectively in complex real-world environments.

Graduation Semester

2025-12

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/132575

Copyright and License Information

Copyright 2025 Haonan Chen

Owning Collections