Bridging prediction and planning for human-centered autonomy

Huang, Zhe

Permalink

https://hdl.handle.net/2142/127228 Copy

Description

Title

Bridging prediction and planning for human-centered autonomy

Author(s)

Huang, Zhe

Issue Date

2024-11-27

Director of Research (if dissertation) or Advisor (if thesis)

Driggs-Campbell, Katherine

Doctoral Committee Chair(s)

Driggs-Campbell, Katherine

Committee Member(s)

• Gupta, Saurabh
• Amato, Nancy
• Mitra, Sayan

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Robotics
• Artificial Intelligence
• Human-Robot Interaction

Abstract

Human-centered autonomy needs to develop a strong understanding of human behavior and use the understanding effectively to accomplish challenging human-involved tasks. This dissertation studies human prediction, robot planning, and integration of both to build human-centered autonomy frameworks, and investigates multiple domains including autonomous driving, social navigation, collaborative manufacturing, and collaborative cooking. This dissertation has three parts of studies which span from human prediction, through integration of prediction and planning, to robot planning across the levels of partially observable intent and fully observable trajectory. In the part of human prediction, I introduce a simultaneous intention estimation and trajectory prediction algorithm by combining particle filtering and recurrent neural networks for modeling pedestrians in autonomous driving applications. I investigate pedestrian attention during crowd motion and observe that an agent pays attention to a limited number of neighbors. I use Gumbel Softmax to learn a sparse interaction graph of agents, which is used to aggregate information from the most important neighbors to make multi-agent trajectory prediction. In the part of integration of prediction and planning, I derive intention tracking on a probabilistic graphical model of intention-evolving human-robot collaboration, and extend the derivation to intention hierarchy. I track human intentions at different levels to switch robot control mode and guide robot task and motion planning in a manufacturing setting. I develop language-driven intention tracking by proposing a language probabilistic graphical model and harnessing large language models and vision language models. I use language-driven intention tracking to build an embodied artificial intelligence agent capable of open world understanding, open human intention tracking, and intention-grounded planning, and demonstrate a robot sous-chef application. In addition, I study how to take human trajectory prediction error into consideration for probabilistically safe crowd navigation. I design interaction-aware conformal prediction algorithm to capture mutual influence between a robot and humans during navigation, by alternating uncertainty-aware model predictive control with calibrated human trajectory prediction error, and decision-dependent conformal prediction with human simulation assuming the robot executes the generated motion plan. In the part of robot planning, I present a motion planning algorithm Neural Informed rapidly-exploring random tree star (NIRRT*) by integrating a point-based neural network for guidance state inference into Informed RRT*. In NIRRT*, the Informed RRT* helps the point-based network focus inference on the region relevant to the planning problem for solution improvement, and the point-based network assists the Informed RRT* to sample critical states in the focused subset for optimal convergence acceleration.

Graduation Semester

2024-12

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/127228

Copyright and License Information

Copyright 2024 Zhe Huang

Owning Collections