Data-driven bimanual catching: Integrating model-based and learning-based approaches for dynamic object grasping

Xu, Yiyang

Permalink

https://hdl.handle.net/2142/127288 Copy

Description

Title

Data-driven bimanual catching: Integrating model-based and learning-based approaches for dynamic object grasping

Author(s)

Xu, Yiyang

Issue Date

2024-12-12

Director of Research (if dissertation) or Advisor (if thesis)

Driggs-Campbell, Katherine

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Bimanual robotic arms
• Robotic catching

Abstract

This thesis presents a novel approach to dynamic object catching using a bimanual robotic system, which integrates both model-based and data-driven methods. Traditional unimanual robotic manipulation systems have limitations in handling large, complex, fast-moving, or irregularly shaped objects. Catching with two hands is more intuitive to humans as it provides greater stability and balance compared to using one hand. We aim to have the robotic system mimic human motion patterns, making it capable of handling a broader range of tasks. We propose a teacher-student policy training framework consisting of a model-based teacher policy and a learning-based student policy. Both policies take raw camera input and output the target grippers' poses for catching objects in flight. The model-based approach utilizes a sampling based technique to generate optimal grasp candidates, and it serves as both a baseline and a foundation for data collection used by the data-driven system. The data-driven approach incorporates PointNet++ and a Transformer encoder to process spatial-temporal point cloud data. Additionally, the network is combined with a physics-based ballistic motion model to estimate the object’s landing position. The deep learning model predicts the 6D rotation for both grippers and 3D translation residuals compensated for object-specific dynamic not captured by the physics-based model. The proposed data-driven system is evaluated in a simulated environment, and it demonstrates significant improvements in grasp success rates and generalization across various object types compared to the baseline model-based method. The results highlight the potential for deploying such a system in real-world applications.

Graduation Semester

2024-12

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/127288

Copyright and License Information

Copyright 2024 Yiyang Xu

Owning Collections