University of Illinois Urbana-Champaign

TranDynaMo: A comparative study of transformer and GRU performance in modeling dynamical systems

Pandey, Amogh

This item's files can only be accessed by the System Administrators group.

Permalink

https://hdl.handle.net/2142/122269

Description

Title
TranDynaMo: A comparative study of transformer and GRU performance in modeling dynamical systems
Author(s)
Pandey, Amogh
Issue Date
2023-12-06
Director of Research (if dissertation) or Advisor (if thesis)
Mehr, Negar
Department of Study
Electrical & Computer Eng
Discipline
Electrical & Computer Engr
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
  • Transformers, Large Language Models
  • ChatGPT
  • GPT
  • LLM
  • Dynamics Learning
  • Dynamics Modeling
  • RNN
  • GRU
  • Control System Dynamics
  • Dynamical Models
Abstract
Dynamics modeling is critical to many robotics and control tasks and applications, such as motion planning and trajectory optimization. However, obtaining a model through current methods is difficult and expensive, so instead, there have been many efforts to come up with data-driven methods to learn dynamics models. With the popularity and success of Large Language Models (LLMs) such as ChatGPT, we were inspired to find a way to leverage transformers, the building blocks of ChatGPT, to solve the dynamics learning problem. In this thesis, we study the viability of utilizing transformers for dynamics modeling. We do this by first introducing transformers and posing dynamics learning as a sequence-to-sequence modeling task. We then create a GPT2-based dynamics learning framework called TranDynaMo and test its dynamics modeling performance for various dynamical systems, such as chaotic systems and systems with limit cycles, by giving it an initial condition and seeing how well it can predict the trajectory. We then compare TranDynaMo against a GRU-based framework. We find that when the dynamics learning framework is required to predict parts of a trajectory significantly beyond the horizon of the training trajectories it was given, TranDynaMo beats the GRU-based framework; however, if the training and testing trajectories are of almost the same horizon, both perform satisfactorily, but the GRU-based framework outperforms TranDynaMo, thus indicating that the generalization of transformers is better.
Graduation Semester
2023-12
Type of Resource
Thesis
Copyright and License Information
Copyright 2023 Amogh Pandey

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois