Learning-based optimal and robust control: A policy optimization perspective

Keivan Esfahani, Darioush

Permalink

https://hdl.handle.net/2142/129357 Copy

Description

Title

Learning-based optimal and robust control: A policy optimization perspective

Author(s)

Keivan Esfahani, Darioush

Issue Date

2024-12-12

Director of Research (if dissertation) or Advisor (if thesis)

• Dullerud, Geir E
• Hu, Bin
• Seiler, Peter J

Doctoral Committee Chair(s)

• Dullerud, Geir E
• Hu, Bin

Committee Member(s)

• West, Mathew
• Salapaka, Srinivasa M

Department of Study

Mechanical Sci & Engineering

Discipline

Mechanical Engineering

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Control Theory
• Policy Optimization
• Reinforcement Learning

Abstract

Reinforcement learning (RL) has achieved remarkable success in domains such as video games, the game of Go, and complex robotic tasks. Central to this success is policy optimization (PO), a subclass of RL where policies—parameterized mappings from observations to actions—are iteratively optimized to enhance system performance. PO’s flexibility, scalability, and data-driven nature make it effective for tackling challenging problems involving nonlinear dynamics, high-dimensional systems, or insufficient parameterization. These qualities also make PO a promising approach for addressing control theory problems, where traditional methods often rely on well-defined models that may not be available. Despite its broad applicability, PO-based control synthesis is inherently nonconvex, posing significant challenges for achieving strong performance guarantees. This dissertation aims to address these challenges in two key areas of control theory: optimal control and robust control. These classical areas provide a solid foundation for evaluating the effectiveness of PO methods relative to traditional approaches and serve as benchmarks for studying the sample complexity of RL algorithms, which learn to control systems through repeated interactions. The first part of this dissertation focuses on two optimal control problems. The first problem addresses the linear quadratic Gaussian (LQG) control problem through a case study on a first-order single-input single-output (SISO) system using PO methods. Despite the inherent nonconvexity of the optimization landscape, we demonstrate that by appropriately parameterizing the policy class and incorporating problem-specific considerations, global convergence can be achieved in this specific scenario. This result provides a foundation for addressing the general LQG problem. Additionally, we investigate the asymptotic stability and region of attraction (ROA) of feedback control systems that utilize large-scale neural network policies and high-dimensional observations. The second part of the dissertation focuses on two cornerstone problems in robust control: structured \(\mathcal{H}_{\infty}\)-synthesis and \(\mu\)-synthesis. Both problems are characterized by nonconvex nonsmooth optimization landscapes, making their solutions particularly challenging. While advanced nonconvex nonsmooth optimization techniques have been developed for the model-based setting, much less is known about their model-free counterparts and the associated sample complexity. In this work, we extend these approaches to the model-free setting, providing theoretical guarantees on sample complexity and convergence performance. We also conduct extensive numerical studies to validate the practical utility of the proposed algorithms. Together, these contributions establish a unified framework that addresses current challenges and paves the way for a comprehensive PO theory for synthesizing dynamical systems with guaranteed stability, robustness, and optimality. Additionally, this work highlights the complexities of solving synthesis problems under partial observations and nonlinearities, offering a vision for scalable and reliable solutions to meet the demands of modern control systems.

Graduation Semester

2025-05

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/129357

Copyright and License Information

Copyright 2025 Darioush Keivan Esfahani

Owning Collections