Dynamical systems perspectives in machine learning

Satpathi, Siddhartha

Permalink

https://hdl.handle.net/2142/113008 Copy

Description

Title

Dynamical systems perspectives in machine learning

Author(s)

Satpathi, Siddhartha

Issue Date

2021-07-12

Director of Research (if dissertation) or Advisor (if thesis)

Srikant, Rayadurgam

Doctoral Committee Chair(s)

Srikant, Rayadurgam

Committee Member(s)

• Beck, Carolyn L
• Chatterjee, Sabyasachi
• Hu, Bin

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Neural Network
• error-log
• LDA
• gradient descent

Abstract

We look at two facets of machine learning from a perspective of dynamical systems, that is, the data generated from a dynamical system and the iterative inference algorithm posed as a dynamical system. In the former, we look at time series data which is generated from a mixture of processes. Each process exists for a fixed duration and generates i.i.d categorical data points during that duration. More than one process can coexist at a particular time. The goal is to find the number of such hidden processes and the characteristic categorical distribution of each. This model is motivated by the problem of finding error events in error-logs from a mobile communication network. In the second direction, we consider the problem of regression using a shallow overparameterized neural network. Broadly, we look at training the neural network with the gradient descent algorithm on the squared loss function and discuss the generalization properties of the output of the gradient descent algorithm on an unseen data point. We look at two problems in this setting. First, we discuss the effect of l2 regularization on the squared loss and discuss how different strength of regularization provides a trade-off on the generalization of the neural network. Second, we look at squared loss without regularization and discuss the generalization properties when the true function we are trying to learn belongs to the class of polynomials in the presence of noisy samples. In both the problems, we consider the gradient descent algorithm as a dynamical system and use tools from control theory to analyze this dynamical system.

Graduation Semester

2021-08

Type of Resource

Thesis

Permalink

http://hdl.handle.net/2142/113008

Copyright and License Information

Copyright 2021 Siddhartha Satpathi

Owning Collections