Files in this item

FilesDescriptionFormat

application/pdf

LIZAMA-THESIS-2020.pdf (1MB)
(no description provided)PDF

Description

 Title: Completion of hinge loss has an implicit bias Author(s): Lizama, Justin N Advisor(s): Telgarsky, Matus J Department / Program: Computer Science Discipline: Computer Science Degree Granting Institution: University of Illinois at Urbana-Champaign Degree: M.S. Genre: Thesis Subject(s): implicit regularization hinge loss Abstract: A new loss function is proposed which learns the hinge loss function an infinite number of times pushing $f(x_i)y_i \to \infty$. It is proven that for a linear model on linearly separable data this modified hinge loss function converges in the direction of the $\ell_2$ max-margin separator at a rate of $\bigO\left( \sqrt{d/t} \right)$ where $d$ is the dimension of the data. Then, an explicit formula for the underlying dynamical system of the gradient descent iterates for two-layer linear networks on the inner product loss function is derived. Using the derived dynamical system, a precise explicit algorithm is developed which when implemented reproduces the gradient descent iterates of two-layer ReLU nets on the inner product exactly. This result is studied further to extrapolate conclusions for neural network optimization. Issue Date: 2020-05-12 Type: Thesis URI: http://hdl.handle.net/2142/108189 Rights Information: Copyright 2020 Justin Lizama Date Available in IDEALS: 2020-08-26 Date Deposited: 2020-05
﻿