Files in this item

FilesDescriptionFormat

application/pdf

application/pdfLIZAMA-THESIS-2020.pdf (1MB)Restricted to U of Illinois
(no description provided)PDF

Description

Title:Completion of hinge loss has an implicit bias
Author(s):Lizama, Justin N
Advisor(s):Telgarsky, Matus J
Department / Program:Computer Science
Discipline:Computer Science
Degree Granting Institution:University of Illinois at Urbana-Champaign
Degree:M.S.
Genre:Thesis
Subject(s):implicit
regularization
hinge
loss
Abstract:A new loss function is proposed which learns the hinge loss function an infinite number of times pushing $f(x_i)y_i \to \infty$. It is proven that for a linear model on linearly separable data this modified hinge loss function converges in the direction of the $\ell_2$ max-margin separator at a rate of $\bigO\left( \sqrt{d/t} \right)$ where $d$ is the dimension of the data. Then, an explicit formula for the underlying dynamical system of the gradient descent iterates for two-layer linear networks on the inner product loss function is derived. Using the derived dynamical system, a precise explicit algorithm is developed which when implemented reproduces the gradient descent iterates of two-layer ReLU nets on the inner product exactly. This result is studied further to extrapolate conclusions for neural network optimization.
Issue Date:2020-05-12
Type:Thesis
URI:http://hdl.handle.net/2142/108189
Rights Information:Copyright 2020 Justin Lizama
Date Available in IDEALS:2020-08-26
Date Deposited:2020-05


This item appears in the following Collection(s)

Item Statistics