Asymptotic Convexity of Wide and Shallow Neural Networks

Borkar, Vivek S.; Pandit, Parthe

Permalink

https://hdl.handle.net/2142/130252 Copy

Description

Title

Asymptotic Convexity of Wide and Shallow Neural Networks

Author(s)

• Borkar, Vivek S.
• Pandit, Parthe

Issue Date

2025-09-17

Keyword(s)

• Shallow and wide networks
• Truncated epigraphs
• Minkowski sums
• Convex minorant
• Stochastic gradient descent

Abstract

There has been considerable interest in analyzing the observed empirical success of wide neural networks, both shallow and deep. A small sample of the enormous activity in this domain can be found in [2], [3], [5]–[8], [10], [14], [15]. In this short note, for a simple model of shallow and wide networks, we establish asymptotic convexity for the maps of error as a function of the parameters, equivalently their truncated epigraphs (defined below) using a known convexification effect of Minkowski sums of compact sets in Rd. In particular, this suggests that the limiting optimization problem implicit in neural network training for the infinitely wide neural network is a simple minimization problem, therefore amenable to SGD: all local minima are global minima. While the latter property is a consequence of convexity, it does not imply convexity and it may begin to hold for neural networks with ‘sufficiently large width’. This leads to a plausible explanation of the empirically observed good performance of shallow and wide networks.

Publisher

Allerton Conference on Communication, Control, and Computing

Series/Report Name or Number

2025 61st Allerton Conference on Communication, Control, and Computing Proceedings

ISSN

2836-4503

Type of Resource

Text

Genre of Resource

Conference Paper/Presentation

Language

eng

Handle URL

https://hdl.handle.net/2142/130252

Copyright and License Information

Copyright 2025 is held by Vivek S. Borkar and Parthe Pandit.

Owning Collections