View a PDF of the paper titled Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization, by Junhyung Park and 2 other authors
Abstract:Benign overfitting occurs when a model fits noisy training data exactly and still generalizes well. We establish this behavior for finite-width two-layer networks with the standard ReLU activation, trained by gradient flow in fixed dimension. Our data-independent initialization combines Gaussian weights with neurons initially active only on shrinking spherical caps. For uniform spherical inputs and any fixed bounded conditional response law, training and excess risks vanish in probability uniformly over all sufficiently late times. The parameters also converge, with probability tending to one, to an interpolating limit whose excess risk vanishes in probability. We use one schedule depending only on sample size and dimension. Our proof connects the network to a kernel flow whose limit combines ridge regression with localized spikes that fit the training residuals. We give explicit sample-size, width and training-time bounds.
Submission history
From: Shiva Kasiviswanathan [view email]
[v1]
Fri, 16 May 2025 18:37:51 UTC (586 KB)
[v2]
Sun, 15 Nov 2026 17:51:45 UTC (55 KB)
![[2505.11621] Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization [2505.11621] Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization](https://techwavedispatch.com/wp-content/uploads/2026/11/250511621-Benign-Overfitting-of-Two-Layer-ReLU-Networks-in-Fixed-Dimension-1024x597.png)