A Generalization Gap Estimation for Overparameterized Models via the Langevin Functional Variance

This article discusses the estimation of the generalization gap, the difference between generalization performance and training performance, for overparameterized models including neural networks. We first show that a functional variance, a key concept in defining a widely-applicable information cri...

Full description

Saved in:
Bibliographic Details
Published in:Journal of computational and graphical statistics 2023-10, Vol.32 (4), p.1287-1295
Main Authors: Okuno, Akifumi, Yano, Keisuke
Format: Article
Language:English
Subjects:
Algorithms
Approximation
Bayesian inference
Estimation
Langevin dynamics
Neural networks
Overparameterization
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This article discusses the estimation of the generalization gap, the difference between generalization performance and training performance, for overparameterized models including neural networks. We first show that a functional variance, a key concept in defining a widely-applicable information criterion, characterizes the generalization gap even in overparameterized settings where a conventional theory cannot be applied. As the computational cost of the functional variance is expensive for the overparameterized models, we propose an efficient approximation of the function variance, the Langevin approximation of the functional variance (Langevin FV). This method leverages only the first-order gradient of the squared loss function, without referencing the second-order gradient; this ensures that the computation is efficient and the implementation is consistent with gradient-based optimization algorithms. We demonstrate the Langevin FV numerically by estimating the generalization gaps of overparameterized linear regression and nonlinear neural network models, containing more than a thousand of parameters therein. Supplementary materials for this article are available online.
ISSN:1061-8600
1537-2715
DOI:10.1080/10618600.2023.2197488