Computing AIC for black-box models using generalized degrees of freedom: A comparison with cross-validation

Generalized degrees of freedom (GDF), as defined by Ye ( 1998 JASA 93:120-131), represent the sensitivity of model fits to perturbations of the data. Such GDF can be computed for any statistical model, making it possible, in principle, to derive the effective number of parameters in machine-learning...

Full description

Saved in:

Bibliographic Details
Published in:	Communications in statistics. Simulation and computation 2018-05, Vol.47 (5), p.1382-1396
Main Authors:	Hauenstein, Severin, Wood, Simon N., Dormann, Carsten F.
Format:	Article
Language:	eng
Subjects:	Binary data Boosted regression trees Data perturbation Data points Degrees of freedom Model complexity Primary 62 Random forest Secondary 07 Statistical models
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Generalized degrees of freedom (GDF), as defined by Ye ( 1998 JASA 93:120-131), represent the sensitivity of model fits to perturbations of the data. Such GDF can be computed for any statistical model, making it possible, in principle, to derive the effective number of parameters in machine-learning approaches and thus compute information-theoretical measures of fit. We compare GDF with cross-validation and find that the latter provides a less computer-intensive and more robust alternative. For Bernoulli-distributed data, GDF estimates were unstable and inconsistently sensitive to the number of data points perturbed simultaneously. Cross-validation, in contrast, performs well also for binary data, and for very different machine-learning approaches.
ISSN:	0361-0918 1532-4141