Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning

We present a new framework to measure the intrinsic properties of (deep) neural networks. While we focus on convolutional networks, our framework can be extrapolated to any network architecture. In particular, we evaluate two network properties, namely, capacity , which is related to expressivity ,...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems 2024-08, Vol.35 (8), p.10380-10393
Main Authors: Badias, Alberto, Banerjee, Ashis G.
Format: Article
Language:English
Subjects:
Algebra
Artificial neural networks
Complexity theory
computational complexity
Deep learning
Image classification
Measurement
Neural networks
Training
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present a new framework to measure the intrinsic properties of (deep) neural networks. While we focus on convolutional networks, our framework can be extrapolated to any network architecture. In particular, we evaluate two network properties, namely, capacity , which is related to expressivity , and compression , which is related to learnability . Both these properties depend only on the network structure and are independent of the network parameters. To this end, we propose two metrics: the first one, called layer complexity , captures the architectural complexity of any network layer; and, the second one, called layer intrinsic power , encodes how data are compressed along the network. The metrics are based on the concept of layer algebra , which is also introduced in this article. This concept is based on the idea that the global properties depend on the network topology, and the leaf nodes of any neural network can be approximated using local transfer functions, thereby allowing a simple computation of the global metrics. We show that our global complexity metric can be calculated and represented more conveniently than the widely used Vapnik-Chervonenkis (VC) dimension. We also compare the properties of various state-of-the-art architectures using our metrics and use the properties to analyze their accuracy on benchmark image classification datasets.
ISSN:2162-237X
2162-2388
2162-2388
DOI:10.1109/TNNLS.2023.3241100