Constrained Dimensionality Reduction Using a Mixed-Norm Penalty Function with Neural Networks

Reducing the dimensionality of a classification problem produces a more computationally-efficient system. Since the dimensionality of a classification problem is equivalent to the number of neurons in the first hidden layer of a network, this work shows how to eliminate neurons on that layer and sim...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on knowledge and data engineering 2010-03, Vol.22 (3), p.365-380
Main Authors: Huiwen Zeng, Trussell, H.J.
Format: Article
Language:English
Subjects:
Algorithms
Applied sciences
Artificial intelligence
Artificial neural networks
Classification
Computer networks
Computer science
control theory
systems
Constraint optimization
Constraints
Data processing. List processing. Character string processing
Degradation
Exact sciences and technology
Hyperspectral imaging
Image classification
Memory organisation. Data processing
mixed-norm penalty
Multi-layer neural network
Multilayers
Neural networks
Neurons
Optimization
Pattern recognition. Digital image processing. Computational geometry
Penalty function
Pruning
Reduction
Software
Studies
Training data
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reducing the dimensionality of a classification problem produces a more computationally-efficient system. Since the dimensionality of a classification problem is equivalent to the number of neurons in the first hidden layer of a network, this work shows how to eliminate neurons on that layer and simplify the problem. In the cases where the dimensionality cannot be reduced without some degradation in classification performance, we formulate and solve a constrained optimization problem that allows a trade-off between dimensionality and performance. We introduce a novel penalty function and combine it with bilevel optimization to solve the constrained problem. The performance of our method on synthetic and applied problems is superior to other known penalty functions such as weight decay, weight elimination, and Hoyer's function. An example of dimensionality reduction for hyperspectral image classification demonstrates the practicality of the new method. Finally, we show how the method can be extended to multilayer and multiclass neural network problems.
ISSN:1041-4347
1558-2191
DOI:10.1109/TKDE.2009.107