Attention-based Knowledge Distillation in Scene Recognition: the Impact of a DCT-driven Loss

Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduce...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on circuits and systems for video technology 2023-09, Vol.33 (9), p.1-1
Main Authors: Lopez-Cifuentes, Alejandro, Escudero-Vinolo, Marcos, Bescos, Jesus, SanMiguel, Juan C.
Format: Article
Language:English
Subjects:
2D frequency transform
Artificial neural networks
Computational modeling
convolutional neural networks
deep learning
Discrete cosine transforms
Distillation
Feature extraction
Knowledge distillation
Knowledge engineering
Multi-attention
Recognition
Scene recognition
Semantics
Task analysis
Training
Two dimensional analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduced via maximum activation maps, as the source knowledge. In this paper, we propose and analyze the use of a 2D frequency transform of the activation maps before transferring them. We pose that-by using global image cues rather than pixel estimates, this strategy enhances knowledge transferability in tasks such as scene recognition, defined by strong spatial and contextual relationships between multiple and varied concepts. To validate the proposed method, an extensive evaluation of the state of the art in scene recognition is presented. Experimental results provide strong evidence that the proposed strategy enables the student network to better focus on the relevant image areas learnt by the teacher network, hence leading to better descriptive features and higher transferred performance than every other state-of-the-art alternative. We publicly release the training and evaluation framework used in this paper at http://www-vpu.eps. uam.es/publications/DCTBasedKDForSceneRecognition.
ISSN:1051-8215
1558-2205
DOI:10.1109/TCSVT.2023.3250031