EAPT: Efficient Attention Pyramid Transformer for Image Processing

Recent transformer-based models, especially patch-based methods, have shown huge potentiality in vision tasks. However, the split fixed-size patches divide the input features into the same size patches, which ignores the fact that vision elements are often various and thus may destroy the semantic i...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2023, Vol.25, p.50-61
Main Authors: Lin, Xiao, Sun, Shuzhou, Huang, Wei, Sheng, Bin, Li, Ping, Feng, David Dagan
Format: Article
Language:English
Subjects:
Ablation
attention mechanism
classification
Communication
Convolutional neural networks
Costs
Encoding
Feature extraction
Formability
Image classification
Image processing
Image segmentation
Modules
object detection
Object recognition
Patches (structures)
pyramid
Semantic segmentation
Semantics
Task analysis
Transformer
Transformers
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recent transformer-based models, especially patch-based methods, have shown huge potentiality in vision tasks. However, the split fixed-size patches divide the input features into the same size patches, which ignores the fact that vision elements are often various and thus may destroy the semantic information. Also, the vanilla patch-based transformer cannot guarantee the information communication between patches, which will prevent the extraction of attention information with a global view. To circumvent those problems, we propose an Efficient Attention Pyramid Transformer (EAPT). Specifically, we first propose the Deformable Attention, which learns an offset for each position in patches. Thus, even with split fixed-size patches, our method can still obtain non-fixed attention information that can cover various vision elements. Then, we design the Encode-Decode Communication module (En-DeC module), which can obtain communication information among all patches to get more complete global attention information. Finally, we propose a position encoding specifically for vision transformers, which can be used for patches of any dimension and any length. Extensive experiments on the vision tasks of image classification, object detection, and semantic segmentation demonstrate the effectiveness of our proposed model. Furthermore, we also conduct rigorous ablation studies to evaluate the key components of the proposed structure.
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2021.3120873