Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning

Deep learning has made tremendous success in computer vision, natural language processing and even visual-semantic learning, which requires a huge amount of labeled training data. Nevertheless, the goal of human-level intelligence is to enable a model to quickly obtain an in-depth understanding give...

Full description

Saved in:
Bibliographic Details
Main Authors: Yin, Chengxiang, Wu, Kun, Che, Zhengping, Jiang, Bo, Xu, Zhiyuan, Tang, Jian
Format: Conference Proceeding
Language:English
Subjects:
Computer vision
Deep learning
Genomics
Integrated circuits
Semantics
Training data
Transfer/Low-shot/Semi/Unsupervised Learning
Vision + language
Visual reasoning and logical representation
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep learning has made tremendous success in computer vision, natural language processing and even visual-semantic learning, which requires a huge amount of labeled training data. Nevertheless, the goal of human-level intelligence is to enable a model to quickly obtain an in-depth understanding given a small number of samples, especially with heterogeneity in the multi-modal scenarios such as visual question answering and image captioning. In this paper, we study the few-shot visual-semantic learning and present the Hierarchical Graph ATtention network (HGAT). This two-stage network models the intra- and inter-modal relationships with limited image-text samples. The main contributions of HGAT can be summarized as follows: 1) it sheds light on tackling few-shot multi-modal learning problems, which focuses primarily, but not exclusively on visual and semantic modalities, through better exploitation of the intra-relationship of each modality and an attention-based co-learning framework between modalities using a hierarchical graph-based architecture; 2) it achieves superior performance on both visual question answering and image captioning in the few-shot setting; 3) it can be easily extended to the semi-supervised setting where image-text samples are partially unlabeled. We show via extensive experiments that HGAT delivers state-of-the-art performance on three widely-used benchmarks of two visual-semantic learning tasks.
ISSN:2380-7504
DOI:10.1109/ICCV48922.2021.00218