In the Eye of the Beholder: Gaze and Actions in First Person Video

We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera. To facilitate our research, we first introduce the EGTEA Gaze+ dataset. Our dataset comes with videos, gaze tracking data, hand masks and action...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2023-06, Vol.45 (6), p.6731-6747
Main Authors: Li, Yin, Liu, Miao, Rehg, James M.
Format: Article
Language:English
Subjects:
Action recognition
Activity recognition
Algorithms
Annotations
Attention
Benchmark testing
Cameras
Convolution
Datasets
deep probabilistic models
Eye movements
Feature recognition
first person vision
Fixation, Ocular
gaze estimation
Gaze tracking
Hand
Humans
Stochastic processes
Three-dimensional displays
video analysis
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera. To facilitate our research, we first introduce the EGTEA Gaze+ dataset. Our dataset comes with videos, gaze tracking data, hand masks and action annotations, thereby providing the most comprehensive benchmark for First Person Vision (FPV). Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV. Our method describes the participant's gaze as a probabilistic variable and models its distribution using stochastic units in a deep network. We further sample from these stochastic units, generating an attention map to guide the aggregation of visual features for action recognition. Our method is evaluated on our EGTEA Gaze+ dataset and achieves a performance level that exceeds the state-of-the-art by a significant margin. More importantly, we demonstrate that our model can be applied to larger scale FPV dataset-EPIC-Kitchens even without using gaze, offering new state-of-the-art results on FPV action recognition.
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2021.3051319