Memories are One-to-Many Mapping Alleviators in Talking Face Generation

Talking face generation aims at generating photorealistic video portraits of a target person driven by input audio. According to the nature of audio to lip motions mapping, the same speech content may have different appearances even for the same person at different occasions. Such one-to-many mappin...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2024-06, Vol.PP, p.1-12
Main Authors: Tang, Anni, He, Tianyu, Tan, Xu, Ling, Jun, Li, Runnan, Zhao, Sheng, Bian, Jiang, Song, Li
Format: Article
Language:English
Subjects:
Faces
Feature extraction
Lips
memory learning
one-to-many mapping
Rendering (computer graphics)
Shape
talking face generation
Training
Visualization
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Talking face generation aims at generating photorealistic video portraits of a target person driven by input audio. According to the nature of audio to lip motions mapping, the same speech content may have different appearances even for the same person at different occasions. Such one-to-many mapping problem brings ambiguity during training and thus causes inferior visual results. Although this one-to-many mapping could be alleviated in part by a two-stage framework (i.e., an audioto- expression model followed by a neural-rendering model), it is still insufficient since the prediction is produced without enough information (e.g., emotions, wrinkles, etc.). In this paper, we propose MemFace to complement the missing information with an implicit memory and an explicit memory that follow the sense of the two stages respectively. More specifically, the implicit memory is employed in the audio-to-expression model to capture high-level semantics in the audio-expression shared space, while the explicit memory is employed in the neural-rendering model to help synthesize pixel-level details. Our experimental results show that our proposed MemFace surpasses all the state-of-theart results across multiple scenarios consistently and significantly.
ISSN:0162-8828
1939-3539
DOI:10.1109/TPAMI.2024.3409380