High‐Resolution Neural Face Swapping for Visual Effects

In this paper, we propose an algorithm for fully automatic neural face swapping in images and videos. To the best of our knowledge, this is the first method capable of rendering photo‐realistic and temporally coherent results at megapixel resolution. To this end, we introduce a progressively trained...

Full description

Saved in:
Bibliographic Details
Published in:Computer graphics forum 2020-07, Vol.39 (4), p.173-184
Main Authors: Naruniec, J., Helminger, L., Schroers, C., Weber, R.M.
Format: Article
Language:English
Subjects:
Ablation
Algorithms
Blending
CCS Concepts
Computing methodologies → Image manipulation
Neural networks
Training
Unsupervised learning
Visual effects
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we propose an algorithm for fully automatic neural face swapping in images and videos. To the best of our knowledge, this is the first method capable of rendering photo‐realistic and temporally coherent results at megapixel resolution. To this end, we introduce a progressively trained multi‐way comb network and a light‐ and contrast‐preserving blending method. We also show that while progressive training enables generation of high‐resolution images, extending the architecture and training data beyond two people allows us to achieve higher fidelity in generated expressions. When compositing the generated expression onto the target face, we show how to adapt the blending strategy to preserve contrast and low‐frequency lighting. Finally, we incorporate a refinement strategy into the face landmark stabilization algorithm to achieve temporal stability, which is crucial for working with high‐resolution videos. We conduct an extensive ablation study to show the influence of our design choices on the quality of the swap and compare our work with popular state‐of‐the‐art methods.
ISSN:0167-7055
1467-8659
DOI:10.1111/cgf.14062