Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models

During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited regions. This leads to a significant waste of computation, especially for minor editing operations. In this work, we present Spatially Sparse Inference (SSI), a general-...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2023-12, Vol.45 (12), p.14465-14480
Main Authors: Li, Muyang, Lin, Ji, Meng, Chenlin, Ermon, Stefano, Han, Song, Zhu, Jun-Yan
Format: Article
Language:English
Subjects:
Algorithms
Computation
Computational modeling
Convolutional codes
Diffusion layers
Diffusion models
Editing
efficiency
Engines
Feature maps
GAN
Graphics processing units
Hardware
Image coding
image editing
Inference
Reduction
Reuse
Silicon germanium
sparse
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited regions. This leads to a significant waste of computation, especially for minor editing operations. In this work, we present Spatially Sparse Inference (SSI), a general-purpose technique that selectively performs computation for edited regions and accelerates various generative models, including both conditional GANs and diffusion models. Our key observation is that users prone to gradually edit the input image. This motivates us to cache and reuse the feature maps of the original image. Given an edited image, we sparsely apply the convolutional filters to the edited regions while reusing the cached features for the unedited areas. Based on our algorithm, we further propose Sparse Incremental Generative Engine (SIGE) to convert the computation reduction to latency reduction on off-the-shelf hardware. With about 1%-area edits, SIGE accelerates DDPM by 3.0\times 3.0× on NVIDIA RTX 3090 and 4.6\times 4.6× on Apple M1 Pro GPU, Stable Diffusion by 7.2\times 7.2× on 3090, and GauGAN by 5.6\times 5.6× on 3090 and 5.2\times 5.2× on M1 Pro GPU. Compared to our conference paper, we enhance SIGE to accommodate attention layers and apply it to Stable Diffusion. Additionally, we offer support for Apple
ISSN:0162-8828
1939-3539
1939-3539
2160-9292
DOI:10.1109/TPAMI.2023.3316020