1MIT 2IIIS, Tsinghua University 3Adobe Research 4CMU
Differentiable Augmentationfor Data-Efficient GAN Training
NeurIPS 2020
Shengyu Zhao1,2 Zhijian Liu1 Ji Lin1 Song Han1Jun-Yan Zhu3,4
Computation AlgorithmComputation Algorithm
Data Is Expensive
FFHQ dataset: 70,000 selective post-processed human faces
Months or even years to collect the data,
along with prohibitive annotation costs.
ImageNet dataset: millions of images from diverse categories
Big Data
Sometimes Not Even Possible
3
Sometimes Not Even Possible
4
“The aim of the new directorate is to support fundamental scientific research ― with specific goals in mind. This is
not about solving incremental technical problems. As one example, in artificial intelligence, the focus would not be
on further refining current algorithms, but rather on developing profoundly new approaches that would enable
machines to "learn" using much smaller data sets ― a fundamental advance that would eliminate the need to
access immense data sets. Success in this work would have a double benefit: seeding economic benefits for the
U.S. while reducing the pressure to weaken privacy and civil liberties in pursuit of more "training" data.”
― L. Rafael Reif
GANs Heavily Deteriorate Given Limited Data
5
Generated samples of StyleGAN2. The quality is poor given limited data.
Ob
am
a
100
imag
es
Generated samples of StyleGAN2 (Karras et al.)
using only hundreds of images
Ca
t (Sim
ard
et a
l.)
160
imag
es
Do
g (S
imard
et a
l.)
389
imag
es
GANs Heavily Deteriorate Given Limited Data
GANs Heavily Deteriorate Given Limited Data
11.1
23.1
36.0
0
5
10
15
20
25
30
35
40
100% training data 20% training data 10% training data
FID↓
StyleGAN2 (baseline) + DiffAugment (ours)
CIFAR-10
Discriminator Overfitting
Data Augmentation
9
Data augmentation: enlarge datasets without collecting new samples.
10
How to Augment GANs?
#1 Approach: Augment reals only
Augment reals only: the same artifacts appear on the generated images.
Artifacts from Color jittering
Artifacts from Translation
Artifacts from Cutout (DeVries et al.)
Generated images
Augment 𝑫 only: the unbalanced optimization cripples training.
#2 Approach: Augment reals & fakes for 𝑫 only
Our approach (DiffAugment): Augment reals + fakes for both 𝐷 and 𝐺
#3 Approach: Differentiable Augmentation (Ours)
Color
Translation
Cutout
Color
Translation
Cutout
fakes reals
11.1
23.1
36.0
9.9 12.2
14.5
0
5
10
15
20
25
30
35
40
100% training data 20% training data 10% training data
FID↓
StyleGAN2 (baseline) + DiffAugment (ours)
11.1
23.1
36.0
9.9 12.2
14.5
0
5
10
15
20
25
30
35
40
100% training data 20% training data 10% training data
FID↓
StyleGAN2 (baseline) + DiffAugment (ours)
Our Results
CIFAR-10
ImageNet Generation (25% training data)
15
Low-Shot Generation
Ob
am
a
100
imag
es
Ca
t (Sim
ard
et a
l.)
160
imag
es
Do
g (S
imard
et a
l.)
389
imag
es
100-Shot Generation
Generated samples of StyleGAN2 (baseline)
Generated samples of StyleGAN2 + DiffAugment (ours)
0
10
20
30
40
50
60
Performance
FID
↓
Scale/Shift (Noguchi et al.) MineGAN (Wang et al.) TransferGAN (Wang et al.) FreezeD (Mo et al.) Ours
1
10
100
1000
10000
100000
Data
# T
rain
ing Im
ages
Fine-Tuning vs. Ours
No pre-training
100-shot Obama
Fine-Tuning vs. Ours
19
TransferGAN (a state-of-the-art fine-tuning method)
70,000 FFHQ faces + 100 Obama portraits
Ours
only 100 Obama portraits
100-Shot Interpolation
The smooth interpolation results suggest little overfitting of our method even given only 100 imagesof Obama, grumpy cat, panda, the Bridge of Sighs, the Medici Fountain, the Temple of Heaven, and Wuzhen.
Data-Efficient Deep Learning
21
Various factual and ethical reasons could cause limited data available.
This research will help alleviate these limitations.
Rare incidents Privacy concerns Under-represented subpopulations
Thanks for listening!
22
Our code, datasets, and models are publicly available at
https://github.com/mit-han-lab/data-efficient-gans.
![An Energy and Bandwidth Efficient Ray Tracing Architecturedkopta/papers/hwrt_hpg13.pdfAila and Karras [Aila and Karras 2010] focus on reducing off-chip bandwidth by partitioning the](https://static.documents.pub/doc/80x56/61468ae77599b83a5f004a06/an-energy-and-bandwidth-efficient-ray-tracing-dkoptapapershwrthpg13pdf-aila.jpg)
