Date post: | 16-Apr-2017 |
Category: |
Data & Analytics |
Upload: | indico-data |
View: | 2,664 times |
Download: | 0 times |
Deep Advances in Generative Modeling
Alec Radford@AlecRad
March 5th 2016
Generative modelingModeling complex high dimensional data is an open problem.
Deep generative models are currently making progress here.
Various areas of study/application:
unsupervised/representation/manifold learning
generative counterparts of discriminative models
density/likelihood estimation
conditional generation
Examples of Generative Modeling
CNNs and RNNs
Useful Generative Model - Skipthought [1506.06726]
Two promising approaches
Variational Autoencoders (VAE) Kingma and Welling [1312.6114]
Generative Adversarial Networks (GAN) Goodfellow et al. [1406.2661]
encoder Z decoder x̂X
z generator x̂discriminato
r
X
Variational Autoencoder
from Kingma and Welling [1312.6114]
● Theoretically elegant autoencoder● Straightforward to implement● Impose a prior on code space
○ regularization○ allows for sampling
● Optimizes variational lower bound on likelihood
encoder Z decoder x̂X
Generative Adversarial Networks z x̂
discriminator
X
generator
Generative Adversarial Networks z x̂
discriminator
X
generator
VAE Extensions
from Kingma et al. [1406.5298]
Semi-Supervised Learning
from Gregor et al. [1502.04623]
DRAW
GAN Extensions - LAPGAN
Deep convolutional GANs (DCGAN) [1511.06434]
Luke Metz Soumith ChintalaAlec Radford
tl;dr add more layers
indico indico FAIR
Deep convolutional GANs (DCGAN) [1511.06434]
DCGAN Architecture tricks
No fully connected layers
Batch Normalization Ioffe and Szegedy [1502.03167]
Leaky Rectifier in the discriminator
Use Adam Kingma and Ba [1412.6980]
Tweak Adam hyperparameters a bit (lr=0.0002, b1=0.5)
Really really really ridiculously good looking samples
on constrained image distributions :(
Interpolation suggests non-overfitting behavior
Vector arithmetic properties of generator
Generator disentangles objects from scene?
Discriminator learns generalizing object detectors
These are responses on validation examples!
Results on standard supervised tasks
Conditional DCGAN
Conditional DCGAN (unpublished)
Sunrise over the ocean
Beautiful falls and stream
sahara desert sand dunes
Tropical rainforest brazil
Stars of the milkyway at night
IssuesStill not completely stable
especially for deep and higher res
Unconstrained natural images
Even the biggest models underfit
Hard to evaluate
no reliable/straightforward metrics
No inference model
limits kinds of analysis
Little work on conv VAE equivalents
makes comparison difficult
Some funky stuff going on
separate data/sample batchnorm statistics
train with heuristic cost not GAN theory
Hybridizing VAEs and GANs (best of both worlds?)
from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]
Hybridizing VAEs and GANs (best of both worlds?)
from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]
Thanks!
Questions?
indico.io