Deep Advances in Generative Modeling

Alec Radford@AlecRad

March 5th 2016

Generative modelingModeling complex high dimensional data is an open problem.

Deep generative models are currently making progress here.

Various areas of study/application:

unsupervised/representation/manifold learning

generative counterparts of discriminative models

density/likelihood estimation

conditional generation

Examples of Generative Modeling

CNNs and RNNs

Useful Generative Model - Skipthought [1506.06726]

Two promising approaches

Variational Autoencoders (VAE) Kingma and Welling [1312.6114]

Generative Adversarial Networks (GAN) Goodfellow et al. [1406.2661]

encoder Z decoder x̂X

z generator x̂discriminato

r

X

Variational Autoencoder

from Kingma and Welling [1312.6114]

● Theoretically elegant autoencoder● Straightforward to implement● Impose a prior on code space

○ regularization○ allows for sampling

● Optimizes variational lower bound on likelihood

encoder Z decoder x̂X

Generative Adversarial Networks z x̂

discriminator

X

generator

Generative Adversarial Networks z x̂

discriminator

X

generator

VAE Extensions

from Kingma et al. [1406.5298]

Semi-Supervised Learning

from Gregor et al. [1502.04623]

DRAW

GAN Extensions - LAPGAN

Deep convolutional GANs (DCGAN) [1511.06434]

Luke Metz Soumith ChintalaAlec Radford

tl;dr add more layers

indico indico FAIR

Deep convolutional GANs (DCGAN) [1511.06434]

DCGAN Architecture tricks

No fully connected layers

Batch Normalization Ioffe and Szegedy [1502.03167]

Leaky Rectifier in the discriminator

Use Adam Kingma and Ba [1412.6980]

Tweak Adam hyperparameters a bit (lr=0.0002, b1=0.5)

Really really really ridiculously good looking samples

on constrained image distributions :(

Interpolation suggests non-overfitting behavior

Vector arithmetic properties of generator

Generator disentangles objects from scene?

Discriminator learns generalizing object detectors

These are responses on validation examples!

Results on standard supervised tasks

Conditional DCGAN

Conditional DCGAN (unpublished)

Sunrise over the ocean

Beautiful falls and stream

sahara desert sand dunes

Tropical rainforest brazil

Stars of the milkyway at night

IssuesStill not completely stable

especially for deep and higher res

Unconstrained natural images

Even the biggest models underfit

Hard to evaluate

no reliable/straightforward metrics

No inference model

limits kinds of analysis

Little work on conv VAE equivalents

makes comparison difficult

Some funky stuff going on

separate data/sample batchnorm statistics

train with heuristic cost not GAN theory

Hybridizing VAEs and GANs (best of both worlds?)

from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]

Hybridizing VAEs and GANs (best of both worlds?)

from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]

Thanks!

Questions?

indico.io