Generative Adversarial NetworksPrem Seetharaman, CS349
1
Generative models
Generative models are unsupervised. Their only task is to generate samples that look like real samples.
Real samples of faces ->
Generative models
Generative models are unsupervised. Their only task is to generate samples that look like real samples.
Real samples of faces -> Generated faces!
Generative models
With generative models, we can leverage unlabeled training data.
Generative models
Some tasks have many solutions.
Example: generating speech for the same text in different styles.
Generative models
Example: Multiple translations for a given text.
Generative models
Central question: how do we model our training data?
Generative models
In these faces for example, there are several “latent” factors at play. What are they?
Distributions
Recall that training example - an image, an audio clip, etc. - can be thought of as a point in space.
Distributions
Recall that training example - an image, an audio clip, etc. - can be thought of as a point in space.
Distributions
Let’s put all of our training data into this space.
Matching distributions
Then, generating data means to sample from this space such that the sampled points are near real points.
Red: sampled points Blue: real points
Matching distributions
Recall that one way we did that was by fitting a GMM to the space and sampling from it.
GMM w/ 1 component
Matching distributions
As the number of components gets higher, the sampled points start to overlap with the real points.
GMM w/ 3 components
15
GMM with 1 component
16
GMM with 10 components
17
GMM with 50 components
Lots of ways to make this happen
Many other methods for generative modeling: 1. Variational autoencoders 2. Autoregressive models 3. Generative adversarial
networks
Abstracting a bitThe goal is to be able to construct points similar to the real points.
Generator
Abstracting a bit
This is a good generator.
Generator
Abstracting a bit
This is a bad generator.
Generator
Abstracting a bitGenerators can produce multiple points that should hopefully be diverse.
Generator
Abstracting a bitA generator that can only produce one “type” of real sample is bad. Called mode collapse.
Generator
Abstracting a bitA generator that covers all of your training data is good! It produces diverse samples.
Generator
A loss function for generators?So we have some definition for what makes a generator “good” or “bad”. How do we turn this into a loss function?
Generator
Enter the discriminatorThe discriminator tries to tell a generated sample apart from a real sample.
Discriminator
Enter the discriminatorThe discriminator tries to tell whether a sample is real or fake.
Discriminator REAL
Enter the discriminatorThe discriminator tries to tell whether a sample is real or fake.
Discriminator FAKE
Enter the discriminatorThe discriminator tries to tell whether a sample is real or fake.
Discriminator REAL
Enter the discriminatorThe discriminator tries to tell whether a sample is real or fake.
Discriminator FAKE
The whole pictureGenerator attempts to create a data point.
DiscriminatorGenerator
The whole pictureIf this generated point is NOT close to the real points in the space…
DiscriminatorGenerator
The whole pictureThe discriminator will easily say that it’s a fake point.
DiscriminatorGenerator FAKE
The whole pictureBut if the generated point IS close to a real point…
DiscriminatorGenerator
The whole pictureThen the discriminator might be fooled!
DiscriminatorGenerator REAL
Generator’s goalFool the discriminator into thinking the generated points are real.
DiscriminatorGenerator REAL
Discriminator’s goalDon’t be fooled by the generator.
DiscriminatorGenerator FAKE
Generator takes as input a latent codeThe output of the generator is conditioned on random noise.
DiscriminatorGenerator REAL
Late
nt c
ode
Generator takes as input a latent codeThe generator interprets this random noise as a point on a manifold.
DiscriminatorGenerator REAL
Late
nt c
ode
Generator takes as input a latent code…and maps points on the manifold to a point in the space of the data.
DiscriminatorGenerator REAL
Late
nt c
ode
Key points
Generator
Latent code
To create good fake data, a generator must know what real data looks like.
The generator attempts to generate samples that are likely under the true distribution.
The distribution is learned implicitly via the discriminator.
Key points
The discriminator learns how to tell real data from fake data.
Thus, a GAN is like learning your loss function, rather than keeping it fixed! Discriminator
REAL or FAKE
This is a game with two players!Player 1: The Generator Player 2: The Discriminator
DiscriminatorGenerator REAL
Late
nt c
ode
This is a game with two players!Player 1: The Generator Player 2: The Discriminator
DiscriminatorGenerator REAL
Late
nt c
ode
Generator is a deep network!
Discriminator is a deep network!
This is a game with two players!Each player seeks to maximize its own success and minimize the success of the other -> a minimax game
DiscriminatorGenerator REAL
Late
nt c
ode
Generator is a deep network!
Discriminator is a deep network!
This is a game with two players!The two players are at odds with one another and are trained in an alternating fashion.
DiscriminatorGenerator REAL
Late
nt c
ode
Generator is a deep network!
Discriminator is a deep network!
Keep the generator fixed. Generate a bunch of fake points.
DiscriminatorGenerator REAL
Late
nt c
ode
At first, the generator has no idea how to create fake points. The data is not close to the real points.
Take a bunch of real points…
DiscriminatorGenerator REAL
Late
nt c
ode
…and feed them to the discriminator. The discriminator should output that these are real points. Use this as a training signal and backprop to train the discriminator.
FAKE
REAL
FAKE
REAL
Take a bunch of real points…
DiscriminatorGenerator REAL
Late
nt c
ode
…and feed them to the discriminator. The discriminator should output that these are real points. Use this as a training signal and backprop to train the discriminator.
FAKE
REAL
FAKE
REAL
Backprop“REAL/FAKE” loss to discriminator
Take a bunch of fake points…
DiscriminatorGenerator FAKE
Late
nt c
ode
…and feed them to the discriminator. The discriminator should output that these are fake points.
REAL
FAKE
REAL
FAKE
Backprop“REAL/FAKE” loss to discriminator
Now keep discriminator fixed. Generate fake points again…
DiscriminatorGenerator
Late
nt c
ode
This time we will update the generator instead. Note how different points may be generated this time - this is due to the randomness of the latent code.
Take these fake points…
DiscriminatorGenerator
Late
nt c
ode
…and feed them to the discriminator. This time, we want the generator to fool the discriminator.
FAKE
FAKE
FAKE
FAKE
FAKE
Take these fake points…
DiscriminatorGenerator
Late
nt c
ode
However the generator doesn’t know the true data distribution yet! So the discriminator easily catches that we are making forgeries.
FAKE
FAKE
FAKE
FAKE
FAKE
Update the generator.
DiscriminatorGenerator
Late
nt c
ode
Update the generator by backpropping through the discriminator.
Backprop negation of “REAL/FAKE” loss to generator.
FAKE
FAKE
FAKE
FAKE
FAKE
Alternate between generator updates and discriminator updates.
DiscriminatorGenerator
Late
nt c
ode
REAL
REAL
FAKE
REAL
FAKE
Backprop“REAL/FAKE” loss to discriminator
Train the discriminator to tell fake points from real points…
Alternate between generator updates and discriminator updates.
DiscriminatorGenerator
Late
nt c
ode
Generate new fake points…
Alternate between generator updates and discriminator updates.
DiscriminatorGenerator
Late
nt c
ode
FAKE
FAKE
REAL
REAL
REAL
Backprop negation of “REAL/FAKE” loss to generator.
Update the generator to fool the discriminator.
Alternate between generator updates and discriminator updates.
DiscriminatorGenerator
Late
nt c
ode
As this process continues…
Alternate between generator updates and discriminator updates.
DiscriminatorGenerator
Late
nt c
ode
…the generator will start to put its generated points near the real points…
Alternate between generator updates and discriminator updates.
DiscriminatorGenerator
Late
nt c
ode
..leading to the discriminator to believe they are real points!
REAL
REAL
REAL
REAL
REAL
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!
Intuition: how do you tell two coins apart?
Flip them and compare them to what you expect.
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!
Green: you want high likelihood that x is real (because it is).
Red: you want low likelihood that G(z) is real (because it isn’t).
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!
D(x) -> coin flip for D on x
D(G(z)) -> coin flip for D on G(x)
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!
Expect 1, try to get 1
Expect 0, try to get 0
Time to add a bit of math
Credit to Bhiksha Raj at CMU for these images!