Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 1
Deep-Learning:
Unsupervised Generative modelsDeep Belief Networks
Deep Stacked AutoEncoders
Generative Adversarial Networks
Pr. Fabien MOUTARDECenter for Robotics
MINES ParisTech
PSL Université Paris
http://people.mines-paristech.fr/fabien.moutarde
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 2
Acknowledgements
During preparation of these slides, I got inspiration and borrowed
some slide content from several sources, in particular:
• Fei-Fei Li & J. Johnson & S. Yeung: course on Generative Models
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture13.pdf
• I. Kokkinos: slides of a CentraleParis course on Deep Belief Networks
http://cvn.ecp.fr/personnel/iasonas/course/DL5.pdf
• I. Goodfellow: NIPS’2016 tutorial on Generative Adversarial Networks (GANs)
https://media.nips.cc/Conferences/2016/Slides/6202-Slides.pdf
• Binglin, Shashank & Bhargav: A short tutorial on Generative Adversarial
Networks (GANs) http://slazebni.cs.illinois.edu/spring17/lec11_gan.pdf
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 3
Outline
• Unsupervised Learning and Generative Models
• Deep Belief Networks (DBN)
and Deep Boltzman Machine (DBM)
• Autoencoders
• Generative Adversarial Networks (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 4
Deep vs Shallow Learning techniques overview
DEEPSHALLOW
GAN
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 5
Supervised vs Unsupervised
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 6
Unsupervised Learning
Examples:
General framework:
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 7
Generative models
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 8
Why Generative?
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 9
Why generative?
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 10
Taxonomy of Generative Models
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 11
Outline
• Unsupervised Learning and Generative Models
• Deep Belief Networks (DBN)
and Deep Boltzman Machine (DBM)
• Autoencoders
• Generative Adversarial Networks (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 12
Deep Belief Networks (DBN)
• One of first Deep-Learning models• Proposed by G. Hinton in 2006• Generative probabilistic model (mostly UNSUPERVISED)
For capturing high-order correlations of observed/visible data (à pattern analysis, or synthesis); and/or characterizingjoint statistical distributions of visible data
Greedy successive UNSUPERVISED learning of layersof Restricted Boltzmann Machine (RBM)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 13
Restricted Boltzmann Machine (RBM)
h, hidden
(~ latent variables)
v, observed
Modelling probability distribution as:
with « Energy » E given by
NB: connections are
BI-DIRECTIONAL
(with same weight)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 14
Training RBM
Finding q=(W,a,b) maximizing likelihood !"#$ %&(v) of dataset S
ó minimize NegLogLikelihood '*+#, log %&(-)
So objective = find ./ = argMin&
'0+#,
01log %&(-2)
Algo: Contrastive Divergence
» Gibbs sampling used inside a gradient descent procedure
In binary input case: with% -3 = 4 5) = 6 73 89:;35 6 < =
>?>? 8 4% 52 = 4 -) = 6 @2 892;:-
Independance within layers è % - 5) = A3% -3 5 % 5 -) = A
2% 52 -and
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 15
Repeat:
1. Take a training sample v, compute B C1 = D +) = E F1 8G1;:+and sample a vector h from this probability distribution
2. Compute positive gradient as outer product HI = +JC = +CK3. From h, compute B +LN = D C) = E ON 8G:;NC and sample reconstructed v',
then resample h' using B CL1 = D +L) = E F1 8G1;:+L[Gibbs sampling single step; should theoretically be repeated until convergence]
4. Compute negative gradient as outer product HP = +LJCL = +LCLK5. Update weight matrix by QG = R HI ' HP = R +CK ' +SCLK6. Update biases a and b analogously: QO = R + ' +L and QF = R C ' CL
Contrastive Divergence algo
Gibbs sampling
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 16
Use of trained RBM
• Input data "completion" : set some vi thencompute h, and generate compatible full samples
• Generating representative samples
• Classification if trainedwith inputs=data+label
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 17
Modeling of input data distribution from trained RBM
Initial data is in blue, reconstructed in red (and green line connects each data point with
reconstructed one).
Learnt energy function:
minima created where data points are
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 18
Interpretation of trained RBM hidden layer
• Look at weights of hidden nodes à low-level features
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 19
Why go deeper with DBN ?
DBN: upper layers à more « abstract » features
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 20
Learning of DBN
Greedy learning of successive layers
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 21
Using low-dim final featuresfor clustering
Much better results than clustering in input space
or using other dimension reduction (PCA, etc…)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 22
Example application of DBN:Clustering of documents in database
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 23
Image Retrievalapplication example of DBN
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 24
DBN supervised tuning
UNSUPERVISED SUPERVISED
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 25
Outline
• Unsupervised Learning and Generative Models
• Deep Belief Networks (DBN)
and Deep Boltzman Machine (DBM)
• Autoencoders
• Generative Adversarial Networks (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 26
Autoencoders
Learn qq
and pF
in order to minimize reconstruction cost:
à unsupervised learning of latent variables,
and of a generative model
T =0UVWU 'WU X =0
UBY Z[ WU 'WU X
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 27
Variants of autoencoders
• Denoising autoencoders
• Sparse autoencoders
• Stochastic autoencoders
• Contractive autoencoders
• VARIATIONAL autoencoders
• …
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 28
Deep Stacked Autoencoders
Proposed by Yoshua Bengio in 2007
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 29
Training of StackedAutoencoers
Greedy layerwise training:
for each layer k, use backpropagation to minimize
|| Ak(h(k))-h(k) ||2 (+ regularization cost l Sij |Wij|
2)
possibly + additional term for "sparsity"
etc…
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 30
Variational AutoEncoders(VAE)
KL = Kullback-Leibler divergence (a.k.a. ‘relative entropy’)
KL(Q || P) measures how different are distributions
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 31
Outline
• Unsupervised Learning and Generative Models
• Deep Belief Networks (DBN)
• Autoencoders
• Generative Adversarial Networks (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 32
Generative Adversarial Network
Goal: generate « artificial » but credible examples
credible = sampled from same probability distribution p(x)
Idea: instead of trying to explicitly estimate p(x),
1. LEARN a transformation G from a simple and known
distribution (e.g. random) into X,
2. then sampling z à generate realistic samples G(z)
[Introduced in 2014 by Ian Goodfellow et al.
(incl. Yoshua Bengio) from University of Montreal]
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 33
GAN’s architecture
(Gaussian/Uniform).
Z ~ latent representation of the image.
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 34
GAN training: minimax two-player game!
Joint training of D and G
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 35
GAN training detail
In practice, alternate Discriminator training
(gradient ascent) and Generator training:
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 36
Training the Discriminator
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 37
Training the Generator
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 38
Convolutional Generatorfor GAN
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 39
Example of fake samplesgenerated by GAN
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 40
Trajectory in latent spaceà continous image transform
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 41
« Arithmetic »of latent vectors
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 42
Image-to-Image translation
Link to an interactive demo of this paper
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 43
GAN for synthesis of realistic images
"Video-to-Video Synthesis", NeurIPS’2018 [Nvidia+MIT]Using Generative Adversarial Network (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 44
Domain transfer!
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 45
Summary and perspectives on DBN/DBM/DSA/VAE/GAN
• Intrinsicly UNSUPERVISED
è can be used on UNLABELLED DATA
• Impressive results in Image Retrieval
• DBN/DBM/VAE = Generative probabilistic models
• GAN = most promising generative model, with
already many remarkable & exciting applications
• Strong potential for enhancement of datasets and
for ultra-realistic synthetic data
• Interest for "creative« /artistic computing?
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 46
Any QUESTIONS ?