InfoGAN : Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
ISL Lab Seminar Hansol Kang
: Mutual Information
Contents
Review
InfoGAN
Experiment
Summary
2019-04-09
2
I. Review
ReviewVanilla GAN, DCGAN
Review• Concept of GAN
2019-04-09
4
VsD GF1F1
F1
F1
FakeR1
Review• Concept of GAN
2019-04-09
5
VsD GFake?R1
@
F1
Review• Adversarial nets
2019-04-09
6
)))]((1[log()]([log),(maxmin )(~)(~ zGDExDEGDV zpzxpxDG zdata−+=
Smart D
)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata−+Real case
Fake case
1
)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata−+
0
should be 0
should be 0
1
Log(x)cf.
Stupid D
)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata−+Real case
Fake case
0
)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata−+
1
should be negative infinity
should be negative infinityD perspective, it should be maximum.
Review• Adversarial nets
2019-04-09
7
)))]((1[log()]([log),(maxmin )(~)(~ zGDExDEGDV zpzxpxDG zdata−+=
Generator
)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata−+
1
should be negative infinity1
Log(x)cf.
G perspective, it should be minimum.
Smart G
Stupid G )))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata−+
0
should be 0
Review2019-04-09
8
• GAN
1) Global Optimality of datag pp =
2) Convergence of Algorithm
D GVs
x
)(xpdata
“Generative Adversarial Networks”
Goal Method
Review2019-04-09
9
• DCGAN : network
DG
“쟤들뭐하냐?”
“CNN이 MLP보다훨씬낫지롱”D“우리가짱이야.” G
Vanilla GAN DCGAN
Review2019-04-09
10
• DCGAN : latent space
0 1
0.1
0.15
0.18
0.143
0.5
0.45
0.47
0.473
0.9
0.95
0.96
0.9370.607±
II. InfoGAN
InfoGANConcept, Mutual Information, Variational method, Results
InfoGAN2019-04-09
12
• Concept
D
GNoise
Real(Kurt Cobain)
Fake(Not Kurt Cobain)
…
Latent code
Dataset
InfoGAN2019-04-09
13
• Concept
D
GNoise
Real(Kurt Cobain)
Fake(Not Kurt Cobain)
…
Latent code
Dataset
0
Simplify
InfoGAN2019-04-09
14
• Concept
D
GNoise
Real(Kurt Cobain)
Fake(Not Kurt Cobain)
…
Latent code
Dataset
0.5
Simplify
InfoGAN2019-04-09
15
• Concept
D
GNoise
Real(Kurt Cobain)
Fake(Not Kurt Cobain)
…
Latent code
Dataset
1
Simplify
InfoGAN2019-04-09
16
• Concept
D
GNoise
Real(Kurt Cobain)
Fake(Not Kurt Cobain)
…
Latent code
Dataset
0.0010.0081.0000.007
…0.005
But…
? : 실제 latent code의 구조는 복잡하여해석이어려움(entangled).
InfoGAN2019-04-09
17
• Concept
GNoise
Latent code
0.0010.0081.0000.007
…0.005
? : 실제 latent code의 구조는 복잡하여해석이어려움(entangled).
Let's make the latent code simple.
The proper generation is difficult.
[0.001, 0.008, …, 005] [005]
Latent code
0.0010.0081.0000.007
…0.005
0000…1
Z C
Z C : Condition
How about adding latent code?Idea
InfoGAN2019-04-09
18
• Concept
GLatent code
Z C
“뭐야? 그러면 C를 Z 옆에바로붙이면되는거야?”
[0.001, 0.008, …, 005 | 0, 0, … 1]
z c[0.001, 0.008, …, 005 | 1, 0, … 0]
z c
[0.001, 0.008, …, 005 ]
z[0.001, 0.008, …, 005 ]
z
Ignore the additional latent code c
Cost function을수정하여 c의영향을만듦.),(maxmin GDVDG
(Mutual Information)
InfoGAN2019-04-09
19
• Mutual Information
)|()();( YXHXHYXI −=)()()();(
YPXPYXPYXI
=
ISL Browserhttps://isl.cnu.ac.kr
Supervised Learning
검색결과약 107,000,000개
Unsupervised Learning
검색결과약 13,400,000개
Clustering
검색결과약 40,900,000개
Supervised Learning Clustering
검색결과약 25,300,000개
Unsupervised Learning Clustering
검색결과약 7,770,000개
Deep Learning
검색결과약 1,380,000,000개
07754.0)( =SLP
00971.0)( =ULP
02964.0)( =CP
0.01833)( =CSLP
0.00563)( =CULP
97551.702964.007754.0
01833.0)()()(
=×
=CPSLPCSLP
56190.1902964.000971.0
00563.0)()()(
=×
=CPULPCULP
“두사건사이의연관성파악”
InfoGAN2019-04-09
20
• Mutual Information
: Generator와 c 사이의연관성을 cost로정의( )),(;),(),(maxmin czGcIGDVGDVIDGλ−=
Maximize
Hard to maximize directly as it requires access to the posterior )|( xcP
( )[ ] ( ))(||)|()(|log),,( )|( zpxzqKLzgxExL xzq φθφθφ +−=
),,(min xL θφReconstruction Error Regularization
VAE Seminar (18.07.23)
InfoGAN2019-04-09
21
• Variational method
)|( xcP )|( xcQIntractable(Very complicated) Tractable(e.g Gaussian)
( )),(;),(),(maxmin czGcIGDVGDVIDGλ−=
( ) ( )),(|)(),(; czGcHcHczGcI −=
( ) ( ) ( ) ),(),(|ln),(,),(| czdcdGczGcPczGcPczGcH ∫∫−=
( ) dydxxyPxyPxyH )|(ln),(| ∫∫−=
dydxxyPxPxyP )|(ln)()|(∫∫−=
c.f Conditional Entropy
Product rule
( ) ( ) ( ) ),(),(|ln),(),(| czdcdGczGcPczGPczGcP∫∫−=
( ) ( ) ( ) ),(),(|ln),(|),( czdcdGczGcPczGcPczGP∫∫−=
( ) [ ] ),()|'(ln),( )|(~' czdGxcPEczGP xcPc∫−=[ ][ ])|'(ln)|(~'),(~ xcPEE xcPcczGx−= ( ) [ ][ ])|'(ln)(),(; )|(~'),(~ xcPEEcHczGcI xcPcczGx+=
(1)
(2)
InfoGAN2019-04-09
22
• Variational method
( ) [ ][ ])|'(ln)(),(; )|(~'),(~ xcPEEcHczGcI xcPcczGx+= (2)
( )
=
)'()|'(ln)|'(||)|'( )|(~' xcQ
xcPExcQxcPD xcPcKL
)|'(ln)'(ln )|(~')|(~' xcQExcPE xcPcxcPc −=
( ) )|'(ln)|'(||)|'()'(ln )|(~')|(~' xcQExcQxcPDxcPE xcPcKLxcPc +=
( ) ( )[ ])|'(ln)|'(||)|'()(),(; )|(~'),(~ xcQExcQxcPDEcHczGcI xcPcKLczGx ++= (3)0≥
( )
=
)()(ln|| ~ xQ
xPEQPD PxKL
c.f KL Divergence
( ) 0|| =QPDKL : 동일분포
[ ])|'(ln)( )|(~'),(~ xcQEEcH xcPcczGx+≥ (4)
)|( xcQ Tractable distribution
InfoGAN2019-04-09
23
• Variational method
( ) [ ])|'(ln)(),(; )|(~'),(~ xcQEEcHczGcI xcPcczGx+≥ (4)
)|( xcP : 여전히남음.
[ ] [ ]),'(),( |~',|~,~|~,~ yxfEyxfE yXxxYyXxxYyXx =
Lemma
[ ])|(ln)(),( ),(~),(~ xcQEcHQGL czGxcPcI +=
[ ])|'(ln)( )|(~'),(~ xcQEEcH xcPcczGx+=
),(),(),,(maxmin QGLGDVQGDV IInfoGANDGλ−=
(5)
InfoGAN2019-04-09
24
• Results
InfoGAN2019-04-09
25
• Results
InfoGAN2019-04-09
26
• Results
InfoGAN2019-04-09
27
• Results
III. Experiment
ExperimentMNIST, FashionMNIST, LSUN
Experiment• Results#1 MNIST (continuous)
2019-04-09
29
Epoch 1 Epoch 5 Epoch 10
Epoch 30 Epoch 50 GIF
Experiment• Results#1 MNIST (categorical)
2019-04-09
30
Epoch 1 Epoch 5 Epoch 10
Epoch 30 Epoch 50 GIF
Experiment• Results#2 Fashion MNIST (continuous)
2019-04-09
31
Epoch 1 Epoch 5 Epoch 10
Epoch 30 Epoch 50 GIF
Experiment• Results#2 Fashion MNIST (categorical)
2019-04-09
32
Epoch 1 Epoch 5 Epoch 10
Epoch 30 Epoch 50 GIF
Experiment• Results#3 LSUN (continuous)
2019-04-09
33
Epoch 1 Epoch 2 Epoch 3
Epoch 4 Epoch 5
Experiment• Results#3 LSUN (categorical)
2019-04-09
34
Epoch 1 Epoch 2 Epoch 3
Epoch 4 Epoch 5
Experiment• Results#3 LSUN (categorical, ep 5)
2019-04-09
35
IV. Summary
SummarySummary, Future Work
Summary2019-04-09
37
• Latent code에추가적인 code를할당하여학습함.
• 기존의 GAN 학습법으로는추가된 code를무시하기에새로운학습방법이필요함.
• Mutual information을통해추가된 code와네트워크간의상호연관성을부여함.
• 주어진 code는그형태에따라 categorical(discrete)or continuous로구분되며, 실제실험을통해적절히학습되는것을확인함.
(cGAN과비슷한접근법)
Future work2019-04-09
38
GAN Research
Vanilla GAN
DCGAN
InfoGAN
LS GAN
BEGAN
Pix2Pix
Cycle GAN
Novel GAN(about depth)
Tools
Document
Programming
PyTorch
Python executable & UI
I Know What You Did Last Faculty
C++ Coding Standard
Mathematical theory
LSM applications
Other Research
Level Processor
Ice Propagation
&