Date post: | 22-Nov-2015 |
Category: |
Documents |
Upload: | sherjil-ozair |
View: | 16 times |
Download: | 0 times |
On the Equivalence Between Deep NADE and Generative
Stochastic Networks Li Yao, Sherjil Ozair, Kyunghyun Cho, Yoshua Bengio
Outline
Deep NADE
How Deep NADE is a GSN
Fast sampling algorithm for Deep NADE
Annealed GSN Sampling
Deep Neural AutoRegressive Distribution Estimators
ith MLP models P(vi|v
Orderless Training
Sample a random permutation of input ordering
Sample a random pivot index j
Backprop through error in indices j to inputs < j
Sampling Sample v1 conditioned on
nothing
Sample v2 conditioned on sampled v1
Sample v3 conditioned on sampled v1, v2
Sample vi conditioned on v1, v2, vi-1
Time complexity of one pass
single-hidden layer: O(#v * #h)
double-hidden layer: O(#v * #h2)
multi-hidden layer: O(#v * k * #h2)
Generative Stochastic Networks
Model P(V | V), where V is a corrupted V
Modeling P(V | V) is easier than modeling P(V)
Gibbs sampling: alternatively sample from P(V | V) and C(V | V)
X
Orderless Training
Sample a random permutation of input ordering
Sample a random pivot index j
Backprop through error in indices j to inputs < j
Also trains a GSN
Corruption process: remove bits j from V
Deep NADE models P(Vj | V
GSN Sampling on NADE
Repeat:
Sample an ordering, and a pivot index j
Sample P(Vj | V
Evaluation: Accuracy
Evaluation: Performance
Evaluation: Qualitative
Thanks!