Bottleneck Conditional Density Estimation Rui Shu 1 Hung H. Bui 2 Mohammad Ghavamzadeh 3 Abstract We introduce a new framework for training deep generative models for high-dimensional condi- tional density estimation. The Bottleneck Con- ditional Density Estimator (BCDE) is a vari- ant of the conditional variational autoencoder (CVAE) that employs layer(s) of stochastic vari- ables as the bottleneck between the input x and target y, where both are high-dimensional. Crucially, we propose a new hybrid training method that blends the conditional generative model with a joint generative model. Hybrid blending is the key to effective training of the BCDE, which avoids overfitting and provides a novel mechanism for leveraging unlabeled data. We show that our hybrid training procedure en- ables models to achieve competitive results in the MNIST quadrant prediction task in the fully- supervised setting, and sets new benchmarks in the semi-supervised regime for MNIST, SVHN, and CelebA. 1. Introduction Conditional density estimation (CDE) refers to the prob- lem of estimating a conditional density p(y|x) for the input x and target y. In contrast to classification where the tar- get y is simply a discrete class label, y is typically contin- uous or high-dimensional in CDE. Furthermore, we want to estimate the full conditional density (as opposed to its conditional mean in regression), an important task the con- ditional distribution has multiple modes. CDE problems in which both x and y are high-dimensional have a wide range of important applications, including video prediction, cross-modality prediction (e.g. image-to-caption), model estimation in model-based reinforcement learning, and so on. 1 Stanford University 2 Adobe Research 3 DeepMind (The work was done when all the authors were with Adobe Research). Cor- respondence to: Rui Shu <[email protected]>. Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017 by the author(s). Classical non-parametric conditional density estimators typically rely on local Euclidean distance in the original in- put and target spaces (Holmes et al., 2012). This approach quickly becomes ineffective in high-dimensions from both computational and statistical points of view. Recent ad- vances in deep generative models have led to new para- metric models for high-dimensional CDE tasks, namely the conditional variational autoencoders (CVAE) (Sohn et al., 2015). CVAEs have been applied to a variety of problems, such as MNIST quadrant prediction, segmentation (Sohn et al., 2015), attribute-based image generation (Yan et al., 2015), and machine translation (Zhang et al., 2016). But CVAEs suffer from two statistical deficiencies. First, they do not learn the distribution of the input x. We ar- gue that in the case of high-dimensional input x where there might exist a low-dimensional representation (such as a low-dimensional manifold) of the data, recovering this structure is important, even if the task at hand is to learn the conditional density p(y|x). Otherwise, the model is suscep- tible to overfitting. Second, for many CDE tasks, the ac- quisition of labeled points is costly, motivating the need for semi-supervised CDE. A purely conditional model would not be able to utilize any available unlabeled data. 1 We note that while variational methods (Kingma & Welling, 2013; Rezende et al., 2014) have been applied to semi- supervised classification (where y is a class label) (Kingma et al., 2014; Maaløe et al., 2016), semi-supervised CDE (where y is high-dimensional) remains an open problem. We focus on a set of deep conditional generative mod- els, which we call bottleneck conditional density estima- tors (BCDEs). In BCDEs, the input x influences the target y via layers of bottleneck stochastic variables z = {z i } in the generative path. The BCDE naturally has a joint gen- erative sibling model which we denote the bottleneck joint density estimator (BJDE), where the bottleneck z gener- ates x and y independently. Motivated by Lasserre et al. (2006), we propose a hybrid training framework that regu- larizes the conditionally-trained BCDE parameters toward the jointly-trained BJDE parameters. This is the key fea- ture that enables semi-supervised learning for conditional density estimation in the BCDEs. 1 We define a “labeled point” to be a paired (x, y) sample, and an “unlabeled point” to be unpaired x or y.
Transcript
Bottleneck Conditional Density Estimation
Rui Shu 1 Hung H. Bui 2 Mohammad Ghavamzadeh 3
Abstract
We introduce a new framework for training deepgenerative models for high-dimensional condi-tional density estimation. The Bottleneck Con-ditional Density Estimator (BCDE) is a vari-ant of the conditional variational autoencoder(CVAE) that employs layer(s) of stochastic vari-ables as the bottleneck between the input x
and target y, where both are high-dimensional.Crucially, we propose a new hybrid trainingmethod that blends the conditional generativemodel with a joint generative model. Hybridblending is the key to effective training of theBCDE, which avoids overfitting and provides anovel mechanism for leveraging unlabeled data.We show that our hybrid training procedure en-ables models to achieve competitive results inthe MNIST quadrant prediction task in the fully-supervised setting, and sets new benchmarks inthe semi-supervised regime for MNIST, SVHN,and CelebA.
1. IntroductionConditional density estimation (CDE) refers to the prob-lem of estimating a conditional density p(y|x) for the inputx and target y. In contrast to classification where the tar-get y is simply a discrete class label, y is typically contin-uous or high-dimensional in CDE. Furthermore, we wantto estimate the full conditional density (as opposed to itsconditional mean in regression), an important task the con-ditional distribution has multiple modes. CDE problemsin which both x and y are high-dimensional have a widerange of important applications, including video prediction,cross-modality prediction (e.g. image-to-caption), modelestimation in model-based reinforcement learning, and soon.
1Stanford University 2Adobe Research 3DeepMind (The workwas done when all the authors were with Adobe Research). Cor-respondence to: Rui Shu <[email protected]>.
Proceedings of the 34 thInternational Conference on Machine
Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017by the author(s).
Classical non-parametric conditional density estimatorstypically rely on local Euclidean distance in the original in-put and target spaces (Holmes et al., 2012). This approachquickly becomes ineffective in high-dimensions from bothcomputational and statistical points of view. Recent ad-vances in deep generative models have led to new para-metric models for high-dimensional CDE tasks, namely theconditional variational autoencoders (CVAE) (Sohn et al.,2015). CVAEs have been applied to a variety of problems,such as MNIST quadrant prediction, segmentation (Sohnet al., 2015), attribute-based image generation (Yan et al.,2015), and machine translation (Zhang et al., 2016).
But CVAEs suffer from two statistical deficiencies. First,they do not learn the distribution of the input x. We ar-gue that in the case of high-dimensional input x wherethere might exist a low-dimensional representation (suchas a low-dimensional manifold) of the data, recovering thisstructure is important, even if the task at hand is to learn theconditional density p(y|x). Otherwise, the model is suscep-tible to overfitting. Second, for many CDE tasks, the ac-quisition of labeled points is costly, motivating the need forsemi-supervised CDE. A purely conditional model wouldnot be able to utilize any available unlabeled data.1 Wenote that while variational methods (Kingma & Welling,2013; Rezende et al., 2014) have been applied to semi-supervised classification (where y is a class label) (Kingmaet al., 2014; Maaløe et al., 2016), semi-supervised CDE(where y is high-dimensional) remains an open problem.
We focus on a set of deep conditional generative mod-els, which we call bottleneck conditional density estima-
tors (BCDEs). In BCDEs, the input x influences the targety via layers of bottleneck stochastic variables z = {z
i
} inthe generative path. The BCDE naturally has a joint gen-erative sibling model which we denote the bottleneck joint
density estimator (BJDE), where the bottleneck z gener-ates x and y independently. Motivated by Lasserre et al.(2006), we propose a hybrid training framework that regu-larizes the conditionally-trained BCDE parameters towardthe jointly-trained BJDE parameters. This is the key fea-ture that enables semi-supervised learning for conditionaldensity estimation in the BCDEs.
1We define a “labeled point” to be a paired (x, y) sample, andan “unlabeled point” to be unpaired x or y.
Bottleneck Conditional Density Estimation
Our BCDE hybrid training framework is a novel approachfor leveraging unlabeled data for conditional density es-timation. Using our BCDE hybrid training framework,we establish new benchmarks for the quadrant predictiontask (Sohn et al., 2015) in the semi-supervised regime forMNIST, SVHN, and CelebA. Our experiments show that1) hybrid training is competitive for fully-supervised CDE,2) in semi-supervised CDE, hybrid training helps to avoidoverfitting, performs significantly better than conditionaltraining with unlabeled data pre-training, and achievesstate-of-the-art results, and 3) hybrid training encouragesthe model to learn better and more robust representations.
2. Background2.1. Variational Autoencoders
Variational Autoencoder (VAE) is a deep generative modelfor density estimation. It consists of a latent variable z withunit Gaussian prior z ⇠ N (0, I
k
), which in turn generatesan observable vector x. The observation is usually con-ditionally Gaussian x|z ⇠ N �
µ
✓
(z), diag(�2✓
(z)
�, where
µ and �
2 are neural networks whose parameters are rep-resented by ✓.2 VAE can be seen as a non-linear gen-eralization of the probabilistic PCA (Tipping & Bishop,1999), and thus, can recover non-linear manifolds in thedata. However, VAE’s flexibility makes posterior inferenceof the latent variables intractable. This inference issue isaddressed via a recognition model q
�
(z|x), which servesas an amortized variational approximation of the intractableposterior p
✓
(z|x). Learning in VAE’s is done by jointly op-timizing the parameters of both the generative and recogni-tion models so as to maximize an objective that resemblesan autoencoder regularized reconstruction loss (Kingma &Welling, 2013), i.e.,
sup
✓,�
Eq�(z|x)
⇥ln p
✓
(x|z)⇤� DKL�q
�
(z|x) || p(z)�. (1)
We note that the objective Eq. (1) can be rewritten in thefollowing form that exposes its connection to the varia-tional lower bound of the log-likelihood
sup
✓
⇣ln p
✓
(x)� inf
�
DKL�q
�
(z|x) || p✓
(z|x)�⌘
= sup
✓,�
Eq�(z|x)
ln
p
✓
(x, z)
p
�
(z|x)�. (2)
We make two remarks regarding the minimization of theterm DKL
�q
�
(z|x) || p✓
(z|x)� in Eq. 2. First, when q(·|·) isa conditionally independent Gaussian, this approximationis at best as good as the mean-field approximation that min-imizes DKL
�q || p
✓
(z|x)� over all independent Gaussian
2For discrete x, one can use a deep network to parameterize aBernoulli or a discretized logistic distribution.
q’s. Second, this term serves as a form of amortized pos-terior regularization that encourages the posterior p
✓
(z|x)to be close to an amortized variational family (Dayan et al.,1995; Ganchev et al., 2010; Hinton et al., 1995). In prac-tice, both ✓ and � are jointly optimized in Eq. (1), andthe reparameterization trick (Kingma & Welling, 2013) isused to transform the expectation over z ⇠ q
�
(z|x) into✏ ⇠ N (0, I
k
); z = µ
�
(x) + diag��
2�
(x)
�✏, which leads to
an easily obtained stochastic gradient.
2.2. Conditional VAEs (CVAEs)
In Sohn et al. (2015), the authors introduce the conditionalversion of variational autoencoders. The conditional gen-erative model is similar to VAE, except that the latent vari-able z and the observed vector y are both conditioned onthe input x. The conditional generative path is
p
✓
(z | x) = N⇣z | µ
z,✓
(x), diag��
2z,✓
(x)
�⌘(3)
p
✓
(y | x, z) = N⇣y | µ
y,✓
(x, z), diag��
2y,✓
(x, z)
�⌘, (4)
and when we use a Bernoulli decoder is
p
✓
(y | x, z) = Ber�y | µ
y,✓
(x, z)
�. (5)
Here, ✓ denotes the parameters of the neural networks usedin the generative path. The CVAE is trained by maximizinga lower bound of the conditional likelihood
ln p
✓
(y|x) � Eq�(z|x,y)
ln
p
✓
(z|x)p✓
(y|x, z)q
�
(z|x, y)�, (6)
but with a recognition network q
�
(z|x, y), which is typi-cally Gaussian N
⇣z|µ
�
(x, y), diag��
2�
(x, y)
�⌘, and takes
both x and y as input.
2.3. Blending Generative and Discriminative
It is well-known that a generative model may yield sub-optimal performance when compared to the same modeltrained discriminatively (Ng & Jordan, 2002), a phe-nomenon attributable to the generative model being mis-specified (Lasserre et al., 2006). However, generative mod-els can easily handle unlabeled data in semi-supervised set-ting. This is the main motivation behind blending genera-tive and discriminative models. Lasserre et al. (2006) pro-posed a principled method for hybrid blending by duplicat-ing the parameter of the generative model into a discrimi-natively trained ✓ and a generatively trained ˜
✓, i.e.,
p(Xl
,Yl
,Xu
,
˜
✓, ✓) = p(
˜
✓, ✓)p(Xu
|˜✓)p(Xl
|˜✓)p(Yl
|Xl
, ✓).
(7)The discriminatively trained parameter ✓ is regularized to-ward the generatively trained parameter ˜✓ via a prior p(˜✓, ✓)that prefers small k✓ � ˜
✓k2. As a result, in addition to
Bottleneck Conditional Density Estimation
z
x
y
BJDE
z
x
y
BCDE
Regularization
Unpaired Data{x
i
} [ {yi
}Paired Data{x
i
, y
i
}
Figure 1. The hybrid training procedure that regularizes BCDE towards BJDE. This regularization enables the BCDE to indirectlyleverage unpaired x and y for conditional density estimation.
learning from the labeled data (Xl
,Yl
), the discrimina-tive parameter ✓ can be informed by the unlabeled dataX
u
via ˜
✓, enabling a form of semi-supervised discrimina-tively trained generative model. However, this approach islimited to simple generative models (e.g., naive Bayes andHMMs), where exact inference of p(y|x, ✓) is tractable.
While Sohn et al. (2015) has successfully applied theCVAE to CDE, CVAE suffers from two limitations. First,the CVAE does not learn the distribution of its input x,and thus, is far more susceptible to overfitting. Second,it cannot incorporate unlabeled data. To resolve these lim-itations, we propose a new approach to high-dimensionalCDE that blends the discriminative model that learns theconditional distribution p(y|x), with a generative modelthat learns the joint distribution p(x, y).
3.1. Overview
Figure 1 provides a high-level overview of our approachthat consists of a new architecture and a new training proce-dure. Our new architecture imposes a bottleneck constraint,resulting a class of conditional density estimators, we call itbottleneck conditional density estimators (BCDEs). UnlikeCVAE, the BCDE generative path prevents x from directlyinfluencing y. Following the conditional training paradigmin Sohn et al. (2015), conditional/discriminative training ofthe BCDE means maximizing the lower bound of a condi-tional likelihood similar to (6),i.e.,
ln p
✓
(y|x) � C(✓,�;x, y)
= Eq�(z|x,y)
ln
p
✓
(z|x)p✓
(y|z)q
�
(z|x, y)�.
When trained over a dataset of paired (X,Y) samples, theoverall conditional training objective is
C(✓,�;X,Y) =
X
x,y2X,Y
C(✓,�;x, y). (8)
However, this approach suffers from the same limitations asCVAE and imposes a bottleneck that limits the flexibility ofthe generative model. Instead, we propose a hybrid trainingframework that takes advantage of the bottleneck architec-ture to avoid overfitting and supports semi-supervision.
One component in our hybrid training procedure tackles theproblem of estimating the joint density p(x, y). To do this,we use the joint counterpart of the BCDE: the bottleneckjoint density estimator (BJDE). Unlike conditional models,the BJDE allows us to incorporate unpaired x and y dataduring training. Thus, the BJDE can be trained in a semi-supervised fashion. We will also show that the BJDE iswell-suited to factored inference (see Section 3.4), i.e., afactorization procedure that makes the parameter space ofthe recognition model more compact.
The BJDE also serves as a way to regularize the BCDE,where the regularization constraint can be viewed as soft-tying between the parameters of these two models’ gen-erative and recognition networks. Via this regularization,BCDE benefits from unpaired x and y for conditional den-sity estimation.
3.2. Bottleneck Joint Density Estimation
In the BJDE, we wish to learn the joint distribution of x
and y. The bottleneck is introduced in the generative pathvia the bottleneck variable z, which points to x and y (seeFigs. 2(a) to 2(c)). Thus, the variational lower bound of the
Bottleneck Conditional Density Estimation
z
x
(a) Joint: (x)
z
y
(b) Joint: (y)
z
x
y
(c) Joint: (x, y)
z1
x
y
(d) Conditional: (x, y)
Figure 2. The joint and conditional components of the BCDE. Dotted lines represent recognition models. The conditional model param-eters are regularized toward the joint model’s. The natural pairing of the conditional and joint parameters is described in Table 1.
Standard BJDE: q
�̃
(z|x, y) q
�̃
(z|y) q
�̃
(z|x) p
✓̃
(y|z) p
✓̃
(x|z)BCDE: q
�
(z|x, y) � p
✓
(z|x) p
✓
(y|z) �
Factored BJDE: � ˆ
`
�̃
(z; y) q
�̃
(z|x) p
✓̃
(y|z) p
✓̃
(x|z)BCDE: � ˆ
`
�̃
(z; y) p
✓
(z|x) p
✓
(y|z) �
Table 1. Soft parameter tying between the BJDE and BCDE. For each network within the BCDE, there is a corresponding network withinthe BJDE. We show the correspondence among the networks with and without the application of factored inference. We regularize allthe BCDE networks to their corresponding BJDE network parameters.
joint likelihood is
ln p
✓̃
(x, y) � Jxy
(
˜
✓,
˜
�;x, y)
= Eq�̃(z|x,y)
"ln
p(z)p
✓̃
(x|z)p✓̃
(y|z)q
�̃
(z|x, y)
#. (9)
We use {˜✓, ˜�} to indicate the parameters of the BJDE net-works and reserve {✓,�} for the BCDE parameters. Forsamples in which x or y is unobserved, we will need tocompute the variational lower bound for the marginal like-lihoods. Here, the bottleneck plays a critical role. If x
were to directly influence y in a non-trivial manner, anyattempt to incorporate unlabeled y would require the recog-nition model to infer the unobserved x from the observedy—a conditional density estimation problem which mightbe as hard as our original task. In the bottleneck archi-tecture, the conditional independence of x and y given z
implies that only the low-dimensional bottleneck needs tobe marginalized. Thus, the usual variational lower boundsfor the marginal likelihoods yield
ln p
✓̃
(x) � Jx
(
˜
✓,
˜
�;x) = Eq�̃(z|x)
"ln
p(z)p
✓̃
(x|z)q
�̃
(z|x)
#,
ln p
✓̃
(y) � Jy
(
˜
✓,
˜
�; y) = Eq�̃(z|y)
"ln
p(z)p
✓̃
(y|z)q
�̃
(z|y)
#.
Since z takes on the task of reconstructing both x and y, theBJDE is sensitive to the distributions of x and y and learnsa joint manifold over the two data sources. Thus, the BJDEprovides the following benefits: 1) learning the distribution
of x makes the inference of z given x robust to perturba-tions in the inputs, 2) z becomes a joint-embedding of xand y, 3) the model can leverage unlabeled data. Followingthe convention in Eq. (8), the joint training objectives is
J (
˜
✓,
˜
�;Xu
,Yu
,Xl
,Yl
) = (10)
Jx
(
˜
✓,
˜
�;Xu
) + Jy
(
˜
✓,
˜
�;Yu
) + Jxy
(
˜
✓,
˜
�;Xl
,Yl
),
where (Xl
,Yl
) is a dataset of paired (x, y) samples, andX
u
and Yu
are datasets of unpaired samples.
3.3. Blending Joint and Conditional Deep Models
Because of potential model mis-specifications, the BJDE isnot expected to yield good performance if applied to theconditional task. Thus, we aim to blend the BJDE andBCDE models in the spirit of Lasserre et al. (2006). How-ever, we note that (7) is not directly applicable since theBCDE and BJDE are two different models, and not twodifferent views (discriminative and generative) of the samemodel. Therefore, it is not immediately clear how to tie theBCDE and BJDE parameters together. Further, these mod-els involve conditional probabilities parameterized by deepnetworks and have no closed form for inference.
Any natural prior for the BCDE parameter ✓ and the BJDEparameter ˜✓ should encourage pBCDE(y|x, ✓) to be close topBJDE(y|x, ˜✓). In the presence of the latent variable z, it isthen natural to encourage p(z|x, ✓) to be close to p(z|x, ˜✓)and p(y|z, ✓) to be close to p(y|z, ˜✓). However, enforc-ing the former condition is intractable as we do not have aclosed form for pBJDE(z|x, ˜✓). Fortunately, an approxima-
Bottleneck Conditional Density Estimation
tion of pBJDE(z|x, ˜✓) is provided by the recognition modelq(z|x, ˜�). Thus, we propose to softly tie together the pa-rameters of networks defining p(z|x, ✓) and q(z|x, ˜�). Thisstrategy effectively leads to a joint prior over the model net-work parameters, as well as the recognition network param-eters p(˜�, ˜✓,�, ✓).
As a result, we arrive at the following hybrid blending ofdeep stochastic models and its variational lower bound
ln p(Xl
,Yl
,Xu
,Yu
,
˜
✓,
˜
�, ✓,�) � ln p(
˜
✓,
˜
�, ✓,�) +
Jx
(
˜
✓,
˜
�;Xu
) + Jy
(
˜
✓,
˜
�;Yu
) +
Jx
(
˜
✓,
˜
�;Xl
) + C(✓,�;Xl
,Yl
). (11)
We interpret ln p(˜✓, ˜�, ✓,�) as a `2-regularization term thatsoftly ties the joint parameters (
˜
✓,
˜
�) and conditional pa-rameters (✓,�) in an appropriate way. For the BCDE andBJDE, there is a natural one-to-one mapping from the con-ditional parameters to a subset of the joint parameters.For the joint model described in Fig. 2(c) and conditionalmodel in Fig. 2(d), the parameter pairings are provided inTable 1. Formally, we define � = {✓,�} and use the index�
a|b to denote the parameter of the neural network on theBayesian network link b ! a in the BCDE. For example�
z|x = ✓
z|x, �z|x,y = �
z|x,y . Similarly, let �̃ = {˜✓, ˜�}.In the BJDE, the same notation yields �̃
z|x =
˜
�
z|x. Thehybrid blending regularization term can be written as
ln p(✓,�,
˜
✓,
˜
�) = ��
2
X
i2I
k�i
� �̃
i
k22 + const, (12)
where I denotes the set of common indices of the joint andconditional parameters. When the index is z|x, it effec-tively means that p(z|x, ✓) is softly tied to q(z|x, ˜✓), i.e.,
k�z|x � �̃
z|xk22 = k✓z|x � ˜
�
z|xk22 .
Setting � = 0 unties the BCDE from the BJDE, and effec-tively yields to a conditionally trained BCDE, while letting� ! 1 forces the corresponding parameters of the BCDEand BJDE to be identical.
Interestingly, Eq. (11) does not contain the term Jxy
. Sinceexplicit training of J
xy
may lead to learning a better jointembedding in the space of z, we note the following general-ization of Eq. (11) that trades off the contribution betweenJxy
and [Jx
+ C],ln p(X
l
,Yl
,Xu
,Yu
,
˜
✓,
˜
�, ✓,�)
� H(
˜
✓,
˜
�, ✓,�;Xl
,Yl
,Xu
,Yu
)
= ln p(
˜
✓,
˜
�, ✓,�) +
Jx
(
˜
✓,
˜
�;Xu
) + Jy
(
˜
✓,
˜
�;Yu
) +
↵ · Jxy
(
˜
✓,
˜
�;Xl
,Yl
) +
(1� ↵) ·hJx
(
˜
✓,
˜
�;Xl
) + C(✓,�;Xl
,Yl
)
i. (13)
Intuitively, the equation computes the lower bound ofp(X
l
,Yl
), either using the joint parameters ˜
✓,
˜
� or factor-izes p(X
l
,Yl
) into p(Xl
)p(Yl
| Xl
) before computing thelower bound of p(Y
l
| Xl
) with the conditional parame-ters. A proof that the lower bound holds for any 0 ↵ 1
is provided in Appendix B. For simplicity, we set ↵ = 0.5
and do not tune ↵ in our experiments.
3.4. Factored Inference
The inference network q
�
(z|x, y) is usually parameterizedas a single neural network that takes both x and y as input.Using the precision-weighted merging scheme proposed bySønderby et al. (2016), we also consider an alternative pa-rameterization of q
�
(z|x, y) that takes a weighted-averageof the Gaussian distribution q
�
(z|x) and a Gaussian like-lihood term ˆ
`(z; y) (see Appendix A). Doing so offers amore compact recognition model and more sharing param-eters between the BCDE and BJDE (e.g., see the bottomtwo rows in Table 1), but at the cost of lower flexibility forthe variational family q
�
(z|x, y).
4. ExperimentsWe evaluated the performance of our hybrid training proce-dure on the permutation-invariant quadrant prediction task(Sohn et al., 2014; Sohn et al., 2015) for MNIST, SVHN,and CelebA. The quadrant prediction task is a conditionaldensity estimation problem where an image data set is par-tially occluded. The model is given the observed region andis evaluated by its perplexity on the occluded region. Thequadrant prediction task consists of four sub-tasks depend-ing on the degree of partial observability. 1-quadrant pre-diction: the bottom left quadrant is observed. 2-quadrantprediction: the left half is observed. 3-quadrant prediction:the bottom right quadrant is not observed. Top-down pre-diction: the top half is observed.
In the fully-supervised case, the original MNIST train-ing set {x0
i
}50000i=1 is converted into our CDE training set
{Xl
,Yl
} = {xi
, y
i
}50000i=1 by splitting each image into
its observed x and unobserved y regions according to thequadrant prediction task. Note that the training set does notcontain the original class label information. In the n
l
-labelsemi-supervised case, we randomly sub-sampled n
l
pairsto create our labeled training set {x
i
, y
i
}nl
i=1. The remain-ing n
u
paired samples are decoupled and put into our un-labeled training sets X
u
= {xi
}nu
i=1 ,Yu
= {yi
}nu
i=1. Testperformance is the conditional density estimation perfor-mance on the entire test set, which is also split into inputx and target y according to the quadrant prediction task.Analogous procedure is used for SVHN and CelebA.
For comparison against Sohn et al. (2015), we evalu-ate the performance of our models on the MNIST 1-
quadrant, 2-quadrant, and 3-quadrant prediction tasks. TheMNIST digits are statically-binarized by sampling fromthe Bernoulli distribution according to their pixel values(Salakhutdinov & Murray, 2008). We use a sigmoid layerto learn the parameter of the Bernoulli observation model.
We provide the performance on the top-down predictiontask for SVHN and CelebA. We used a discretized logisticobservation model Kingma et al. (2016) to model the pixelvalues for SVHN and a Gaussian observation model withfixed variance for CelebA. For numerical stability, we relyon the implementation of the discretized logistic distribu-tion described in Salimans et al. (2017).
In all cases, we extracted a validation set of 10000 samplesfor hyperparameter tuning. While our training objectiveuses a single (IW=1) importance-weighted sample (Burdaet al., 2015), we measure performance using IW=100 toget a tighter bound on the test log-likelihood (Sohn et al.,2015). We run replicates of all experiments and report themean performance with standard errors. For a more ex-pressive variational family (Ranganath et al., 2015), we usetwo stochastic layers in the BCDE and perform inference
via top-down inference (Sønderby et al., 2016). We usemulti-layered perceptrons (MLPs) for MNIST and SVHN,and convolutional neural networks (CNNs) for CelebA. Allneural networks are batch-normalized (Ioffe & Szegedy,2015) and updated with Adam (Kingma & Ba, 2014). Thenumber of training epochs is determined based on the val-idation set. The dimensionality of each stochastic layer is50, 100, and 300 for MNIST, CelebA, and SVHN respec-tively. All models were implemented in Python3 using Ten-sorflow (Abadi, 2015).
4.1. Conditional Log-Likelihood Performance
Tables 2 to 6 show the performance comparisons betweenthe CVAE and the BCDE. For baselines, we use the CVAE,the BCDE trained with the conditional objective, and theBCDE initialized via pre-training J
x
(·) and Jy
(·) using theavailable x and y data separately (and then trained condi-tionally). Against these baselines, we measure the perfor-mance of the BCDE (with and without factored inference)
3github.com/ruishu/bcde
Bottleneck Conditional Density Estimation
trained with the hybrid objective H(·). We tuned the regu-larization hyperparameter � =
�10
�3, 10
�2, . . . , 10
3
onthe MNIST 2-quadrant semi-supervised tasks and settledon using � = 10
�2 for all tasks.
Fully-supervised regime. By comparing in the fully-supervised regime for MNIST (Tables 2 to 4, n
l
= 50000),we show that the hybrid BCDE achieves competitive per-formance against the pretrained BCDE and out-performspreviously reported results for CVAE (Sohn et al., 2015).
Semi-supervised regime. As the labeled training size n
l
reduces, the benefit of having the hybrid training procedurebecomes more apparent. The BCDEs trained with the hy-brid objective function tend to significantly improve uponits conditionally-trained counterparts.
On MNIST, hybrid training of the factored BCDE achievesthe best performance. Both hybrid models achieve overa 1-nat difference than the pre-trained baseline in somecases—a significant difference for binarized MNIST (Wuet al., 2016). Conditional BCDE performs very poorly inthe semi-supervised tasks due to overfitting.
On CelebA, hybrid training of the factored BCDE alsoachieves the best performance. Both hybrid models sig-nificantly out-perform the conditional baselines and yieldbetter visual predictions than conditional BCDE (see Ap-pendix C). The hybrid models also outperform pre-trainedBCDE with only half the amount of labeled data.
On SVHN, the hybrid BCDE with standard inferencemodel significantly out-performs the conditional baselines.However, the use of factored inference results in muchpoorer performance. Since the decoder is a discretized lo-gistic distribution with learnable scale, it is possible that thefactored inference model is not expressive enough to modelthe posterior distribution.
Model entropy. In Figure 3, we sample from p
✓
(y|x) forthe conditional BCDE and the hybrid BCDE. We showthat the conditionally-trained BCDE achieves poorer per-formance because it learns a lower-entropy model. In con-trast, hybrid training learns a lower perplexity model, re-sulting in a high-entropy conditional image generator thatspreads the conditional probability mass over the target out-put space (Theis et al., 2015).
4.2. Conditional Training Overfits
To demonstrate the hybrid training’s regularization be-havior, we show the test set performance during training(Fig. 4) on the 2-quadrant MNIST task (n
l
= 10000). Evenwith pre-trained initialization of parameters, models thatwere trained conditionally quickly overfit, resulting in poortest set performance. In contrast, hybrid training regular-izes the conditional model toward the joint model, which is
(a) Conditional (b) Hybrid
Figure 3. Comparison of conditional image generation for theconditional versus hybrid BCDE on the semi-supervised 1-quadrant task. Row 1 shows the original images. Rows 2-4 showthree attempts by each model to sample y according to x (thebottom-left quadrant, indicated in gray). Hybrid training yields ahigher-entropy model that has lower perplexity.
much more resilient against overfitting.
Figure 4. Comparison of the BCDE variants on the 2-quadrantMNIST prediction task with nl = 10000 labeled points. In con-trast to conditional training, hybrid training is less susceptible tooverfitting.
4.3. Robustness of Representation
Since hybrid training encourages the BCDE to consider thedistribution of x, we can demonstrate that models trainedin a hybrid manner are robust against structured perturba-tions of the data set. To show this, we experimented withtwo variants of the MNIST quadrant task called the shift-
sensitive and shift-invariant top-bottom prediction tasks. Inthese experiments, we set � = 0.1.
4.3.1. SHIFT-SENSITIVE ESTIMATION
In the shift-sensitive task, the objective is to learn to predictthe bottom half of the MNIST digit (y) when given the tophalf (x). However, we introduce structural perturbation tothe top and bottom halves of the image in our training, val-idation, and test sets by randomly shifting each pair (x, y)horizontally by the same number of pixels (shift varies be-tween {�4,�3, . . . , 3, 4}). We then train the BCDE us-ing either the conditional or hybrid objective in the fully-
Bottleneck Conditional Density Estimation
supervised regime. Note that compared to the original top-down prediction task, the perplexity of the conditional taskremains the same after the perturbation is applied.
Table 7. Shift-sensitive top-bottom MNIST prediction. Perfor-mance with and without structural corruption reported, along withthe performance difference. Hybrid training is robust againststructural perturbation of (x, y).
Table 7 shows that hybrid training consistently achievesbetter performance than conditional training. Furthermore,the hybridly trained models were less affected by the in-troduction of the perturbation, demonstrating a higher de-gree of robustness. Because of its more compact recogni-tion model, hybrid + factored is less vulnerable to overfit-ting, resulting in a smaller performance gap between per-formance on the shifted and original data.
4.3.2. SHIFT-INVARIANT ESTIMATION
The shift-invariant task is similar to the shift-sensitive top-bottom task, but with one key difference: we only introducestructural noise to the top half of the image in our training,validation, and test sets. The goal is thus to learn that theprediction of y (which is always centered) is invariant tothe shifted position of x.
Table 8. Shift-invariant top-bottom MNIST prediction. Perfor-mance with and without structural corruption reported, along withthe performance difference. Hybrid training is robust againststructural corruption of x.
Figure 5. Visualization of the latent space of hybrid andconditionally-trained BCDEs. PCA plots of the latent space sub-region for all x’s whose class label = 2 are shown. Fill colorindicates the degree of shift: blue = �4, orange = +4.
Table 8 shows similar behavior to Table 7. Hybrid training
continues to achieve better performance than conditionalmodels and suffer a much smaller performance gap whenstructural corruption in x is introduced.
In Fig. 5, we show the PCA projections of the latent spacesub-region populated by digits 2 and color-coded all pointsbased on the degree of shift. We observe that hybrid train-ing versus conditional training of the BCDE result in verydifferent learned representations in the stochastic layer. Be-cause of regularization toward the joint model, the hybridBCDE’s latent representation retrains information about xand learns to untangle shift from other features. And as ex-pected, conditional training does not encourage the BCDEto be aware of the distribution of x, resulting in a latentrepresentation that is ignorant of the shift feature of x.
5. ConclusionWe presented a new framework for high-dimensional con-ditional density estimation. The building blocks of ourframework are a pair of sibling models: the BottleneckConditional Density Estimator (BCDE) and the BottleneckJoint Density Estimator (BJDE). These models use layersof stochastic neural networks as bottleneck between the in-put and output data. While the BCDE learns the conditionaldistribution p(y|x), the BJDE learns the joint distributionp(x, y). The bottleneck constraint implies that only the bot-tleneck needs to be marginalized when either the input x orthe output y are missing during training, thus, enabling theBJDE to be trained in a semi-supervised fashion.
The key component of our framework is our hybrid objec-tive function that regularizes the BCDE towards the BJDE.Our new objective is a novel extension of Lasserre et al.(2006) that enables the principle of hybrid blending to beapplied to deep variational models. Our framework pro-vides a new mechanism for the BCDE, a conditional model,to become more robust and to learn from unlabeled data insemi-supervised conditional density estimation.
Our experiments showed that hybrid training is compet-itive in the fully-supervised regime against pre-training,and achieves superior performance in the semi-supervisedquadrant prediction task in comparison to conditionalmodels, achieving new state-of-the-art performances onMNIST, SVHN, and CelebA. Even with pre-trained weightinitializations, the conditional model is still susceptible tooverfitting. In contrast, hybrid training is significantly morerobust against overfitting. Furthermore, hybrid trainingtransfers the nice embedding properties of the BJDE to theBCDE, allowing the BCDE to learn better and more robustrepresentation of the input x. The success of our hybridtraining framework makes it a prime candidate for otherhigh-dimensional conditional density estimation problems,especially in semi-supervised settings.
learning on heterogeneous systems, 2015. URL http:
//tensorflow.org/. Software available from ten-sorflow.org.
Burda, Y., Grosse, R., and Salakhutdinov, R. ImportanceWeighted Autoencoders. arXiv preprints:1509.00519,2015.
Dayan, P., Hinton, G., Neal, R., and Zemel, R. TheHelmholtz Machine. Neural computation, 1995.
Ganchev, K., Graca, J., Gillenwater, J., and Taskar, B. Pos-terior regularization for structured latent variable mod-els. JMLR, 2010.
Hinton, G., Dayan, P., Frey, B., and Radford, R. The“wake-sleep” algorithm for unsupervised neural net-works. Science, 1995.
Holmes, M. P., Gray, A. G., and Isbell, C. L.Fast Nonparametric Conditional Density Estimation.arXiv:1206.5278, 2012.
Ioffe, S. and Szegedy, C. Batch Normalization: Acceler-ating Deep Network Training by Reducing Internal Co-variate Shift. arXiv:1502.03167, 2015.
Kingma, D. and Ba, J. Adam: A Method for StochasticOptimization. arXiv:1412.6980, 2014.
Kingma, D. P and Welling, M. Auto-Encoding VariationalBayes. arXiv:1312.6114, 2013.
Kingma, D. P., Rezende, D. J., Mohamed, S., and Welling,M. Semi-Supervised Learning with Deep GenerativeModels. arXiv:1406.5298, 2014.
Kingma, Diederik P., Salimans, Tim, and Welling, Max.Improving variational inference with inverse autoregres-sive flow. CoRR, abs/1606.04934, 2016. URL http:
//arxiv.org/abs/1606.04934.
Lasserre, J., Bishop, C., and Minka, T. Principled hybridsof generative and discriminative models. In The IEEE
Conference on Computer Vision and Pattern Recogni-
tion, 2006.
Maaløe, L., Kaae Sønderby, C., Kaae Sønderby, S.,and Winther, O. Auxiliary Deep Generative Models.arXiv:1602.05473, 2016.
Ng, A. and Jordan, M. On discriminative vs. generativeclassifiers: A comparison of logistic regression and naivebayes. Neural Information Processing Systems, 2002.
Ranganath, R., Tran, D., and Blei, D. M. Hierarchical Vari-ational Models. ArXiv e-prints, 1511.02386, November2015.
Rezende, D., Mohamed, S., and Wierstra, D. Stochas-tic Backpropagation and Approximate Inference in DeepGenerative Models. ArXiv e-prints, 1401.4082, January2014.
Salakhutdinov, R. and Murray, I. On the quantitative anal-ysis of deep belief networks. International Conference
on Machine Learning, 2008.
Salimans, Tim, Karpathy, Andrej, Chen, Xi, and Kingma,Diederik P. Pixelcnn++: Improving the pixelcnn withdiscretized logistic mixture likelihood and other modi-fications. CoRR, abs/1701.05517, 2017. URL http:
//arxiv.org/abs/1701.05517.
Sohn, K., Shang, W., and H., Lee. Improved multimodaldeep learning with variation of information. Neural In-
formation Processing Systems, 2014.
Sohn, K., Yan, X., and Lee, H. Learning structured outputrepresentation using deep conditional generative models.Neural Information Processing Systems, 2015.
Sønderby, C. K., Raiko, T., Maaløe, L., Kaae Sønderby,S., and Winther, O. Ladder Variational Autoencoders.arXiv:1602.02282, 2016.
Theis, L., van den Oord, A., and Bethge, M. A note onthe evaluation of generative models. arXiv:1511.01844,2015.
Tipping, M. and Bishop, C. Probabilistic Principal Com-ponent Analysis. J. R. Statist. Soc. B, 1999.
Wu, Y., Burda, Y., Salakhutdinov, R., and Grosse, R. Onthe Quantitative Analysis of Decoder-Based GenerativeModels. arXiv:1611.04273, 2016.
Yan, X., Yang, J., Sohn, K., and Lee, H. Attribute2Image:Conditional Image Generation from Visual Attributes.arXiv:1512.00570, 2015.
Zhang, B., Xiong, D., Su, J., Duan, H., and Zhang, M. Vari-ational Neural Machine Translation. arXiv:1605.07869,2016.
