Sylvester Normalizing Flows for Variational Inference Rianne van den Berg * University of Amsterdam Leonard Hasenclever * University of Oxford Jakub M. Tomczak University of Amsterdam Max Welling University of Amsterdam Abstract Variational inference relies on flexible ap- proximate posterior distributions. Normaliz- ing flows provide a general recipe to con- struct flexible variational posteriors. We in- troduce Sylvester normalizing flows, which can be seen as a generalization of planar flows. Sylvester normalizing flows remove the well-known single-unit bottleneck from planar flows, making a single transformation much more flexible. We compare the performance of Sylvester normalizing flows against pla- nar flows and inverse autoregressive flows and demonstrate that they compare favorably on several datasets. The code of our model is pub- licly available at https://github.com/ riannevdberg/sylvester-flows. 1 INTRODUCTION Stochastic variational inference (Hoffman et al., 2013) allows for posterior inference in increasingly large and complex problems using stochastic gradient ascent. In continuous latent variable models, variational inference can be made particularly efficient through the amortized inference, in which inference networks amortize the cost of calculating the variational posterior for a data point (Gershman and Goodman, 2014). A particularly suc- cessful class of models is the variational autoencoder (VAE) in which both the generative model and the infer- ence network are given by neural networks, and sampling from the variational posterior is efficient through the non- centered parameterization (Kingma and Welling, 2014), also known as the reparameterization trick (Kingma and Welling, 2013; Rezende et al., 2014). * Equal contribution. Despite its success, variational inference has drawbacks compared to other inference methods such as MCMC. Variational inference searches for the best posterior ap- proximation within a parametric family of distributions. Hence, the true posterior distribution can only be recov- ered exactly if it happens to be in the chosen family. In particular, with widely used simple variational fami- lies such as diagonal covariance Gaussian distributions, the variational approximation is likely to be insufficient. More complex variational families enable better poste- rior approximations, resulting in improved model perfor- mance. Therefore, designing tractable and more expres- sive variational families is an important problem in vari- ational inference (Nalisnick et al., 2016; Salimans et al., 2015; Tran et al., 2015). Rezende and Mohamed (2015) introduced a general framework for constructing more flexible variational dis- tributions, called normalizing flows. Normalizing flows transform a base density through a number of invert- ible parametric transformations with tractable Jacobians into more complicated distributions. They proposed two classes of normalizing flows: planar flows and radial flows. While effective for small problems, these can be hard to train and often many transformations are required to get good performance. For planar flows, Kingma et al. (2016) argue that this is due to the fact that the transformation used acts as a bottleneck, warping one direction at a time. Having a large number of flows makes the inference network very deep and harder to train, empirically resulting in suboptimal performance. Kingma et al. (2016) proposed inverse auto-regressive flows (IAF), achieving state of the art results on dy- namically binarized MNIST at the time of publication. While very successful, each transformation in IAF only depends on the datapoint x through a context vector, with flow parameters that are independent of the datapoint. Paper contribution In this paper, we use Sylvester’s determinant identity to introduce Sylvester normalizing flows (SNFs). This family of flows is a generalization arXiv:1803.05649v2 [stat.ML] 20 Feb 2019
Transcript
Sylvester Normalizing Flows for Variational Inference
Rianne van den Berg∗University of Amsterdam
Leonard Hasenclever∗University of Oxford
Jakub M. TomczakUniversity of Amsterdam
Max WellingUniversity of Amsterdam
Abstract
Variational inference relies on flexible ap-proximate posterior distributions. Normaliz-ing flows provide a general recipe to con-struct flexible variational posteriors. We in-troduce Sylvester normalizing flows, whichcan be seen as a generalization of planarflows. Sylvester normalizing flows remove thewell-known single-unit bottleneck from planarflows, making a single transformation muchmore flexible. We compare the performanceof Sylvester normalizing flows against pla-nar flows and inverse autoregressive flows anddemonstrate that they compare favorably onseveral datasets. The code of our model is pub-licly available at https://github.com/riannevdberg/sylvester-flows.
1 INTRODUCTION
Stochastic variational inference (Hoffman et al., 2013)allows for posterior inference in increasingly large andcomplex problems using stochastic gradient ascent. Incontinuous latent variable models, variational inferencecan be made particularly efficient through the amortizedinference, in which inference networks amortize the costof calculating the variational posterior for a data point(Gershman and Goodman, 2014). A particularly suc-cessful class of models is the variational autoencoder(VAE) in which both the generative model and the infer-ence network are given by neural networks, and samplingfrom the variational posterior is efficient through the non-centered parameterization (Kingma and Welling, 2014),also known as the reparameterization trick (Kingma andWelling, 2013; Rezende et al., 2014).
∗ Equal contribution.
Despite its success, variational inference has drawbackscompared to other inference methods such as MCMC.Variational inference searches for the best posterior ap-proximation within a parametric family of distributions.Hence, the true posterior distribution can only be recov-ered exactly if it happens to be in the chosen family.In particular, with widely used simple variational fami-lies such as diagonal covariance Gaussian distributions,the variational approximation is likely to be insufficient.More complex variational families enable better poste-rior approximations, resulting in improved model perfor-mance. Therefore, designing tractable and more expres-sive variational families is an important problem in vari-ational inference (Nalisnick et al., 2016; Salimans et al.,2015; Tran et al., 2015).
Rezende and Mohamed (2015) introduced a generalframework for constructing more flexible variational dis-tributions, called normalizing flows. Normalizing flowstransform a base density through a number of invert-ible parametric transformations with tractable Jacobiansinto more complicated distributions. They proposed twoclasses of normalizing flows: planar flows and radialflows. While effective for small problems, these can behard to train and often many transformations are requiredto get good performance. For planar flows, Kingmaet al. (2016) argue that this is due to the fact that thetransformation used acts as a bottleneck, warping onedirection at a time. Having a large number of flowsmakes the inference network very deep and harder totrain, empirically resulting in suboptimal performance.Kingma et al. (2016) proposed inverse auto-regressiveflows (IAF), achieving state of the art results on dy-namically binarized MNIST at the time of publication.While very successful, each transformation in IAF onlydepends on the datapoint x through a context vector, withflow parameters that are independent of the datapoint.
Paper contribution In this paper, we use Sylvester’sdeterminant identity to introduce Sylvester normalizingflows (SNFs). This family of flows is a generalization
of planar flows, removing the bottleneck. We compare anumber of different variants of SNFs and show that theycompare favorably against planar flows and IAFs. Weshow that one specific variant of SNF is related to IAF,with the main difference being the amortization strategyof the flow parameters. Besides the usual requirementof having flexible transformations, this demonstrates theimportance of having data-dependent flow parameters.Note that this concept generalizes to applying normal-izing flows to any conditional distribution, in the sensethat the transformation parameters should be functionsof the conditioning variable.
2 VARIATIONAL INFERENCE
Consider a probabilistic model with observations x andcontinuous latent variables z and model parameters θ. Ingenerative modeling we are often interested in perform-ing maximum (marginal) likelihood learning of the pa-rameters θ of the latent-variable model pθ(x, z). Thisrequires marginalization over the unobserved latent vari-ables z. Unfortunately, this integration is generally in-tractable. Variational inference (Jordan et al., 1999) in-stead introduces a variational approximation q(z|x) tothe posterior, to construct a lower bound on the logmarginal likelihood:
This bound is known as the evidence lower bound(ELBO) and F is referred to as the variational free en-ergy. In equation (2), the first term represents the re-construction error, and the second term is the Kullback-Leibler (KL) divergence from the approximate posteriorto the prior distribution, which acts as a regularizer. Inthis paper we consider variational autoencoders (VAEs),where both pθ(x|z) and q(z|x) are distributions whoseparameters are given by neural networks. That is, weperform amortized inference such that q(z|x) = qφ(z|x)with φ the parameters of the encoder neural network. Theparameters θ and φ of the generative model and inferencemodel, respectively, are trained jointly through stochas-tic minimization of F(θ, φ), which can be made effi-cient through the reparameterization trick (Kingma andWelling, 2013; Rezende et al., 2014).
From equation (1) we see that the better the variationalapproximation to the posterior the tighter the ELBO. Thesimplest, but probably most widely used choice of vari-ation distribution qφ(z|x) is diagonal-covariance Gaus-sians of the form N (z|µµµφ(x), σσσ2
φ(x))
However, with such simple variational distributions the
Figure 1: Since the ELBO is only a lower bound on thelog marginal likelihood, they do not share the same localmaxima. The looser the ELBO is the more this can biasmaximum likelihood estimates of the model parameters.
ELBO will be fairly loose, resulting in biased maximumlikelihood estimates of the model parameters θ (see Fig.1) and harming generative performance. Thus, for varia-tional inference to work well, more flexible approximateposterior distributions are needed.
2.1 NORMALIZING FLOWS
Rezende and Mohamed (2015) propose a way to con-struct more flexible posteriors by transforming a sim-ple base distribution with a series of invertible transfor-mations (known as normalizing flows) with easily com-putable Jacobians. The resulting transformed density af-ter one such transformation f is as follows (Tabak andTurner, 2013; Tabak and Vanden-Eijnden, 2010):
p1(z′) = p0(z)
∣∣∣∣det
(∂f(z)
∂z
)∣∣∣∣−1 , (4)
where z′ = f(z), z, z′ ∈ RD and f : RD 7→ RD isan invertible function. In general the cost of computingthe Jacobian will be O(D3). However, it is possible todesign transformations with more efficiently computableJacobians.
This strategy is used in variational inference as fol-lows: first, a stochastic variable is drawn from a simplebase posterior distribution such as a diagonal GaussianN (z0|µ(x),σ2(x)). The sample is then transformedwith a number of flows. After applying K flows, thefinal latent stochastic variables are given by zK = fK ◦. . . f2 ◦ f1(z0). The corresponding log-density is thengiven by:
log qK(zK |x) = log q0(z0|x)
−K∑k=1
log
∣∣∣∣det
(∂fk(zk−1;λk(x))
∂zk−1
)∣∣∣∣, (5)
where λk are the parameters of the k-th transformation.Note that in order to achieve a flexible amortization strat-egy, the flow parameters λk can be made dependent onthe input data: λk = λk(x) (Rezende and Mohamed,2015). Given a variational posterior qφ(z|x) = qK(z|x)parametrized by a normalizing flow of length K, the vari-ational objective can be rewritten as:
Normalizing flows are frequently applied to amortizedvariational inference. Instead of learning the param-eters of the posterior distribution for each data point,(such as µ and σ for a Gaussian posterior), the input-dependence of the posterior distribution parameters ismodeled through an encoder/inference network. Whenperforming amortized inference for normalizing flows,the flow parameters determine the final distribution, andshould thus also be considered functions of the datapointx. This can be achieved through the use of hypernet-works (Ha et al., 2016).
Rezende and Mohamed (2015) introduced a normalizingflow, called planar flow, for which the Jacobian determi-nant could be computed efficiently. A single transforma-tion of the planar flow is given by:
z′ = z + uh(wT z + b). (8)
Here, u,w ∈ RD, b ∈ R and h is a suitable smooth ac-tivation function. Rezende and Mohamed (2015) showthat for h = tanh, transformations of this kind are in-vertible as long as uTw ≥ −1.
By the Matrix determinant lemma the Jacobian of thistransformation is given by:
det∂z′
∂z= det
(I + uh′(wT z + b)wT
)= 1 + uTh′(wT z + b)w, (9)
where h′ denotes the derivative of h and which can becomputed in O(D) time.
In practice, many planar flow transformations are re-quired to transform a simple base distribution into a flex-ible distribution, especially for high dimensional latentspaces. Kingma et al. (2016) argue that this is relatedto the term uh(wT z + b) in Eq. (8), which effectivelyacts as a single-neuron MLP. In the next section we willderive a generalization of planar flows, which does not
have a single-neuron bottleneck, while still maintainingthe property of an efficiently computable Jacobian deter-minant.
3 SYLVESTER NORMALIZINGFLOWS
Consider the following more general transformation sim-ilar to a single layer MLP with M hidden units and aresidual connection:
z′ = z + Ah(Bz + b), (10)
with A ∈ RD×M ,B ∈ RM×D, b ∈ RM , and M ≤ D.The Jacobian determinant of this transformation can beobtained using Sylvester’s determinant identity, which isa generalization of the matrix determinant lemma.Theorem 1 (Sylvester’s determinant identity). For allA ∈ RD×M ,B ∈ RM×D,
det (ID + AB) = det (IM + BA) , (11)
where IM and ID are M and D-dimensional identitymatrices, respectively.
When M < D, the computation of the determinant of aD ×D matrix is thus reduced to the computation of thedeterminant of an M ×M matrix.
Using Sylvester’s determinant identity, the Jacobian de-terminant of the transformation in Eq. (10) is given by:
det
(∂z′
∂z
)= det (IM + diag (h′(Bz + b)) BA) .
(12)Since Sylvester’s determinant identity plays a crucialrole in the proposed family of normalizing flows, we willrefer to them as Sylvester normalizing flows.
3.1 PARAMETERIZATION OF A AND B
In general, the transformation in (10) will not be invert-ible. Therefore, we propose the following special case ofthe above transformation:
z′ = z + QRh(RQT z + b) = φ(z), (13)
where R and R are upper triangular M ×M matrices,and
Q = (q1 . . .qM )
with the columns qm ∈ RD forming an orthonormal setof vectors. By theorem 1, the determinant of the JacobianJ of this transformation reduces to:
det J = det(IM + diag
(h′(RQT z + b)
)RQTQR
)= det
(IM + diag
(h′(RQT z + b)
)RR
),
(14)
which can be computed in O(M), since RR is also up-per triangular. The following theorem gives a sufficientcondition for this transformation to be invertible.
Theorem 2. Let R and R be upper triangular matrices.Let h : R −→ R be a smooth function with bounded,positive derivative. Then, if the diagonal entries of Rand R satisfy riirii > −1/‖h′‖∞ and R is invertible,the transformation given by (13) is invertible.
Proof. Case 1: R and R diagonal
Recall that one-dimensional real functions with strictlypositive derivatives are invertible. The columns ofQ are orthonormal and span a subspace W =span{q1, . . . ,qM} of RD. Let W⊥ denote its orthog-onal complement. We can decompose z = z‖ + z⊥,where z‖ ∈ W and z⊥ ∈ W⊥. Similarly we can decom-pose z′ = z′‖ + z′⊥. Clearly, QRh(RQT z + b) ∈ W .Hence φ only acts on z‖ and z⊥ = φ(z)⊥ = z′⊥. Thus,it suffices to consider the effect of φ on z‖. Multiplying(13) by QT from the left gives:
QT z′︸ ︷︷ ︸v′
= QT z︸︷︷︸v
+Rh(R QT z︸︷︷︸v
+b)
= (f1(v1), . . . , fM (vM ))T , (15)
where the vectors v and v′ are the respective coordi-nates of z‖ and z′‖ w.r.t. q1, . . . ,qM . The dimen-sions in (15) are completely independent and each di-mension is transformed by a real function fi(v) = v +riih(riiv + bi). Consider a single dimension i of (15).Since ‖h′‖∞riirii > −1, we have f ′i(v) > 0 and thusfi is invertible. Since all dimensions are independent andthe transformation is invertible in each dimension we canfind f−1 : W → W such that z‖ = f−1(z′‖). Hence wecan write the inverse of φ as:
φ−1(z′) = z′⊥︸︷︷︸z⊥
+ f−1(z′‖)︸ ︷︷ ︸z‖
= z, (16)
Case 2: R triangular, R diagonal
Let us now consider the case when R is an upper triangu-lar matrix. By the argument for the diagonal case above,it suffices to consider the effect of the transformation inW . Multiplying (13) by QT from the left gives:
QT z′︸ ︷︷ ︸v′
= QT z︸︷︷︸v
+Rh(R QT z︸︷︷︸v
+b) (17)
where the vectors v and v′ contain the respective coordi-nates of z‖ and z′‖ w.r.t. q1, . . . ,qM . As in the diagonalcase consider the functions fi(v) = v + riih(riiv + bi).
Since ‖h′‖∞riirii > −1, we have f ′i(v) > 0 and thus fiis invertible. Let us rewrite (17) in terms of fi:
v′1 = f1(v1) +
M∑j=2
r1jh(rjjvj + bj) (18)
. . .
v′k = fk(vk) +
M∑j=k+1
rkjh(rjjvj + bj) (19)
. . .
v′M = fM (vM ) (20)
Since fM is invertible we can write vM = f−1M (v′M ).Now suppose we have expressed {vj ,∀j > k} in termsof {v′j ,∀j > k}. Then
fk(vk) = v′k −M∑
j=k+1
rkjh(rjjvj + bj)︸ ︷︷ ︸some function of {v′j ,∀j>k}
=: gk(v′k, v′k+1, . . . , v
′M ) (21)
vk = f−1k (gk(v′k, v′k+1, . . . , v
′M )).
Thus we have expressed {vj ,∀j ≥ k} in terms of{v′j ,∀j ≥ k}. By induction, we can express {vj ,∀j}in terms of {v′j ,∀j} and hence the transformation is in-vertible.
Case 3: R and R triangular
Now consider the general case when R is triangular. Asbefore we only need to consider the effect of the trans-formation inW .
QT z′︸ ︷︷ ︸v′
= QT z︸︷︷︸v
+Rh(R QT z︸︷︷︸v
+b) (22)
Let g be the function g(v) = Rv. By assumption, g isinvertible with inverse g−1. Multiplying (22) by R gives:
g(v′) = g(v) + RRh(g(v) + b)︸ ︷︷ ︸=:f(g(v))
(23)
Since RR is upper triangular with diagonal entriesrjjrjj , f is covered by case 2 considered before and isinvertible. Thus, v can be written as:
v = g−1(f−1(g(v′))). (24)
Hence the transformation in (22) is invertible.
3.2 PRESERVING ORTHOGONALITY OF Q
Orthogonality is a convenient property, mathematically,but hard to achieve in practice. In this paper we consider
three different flows based on the theorem above and var-ious ways to preserve the orthogonality of Q. The firsttwo use explicit differentiable constructions of orthogo-nal matrices, while the third variant assumes a specificfixed permutation matrix as the orthogonal matrix.
Orthogonal Sylvester flows. First, we consider aSylvester flow using matrices with M orthogonalcolumns (O-SNF). In this flow we can choose M < D,and thus introduce a flexible bottleneck. Similar to(Hasenclever et al., 2017), we ensure orthogonality ofQ by applying the following differentiable iterative pro-cedure proposed by (Bjorck and Bowie, 1971; Kovarik,1970):
Q(k+1) = Q(k)
(I +
1
2
(I−Q(k)>Q(k)
)). (25)
with a sufficient condition for convergence given by‖Q(0)>Q(0) − I‖2 < 1. Here, the 2-norm of a matrixX refers to ‖X‖2 = λmax(X), with λmax(X) repre-senting the largest singular value of X. In our experi-mental evaluations we ran the iterative procedure until‖Q(k)>Q(k)−I‖F ≤ ε, with ‖X‖F the Frobenius norm,and ε a small convergence threshold. We observed thatrunning this procedure up to 30 steps was sufficient to en-sure convergence with respect to this threshold. To min-imize the computational overhead introduced by orthog-onalization we perform this orthogonalization in parallelfor all flows.
Since this orthogonalization procedure is differentiable,it allows for the calculation of gradients with respect toQ(0) by backpropagation, allowing for any standard op-timization scheme such as stochastic gradient descent tobe used for updating the flow parameters.
Householder Sylvester flows. Second, we studyHouseholder Sylvester flows (H-SNF) where the orthog-onal matrices are constructed by products of House-holder reflections. Householder transformations are re-flections about hyperplanes. Let v ∈ RD, then the re-flection about the hyperplane orthogonal to v is givenby:
H(z) = z− 2vvT
‖v‖2 z (26)
It is worth noting that performing a single Householdertransformation is very cheap to compute, as it only re-quires D parameters. Chaining together several House-holder transformations results in more general orthog-onal matrices, and it can be shown (Bischof and Sun,1997; Sun and Bischof, 1995) that any M ×M orthogo-nal matrix can be written as the product ofM−1 House-holder transformations. In our Householder Sylvester
flow, the number of Householder transformations H isa hyperparameter that trades off the number of parame-ters and the generality of the orthogonal transformation.Note that the use of Householder transformations forcesus to use M = D, since Householder transformation re-sult in square matrices.
Triangular Sylvester flows. Third, we consider a tri-angular Sylvester flow (T-SNF), in which all orthogo-nal matrices Q alternate per transformation between theidentity matrix and the permutation matrix correspond-ing to reversing the order of z. This is equivalent to al-ternating between lower and upper triangular R and Rfor each flow.
3.3 DATA-DEPENDENT FLOW PARAMETERS
As previously mentioned, the parameters of the base dis-tribution as well as the flow parameters can be func-tions of the data point x (Rezende and Mohamed,2015). Figure 2 (left) shows a diagram of one SNFstep and the amortization procedure. The inference net-work takes datapoints x as input, and provides as anoutput the mean and variance of z0 such that z0 ∼N (z|µµµ0(x), (σσσ0)2(x)). Several SNF transformations arethen applied, such that z0 → z1 → . . . zK , producinga flexible posterior distribution for zK . All of the flowparameters (R, R and Q for each transformation) areproduced as an output of a hypernetwork (attached to theinference network) and are thus functions of x.
4 RELATED WORK
4.1 NORMALIZING FLOWS FORVARIATIONAL INFERENCE
A number of invertible transformations with tractable Ja-cobians have been proposed in recent years. Rezende andMohamed (2015) first discussed such transformations inthe context of stochastic variation inference, coining theterm normalizing flows.
Rezende and Mohamed (2015) proposed two differentparametric families of transformations with tractable Ja-cobians: planar and radial flows. While effective forsmall problems, these transformations are hard to scaleto large latent spaces and often require a large numberof transformations. The transformation corresponding toplanar flows is given in Eq. (8).
More recently, a successful class of flows called InverseAutoregressive Flows was introduced in (Kingma et al.,2016). As the name suggests, one IAF transformationcan be seen as the inverse of an autoregressive transfor-mation. Consider the following autoregressive transfor-
Figure 2: Different amortization strategies for Sylvester normalizing flows and Inverse Autoregressive Flows. Left:our inference network produces input-dependent flow parameters through a hypernetwork (Ha et al., 2016). Thisstrategy is also employed by planar flows. Right: IAF introduces a measure of x dependence through a context h(x).This context acts as an additional input for each transformation. The flow parameters themselves are independent ofx, but the number of hidden units per flow is larger than in SNF.
with ε ∼ N (0, I). This transformation models the dis-tribution over the variable z with an autoregressive fac-torization p(z) = p(z0)
∏Di=1 p(zi|zi−1, . . . , z0). Since
the parameters of transformation for zi are dependent onz1:i−1, this procedure requiresD sequential steps to sam-ple a single vector z. This is undesirable for variationalinference, where sampling occurs for every forward pass.
However, the inverse transformation (which exists ifσi > 0 ∀i) is easy to sample from:
εi =zi − µi(z1:i−1)
σi(z1:i−1). (28)
For this inverse transformation, εi is no longer depen-dent on the transformation of εj for j 6= i. Hence,this transformation can be computed in parallel: ε =(z − µ(z))/σ(z). Rewriting σi(z1:i−1) = 1/σi(z1:i−1)and µi(z1:i−1) = −µ(z1:i−1)/σi(z1:i−1), yields the IAFtransformation:
zti = µti(zt−11:i−1) + σti(z
t−11:i−1) · zt−1i , i = 1, ..., D.
(29)
Starting from z0 ∼ N (0, I), multiple IAF transforma-tions can be stacked on top of each other to produce flex-ible probability distributions.
If µt and σt depend on zt−1 linearly, IAF can modelfull covariance Gaussian distributions. In order to moveaway from Gaussian distributions to more flexible dis-tributions, it is important that µt and σt are nonlinearfunctions of zt−1.
In practice, wide MADEs (Germain et al., 2015) or deepPixelCNN layers (van den Oord et al., 2016) are neededto increase the flexibility of IAF transformations. Thisresults in transformations with a large number of pa-rameters. As shown in Figure 2 (right), amortization is
achieved through a context h(x) that is fed into the au-toregressive networks as an additional input at every IAFstep.
Our Triangular Sylvester flows are strongly related tomean-only IAF transformations (σt = 1). As mentionedin Kingma et al. (2016), between every IAF transforma-tion the order of z is reversed, in order to ensure that onaverage all dimensions get warped equally. In T-SNF, thesame effect is achieved by using the permutation matrixthat reverses the order of z in every other transformationas the orthogonal matrix. However, mean-only IAF is avolume-preserving transformation, i.e. the determinantof the Jacobian has absolute value one. T-SNF is not vol-ume preserving due to the nonzero elements on the di-agonals of R and R. Note, that in Kingma et al. (2016)it was shown that the difference in performance betweenmean-only IAF and the general IAF transformation wasnegligible.
The most important difference between IAF and T-SNFis the way parameters are amortized. In T-SNF, R and Rare directly amortized functions of the input x (see Fig.2). This is equivalent to amortizing the MADE parame-ters in mean-only IAF. Having input dependent MADEparameters allows for flexible transformations with fewerparameters.
Householder Sylvesters flows can also be seen as anon-linear extension of Householder flows (Tomczakand Welling, 2016). Householder flows are volume-preserving flows, which transform the variational pos-terior with a diagonal covariance matrix to a full-covariance posterior. Householder flows are a specialcase of H-SNF if h(z) = z, R is the identity matrix,and the residual connection in Eq. (13) is left out.
4.2 NORMALIZING FLOWS FOR DENSITYESTIMATION
A number of invertible transformations have been pro-posed in the context of density estimation. Note that
density estimation requires the inverse of the flow to betractable. Having a provably invertible transformation isnot the same as being able to compute the inverse.
For density estimation with normalizing flows, we areinterested maximizing the log-likelihood of the data:
log p(x) = log p0(f−1(x)) + log
∣∣∣∣det
(∂f−1(x)
∂x
)∣∣∣∣ .(30)
Thus, the goal is to transform a complicated data dis-tribution back to a simple distribution. In general, bothdirections of an invertible transformations need not betractable. Hence, methods developed for density estima-tion are generally not directly applicable to variationalinference.
Non-linear independent component estimation (NICE,Dinh et al. (2014)) and the related Real NVP (Dinh et al.,2016), and Masked Autoregressive Flow (MAF, Papa-makarios et al. (2017)) are recent examples of normal-izing flows for density estimation.
In NICE, each transformation splits the variables intotwo disjoint subsets zA, zB . One of the subsets is trans-formed as z′A = zA + f(zB), while zB is left un-changed. In the next transformation a different subset ofvariables is transformed. This results in a transformationwhich is trivially invertible and has a tractable Jacobian.Real NVP uses the same fundamental idea. Appealingly,because of the tractable inverse, NICE and real NVPcan generate data and estimate density with one forwardpass. However due to fact that only a subset of variablesis updated in each transformation many transformationsare needed in practice. Rezende and Mohamed (2015)compared NICE to planar flows in the context of varia-tional inference and found that planar flows empiricallyperform better.
Finally, Papamakarios et al. (2017) showed that fitting anMAF can be seen as fitting an implicit IAF from the datadistribution to the base distribution. However, generat-ing data from an MAF density model requires D passes,making it unappealing for variational inference.
5 NUMBER OF PARAMETERS
Here, we briefly compare the number of parametersneeded by planar flows, IAF and the three Sylvester nor-malizing flows. We denote the size of the stochastic vari-ables z with D, and the number of output units of theinference network with E.
Planar flows use amortized parameters u,w ∈ RD andb ∈ R for each flow transformation. Therefore, the num-ber of parameters related to K flow transformations is
equal to 2EDK + EK.
For the implementation of IAF as described in Section 6,the inference network needs to produce a context of sizeC, where C denotes the width of the MADE layers. Thetotal number of flow related learnable parameters thencomes down to EC +K × (C2 + 3CD).
In the case of Orthogonal Sylvester flows with a bottle-neck of size M , we require KE × (MD + 2M2 + M)parameters. For Householder Sylvester flows with HHouseholder reflections per flow transformation, KE ×(HD + 2D2 + D) parameters are needed. Finally, fortriangular Sylvester flows KE × (2D2 +D) parametersrequire optimization.
Planar flows require the smallest number of parametersbut generally result in worse results. IAFs on the otherhand require a number of parameters that is quadraticin the width of the MADE layers. For good results thishas to be quite large. In contrast, for SNFs the numberof parameters is quadratic in the dimension of the latentspace and while large, this can still be amortized.
6 EXPERIMENTS
We perform empirical studies of the performance ofSylvester flows on four datasets: statically binarizedMNIST, Freyfaces, Omniglot and Caltech 101 Silhou-ettes. The baseline model is a plain VAE with a fully fac-torized Gaussian distribution. We furthermore compareagainst planar flows and Inverse Autoregressive Flows ofdifferent sizes.
We use annealing to optimize the lower bound, wherethe prefactor of the KL divergence is linearly increasedfrom 0 to 1 during 100 epochs as suggested by Bowmanet al. (2015) and Sønderby et al. (2016). A learning rateof 0.0005 was used in all experiments. In order to obtainestimates for the negative log likelihood we used impor-tance sampling (as proposed in (Rezende et al., 2014)).Unless otherwise stated, 5000 importance samples wereused.
In order to assess the performance of the different flowsproperly, we use the same base encoder and decoder ar-chitecture for all models. We use gated convolutions andtransposed convolutions as base layers for the encoderand decoder architecture respectively. The inference net-work consists of several gated convolution layers thatproduce a hidden unit vector. After being flattened, thesehidden units act as an input to two fully connected layersthat predict the mean and variance of z0.
For planar and Sylvester flows, the flattened hidden unitsare passed to a separate linear layer that output the amor-tized flow parameters. For IAF, the flattened hidden units
are also passed to a linear layer to produce the contextvector hcontext(x). For details of the architecture seeSection A of the appendix. In all models the latent spaceis of dimension 64.
4 8 16Number of flows
83.5
84.0
84.5
85.0
85.5
86.0
86.5
-ELB
O
VAEPlanarIAF(320)
IAF(640)IAF(1280)O-SNF(16)
O-SNF(32)H-SNF(8)T-SNF
Figure 3: The negative evidence lower bound for staticMNIST. The results for H-SNF with 4 reflections per or-thogonal matrix are left out for clarity, as they are verysimilar to the results with 8 reflections. Each model isevaluated 3 times. The shaded areas indicate± one stan-dard deviation.
We use the following implementation for each IAF trans-formation1: one IAF transformation first applies oneMADE Layer (denoted as MaskedLinear) followed bya nonlinearity to the input z, upscaling it to a hiddenvariable of size M . At this point the context vectorhcontext(x) is added to the hidden units, after which twomore masked layers are applied to produce the mean andscale of the IAF transformation:
hz ← ELU(MaskedLinear(z))
h← hz + hcontext(x)
h← ELU(MaskedLinear(h))
µ← MaskedLinear(h), s← MaskedLinear(h)
z′ ← σ(s)� z + (1− σ(s))� µ. (31)
Here, σ( ) denotes the sigmoid activation function. InKingma et al. (2016) it was mentioned that the gatedform of IAF in Eq. (31) is more stable than the formof Eq. (29). Note that the size of hcontext(x) scales withthe width of the MADE layers C.
1This implementation is based on the open source code forIAF available at https://github.com/openai/iaf
Table 1: Negative log-likelihood and free energy (nega-tive evidence lower bound) for static MNIST. Numbersare produced with 3 runs per model with different ran-dom initializations. Standard deviations over the 3 dif-ferent runs are also shown.
Figure 3 shows the dependence of the negative evidencelower bound (or free energy) on the number of flows andthe type of flow for static MNIST. The exact numberscorresponding to the figure are shown in Section B in theappendix.
For all models the performance improves as a functionsof the number of flows. For 4 flows the difference be-tween the baseline VAE and planar flows is very small.However, planar flows clearly benefit from more flowtransformations.
For IAF three different widths of the MADE layers wereused: C = 320, 640 and 1280. Surprisingly, for 4 flowsthe widest IAF with 1280 hidden units is outperformedby an IAF with 640 hidden units in the MADE layers.We expect this to be due to the fact that this model hasmore parameters and can therefore be harder to train, asindicated by the larger standard deviation for this model.
All three Sylvester flows outperform IAF and planarflows. For Orthogonal Sylvester flows, we show resultsfor M = 16 and M = 32 orthogonal vectors per or-thogonal matrix, thus corresponding to bottlenecks ofsize 16 and 32 respectively for a latent space of sizeD = 64. Clearly, a larger bottleneck improves per-formance. For Householder Sylvester flows we experi-mented with H = 4 and H = 8 Householder reflectionsper orthogonal matrix. Since the results were nearly in-distinguishable between these two variants, we have leftout the curve for H = 4 to avoid clutter. O-SNF withM = 32, H-SNF and T-SNF seem to perform on par.
In Table 1, the negative evidence lower bound and theestimated negative log-likelihood are shown for the base-line VAE, together with all flow models for 16 flows. Thereported result for IAF is for a MADE width of 1280.The O-SNF model has a bottleneck of M = 32, and
Table 2: Results for Freyfaces, Omniglot and Caltech 101 Silhouettes datasets. For the Freyfaces dataset the resultsare reported in bits per dim. For the other datasets the results are reported in nats. For each flow model 16 flows areused. For IAF a MADE width of 1280 was used, and for O-SNF flow a bottleneck of M = 32 was used. For H-SNF 8householder reflections were used to construct orthogonal matrices. For all datasets 3 runs per model were performed.
Model Freyfaces Omniglot Caltech 101-ELBO NLL -ELBO NLL -ELBO NLL
H-SNF contains 8 Householder reflections per orthogo-nal matrix. Again, all Sylvester flows outperform planarflows and IAF, both in terms of the free energy and thenegative log-likelihood.
As discussed in Section 4, T-SNF is closely related tomean-only IAF, but with the MADE parameters pro-duced by a hypernetwork that depends on the input datax. The fact that T-SNF outperforms IAF indicates thathaving data-dependent flow parameters directly leads toa more flexible transformation compared to taking a verywide MADE with a data-dependent context as an addi-tional input.
6.2 FREYFACES, OMNIGLOT AND CALTECH101 SILHOUETTES
We further assess the performance of the different mod-els on Freyfaces, Omniglot and Caltech 101 Silhouettes.The results are shown in Table 2. The model settings arethe same2 as those used for Table 1.
Freyfaces is a very small dataset of around 2000 faces.All normalizing flows increase the performance, withplanar flows yielding the best result, closely followed byTriangular and Householder Sylvester flows. We expectplanar flows to perform the best in this case since it is theleast sensitive to overfitting.
For Omniglot and Caltech 101 Silhouettes the results areclearer, with the Sylvester normalizing flows family re-sulting in the best performance. Both H-SNF and T-SNFperform better than O-SNF. This could be attributed tothe fact that O-SNF has a bottleneck of M = 32 for alatent space size of D = 64. The IAF scores for Cal-tech 101 are surprisingly bad. We expect this could bethe case due to the large number of parameters that need
2For Caltech 101 Silhouettes we used 2000 importancesamples for the estimation of the negative log-likelihood.
to be trained for IAF(1280). Therefore we also evaluatedthe result for MADEs of width 320 for 16 flows. The re-sulting free energy and estimated negative log-likelihoodare 111.23 ± 0.45 and 99.74 ± 0.28 respectively, onlyslightly improving on the results of 1280 wide IAFs.
7 CONCLUSION
We present a new family of normalizing flows: Sylvesternormalizing flows. These flows generalize planar flows,while maintaining an efficiently computable Jacobiandeterminant through the use of Sylvester’s determinantidentity. We ensure invertibility of the flows throughthe use of orthogonal and triangular parameter matri-ces. Three variants of Sylvester flows are investigated.First, orthogonal Sylvester flows use an iterative pro-cedure to maintain orthogonality of parameter matrices.Second, Householder Sylvester flows use Householderreflections to construct orthogonal matrices. Third, tri-angular Sylvester flows alternate between fixed permu-tation and identity matrices for the orthogonal matrices.We show that the triangular Sylvester flows are closelyrelated to mean-only IAF, with data-dependent MADEparameters. While performing comparably with planarflows and IAF for the Freyfaces dataset, our proposedfamily of flows improve significantly upon planar flowsand IAF on the three other datasets.
Acknowledgements
We would like to thank Christos Louizos for helpingwith the implementation of inverse autoregressive flows,and Diederik Kingma for fruitful discussions. LH isfunded by the UK EPSRC OxWaSP CDT through grantEP/L016710/1. JMT is funded by the European Com-mission within the MSC-IF (Grant No. 702666). RvdBis funded by SAP SE.
ReferencesChristian Bischof and Xiaobai Sun. On orthogonal block
elimination. Technical Report MCS-P450-0794, Ar-gonne National Laboratory, Argonne, IL, 10 1997.
Ake Bjorck and Clazett Bowie. An iterative algorithm forcomputing the best estimate of an orthogonal matrix.SIAM Journal on Numerical Analysis, 8(2):358–364,1971.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, An-drew M. Dai, Rafal Jozefowicz, and Samy Bengio.Generating Sentences from a Continuous Space. nov2015. URL http://arxiv.org/abs/1511.06349.
Laurent Dinh, David Krueger, and Yoshua Bengio.NICE: non-linear independent components estima-tion. abs/1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.Density estimation using Real NVP. arXiv preprintarXiv:1605.08803, 2016.
Mathieu Germain, Karol Gregor, Iain Murray, and HugoLarochelle. MADE: Masked Autoencoder for Distri-bution Estimation. ICML, pages 881–889, 2015.
Samuel Gershman and Noah Goodman. Amortized in-ference in probabilistic reasoning. In Proceedings ofthe Annual Meeting of the Cognitive Science Society,volume 36, 2014.
David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks.arXiv preprint arXiv:1609.09106.
Leonard Hasenclever, Jakub Tomczak, Rianne van denBerg, and Max Welling. Variational inference withorthogonal normalizing flows. 2017.
Matthew D. Hoffman, David M. Blei, Chong Wang, andJohn Paisley. Stochastic variational inference. Journalof Machine Learning Research, 14:1303–1347, 2013.
Michael I. Jordan, Zoubin Ghahramani, Tommi S.Jaakkola, and Lawrence K. Saul. An introduction tovariational methods for graphical models. MachineLearning, 37(2):183–233, 1999.
Diederik Kingma and Max Welling. Efficient gradient-based inference through transformations betweenbayes nets and neural nets. ICML, pages 1782–1790,2014.
Diederik P Kingma and Max Welling. Auto-encodingvariational bayes. arXiv preprint arXiv:1312.6114,2013.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz,Xi Chen, Ilya Sutskever, and Max Welling. Im-proved Variational Inference with Inverse Autoregres-sive Flow. NIPS, pages 4743–4751, 2016.
Zdislav Kovarik. Some iterative methods for improvingorthonormality. SIAM Journal on Numerical Analysis,7(3):386–389, 1970.
Eric Nalisnick, Lars Hertel, and Padhraic Smyth. Ap-proximate inference for deep latent gaussian mixtures.In NIPS Workshop on Bayesian Deep Learning, 2016.
George Papamakarios, Iain Murray, and Theo Pavlakou.Masked Autoregressive Flow for Density Estimation.NIPS, pages 2335–2344, 2017.
Danilo Rezende and Shakir Mohamed. Variational in-ference with normalizing flows. ICML, pages 1530–1538, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and DaanWierstra. Stochastic backpropagation and approx-imate inference in deep generative models. arXivpreprint arXiv:1401.4082, 2014.
Tim Salimans, Diederik Kingma, and Max Welling.Markov Chain Monte Carlo and variational inference:Bridging the gap. ICML, pages 1218–1226, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe,Søren Kaae Sønderby, and Ole Winther. Ladder Vari-ational Autoencoders. feb 2016. URL http://arxiv.org/abs/1602.02282.
Xiaobai Sun and Christian Bischof. A basis-kernel rep-resentation of orthogonal matrices. SIAM Journal onMatrix Analysis and Applications, 16(4):1184–1196,1995.
EG Tabak and Cristina V Turner. A family of nonpara-metric density estimation algorithms. Communica-tions on Pure and Applied Mathematics, 66(2):145–164, 2013.
Esteban G Tabak and Eric Vanden-Eijnden. Density es-timation by dual ascent of the log-likelihood. Com-munications in Mathematical Sciences, 8(1):217–233,2010.
Jakub M Tomczak and Max Welling. Improving Varia-tional Auto-encoders using Householder Flow. arXivpreprint arXiv:1611.09630, 2016.
Dustin Tran, Rajesh Ranganath, and David M Blei.The variational Gaussian process. arXiv preprintarXiv:1511.06499, 2015.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt,koray kavukcuoglu, Oriol Vinyals, and Alex Graves.Conditional image generation with pixelcnn decoders.NIPS, pages 4790–4798, 2016.
In the experiments we used convolutional layers for boththe encoder and the decoder. Moreover, we used thegated activation function for convolutional layers:
hl = (Wl ∗ hl−1 + bl)� σ(Vl ∗ hl−1 + cl),
where hl−1 and hl are inputs and outputs of the l-thlayer, respectively, Wl,Vl are weights of the l-th layer,bl, cl denote biases, ∗ is the convolution operator, andσ(·) is the sigmoid activation function.
We used the following architecture of the encoder (k is akernel size, p is a padding size, and s is a stride size):3
Conv(in = 1, out = 32, k = 5,p = 2, s = 1)
Conv(in = 32, out = 32, k = 5,p = 2, s = 2)
Conv(in = 32, out = 64, k = 5,p = 2, s = 1)
Conv(in = 64, out = 64, k = 5,p = 2, s = 2)
Conv(in = 64, out = 64, k = 5,p = 2, s = 1)
Conv(in = 64, out = 64, k = 5,p = 2, s = 1)
Conv(in = 64, out = 256, k = 7,p = 0, s = 1)
Notice the last layer acts as a fully-connected layer.Eventually, fully-connected linear layers were used toparameterized diagonal Gaussian distribution and amor-tized parameters of a flow.
The decoder mirrors the structure of the encoder withtransposed convolutional layers (op is an outer padding):
ConvT(in = 64, out = 64, k = 7,p = 0, s = 1)
ConvT(in = 64, out = 64, k = 5,p = 2, s = 1)
ConvT(in = 64, out = 32, k = 5,p = 2, s = 2, op = 1)
ConvT(in = 32, out = 32, k = 5,p = 2, s = 1)
ConvT(in = 32, out = 32, k = 5,p = 2, s = 2, op = 1)
ConvT(in = 32, out = 32, k = 5,p = 2, s = 1)
ConvT(in = 32, out = 1, k = 1,p = 0, s = 1)
A.1 Description of datasets
In the experimetns we used the following four imagedatasets: static MNIST4, OMNIGLOT5, Caltech 101 Sil-houettes6, and Frey Faces7. Frey Faces contains images
3We use a PyTorch convention of defining convolutionallayers.
of size 28 × 20 and all other datasets contain 28 × 28images.
MNIST consists of hand-written digits split into 60,000training datapoints and 10,000 test sample points. In or-der to perform model selection we put aside 10,000 im-ages from the training set.
OMNIGLOT is a dataset containing 1,623 hand-writtencharacters from 50 various alphabets. Each character isrepresented by about 20 images that makes the problemvery challenging. The dataset is split into 24,345 train-ing datapoints and 8,070 test images. We randomly pick1,345 training examples for validation. During trainingwe applied dynamic binarization of data similarly to dy-namic MNIST.
Caltech 101 Silhouettes contains images representing sil-houettes of 101 object classes. Each image is a filled,black polygon of an object on a white background. Thereare 4,100 training images, 2,264 validation datapointsand 2,307 test examples. The dataset is characterized bya small training sample size and many classes that makesthe learning problem ambitious.
Frey Faces is a dataset of faces of a one person withdifferent emotional expressions. The dataset consistsof nearly 2,000 gray-scaled images. We randomly splitthem into 1,565 training images, 200 validation imagesand 200 test images. We repeated the experiment 3 times.
B MNIST experiments
The exact numbers for the evidence lower bound asshown in Fig. 3 are listed in Table 3.
![Page 1: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/1.jpg)
![Page 2: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/2.jpg)
![Page 3: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/3.jpg)
![Page 4: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/4.jpg)
![Page 5: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/5.jpg)
![Page 6: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/6.jpg)
![Page 7: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/7.jpg)
![Page 8: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/8.jpg)
![Page 9: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/9.jpg)
![Page 10: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/10.jpg)
![Page 11: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/11.jpg)
![Page 12: arXiv:1803.05649v2 [stat.ML] 20 Feb 2019](https://reader031.documents.pub/reader031/viewer/2022012408/616a360011a7b741a350055e/html5/thumbnails/12.jpg)
![arXiv:1308.5609v2 [stat.ML] 16 Jan 2014](https://static.documents.pub/doc/80x56/621862b67053ff754469871c/arxiv13085609v2-statml-16-jan-2014.jpg)
![arXiv:2006.04804v3 [stat.ML] 2 Oct 2020](https://static.documents.pub/doc/80x56/6233ea18bf6eba10a45da8b4/arxiv200604804v3-statml-2-oct-2020.jpg)
![arXiv:1606.01822v3 [stat.ML] 22 Aug 2016](https://static.documents.pub/doc/80x56/625c0fc4c4db9307bb6a41d6/arxiv160601822v3-statml-22-aug-2016.jpg)
![arXiv:1402.0422v1 [stat.ML] 3 Feb 2014](https://static.documents.pub/doc/80x56/61b5a98fbc59e20b3814e9fe/arxiv14020422v1-statml-3-feb-2014.jpg)
![arXiv:1801.09185v2 [stat.ML] 17 Nov 2018](https://static.documents.pub/doc/80x56/6169b64011a7b741a34a85bb/arxiv180109185v2-statml-17-nov-2018.jpg)
![arXiv:0807.3423v1 [stat.ML] 22 Jul 2008](https://static.documents.pub/doc/80x56/623c97e3deff0758583d446a/arxiv08073423v1-statml-22-jul-2008.jpg)
![arXiv:1810.09920v4 [stat.ML] 4 Mar 2019](https://static.documents.pub/doc/80x56/624d37d63b62964d4b60318e/arxiv181009920v4-statml-4-mar-2019.jpg)
![arXiv:1012.3407v1 [stat.ML] 15 Dec 2010](https://static.documents.pub/doc/80x56/6169e8f011a7b741a34cb778/arxiv10123407v1-statml-15-dec-2010.jpg)
![arXiv:1606.00931v2 [stat.ML] 25 Oct 2016](https://static.documents.pub/doc/80x56/586767d41a28ab96408b96d9/arxiv160600931v2-statml-25-oct-2016.jpg)
![arXiv:1306.0895v1 [stat.ML] 4 Jun 2013](https://static.documents.pub/doc/80x56/616e1cc05081f1393b73ed1c/arxiv13060895v1-statml-4-jun-2013.jpg)
![arXiv:1705.04790v2 [stat.ML] 16 May 2017](https://static.documents.pub/doc/80x56/617095f8ba14e8120072221b/arxiv170504790v2-statml-16-may-2017.jpg)
![arXiv:1302.6766v3 [stat.ML] 14 Dec 2016](https://static.documents.pub/doc/80x56/61abeeabe5890900281b4b69/arxiv13026766v3-statml-14-dec-2016.jpg)
![arXiv:1708.05894v1 [stat.ML] 19 Aug 2017](https://static.documents.pub/doc/80x56/61c2fe3f145932276c78d6d2/arxiv170805894v1-statml-19-aug-2017.jpg)
![arXiv:1805.07226v2 [stat.ML] 21 Jan 2019](https://static.documents.pub/doc/80x56/61d025961c3df33c7d0df059/arxiv180507226v2-statml-21-jan-2019.jpg)
![arXiv:1706.08110v2 [stat.ML] 14 Nov 2017](https://static.documents.pub/doc/80x56/616a58c311a7b741a3518018/arxiv170608110v2-statml-14-nov-2017.jpg)
![arXiv:2010.00462v1 [stat.ML] 1 Oct 2020](https://static.documents.pub/doc/80x56/624c6f3eecd66917403c4b7a/arxiv201000462v1-statml-1-oct-2020.jpg)
![arXiv:1706.06066v3 [stat.ML] 15 Feb 2018](https://static.documents.pub/doc/80x56/616a264011a7b741a34f5683/arxiv170606066v3-statml-15-feb-2018.jpg)
![arXiv:2109.05175v1 [stat.ML] 11 Sep 2021](https://static.documents.pub/doc/80x56/617937f62719ed6ffb279584/arxiv210905175v1-statml-11-sep-2021.jpg)