1

Accelerated Variance Reduction StochasticADMM for Large-Scale Machine Learning

Yuanyuan Liu, Member, IEEE, Fanhua Shang, Senior Member, IEEE, Hongying Liu, Member, IEEE,Lin Kong, Licheng Jiao, Fellow, IEEE, and Zhouchen Lin, Fellow, IEEE

Abstract—Recently, many stochastic variance reduced alternating direction methods of multipliers (ADMMs) (e.g., SAG-ADMM andSVRG-ADMM) have made exciting progress such as linear convergence rate for strongly convex (SC) problems. However, theirbest-known convergence rate for non-strongly convex (non-SC) problems is O(1/T ) as opposed to O(1/T 2) of accelerateddeterministic algorithms, where T is the number of iterations. Thus, there remains a gap in the convergence rates of existing stochasticADMM and deterministic algorithms. To bridge this gap, we introduce a new momentum acceleration trick into stochastic variancereduced ADMM, and propose a novel accelerated SVRG-ADMM method (called ASVRG-ADMM) for the machine learning problemswith the constraint Ax+By=c. Then we design a linearized proximal update rule and a simple proximal one for the two classes ofADMM-style problems with B=τI and B 6=τI, respectively, where I is an identity matrix and τ is an arbitrary bounded constant. Notethat our linearized proximal update rule can avoid solving sub-problems iteratively. Moreover, we prove that ASVRG-ADMM convergeslinearly for SC problems. In particular, ASVRG-ADMM improves the convergence rate from O(1/T ) to O(1/T 2) for non-SC problems.Finally, we apply ASVRG-ADMM to various machine learning problems, e.g., graph-guided fused Lasso, graph-guided logisticregression, graph-guided SVM, generalized graph-guided fused Lasso and multi-task learning, and show that ASVRG-ADMMconsistently converges faster than the state-of-the-art methods.

Index Terms—Stochastic optimization, ADMM, variance reduction, momentum acceleration, strongly convex and non-strongly convex,smooth and non-smooth

F

1 INTRODUCTION

THIS paper mainly considers the following compositefinite-sum equality-constrained optimization problem,

minx∈Rdx,y∈Rdy

{f(x) + h(y), s.t., Ax+By = c

}(1)

where c∈Rdc , A∈Rdc×dx , B∈Rdc×dy , f(x) := 1n

∑ni=1fi(x),

each component function fi(·) is convex, and h(·) is convexbut possibly non-smooth. For instance, a popular choice offi(·) in binary classification problems is the logistic loss,i.e., fi(x) = log(1 + exp(−biaTi x)), where (ai, bi) is thefeature-label pair, and bi∈{±1}. With regard to h(·), we areinterested in a sparsity-inducing regularizer, e.g., `1-norm[1, 2], group Lasso [3, 4] and nuclear norm [5–7].

Problem (1) arises in many places in machine learning,pattern recognition, computer vision, statistics, and opera-tions research [8]. When the constraint in Eq. (1) is Ax= y,the formulation (1) becomes

minx∈Rdx,y∈Rdy

{f(x) + h(y), s.t., Ax = y

}(2)

where A∈Rdy×dx . Recall that this class of problems includethe graph-guided fused Lasso [3], generalized Lasso [4] and

• Y. Liu, F. Shang, H. Liu L. Kong, and L. Jiao are with the Key Labora-tory of Intelligent Perception and Image Understanding of Ministry ofEducation, School of Artificial Intelligence, Xidian University, China. E-mails: {yyliu, fhshang, hyliu}@xidian.edu.cn; xdkonglin0511@163.com;lchjiao@mail.xidian.edu.cn.

• Z. Lin is with the Key Laboratory of Machine Perception (MOE),School of EECS, Peking University, Beijing 100871, P.R. China. E-mail:zlin@pku.edu.cn.

Manuscript received August 8, 2019.

graph-guided SVM [9] as notable examples. If the constraintdegenerates x = y, this class of problems include the reg-ularized empirical risk minimization (ERM) problem, e.g.,logistic regression, Lasso and linear support vector machine.

For solving the large-scale optimization problem in-volving a large sum of n component functions, stochasticgradient descent (SGD) [10] uses only one or a mini-batchof gradients in each iteration, and thus enjoys a significantlylower per-iteration complexity than deterministic methodsincluding Nesterov’s accelerated gradient descent (AGD)[11, 12] and accelerated proximal gradient (APG) [13, 14],i.e., O(dx) vs. O(ndx). Therefore, SGD has been successfullyapplied to many large-scale machine learning problems[9, 15, 16], especially training deep network models [17].However, the variance of the stochastic gradient estimatormay be large, and thus we need to gradually reduce its step-size, which leads to slow convergence [18], especially forequality-constrained composite convex problems [19].

This paper mainly focuses on the large sample regime.In this regime, even first-order deterministic methods suchas FISTA [14] become computationally burdensome dueto their per-iteration complexity of O(ndx). As a result,SGD with low per-iteration complexityO(dx) has witnessedtremendous progress in the recent years. Recently, a numberof stochastic variance reduced methods such as SAG [23],SDCA [24], SVRG [18], Prox-SVRG [25] and VR-SGD [26]have been proposed to successfully address the problemof high variance of stochastic gradient estimators in or-dinary SGD, resulting in linear convergence for stronglyconvex problems as opposed to sub-linear rates of SGD.More recently, the Nesterov’s acceleration technique [27]was introduced in [28–31] to further speed up the stochastic

2

TABLE 1Comparison of convergence rates and memory requirements of various stochastic ADMM algorithms, including stochastic ADMM

(STOC-ADMM) [9], stochastic average gradient ADMM (SAG-ADMM) [19], stochastic dual coordinate ascent ADMM (SDCA-ADMM) [20], scalablestochastic ADMM (SCAS-ADMM) [21], stochastic variance reduced gradient ADMM (SVRG-ADMMM) [22], and our ASVRG-ADMM. It should be

noted although all the methods except SDCA-ADMM apply the same update rule in (4), their algorithms do not actually work for solving theproblem (1) with the constraint Ax+By=c, where B 6= τI, τ is an arbitrary bounded constant, and I is an identity matrix.

Non-strongly convex Strongly convex Constraints Space requirement

STOC-ADMM [9] O(1/√T ) O(log T/T ) Ax = y O(dxdy+ d2x)

SAG-ADMM [19] O(1/T ) unknown Ax = y O(dxdy+ ndx)

SDCA-ADMM [20] unknown linear rate Ax+By = c O(dxdy+ n)

SCAS-ADMM [21] O(1/T ) O(1/T ) Ax = y O(dxdy)

SVRG-ADMM [22] O(1/T ) linear rate Ax = y O(dxdy)

ASVRG-ADMM (ours) O(1/T 2) linear rate Ax+By = c O(dxdy)

variance reduced algorithms, which results in the best-known convergence rates for both strongly convex (SC) andnon-strongly convex (non-SC) problems, e.g., Katyusha [29].This also motivates us to integrate the momentum accelera-tion trick into the stochastic alternating direction method ofmultipliers (ADMM) below.

1.1 Review of Stochastic ADMMs

It is well known that the ADMM is an effective optimiza-tion tool [32] to solve this class of composite optimizationproblems (1). The ADMM has shown attractive performancein a wide range of real-world problems, such as big dataclassification [33] and matrix and tensor recovery [5, 34, 35].We refer the reader to [36–39] for some review papers onthe ADMM. Recently, several faster deterministic ADMMalgorithms have been proposed to solve some special casesof Problem (1). For instance, [40] proposed an acceleratedADMM, and proved that their algorithm has an O(1/T 2)convergence rate for SC problems, similar to [37, 41]1.[42, 43] proposed a faster ADMM algorithm with a conver-gence rate O(1/T 2) for solving the special case of Problem(1) with the constraint Ax = y. However, the per-iterationcomplexity of all the full-batch ADMMs is O(ndx), and thusthey become very slow and are not suitable for large-scalemachine learning problems.

To tackle the issue of high per-iteration complexity ofdeterministic ADMM, [9, 44, 45] proposed some online orstochastic ADMM algorithms. However, all these variantsonly achieve the convergence rate of O(log T/T ) for SCproblems and O(1/

√T ) for non-SC problems, respectively,

as compared with the linear convergence and O(1/T 2)rates of the accelerated deterministic ADMM algorithmsmentioned above. Recently, several accelerated and fasterconverging versions of stochastic ADMMs such as SAG-ADMM [19], SDCA-ADMM [20] and SVRG-ADMM [22],which are all based on variance reduction techniques, havebeen proposed. With regard to strongly convex problems,[20, 22] proved that linear convergence can be obtained forthe special ADMM form (i.e., Problem (2)) and the generalADMM form, respectively. [46] also proposed a fast stochas-tic variance reduced ADMM for stochastic composition

1. Note that, for simplicity, we do not differentiate the O(1/T 2) ando(1/T 2) because they are of the same order in the worst-case natureand their difference is insignificant in general, where T is the numberof iterations.

optimization problems. More recently, [47, 48] proposed twoaccelerated stochastic ADMM algorithms for the problem (2)and four-composite optimization problems, respectively. ForSAG-ADMM and SVRG-ADMM, an O(1/T ) convergencerate can be guaranteed for non-strongly convex problems,which implies that there remains a gap in convergence ratesbetween the stochastic ADMM and accelerated determinis-tic algorithms, i.e., O(1/T ) vs. O(1/T 2).

1.2 ContributionsTo fill in this gap, we design a new momentum accelerationtrick similar to the ones in deterministic optimization andincorporate it into the stochastic variance reduction gradi-ent (SVRG) based stochastic ADMM (SVRG-ADMM) [22].Naturally, the proposed method has a low per-iteration costas existing stochastic ADMM algorithms such as SVRG-ADMM, and does not require the storage of all gradients(or dual variables) as in SAG-ADMM [19] and SCAS-ADMM [21], as shown in Table 1.

The main differences between this paper and our pre-vious conference paper [49] are listed as follows: 1) Webriefly review recent work on stochastic ADMM for solvingProblems (1) and (2). 2) When B 6= τI in Eq. (1), where τ isan arbitrary bounded constant and I is an identity matrix,the sub-problem with respect to y (see Eq. (4) below) hasno closed-form solution and has to be solved iteratively.To overcome this difficulty, we present a new linearizedproximal update rule for both SC and non-SC problems (1)with the constraintAx+By=cwhenB 6=τI . In other words,the existing stochastic ADMM algorithms including the pro-posed ones in our previous work [49] do not work for thiscase. Although the theoretical guarantees of existing vari-ance reduced stochastic ADMMs except SDCA-ADMM [20]are for Problem (1) with the general constraint Ax+By= c,they do not actually work for solving such problems. 3)For the case of B = τI , we use a simple proximal updaterule as in our previous work [49] instead of the linearizedproximal one. Then we propose two novel acceleratedSVRG-ADMM algorithms (called ASVRG-ADMM) for bothSC and non-SC problems. 4) We also theoretically analyzethe convergence properties of the proposed ASVRG-ADMMalgorithms for both SC and non-SC problems and the twocases of B 6= τI and B = τI , respectively. 5) We furtherimprove the theoretical results in our previous work [49]by removing the boundedness assumption. 6) Finally, wereport more experimental results especially for the ADMM

3

problem (1) with the constraint Ax+By = c to verify boththe effectiveness and efficiency of ASVRG-ADMM.

The main contributions of this paper are summarized asfollows.

• We propose an efficient accelerated variance re-duced stochastic ADMM (ASVRG-ADMM) method,which integrates both our momentum accelerationtrick and the variance reduction technique of SVRG-ADMM [22]. Moreover, ASVRG-ADMM has a lin-earized proximal rule and a simple proximal one forboth cases of B 6=τI and B=τI , respectively.

• We prove that ASVRG-ADMM achieves a linear con-vergence rate for SC problems, which is consistentwith the best-known result in SDCA-ADMM [20]and SVRG-ADMM [22]. Besides, when ASVRG-ADMM uses its linearized proximal rule, it becomesmore practical than existing algorithms, which haveto solve the sub-problems iteratively.

• In particular, for the more general problem (1) withthe constraint Ax+By = c and B 6= τI , we alsodesign a novel epoch initialization technique for thevariable y at each epoch of our linearized proximalacceleration algorithm for SC problems.

• We also prove that ASVRG-ADMM has a conver-gence rate O(1/T 2) for non-SC problems, whichmeans that ASVRG-ADMM is a factor T fasterthan SAG-ADMM and SVRG-ADMM, whose con-vergence rate is O(1/T ). In particular, we designan adaptive increasing epoch length strategy andfurther improve the theoretical results by using thisstrategy and removing boundedness assumptions.

• Various experimental results on synthetic and real-world datasets further verify that our ASVRG-ADMM converges consistently much faster than thestate-of-the-art stochastic ADMM methods.

The remainder of this paper is organized as follows.Section 2 discusses some recent advances in stochasticADMM. Section 3 proposes a new accelerated stochasticvariance reduction ADMM method (called ASVRG-ADMM)with the proposed momentum acceleration trick. Moreover,we analyze the convergence properties of ASVRG-ADMMin Section 4. Experimental results in Section 5 show theeffectiveness of ASVRG-ADMM. In Section 6, we concludethis paper and discuss the future work.

2 RELATED WORK

This section reveals recent progresses and efforts in stochas-tic optimization methods that are based on the stochasticalternating direction method of multipliers (ADMM).

2.1 Notation

Throughout this paper, the norm ‖·‖ denotes the standardEuclidean norm, and ‖·‖1 is the `1-norm, i.e., ‖x‖1 =

∑i|xi|.

We denote by ∇f(x) the gradient of f(x) if it is differen-tiable, or ∂f(x) any of the subgradients of f(·) at x if f(·)is only Lipschitz continuous. To facilitate our discussion, wefirst make the following basic assumptions.

2.2 Basic AssumptionsAssumption 1 (Smoothness). Each convex component functionfi(·) is L-smooth if its gradients are L-Lipschitz continuous, thatis

‖∇fi(x)−∇fi(y)‖ ≤ L‖x− y‖, for all x, y ∈ Rd.

Assumption 2 (Strong Convexity). A convex function g(·) :Rd→R is µ-strongly convex, if there exists a constant µ>0 suchthat

g(y) ≥ g(x) + 〈∇g(x), y−x〉+µ

2‖y−x‖2, for all x, y∈Rd.

If g(·) is non-smooth, we modify the above inequality by simplyreplacing ∇g(x) with an arbitrary sub-gradient ∂g(x).

2.3 Stochastic ADMMIt is easy to see that Problem (2) is only a special case of thegeneral ADMM form (1) when B=−Id2

and c=0. Thus, thepurpose of this paper is to propose an accelerated stochasticvariance reduced ADMM method for solving the more gen-eral problem (1). Although the stochastic (or online) ADMMalgorithms and theoretical results in [9, 19, 22, 44] are all forthe problem (1), they do not actually work.

The augmented Lagrangian function of Problem (1) is

L(x, y, λ) = f(x) + h(y) + 〈λ,Ax+By − c〉

+β

2‖Ax+By − c‖2

(3)

where λ is the vector of Lagrangian multipliers (also calledthe dual variable), and β > 0 is a penalty parameter. Tominimize Problem (1), together with the dual variable λ, theupdate steps of deterministic ADMM are

yk = arg miny{h(y) + β

2 ‖Axk−1+By−c+λk−1‖2}, (4)

xk = arg minx{f(x) + β

2 ‖Ax+Byk−c+λk−1‖2}, (5)

λk = λk−1 +Axk +Byk − c. (6)

To extend the deterministic ADMM to the online andstochastic settings, the update rules for yk and λk remainunchanged, while in [9, 44], the update rule of xk is approx-imated as follows:

xk = arg minx

{〈x,∇fik(xk−1)〉+

1

2ηk‖x− xk−1‖2G

+β

2‖Ax+Byk−c+λk−1‖2

} (7)

where we draw ik uniformly at random from [n] :={1, . . . , n}, ηk ∝ 1/

√k is the learning rate or step-size, and

‖z‖2G = zTGz with a given positive semi-definite matrix G,e.g.,G�Id1 as in [22]. Analogous to SGD, the stochastic AD-MM variants also use an unbiased estimate of the gradientat each iteration, i.e., E[∇fik(xk−1)] =∇f(xk−1). However,all those algorithms have much slower convergence ratesthan their deterministic counterparts mentioned above. Thisbarrier is mainly due to the large variance introduced by thestochasticity of the gradients [18]. Essentially, to guaranteeconvergence of SGD and its ADMM variants, we need toemploy a decaying sequence of step-sizes {ηk}, which inturn leads to slower convergence rates.

Recently, a number of variance reduced stochastic AD-MM methods (e.g., SAG-ADMM and SVRG-ADMM) have

4

Algorithm 1 ASVRG-ADMM for strongly-convex problemsInput: m, η, β > 0, 1 ≤ b ≤ n.Initialize: x0 = z0, y0, θ, λ0 = − 1

β (AT )†∇f(x0),

ν= 1+ ηβ‖BTB‖2θ , γ=1+ ηβ‖ATA‖2

θ ;1: for s = 1, 2, . . . , T do2: p = ∇f(xs−1), xs0 = zs0 = xs−1, λs0 = λs−1;3: ys0 = ys−1 for the case of B = τI , or ys0 =

− 1βB†(Azs0 − c) for the case of B 6= τI ;

4: for k = 1, 2, . . . ,m do5: Choose Ik⊆ [n]of size b, uniformly at random;6: ∇fIk(xsk−1)= 1

|Ik|∑ik∈Ik

[∇fik(xsk−1)−∇fik(xs−1)

]+p;

7: ysk=Prox1βτ2

h

((−Azsk−1 + c− λsk−1)/τ

)for the case of B = τI ,

ysk=Proxηβθν

h

[ysk−1−

ηβθνB

T(Azsk−1+Bysk−1−c+λsk−1)]

for the case of B 6= τI ;8: zsk=zsk−1−

ηγθ

[∇fIk(xsk−1)+βAT(Azsk−1+By

sk−c+λsk−1)

];

9: xsk=(1− θ)xs−1 + θzsk;10: λsk=λsk−1 +Azsk +Bysk − c;11: end for12: xs= 1

m

∑mk=1x

sk, ys=(1−θ)ys−1+ θ

m

∑mk=1y

sk;

13: λs=− 1β (AT )†∇f(xs);

14: end forOutput: xT , yT .

been proposed and made exciting progress such as linearconvergence rates. SVRG-ADMM [22] is particularly attrac-tive here because of its low storage requirement comparedwith the algorithms in [19, 20]. Within each epoch of mini-batch SVRG-ADMM, the full gradient p = ∇f(x) is firstcomputed, where x is the average point of the previousepoch. Then ∇fik(xk−1) and ηk in (7) are replaced by

∇fIk(xk−1) =1

|Ik|∑ik∈Ik

(∇fik(xk−1)−∇fik(x)) + p (8)

and a constant step-size η, respectively, where Ik ⊂ [n] isa mini-batch of size b. Note that mini-batching is a usefultechnique to reduce the variance of the stochastic gradi-ents [26, 50]. In fact, ∇fIk(xk−1) is also an unbiased estimatorof the gradient ∇f(xk−1), i.e., E[∇fIk(xk−1)]=∇f(xk−1).

For the equality-constrained composite convex problem(2), Xu et al. [47] proposed a faster variant of SVRG-ADMMwith an adaptive penalty parameter scheme. Fang et al. [48]proposed an accelerated stochastic ADMM with Nesterov’sextrapolation and variance reduction techniques for solvingfour-composite optimization problems. Moreover, Huang etal. [51], and Huang and Chen [52] proposed several variantsof SVRG-ADMM for solving non-smooth and non-convexoptimization problems.

3 MOMENTUM ACCELERATED VARIANCE REDUC-TION STOCHASTIC ADMMIn this section, we propose an efficient accelerated variancereduced stochastic ADMM (ASVRG-ADMM) method forsolving both SC and non-SC problems (1). In particular,we design two new linearized proximal accelerated algo-rithms for both SC and non-SC problems with the constraintAx+By=c and B 6=τI , respectively.

3.1 ASVRG-ADMM for Strongly Convex Problems

In this part, we first consider the case of Problem (1)when each fi(·) is convex, L-smooth, and f(·) is µ-stronglyconvex. Recall that this class of problems include graph-guided logistic regression and support vector machines(SVM) as notable examples. To efficiently solve this class ofproblems, we incorporate both the momentum accelerationtrick proposed in our previous work [49] and the variancereduced stochastic ADMM [22], as shown in Algorithm 1.All our algorithms including Algorithm 1 are divided intoT epochs, and each epoch consists of m stochastic updates,where m is usually chosen to be m=Θ(n) as in [18, 49].

3.1.1 Update Rule of yAs in both SVRG-ADMM [22] and ASVRG-ADMM [49], thevariable y is updated by solving the following problem forboth strongly convex and non-strongly convex cases:

ysk = arg miny

{h(y) +

β

2‖Azsk−1+By − c+ λsk−1‖2

}(9)

where the superscript s indicates the s-th epoch, the sub-script k denotes the k-th inner-iteration, zsk−1 is an auxiliaryvariable and its update rule is given in Section 3.1.2.

When B=τI (e.g., B is an identity matrix), the solutionto the problem in Eq. (9) can be relatively easily obtained.In other words, we still apply the simple proximal ruleproposed in our previous work [49] to solve such problems.For this case, we give the following proximal update rule:

ysk = Prox1βτ2

h

((−Azsk−1 + c− λsk−1)/τ

)where the proximal operator Proxδh(·) is defined as

Proxδh(w) = arg minx

{1

2δ‖x− w‖2 + h(x)

}.

However, when B 6=τI (e.g., B is not a diagonal matrix),it is often hard to solve the problem (9) in practice [32]. Toaddress this issue, in this paper we design the followinglinearized proximal rule to update y,

ysk = arg miny

{h(y) +

β

2

∥∥Azsk−1 +By − c+ λsk−1

∥∥2

+θs−1

2η‖y − ysk−1‖2Qs

}where Qs = νId2 −

ηβθs−1

BTB with ν ≥ 1 + ηβ‖BTB‖2θs−1

toensure that Qs � I , where ‖ · ‖2 is the spectral norm, i.e.,the largest singular value of the matrix. The above problemis equivalent to the following problem,

ysk = arg miny

{h(y) +

νθs−1

2η

∥∥∥∥y − ysk−1 +ηβ

θs−1νpsk

∥∥∥∥2}

(10)

where psk = BT(Azsk−1 +Bysk−1− c +λsk−1). We can easilyobtain the following proximal update rule for Problem (10):

ysk=Proxηβ

θs−1ν

h

[ysk−1−

ηβ

θs−1νBT(Azsk−1+Bysk−1−c+λsk−1)

].

From the above analysis, it is clear that we introduce thelinearized proximal operation into the proposed algorithms(including Algorithm 1 and Algorithm 2 below) and make

5

our algorithms much more practical than existing stochasticADMM algorithms including SVRG-ADMM [22] and the al-gorithms proposed in [49]. Besides, the proposed linearizedproximal rule can also avoid the calculation of the pseudo-inverse matrix at each inner-iteration. Then the new algo-rithms proposed in this paper as well as their convergenceanalysis are different from those in our previous work [49].To ensure linear convergence of the proposed linearizedproximal algorithm for strongly convex problems as SVRG-ADMM, we also design the following new epoch initializa-tion strategy for ys0 at each epoch instead of ys0 = ys−1 in [49],where the snapshot point ys−1 is defined in Algorithm 1.

ys0 = −B†(Azs0 − c) (11)

where B is required to be a matrix of full column rank, and(·)† denotes the pseudo-inverse of a matrix. Note that theepoch initialization strategy in Eq. (11) plays a key role inour linear convergence guarantees of our linearized proxi-mal acceleration algorithm for the general case of B 6= τI .For the case of B = τI , we still use the proximal ruleand the initialization strategy (i.e., λs = − 1

β (AT )†∇f(xs))in our previous work [49] to guarantee linear convergence,while only this strategy cannot guarantee the convergenceof our algorithm for the general case of B 6= τI . Therefore,we require both the initialization strategies of ys0 and λs toguarantee linear convergence of our algorithm. Note that theinitialization techniques involve the pseudo-inverses of AT

and B. As A and B are often sparse, these can be efficientlycomputed by the Lanczos algorithm [53].

3.1.2 Update Rule of zz is an auxiliary variable, and its update rule is given asfollows. Similar to [19, 22], we also use the inexact Uzawamethod [54] to approximate (7), which can avoid computingthe inverse of the matrix ( 1

η Id1 + βATA). Moreover, themomentum parameter θs (0≤ θs ≤ 1 and its update rule isprovided in Section 3.1.4) is introduced into the proximalterm 1

2η‖z− zsk−1‖2Gs similar to that of (7), and then the

problem with respect to z is formulated as follows:

minz

{⟨z − zsk−1, ∇fIk(xsk−1)

⟩+θs−1

2η‖z − zsk−1‖2Gs

+β

2‖Az +Bysk − c+ λsk−1‖2

} (12)

where ∇fIk(xsk−1) is the stochastic variance reduced gradientestimator independently introduced in [18, 55], and Gs =

γId1− ηβθs−1

ATA with γ > 1+ ηβ‖ATA‖2θs−1

to ensure that Gs� Isimilar to [22]. In fact, there is also an alternative to set Gs asan identity matrix, and then the problem (12) can be solvedthrough matrix inversion [9, 19].

3.1.3 Our Momentum Accelerated Update Rule for xIn particular, our momentum accelerated update rule for xis defined as follows:

xsk= xs−1+ θs−1(zsk− xs−1)=(1− θs−1)xs−1+ θs−1zsk (13)

where θs−1(zsk− xs−1) is a new momentum term similar tothose as in accelerated deterministic methods [27], whichhelps accelerate the convergence speed of our algorithms

by using the iterate of the previous epoch, i.e., xs−1. Notethat θs−1 is a momentum parameter, and its update rule isgiven below. The momentum term, θs−1(zsk−xs−1), plays akey role as the Katyusha momentum in [29]. Different fromKatyusha [29], which uses both the Nesterov’s momentumand Katyusha momentum, our ASVRG-ADMM algorithms(including Algorithm 1 and Algorithm 2 below) have onlyone momentum term.

3.1.4 Update Rule of θs

In all epochs of Algorithm 1, the momentum parameter θscan be set to a constant θ, which must satisfy the condition0 ≤ θ ≤ 1−δ(b)/(α−1), where α = 1

Lη and δ(b) = n−bb(n−1) .

In particular, we also provide the selecting schemes forthe momentum parameter θ and corresponding theoreticalanalysis for the two cases of B = τI and B 6= τI , which allare presented in the Supplementary Material.

The detailed procedure for solving the strongly convexproblem (1) is shown in Algorithm 1, where we use the sameepoch initialization technique for λs as in [22]. Similar to xsk,ys=(1−θs−1)ys−1+θs−1

m

∑mk=1y

sk. When θ=1, ASVRG-ADMM

degenerates to the linearized proximal variant of SVRG-ADMM in [22], as shown in the Supplementary Material.

3.2 ASVRG-ADMM for Non-Strongly Convex Problems

In this part, we consider the non-strongly convex (non-SC)problems of the form (1) when each fi(·) is convex, L-smooth, and h(·) is not necessarily strongly convex (possiblynon-smooth), e.g., graph-guided fused Lasso. The detailedprocedure for solving the non-SC problem (1) is shown inAlgorithm 2, which has slight differences in the initializationand output of each epoch from Algorithm 1. In addition,the key difference between them is the update rule forthe momentum parameter θs. Different from the stronglyconvex case, the momentum parameter θs for the non-SCcase is required to satisfy the following inequalities:

1− θsθ2s

=1

θ2s−1

and 0 ≤ θs ≤ 1− δ(b)

α− 1(14)

where δ(b) := n−bb(n−1) is a decreasing function with respect

to the mini-batch size b. The condition (14) allows themomentum parameter to decease, but not too fast, similarto the requirement on the step-size ηk in classical SGD andstochastic ADMM [56]. Unlike deterministic accelerationmethods, θs must satisfy both inequalities in (14).

Motivated by the momentum acceleration techniquesin [27, 57] for deterministic optimization, we give the updaterule of the momentum parameter θs for the mini-batch case:

θs =

√θ4s−1 + 4θ2

s−1 − θ2s−1

2and θ0 = 1− δ(b)

α− 1. (15)

For the special case of b = 1, we have δ(1) = 1 and θ0 =1− 1

α−1 , while b=n (i.e., the deterministic version), δ(n) =0 and θ0 = 1. Since the sequence {θs} is decreasing, θs ≤1− δ(b)

α−1 is satisfied. That is, θs in Algorithm 2 is adaptivelyadjusted as in (15).

6

Algorithm 2 ASVRG-ADMM for non-SC problemsInput: m, η, β > 0, 1 ≤ b ≤ n.Initialize: x0 = z0, y0 = y0

m, λ0, θ0 = 1− Lηδ(b)1−Lη .

1: for s = 1, 2, . . . , T do2: xs0 =(1−θs−1)xs−1+θs−1z

s−1, ys0 =ys−1m , λs0 = λs−1;

3: p= ∇f(xs−1), zs0 = zs−1;4: ν= 1+ ηβ‖BTB‖2

θs−1, γ=1+ ηβ‖ATA‖2

θs−1;

5: for k = 1, 2, . . . ,m do6: Choose Ik⊆ [n]of size b, uniformly at random;7: ∇fIk(xsk−1)= 1

|Ik|∑ik∈Ik

[∇fik(xsk−1)−∇fik(xs−1)

]+p;

8: ysk = Prox1βτ2

h

((−Azsk−1 + c− λsk−1)/τ

)for the case of B = τI ,

ysk=Proxηβs−1νθs−1

h

[ysk−1−

ηβBT(Azsk−1+Bysk−1−c+λsk−1)

νθs−1

]for the case of B 6= τI ;

9: zsk=zsk−1−η

γθs−1

[∇fIk(xsk−1)+βAT(Azsk−1+By

sk−c+λsk−1)

];

10: xsk=(1− θs−1)xs−1 + θs−1zsk;

11: λsk=λsk−1 +Azsk +Bysk − c;12: end for13: xs= 1

m

∑mk=1x

sk, ys=(1−θs−1)ys−1+ θs−1

m

∑mk=1y

sk;

14: λs=λsm, zs=zsm, θs=

√θ4s−1+4θ2

s−1−θ2s−1

2 ;15: end forOutput: xT , yT .

4 CONVERGENCE ANALYSIS

In this section, we theoretically analyze the convergenceproperties of our ASVRG-ADMM algorithms (i.e., Algo-rithms 1 and 2) for SC and non-SC problems with the casesof B 6= τI and B = τI , respectively. We first make thefollowing assumption for the case of SC problems.

Assumption 3. The matrices A and BT both have full row rank.

The first two assumptions (i.e., Assumptions 1 and 2)are common in the convergence analysis of first-order op-timization methods, while the last one (i.e., Assumption 3)has been used in the convergence analysis of deterministicADMM [7, 58, 59] and stochastic ADMM [22] for only thestrongly convex case. Following [22], we first introduce thefollowing function as a convergence criterion, where h′(y) 2

is the (sub)gradient of h(·) at y.

P (x, y) := f(x)− f(x∗)− 〈∇f(x∗), x−x∗〉+ h(y)− h(y∗)− 〈h′(y∗), y−y∗〉

where (x∗, y∗) denotes an optimal solution of Problem (1).By the convexity of f(·) and h(·), P (x, y)≥0 for all x, y∈Rd.

Note that we present a new linearized proximal tech-nique in (10) to update ysk, and thus we need to provide newconvergence guarantees for our algorithms (i.e., Algorithm-s 1 and 2), which are different from those in our previouswork [49]. Next, we present five main theoretical results forthe convergence properties of Algorithms 1 and 2. And thedetailed proofs of all the theoretical results are provided inthis paper or the Supplementary Material.

2. Note that ∇f(x) is the gradient of a smooth function f(·) at x,while h′(y) denotes a subgradient (or the gradient) of a non-smooth(or smooth) function h(·) at y.

We first sketch the proofs of our main theoretical resultsas follows: The proofs of our main results rely on theone-epoch inequalities in Lemma 4 (B 6= τI) below andLemma 7 (B = τI) in the Supplementary Material. That is,the proofs of Theorems 1-5 below rely on the one-epochinequalities in Lemmas 4 and 7, but require telescopingsuch inequalities in different manners. Furthermore, P (x, y)in Lemma 4 consists of two terms, and thus we give theupper bounds of the two terms in Lemmas 2 and 3 toobtain Lemma 4, as well as applying Lemmas 2 and 6 toget Lemma 7 in the Supplementary Material. In addition,to remove the strong assumption used in Theorems 3 and4, we also design an adaptive strategy of increasing epochlength for Algorithm 2, and the corresponding theoreticalresult is given in Theorem 5, which shows that Algorithm 2with an adaptive increasing epoch length attains the sameconvergence rate without the boundedness assumption.

4.1 Key LemmasIn this part, we give and prove some intermediate keyresults for our convergence analysis.

Lemma 1.

E[‖∇fIk(xsk−1)−∇f(xsk−1)‖2]

≤ 2Lδ(b)[f(xs−1)−f(xsk−1)+〈∇f(xsk−1, x

sk−1− xs−1〉

]where δ(b)= n−b

b(n−1)≤1 and 1 ≤ b ≤ n.

The proofs of Lemmas 1, 2 and all the theorems beloware provided in the Supplementary Material. Lemma 1provides an upper bound on the expected variance of themini-batch SVRG estimator ∇fIk(xsk−1).

Lemma 2. Let (x∗, y∗) be an optimal solution of Problem (1),and λ∗ be the corresponding Lagrange multiplier that maximizesthe dual. Let ϕsk = β(λsk − λ∗), and suppose that each fi(·) isL-smooth. If the inequality 1− θs−1≥ δ(b)

α−1 is satisfied, then

E[f(xs)− f(x∗)− 〈∇f(x∗), xs − x∗〉]

− E

[θs−1

m

m∑k=1

⟨ATϕsk, x

∗−zsk⟩]

≤ (1−θs−1)E[f(xs−1)− f(x∗)− 〈∇f(x∗), xs−1− x∗〉

]+θ2s−1

2mηE[‖x∗ − zs0‖2Gs − ‖x

∗ − zsm‖2Gs].

For the case of B 6= τI , we have the following result,which corresponds to Lemma 7 in the Supplementary Ma-terial for the case of B=τI .

Lemma 3. Let {(ys, ysk)} be the sequence generated by Algorith-m 1 (or Algorithm 2), we have

E[h(ys)− h(y∗)− 〈h′(y∗), ys− y∗〉]

− θs−1

m

m∑k=1

E[〈BTϕsk, y∗−ysk〉

]≤(1−θs−1)E

[h(ys−1)−h(y∗)−〈h′(y∗), ys−1 − y∗〉

]+βθs−1

2mE

[‖Azs0−Ax∗‖2−‖Azsm−Ax∗‖2+

m∑k=1

‖λsk−λsk−1‖2]

+θ2s−1

2mηE[‖y∗ − ys0‖2Qs−‖y

∗ − ysm‖2Qs].

7

Since a new linearized proximal rule is proposed toupdate the variable y for Algorithms 1 and 2 in the caseof B 6=τI , we need to give the following proof for Lemma 3,which is different from Lemma 6 in the SupplementaryMaterial for the case of B=τI .

Proof: Since λsk=λsk−1+Azsk+By

sk−c, and using the op-

timality condition of Problem (10) (i.e., h′(ysk)+βBT(Azsk−1+

Byk−c+λk−1)+ θs−1

η Qs(ysk−ysk−1)=0), we have

h(ysk)− h(y∗)

≤〈h′(ysk), ysk − y∗〉

=

⟨βBT(Azsk−1+Byk−c+λk−1)+

θs−1Qs(ysk−ysk−1)

η, y∗−ysk

⟩=

⟨βBTλsk +

θs−1Qsη

(ysk−ysk−1), y∗−ysk⟩

+⟨βBT (Azsk−1 −Azsk), y∗ − ysk

⟩=β

2

⟨BTλsk, y

∗−ysk⟩

+θs−1

2η

(‖y∗−ysk−1‖2Qs−‖y

∗−ysk‖2Qs)

+β

2

(‖Azsk−1−Ax∗‖2−‖Azsk−Ax∗‖2+‖λsk−λsk−1‖2

)where the last equality follows from Ax∗+By∗ = c andProperty 1 in the Supplementary Material. Taking expecta-tion over the random choice of ik, we have

E[h(ysk)−h(y∗)−〈h′(y∗), ysk−y∗〉−〈BTϕsk, y∗−ysk〉

]≤ β

2E[‖Azsk−1 −Ax∗‖2 − ‖Azsk −Ax∗‖2

]+

1

2E[β‖λsk−λsk−1‖2+

θs−1

η

(‖y∗−ysk−1‖2Qs−‖y

∗−ysk‖2Qs)].

Using the update rule of ys = (1−θs−1)ys−1 + θs−1

m

∑mk=1y

sk,

h(ys) ≤ (1−θs−1)h(ys−1)+ θs−1

m

∑mk=1h(ysk), and taking ex-

pectation over whole history and summing up the aboveinequality for all k = 1, . . . ,m, we have

E

[h(ys)−h(y∗)−〈h′(y∗), ys−y∗〉− θs−1

m

m∑k=1

〈BTϕsk, y∗−ysk〉]

≤θs−1

mE

[m∑k=1

(h(ysk)−h(y∗)+ 〈h′(y∗)−θs−1B

Tϕsk, y∗−ysk〉

)]

+ E

[θ2s−1

2η

(‖y∗ − ysk−1‖2Qs − ‖y

∗ − ysk‖2Qs)]

+ (1−θs−1)E[h(ys−1)− h(y∗)− 〈h′(y∗), ys−1 − y∗〉

]≤ βθs−1

2mE

[‖Azs0−Ax∗‖2−‖Azsm−Ax∗‖2+

m∑k=1

‖λsk−λsk−1‖2]

+ (1− θs−1)E[h(ys−1)− h(y∗)− 〈h′(y∗), ys−1 − y∗〉

]+θ2s−1

2η

[‖y∗ − ysk−1‖2Qs − ‖y

∗ − ysk‖2Qs].

This completes the proof.For the case of B 6= τI , we also have the following one-

epoch inequality, which is a key lemma to prove Theorems2, 4 and 5 below and is corresponding to Lemma 7 in theSupplementary Material for the case of B=τI , and Lemma7 is also a main result to prove Theorems 1, 3 and 6 below.

Lemma 4. (One-epoch Upper Bound) Using the same notationas in Lemma 2, let {(zsk, xsk, ysk, λsk, xs, ys)} be the sequence

generated by Algorithm 1 (or Algorithm 2) with θs ≤ 1− δ(b)α−1 .

Then the following inequality holds for all k,

E[P (xs, ys)− θs−1

m

m∑k=1

((x∗−zsk)TATϕsk+(y∗−ysk)TBTϕsk

)]≤E[P (xs−1, ys−1)

1/(1−θs−1)+θ2s−1

2mη

(‖x∗− zs0‖2Gs−‖x

∗− zsm‖2Gs)]

+βθs−1

2mE[‖Azs0−Ax∗‖2−‖Azsm−Ax∗‖2+

m∑k=1

‖λsk−λsk−1‖2]

+θ2s−1

2mηE[Rs −‖y∗ − ysm‖2Qs

]where Rs is defined as follows:

Rs =

{σ‖Ax∗ −Azs0‖2, if f(x) is SC,

‖y∗ − ys0‖2Qs , if f(x) is non-SC(16)

and σ = ‖B†‖22( 2ηβ‖BTB‖2θs−1

+ 1).

Proof: Using Lemmas 2 and 3 and the definition ofP (x, y), we have

E

[P (xs, ys)− θs−1

m

m∑k=1

((x∗−zsk)TATϕsk+(y∗−ysk)TBTϕsk

)]

≤E[P (xs−1, ys−1)

1/(1−θs−1)+θ2s−1

(‖x∗− zs0‖2Gs−‖x

∗− zsm‖2Gs)

2mη

]

+βθs−1

2mE

[‖Azs0−Ax∗‖2−‖Azsm−Ax∗‖2+

m∑k=1

‖λsk−λsk−1‖2]

+θ2s−1

2mηE[‖y∗−ys0‖2Qs−‖y

∗ − ysm‖2Qs].

When f(·) is µ-strongly convex and Ax∗+By∗ = c, wehave y∗=B†(c−Ax∗). Using the update rule of ys0 =B†(c−Azs0) and ν= 1+ ηβ‖BTB‖2

θs−1, we have

‖y∗ − ys0‖2Qs = ‖B†(Azs0 −Ax∗)‖2Qs≤ ‖B†‖22‖Azs0 −Ax∗‖2Qs

≤ ‖B†‖22∥∥∥∥νI − η

θs−1BTB

∥∥∥∥2

‖Azs0 −Ax∗‖2

≤ ‖B†‖22(

2ηβ‖BTB‖2θs−1

+ 1

)‖Azs0 −Ax∗‖2.

Therefore, the result of Lemma 4 holds.

4.2 Linear ConvergenceFor Algorithm 1, we first give the following results for thetwo cases of B=τI and B 6=τI , respectively.

Theorem 1. (Case of B = τI) Using the same notation as inLemma 2 with θ ≤ 1− δ(b)

α−1 , suppose that f(·) is µ-stronglyconvex, each fi(·) is L-smooth and Assumption 3 holds, and mis sufficiently large so that

ρ1 =θ‖θG+ηβATA‖2

ηmµ︸ ︷︷ ︸1

+ (1−θ)︸ ︷︷ ︸2

+Lθ

βmσmin(AAT )︸ ︷︷ ︸3

< 1 (17)

where σmin(AAT ) is the smallest eigenvalue of the positive semi-definite matrix AAT , and Gs≡G as in Eq. (12). Then

E[P (xT, yT )

]≤ ρT1 P (x0, y0).

8

The theoretical result in our previous work [49] can beviewed as the special case of Theorem 1 when B= I . FromTheorem 1, we can see that ASVRG-ADMM achieves linearconvergence, which is consistent with that of SVRG-ADMM,while SCAS-ADMM has only an O(1/T ) convergence rate.

Remark 1. Theorem 1 shows that our result improves slightlyupon the rate ρ1 in SVRG-ADMM [22] with the same η and β.Specifically, ρ1 in Eq. (17) consists of three components, whichare corresponding to those of Theorem 1 in [22]. In Algorithm 1,recall that θ ≤ 1 and G is defined in Eq. (12). Thus, the upperbound of Eq. (17) is slightly smaller than that of Theorem 1in [22]. In particular, we can set η = 1/8L (i.e., α = 8) andθ = 1− δ(b)/(α−1) = 1− δ(b)/7. Therefore, the second termin Eq. (17) equals to δ(b)/7, while that of SVRG-ADMM isapproximately equal to 4Lηδ(b)/(1− 4Lηδ(b)) ≥ δ(b)/2. Insummary, the convergence speed of SVRG-ADMM can be slightlyimproved by ASVRG-ADMM.

Theorem 2. (Case of B 6= τI) Using the same notation as inLemma 2 with θ ≤ 1− δ(b)

α−1 , suppose that f(·) is µ-stronglyconvex, each fi(·) is L-smooth and Assumption 3 holds, and mis sufficiently large so that

ρ2 =θ%

ηmµ+ (1− θ) +

Lθ

βmσmin(AAT )< 1 (18)

where % = ‖θG+η(β+θσ)ATA)‖2. Then

E[P (xT, yT )

]≤ ρT2 P (x0, y0).

From Theorem 2, ASVRG-ADMM can also achieve linearconvergence for the more complex ADMM-style problem(1) with B 6= τI . It is not hard to see that the convergencerate ρ2 in Theorem 2 is slightly larger than that (i.e., ρ1) ofTheorem 1, meaning slow convergence for more complexoptimization problems, as verified by our experiments.

4.3 Convergence Rate of O(1/T 2)

We first assume that y ∈ Y and z ∈ Z , where Y andZ are the convex compact sets with diameters DY =supy1,y2∈Y‖y1−y2‖ and DZ = supz1,z2∈Z‖z1−z2‖, respec-tively, and DΛ = supλ1,λ2∈Λ‖λ1−λ2‖. The above assump-tion is called the boundedness assumption. We also denoteDx∗=‖x0−x∗‖, Dy∗=‖y0−y∗‖ and Dλ∗=‖λ0−λ∗‖, where(x0, y0, λ0) are initial points, (x∗, y∗) is an optimal solutionof Problem (1) and λ∗ is the corresponding dual variable.The boundedness of Dx∗ , Dy∗ and Dλ∗ are easily satisfied,which is called the basic conditions in this paper.

For Algorithm 2, we give the following results for thecases of B= τI and B 6= τI , respectively, whose proofs areprovided in the Supplementary Material.

Theorem 3. (Case of B = τI) Let ς be a positive constant,suppose that each fi(·) is L-smooth, Z and Λ are the convexcompact sets with diameters DZ and DΛ, then

E[P (xT, yT )+ς‖AxT +τ yT − c‖

]≤

4(α−1)δ(b)(P (x0, y0)+ς‖Ax0 + τ y0 − c‖

)(α− 1− δ(b))2(T+1)2

+2LαD2

x∗

m(T+1)2+

4αβ(‖ATA‖2D2Z+D2

Λ)

m(α−1)(T + 1).

(19)

Remark 2. With m= Θ(n), Theorem 3 shows that the conver-gence bound consists of the three components, which converge asO(1/T 2),O(1/nT 2) andO(1/nT ), respectively, while the threecomponents of SVRG-ADMM converge as O(1/T ), O(1/nT )and O(1/nT ). Clearly, ASVRG-ADMM achieves the conver-gence rate of O(1/T 2) as opposed to O(1/T ) of SVRG-ADMMand SAG-ADMM (m� T in general). All the components inthe bound of SCAS-ADMM converge as O(1/T ). Thus, it isclear that ASVRG-ADMM is at least a factor T faster thanexisting stochastic ADMM algorithms including SAG-ADMM,SVRG-ADMM and SCAS-ADMM. Theorem 3 shows that theconvergence result in our previous work [49] can be viewed as thespecial case of Theorem 3. In addition, Theorems 3 and 4 belowrequire the boundedness assumption and the basic conditions (i.e.,Dx∗ , Dy∗ and Dλ∗ are bounded by some constants).

Theorem 4. (Case of B 6= τI) Using the same notation as inLemma 2, and suppose that each fi(·) is L-smooth, and Y , Z andΛ are the convex compact sets with diameters DY , DZ and DΛ,then we have

E[P (xT, yT )+ ς‖AxT +ByT− c‖

]≤

4(α−1)δ(b)(P (x0, y0) + ς‖Ax0+By0− c‖

)(α− 1− δ(b))2(T+1)2

+2αβ

(2‖ATA‖2D2

Z+‖BTB‖2D2Y+2D2

Λ

)m(α−1)(T + 1)

+2Lα(D2

x∗ +D2y∗)

m(T + 1)2.

(20)

4.4 O(1/T 2) without Boundedness Assumption

The result in Theorem 4 shows that ASVRG-ADMM attainsthe optimal convergence rate O(1/T 2) for the non-SC prob-lem (1) with B 6= τI . Compared with SVRG-ADMM andSAG-ADMM, ASVRG-ADMM attains a better convergencerate for non-SC problems, but with the price on the bound-edness of the feasible primal sets Z , Y , and the feasible dualset Λ. Note that many previous works such as [60, 61] alsorequire such assumptions of boundedness when provingthe convergence of ADMMs. In order to remove the strongassumption and further improve our theoretical results, wedesign an adaptive strategy of increasing epoch length,i.e., ms+1 = d θs−1θs mse, while a constant epoch length m isused in original Algorithm 2. The increasing epoch lengthstrategy is similar to that in [62], that is, ms+1 = d θs−1θs mseinstead of ms+1 =2ms in [62]. By replacing the epoch lengthm in Algorithm 2 with ms, we can obtain the followingimproved theoretical result. It should be noted that theincreasing factor θs−1

θsapproaches 1 as the number of epochs

increases, which means that the epoch length increases veryslowly. Below we only present the convergence result forthe general case of B 6=τI , the theoretical result for the caseof B = τI and the detailed proofs for all the results areprovided in the Supplementary Material.

Theorem 5. (Without boundedness assumption) Using the samenotation as in Lemma 2, suppose that each fi(·) is L-smooth. Let{(xs, ys, λs)} be the sequence generated by Algorithm 2 with our

9

adaptive increasing epoch length strategy for the case of B 6= τI ,then

E[P (xT, yT )+ ς‖AxT +ByT− c‖

]≤

4(α−1)δ(b)(P (x0, y0) + ς‖Ax0+By0− c‖

)(α− 1− δ(b))2(T+1)2

+2(α−1)β

(2‖ATA‖2D2

x∗+‖BTB‖2D2y∗+2D2

λ∗)

(α−1−δ(b))m1(T+1)2

+2Lα(D2

x∗+D2y∗)

m1(T+1)2.

(21)

Remark 3. With the setting m1 = Θ(n), Theorem 5 shows thatASVRG-ADMM with our adaptive epoch length strategy obtainsthe rate ofO(1/T 2). The upper bound only relies on the constantsDx∗ , Dy∗ and Dλ∗ , while the theoretical result in Theorem 4requires that Y , Z and Λ are all bounded with the diameters DY ,DZ and DΛ. That is, ASVRG-ADMM with our adaptive epochlength strategy achieves the convergence rate O(1/T 2) withoutthe boundedness assumption.

4.5 Discussion

All our algorithms and convergence results can be extendedto the following settings: When the mini-batch size b = nand m = 1, then δ(n) = 0, that is, the first term of both(19) and (20) vanishes, and ASVRG-ADMM degeneratesto the deterministic two-block3 ADMM version [63]. Theconvergence rate of (20) becomes O

(D2x∗+D

2y∗

(T+1)2 +D2Z+D

2Y+D

2Λ

T+1

),

which is consistent with the result for accelerated deter-ministic ADMM [34, 37]. Many empirical risk minimizationproblems can be viewed as the special case of Problem (2)when A = I . Thus, our method can be extended to solvethem, and has anO(1/T 2+1/(nT 2)) rate, which is consistentwith the best-known result as in [29, 30].

5 EXPERIMENTAL RESULTS

In this section, we apply ASVRG-ADMM to solve vari-ous machine learning problems, e.g., non-SC graph-guidedfused Lasso, SC and non-SC graph-guided logistic regres-sion, and SC graph-guided SVM problems. We compareASVRG-ADMM with the state-of-the-art methods: STOC-ADMM [9], OPG-ADMM [45], SAG-ADMM [19], SCAS-ADMM [21] and SVRG-ADMM [22]. All methods wereimplemented in MATLAB, and the experiments were per-formed on a PC with an Intel i5-2400 CPU and 16GB RAM.

5.1 Synthetic Data

We first evaluate the empirical performance of the pro-posed algorithms for solving both SC and non-SC prob-lems (1) on some synthetic data. Here, each fi(x) is thelogistic loss function on the feature-label pair (ai, bi), i.e.,fi(x) = log(1+exp(−biaTi x)) (i = 1, 2, . . . , n) for the non-SC case and fi(x) = log(1+exp(−biaTi x))+ λ2

2 ‖x‖22 for the

SC case, where λ2 ≥ 0 is the regularization parameter. Weused a relatively small data set, a9a (about 733K), and arelatively large data set, epsilon (about 11G), as listed in

3. Note that the formulation (1) is called two-block because of thetwo sets of variables (x, y), which are updated alternately.

0 1 2 3 4 5 6

CPU time (s)

10-4

10-3

10-2

10-1

Obj

ectiv

e m

inus

min

imum

val

ue

SVRG-ADMM+ASVRG-ADMM

(a) SC, a9a

0 1 2 3 4 5 6

CPU time (s)

10-2

10-1

Obj

ectiv

e m

inus

min

imum

val

ue

SVRG-ADMM+ASVRG-ADMM

(b) Non-SC, a9a

0 500 1000 1500

CPU time (s)

10-3

10-2

Obj

ectiv

e m

inus

min

imum

val

ue

SVRG-ADMM+ASVRG-ADMM

(c) SC, epsilon

0 500 1000 1500

10−2

CPU time (s)

Obj

ectiv

e m

inus

min

imum

val

ue

SVRG−ADMM+APSVRG−ADMM

(d) Non-SC, epsilon

Fig. 1. Comparison of the linearized proximal SVRG-ADMM and ourASVRG-ADMM algorithms for both SC and non-SC problems on thetwo data sets: a9a (top) and epsilon (bottom).

Table 2. Note that the constraint matrix A is set to A = [G; I]as in [9, 19, 22, 61], where G is the sparsity pattern of thegraph obtained by sparse inverse covariance selection [64],while both B and c are randomly generated. In particular,the generated matrix B has full column rank, but is not anidentity matrix. Since the original SVRG-ADMM [22] cannotbe used to solve the general problem (1) with B 6= τI , wealso present its linearized proximal variant (called SVRG-ADMM+), as shown in the Supplementary Material.

Fig. 1 shows the training loss (i.e., the training objectivevalue minus the minimum) of SVRG-ADMM+ and ASVRG-ADMM for both SC and non-SC problems on a9a andepsilon. All the experimental results show that our ASVRG-ADMM method (i.e., Algorithms 1 and 2) converges consis-tently much faster than SVRG-ADMM+, which empiricallyverifies our theoretical results of ASVRG-ADMM.

5.2 Real-world ApplicationsWe also apply our ASVRG-ADMM method to solve anumber of real-world machine learning problems such asgraph-guided fused Lasso, graph-guided logistic regression,graph-guided SVM, generalized graph-guided logistic re-gression and multi-task learning.

5.2.1 Graph-Guided Fused LassoWe evaluate the empirical performance of ASVRG-ADMMfor solving the non-SC graph-guided fused Lasso problem:

minx,y

{ 1

n

n∑i=1

fi(x) + λ1‖y‖1, s.t., Ax = y}

(22)

where fi(·) is the logistic loss function on the feature-labelpair (ai, bi), i.e., fi(x) = log(1+exp(−biaTi x)), and λ1 ≥ 0is the regularization parameter. As in [9, 22, 61], we set A=[G; I], where G is the sparsity pattern of the graph obtainedby sparse inverse covariance selection [64]. We used four

10

0 1 2 3 4 5

10−4

10−3

10−2

10−1

CPU time (s)

Obj

ectiv

e m

inus

min

imum

val

ue

0 10 20 30 40 50

10−3

10−2

10−1

CPU time (s)

Obj

ectiv

e m

inus

min

imum

val

ue

0 20 40 60 80

10−4

10−2

CPU time (s)

Obj

ectiv

e m

inus

min

imum

val

ue

0 50 100 150 200 250 300

10−4

10−2

CPU time (s)

Obj

ectiv

e m

inus

min

imum

val

ue

0 1 2 3 4 50.32

0.33

0.34

0.35

0.36

CPU time (s)

Tes

t los

s

(a) a9a

0 10 20 30 40 50

0.14

0.16

0.18

0.2

0.22

0.24

0.26

0.28

CPU time (s)

Tes

t los

s

(b) w8a

0 20 40 60 80

0.3765

0.377

0.3775

0.378

0.3785

CPU time (s)

Tes

t los

s

(c) SUSY

0 50 100 150 200 250 3000.326

0.3265

0.327

0.3275

0.328

0.3285

0.329

0.3295

CPU time (s)

Tes

t los

s

(d) HIGGS

STOC−ADMM OPG−ADMM SAG−ADMM SCAS−ADMM SVRG−ADMM ASVRG−ADMM

Fig. 2. Comparison of different stochastic ADMM methods for non-SC graph-guided fused Lasso problems on the four data sets. The y-axisrepresents the objective value minus the minimum value (top) or test loss (bottom), and the x-axis corresponds to the running time (seconds).

publicly available data sets4 in our experiments, as listedin Table 2. The parameter m of ASVRG-ADMM is set tom= d2n/be as in [19, 22], as well as η and β. All the otheralgorithms except STOC-ADMM adopted the linearizationof the penalty term β

2 ‖Ax−y+z‖2 to avoid the inversion of1ηkId1

+βATA at each iteration, which can be computationallyexpensive for large matrices.

TABLE 2Summary of data sets and regularization parameters λ1 and λ2 used in

our experiments.

Data sets ] training ] test ] mini-batch λ1 λ2

a9a 16,281 16,280 20 1e-5 1e-2epsilon 400,000 100,000 30 1e-4 1e-4w8a 32,350 32,350 20 1e-5 1e-2SUSY 3,500,000 1,500,000 100 1e-5 1e-2HIGGS 7,700,000 3,300,000 150 1e-5 1e-2

Fig. 2 shows the training loss (i.e., the training objectivevalue minus the minimum) and test error of all the algo-rithms for non-SC problems on the four data sets. SAG-ADMM could not generate experimental results on theHIGGS data set because it ran out of memory. These figuresclearly indicate that the variance reduced stochastic AD-MM algorithms (i.e., SAG-ADMM, SCAS-ADMM, SVRG-ADMM and ASVRG-ADMM) converge much faster thanthose without variance reduction techniques, e.g., STOC-ADMM and OPG-ADMM. In particular, ASVRG-ADMMconsistently outperforms the other algorithms in terms ofconvergence speed in all the settings, which empirically ver-ifies our theoretical result that ASVRG-ADMM has a fasterconvergence rate ofO(1/T 2), as opposed to the best-knownrate of O(1/T ). Moreover, the test error of ASVRG-ADMMis consistently better than those of the other methods.

4. http://www.csie.ntu.edu.tw/∼cjlin/libsvmtools/datasets/

0 1 2 3 4 510

−10

10−8

10−6

10−4

10−2

CPU time (s)

Obj

ectiv

e m

inus

min

imum

val

ue

0 1 2 3 4 50.37

0.375

0.38

0.385

0.39

0.395

0.4

0.405

CPU time (s)

Tes

t los

s

0 5 10 15 20 25 30

10−10

10−5

CPU time (s)

Obj

ectiv

e m

inus

min

imum

val

ue

0 5 10 15 20 25 300.25

0.3

0.35

CPU time (s)

Tes

t los

s

STOC−ADMM OPG−ADMM SAG−ADMM

SCAS−ADMM SVRG−ADMM ASVRG−ADMM

Fig. 3. Comparison of the stochastic ADMM methods for SC graph-guided logistic regression problems on a9a (top) and w8a (bottom).

5.2.2 Graph-Guided Logistic Regression

We also discuss the performance of ASVRG-ADMM for theSC graph-guided logistic regression problem:

minx,y

{ 1

n

n∑i=1

(fi(x)+

λ2

2‖x‖2

)+λ1‖y‖1, s.t., Ax=y

}. (23)

Due to limited space and similar experimental phenomenaon the four data sets, we only report the experimental resultson the a9a and w8a data sets in Fig. 3, from which wecan see that SVRG-ADMM and ASVRG-ADMM achievecomparable performance, and they significantly outperformthe other methods in terms of convergence speed, which isconsistent with their linear (geometric) convergence guar-antees. Moreover, ASVRG-ADMM converges slightly fasterthan SVRG-ADMM, which shows the effectiveness of the

11

2 4 6 8 10

0.83

0.835

0.84

0.845

0.85

0.855

Running time (s)

Tes

ting

accu

racy

%

SVMSTOC−ADMMSVRG−ADMMASVRG−ADMM

2 4 6 80.84

0.845

0.85

0.855

# of epochs

Tes

ting

accu

racy

%

Fig. 4. Accuracy comparison of multi-class classification on 20news-groups: accuracy vs running time (left) or number of epochs (right).

proposed momentum trick to accelerate variance reducedstochastic ADMM, as we expected.

5.2.3 Graph-Guided SVMWe also evaluate the performance of ASVRG-ADMM forsolving the SC graph-guided SVM problem,

minx,y

{ 1

n

n∑i=1

([1− biaTi x]+ +

λ2

2‖x‖22

)+ λ1‖y‖1

},

s.t., Ax = y

(24)

where [x]+ =max(0, x) is the non-smooth hinge loss. To ef-fectively solve (24), we use the smooth Huberized hinge lossin [65] to approximate the hinge loss. For the 20newsgroupsdata set5, we randomly divide it into 80% training set and20% test set. Following [9], we set λ1 = λ2 = 10−5, and usethe one-vs-rest scheme for the multi-class classification.

Fig. 4 shows the average prediction accuracies and s-tandard deviations of testing accuracies over 10 differen-t runs. Since STOC-ADMM, OPG-ADMM, SAG-ADMMand SCAS-ADMM consistently perform worse than SVRG-ADMM and ASVRG-ADMM in all settings, we only reportthe results of STOC-ADMM. We can see that SVRG-ADMMand ASVRG-ADMM consistently outperform the classicalSVM and STOC-ADMM. Moreover, ASVRG-ADMM per-forms much better than the other methods in all settings,which further verifies the effectiveness of ASVRG-ADMM.

5.2.4 Generalized Graph-Guided Logistic RegressionMoreover, we apply ASVRG-ADMM to solve the non-SCgraph-guided logistic regression problem as in [66]:

minx,y

{ 1

n

n∑i=1

fi(x) + λ1‖x‖1 + λ2‖y‖1, s.t., Ax = y}. (25)

All the problems in (22), (23) and (24) can be cast as theform (2), while Problem (25) can be cast as the form (1), i.e.,minx,v{f(x)+‖v‖1, s.t.Cx+Bv=0}, where v=[λ1z

T, λ2yT ]T

and z are slack variables, C=[Idx, AT ]T,B=−

[1λ1Idx 0

0 1λ2Idy

].

The experimental results on the a9a data set are shownin Fig. 5, from which we can see that SVRG-ADMM+ andASVRG-ADMM converge significantly faster than STOC-ADMM+. Note that SVRG-ADMM+ and STOC-ADMM+are the linearized proximal variants of SVRG-ADMMand STOC-ADMM. Moreover, ASVRG-ADMM outperformsthem in terms of both convergence speed and test error,which shows the effectiveness of our momentum trick toaccelerate variance reduced stochastic ADMM.

5. http://www.cs.nyu.edu/∼roweis/data.html

0 1 2 3 4 5 6Running time (s)

10-6

10-4

10-2

Obj

ectiv

e m

inus

min

imum

val

ue STOC-ADMM+SVRG-ADMM+ASVRG-ADMM

0 1 2 3 4 5 6Running time (s)

0.227

0.228

0.229

Tes

t err

or

STOC-ADMM+SVRG-ADMM+ASVRG-ADMM

Fig. 5. Comparison of all the methods for generalized graph-guidedfused Lasso on a9a, where regularization parameters λ1=λ2=10−5.

0 5 10 15 20 25 30Running time (s)

10-4

10-2

100

Obj

ectiv

e m

inus

min

imum

val

ue

STOC-ADMMSVRG-ADMMASVRG-ADMM

0 5 10 15 20 25 30Running time (s)

10-2

10-1

Tes

t err

or

STOC-ADMMSVRG-ADMMASVRG-ADMM

Fig. 6. Comparison of all the methods for multi-task learning problemson 20newsgroups, where the regularization parameter λ1=10−4.

5.2.5 Multi-Task Learning

Finally, we consider the multi-task learning problemand can cast it as the non-SC constrained problem:minX,Y {

∑Ni=1fi(X)+λ1‖Y ‖∗, s.t., X = Y }, where X,Y ∈

Rd×N , N is the number of tasks, fi(X) is the multinomiallogistic loss on the i-th task, and ‖Y ‖∗ is the nuclear norm.The experimental results in Fig. 6 show that ASVRG-ADMMoutperforms the other methods including SVRG-ADMM interms of both convergence speed and test error.

6 CONCLUSIONS AND FURTHER WORK

In this paper, we proposed an efficient accelerated stochas-tic variance reduced ADMM (ASVRG-ADMM) method, inwhich we combined both our proposed momentum accel-eration trick and the variance reduction stochastic ADM-M [22]. We also designed two different update rules forthe general ADMM (i.e., B 6= τI) and special ADMM (i.e.,B = τI) problems, respectively. That is, we presented anew linearized proximal scheme for the case of B 6= τI ,and adopted a simple proximal scheme in our previouswork [49] for the case of B = τI . Moreover, we theoreti-cally analyzed the convergence properties of the proposedlinearized proximal accelerated SVRG-ADMM algorithms,which show that ASVRG-ADMM achieves linear conver-gence and O(1/T 2) rates for strongly convex and non-strongly convex cases, respectively. In particular, ASVRG-ADMM is at least a factor T faster than existing stochasticADMM methods for non-strongly convex problems.

Our empirical study showed that the convergence speedof ASVRG-ADMM is much faster than those of the state-of-the-art stochastic ADMM methods such as SVRG-ADMM.We can apply our proposed momentum acceleration trick toaccelerate existing incremental gradient descent algorithmssuch as [67, 68] for solving regularized empirical risk mini-mization problems. An interesting direction of future workis the research of our proposed momentum acceleration trick

12

for accelerating incremental gradient descent ADMM algo-rithms such as SAG-ADMM [19] and SAGA-ADMM [52]. Inaddition, it is also interesting to extend our algorithms andtheoretical results from the two-block version to the multi-block ADMM case [69].

ACKNOWLEDGMENTS

We thank all the reviewers for their valuable comments.This work was supported by the National Natural ScienceFoundation of China (Nos. 61876221, 61876220, 61976164,61836009, U1701267, 61871310, and 61801353), the Projectsupported the Foundation for Innovative Research Groupsof the National Natural Science Foundation of China (No.61621005), the Major Research Plan of the National NaturalScience Foundation of China (Nos. 91438201 and 91438103),the Program for Cheung Kong Scholars and InnovativeResearch Team in University (No. IRT 15R53), the Fundfor Foreign Scholars in University Research and TeachingPrograms (the 111 Project) (No. B07048), and the ScienceFoundation of Xidian University (Nos. 10251180018 and10251180019). Z. Lin is supported by NSF China (grant no.s61625301 and 61731018), Major Scientific Research Project ofZhejiang Lab (grant no.s 2019KB0AC01 and 2019KB0AB02),Beijing Academy of Artificial Intelligence, and Qualcomm.

REFERENCES

[1] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma,“Robust face recognition via sparse representation,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2,pp. 210–227, Feb. 2009.

[2] W. Zhang, L. Zhang, Z. Jin, R. Jin, D. Cai, X. Li, R. Liang,and X. He, “Sparse learning with stochastic compositeoptimization,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 39, no. 6, pp. 1223–1236, Jun. 2017.

[3] S. Kim, K. A. Sohn, and E. P. Xing, “A multivariateregression approach to association analysis of a quan-titative trait network,” Bioinformatics, vol. 25, pp. i204–i212, 2009.

[4] R. J. Tibshirani and J. Taylor, “The solution path of thegeneralized lasso,” Annals of Statistics, vol. 39, no. 3, pp.1335–1371, 2011.

[5] J. Liu, P. Musialski, P. Wonka, and J. Ye, “Tensor com-pletion for estimating missing values in visual data,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp.208–220, Jan. 2013.

[6] C. Lu, J. Feng, S. Yan, and Z. Lin, “A unified alter-nating direction method of multipliers by majorizationminimization,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 40, no. 3, pp. 527–541, Mar. 2018.

[7] F. Shang, J. Cheng, Y. Liu, Z.-Q. Luo, and Z. Lin,“Bilinear factor matrix norm minimization for robustPCA: Algorithms and applications,” IEEE Trans. PatternAnal. Mach. Intell., vol. 40, no. 9, pp. 2066–2080, Sep.2018.

[8] S. Bubeck, “Convex optimization: Algorithms and com-plexity,” Found. Trends Mach. Learn., vol. 8, pp. 231–358,2015.

[9] H. Ouyang, N. He, L. Q. Tran, and A. Gray, “Stochasticalternating direction method of multipliers,” in Proc.30th Int. Conf. Mach. Learn., 2013, pp. 80–88.

[10] H. Robbins and S. Monro, “A stochastic approximationmethod,” Ann. Math. Statist., vol. 22, no. 3, pp. 400–407,1951.

[11] Y. Nesterov, “A method of solving a convex program-ming problem with convergence rate O(1/k2),” SovietMath. Doklady, vol. 27, pp. 372–376, 1983.

[12] ——, “Gradient methods for minimizing compositefunctions,” Math. Program., vol. 140, pp. 125–161, 2013.

[13] P. Tseng, “On accelerated proximal gradient methodsfor convex-concave optimization,” Technical report, Uni-versity of Washington, 2008.

[14] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,”SIAM J. Imaging Sci., vol. 2, no. 1, pp. 183–202, 2009.

[15] T. Zhang, “Solving large scale linear prediction prob-lems using stochastic gradient descent algorithms,” inProc. 21st Int. Conf. Mach. Learn., 2004, pp. 919–926.

[16] C. Hu, J. T. Kwok, and W. Pan, “Accelerated gradientmethods for stochastic optimization and online learn-ing,” in Proc. Adv. Neural Inf. Process. Syst., 2009, pp.781–789.

[17] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Im-ageNet classification with deep convolutional neuralnetworks,” in Proc. Adv. Neural Inf. Process. Syst., 2012,pp. 1097–1105.

[18] R. Johnson and T. Zhang, “Accelerating stochastic gra-dient descent using predictive variance reduction,” inProc. Adv. Neural Inf. Process. Syst., 2013, pp. 315–323.

[19] L. W. Zhong and J. T. Kwok, “Fast stochastic alternatingdirection method of multipliers,” in Proc. 31st Int. Conf.Mach. Learn., 2014, pp. 46–54.

[20] T. Suzuki, “Stochastic dual coordinate ascent with al-ternating direction method of multipliers,” in Proc. 31stInt. Conf. Mach. Learn., 2014, pp. 736–744.

[21] S.-Y. Zhao, W.-J. Li, and Z.-H. Zhou, “Scalable stochas-tic alternating direction method of multipliers,” arX-iv:1502.03529v3, 2015.

[22] S. Zheng and J. T. Kwok, “Fast-and-light stochasticADMM,” in Proc. 25th Int. Joint Conf. Artif. Intell., 2016,pp. 2407–2613.

[23] N. L. Roux, M. Schmidt, and F. Bach, “A stochasticgradient method with an exponential convergence ratefor finite training sets,” in Proc. Adv. Neural Inf. Process.Syst., 2012, pp. 2672–2680.

[24] S. Shalev-Shwartz and T. Zhang, “Stochastic dual coor-dinate ascent methods for regularized loss minimiza-tion,” J. Mach. Learn. Res., vol. 14, pp. 567–599, 2013.

[25] L. Xiao and T. Zhang, “A proximal stochastic gradientmethod with progressive variance reduction,” SIAM J.Optim., vol. 24, no. 4, pp. 2057–2075, 2014.

[26] F. Shang, K. Zhou, H. Liu, J. Cheng, I. Tsang, L. Zhang,D. Tao, and L. Jiao, “VR-SGD: A simple stochasticvariance reduction method for machine learning,” IEEETrans. Knowl. Data Eng., vol. 32, no. 1, pp. 188–202, Jan.2020.

[27] Y. Nesterov, Introductory Lectures on Convex Optimiza-tion: A Basic Course. Boston: Kluwer Academic Publ.,2004.

[28] A. Nitanda, “Stochastic proximal gradient descent withacceleration techniques,” in Proc. Adv. Neural Inf. Pro-cess. Syst., 2014, pp. 1574–1582.

13

[29] Z. Allen-Zhu, “Katyusha: The first direct accelerationof stochastic gradient methods,” J. Mach. Learn. Res.,vol. 18, no. 221, pp. 1–51, 2018.

[30] L. T. K. Hien, C. Lu, H. Xu, and J. Feng, “Ac-celerated stochastic mirror descent algorithms forcomposite non-strongly convex optimization,” arX-iv:1605.06892v2, 2016.

[31] F. Shang, Y. Liu, J. Cheng, K. W. Ng, and Y. Yoshida,“Guaranteed sufficient decrease for stochastic variancereduced gradient optimization,” in Proc. 21st Int. Conf.Artif. Intell. Statist., 2018, pp. 1027–1036.

[32] Z. Lin, R. Liu, and Z. Su, “Linearized alternating di-rection method with adaptive penalty for low-rankrepresentation,” in Proc. Adv. Neural Inf. Process. Syst.,2011, pp. 612–620.

[33] F. Nie, Y. Huang, XiaoqianWang, and H. Huang, “Lin-ear time solver for primal SVM,” in Proc. 31st Int. Conf.Mach. Learn., 2014, pp. 505–513.

[34] C. Lu, H. Li, Z. Lin, and S. Yan, “Fast proximal lin-earized alternating direction method of multiplier withparallel splitting,” in Proc. AAAI Conf. Artif. Intell., 2016,pp. 739–745.

[35] M. Hong and Z.-Q. Luo, “On the linear convergence ofthe alternating direction method of multipliers,” Math.Program., vol. 162, pp. 165–199, 2017.

[36] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein,“Distributed optimization and statistical learning viathe alternating direction method of multipliers,” Found.Trends Mach. Learn., vol. 3, no. 1, pp. 1–122, 2011.

[37] T. Goldstein, B. O’Donoghue, S. Setzer, and R. Bara-niuk, “Fast alternating direction optimization method-s,” SIAM J. Imaging Sci., vol. 7, no. 3, pp. 1588–1623,2014.

[38] J. Eckstein and W. Yao, “Understanding the conver-gence of the alternating direction method of multipli-ers: Theoretical and computational perspectives,” Pac.J. Optim., vol. 11, pp. 619–644, 2015.

[39] H. Li and Z. Lin, “Accelerated alternating directionmethod of multipliers: an optimal O(1/K) nonergodicanalysis,” J Sci. Comput., vol. 79, no. 2, pp. 671–699,2019.

[40] M. Kadkhodaie, K. Christakopoulou, M. Sanjabi, andA. Banerjee, “Accelerated alternating direction methodof multipliers,” in Proc. SIGKDD Conf. Knowl. Disc. DataMin., 2015, pp. 497–506.

[41] D. Davis and W. Yin, “Faster convergence rates ofrelaxed Peaceman-Rachford and ADMM under regu-larity assumptions,” Math. Oper. Res., vol. 42, no. 3, pp.783–805, 2017.

[42] W. Tian and X. Yuan, “An alternating direction methodof multipliers with a worst-case O(1/n2) convergencerate,” Math. Comp., vol. 88, pp. 1685–1713, 2019.

[43] G. Franca, D. P. Robinson, and R. Vidal, “ADMM andaccelerated ADMM as continuous dynamical systems,”in Proc. Int. Conf. Mach. Learn., 2018, pp. 1554–1562.

[44] H. Wang and A. Banerjee, “Online alternating directionmethod,” in Proc. 29th Int. Conf. Mach. Learn., 2012, pp.1119–1126.

[45] T. Suzuki, “Dual averaging and proximal gradien-t descent for online alternating direction multipliermethod,” in Proc. 30th Int. Conf. Mach. Learn., 2013, pp.

392–400.[46] Y. Yu and L. Huang, “Fast stochastic variance reduced

admm for stochastic composition optimization,” inProc. Int. Joint Conf. Artif. Intell., 2017, pp. 3364–3370.

[47] Y. Xu, M. Liu, Q. Lin, and T. Yang, “ADMM without afixed penalty parameter: Faster convergence with newadaptive penalization,” in Proc. Adv. Neural Inf. Process.Syst., 2017, pp. 1267–1277.

[48] C. Fang, F. Cheng, and Z. Lin, “Faster and non-ergodicO(1/K) stochastic alternating direction method ofmultipliers,” in Proc. Adv. Neural Inf. Process. Syst., 2017,pp. 4479–4488.

[49] Y. Liu, F. Shang, and J. Cheng, “Accelerated variancereduced stochastic ADMM,” in Proc. AAAI Conf. Artif.Intell., 2017, pp. 2287–2293.

[50] J. Koneeny, J. Liu, P. Richtarik, , and M. Takae, “Mini-batch semi-stochastic gradient descent in the proximalsetting,” IEEE J. Sel. Top. Sign. Proces., vol. 10, no. 2, pp.242–255, 2016.

[51] F. Huang, S. Chen, and H. Huang, “Faster stochasticalternating direction method of multipliers for noncon-vex optimization,” in Proc. Int. Conf. Mach. Learn., 2019,pp. 2839–2848.

[52] F. Huang and S. Chen, “Mini-batch stochastic AD-MMs for nonconvex nonsmooth optimization,” arX-iv:1802.03284v3, 2019.

[53] G. H. Golub and C. F. V. Loan, Matrix Computions.Maryland: Johns Hopkins University Press, 2013.

[54] X. Zhang, M. Burger, and S. Osher, “A unified primal-dual algorithm framework based on Bregman itera-tion,” J. Sci. Comput., vol. 46, no. 1, pp. 20–46, 2011.

[55] L. Zhang, M. Mahdavi, and R. Jin, “Linear convergencewith condition number independent access of full gra-dients,” in Proc. Adv. Neural Inf. Process. Syst., 2013, pp.980–988.

[56] P. Tseng, “An incremental gradient(-projection) methodwith momentum term and adaptive step size rule,”SIAM J. Optim., vol. 8, no. 2, pp. 506–531, 1998.

[57] ——, “Approximation accuracy, gradient methods, anderror bound for structured convex optimization,” Math.Program., vol. 125, pp. 263–295, 2010.

[58] R. Nishihara, L. Lessard, B. Recht, A. Packard, andM. I. Jordan, “A general analysis of the convergenceof ADMM,” in Proc. 32nd Int. Conf. Mach. Learn., 2015,pp. 343–352.

[59] W. Deng and W. Yin, “On the global and linear conver-gence of the generalized alternating direction methodof multipliers,” J. Sci. Comput., vol. 66, pp. 889–916,2016.

[60] B. He and X. Yuan, “Convergence analysis of primal-dual algorithms for a saddle-point problem: from con-traction perspective,” SIAM J. Imaging Sciences, vol. 5,no. 1, p. 119149, 2012.

[61] S. Azadi and S. Sra, “Towards an optimal stochasticalternating direction method of multipliers,” in Proc.31st Int. Conf. Mach. Learn., 2014, pp. 620–628.

[62] Z. Allen-Zhu and Y. Yuan, “Improved SVRG for non-strongly-convex or sum-of-non-convex objectives,” inProc. 33rd Int. Conf. Mach. Learn., 2016, p. 10801089.

[63] Q. Tran-Dinh, “Non-ergodic alternating proximal aug-mented lagrangian algorithms with optimal rates,” in

14

Proc. Adv. Neural Inf. Process. Syst., 2018, pp. 4816–4824.[64] O. Banerjee, L. E. Ghaoui, and A. d’Aspremont, “Model

selection through sparse maximum likelihood estima-tion for multivariate Gaussian or binary data,” J. Mach.Learn. Res., vol. 9, pp. 485–516, 2008.

[65] S. Rosset and J. Zhu, “Piecewise linear regularizedsolution paths,” Ann. Statist., vol. 35, no. 3, pp. 1012–1030, 2007.

[66] S. Yang, L. Yuan, Y.-C. Lai, X. Shen, P. Wonka, and J. Ye,“Feature grouping and selection over an undirectedgraph,” in Proc. SIGKDD Conf. Knowl. Disc. Data Min.,2012, pp. 922–930.

[67] K. Zhou, Q. Ding, F. Shang, J. Cheng, D. Li, and Z. Q.Luo, “Direct acceleration of SAGA using sampled neg-ative momentum,” in Proc. Int. Conf. Artif. Intell. Statist.,2019, pp. 1602–1610.

[68] Y. Liu, F. Shang, and L. Jiao, “Accelerated incrementalgradient descent using momentum acceleration withscaling factor,” in Proc. Int. Joint Conf. Artif. Intell., 2019,pp. 3045–3051.

[69] C. Chen, B. He, Y. Ye, and X. Yuan, “The direct exten-sion of ADMM for multi-block convex minimizationproblems is not necessarily convergent,” Math. Comp.,vol. 155, pp. 57–79, 2016.

Yuanyuan Liu received the Ph.D. degree in Pat-tern Recognition and Intelligent System from Xi-dian University, Xi’an, China, in 2013.

She is currently a professor with the Schoolof Artificial Intelligence, Xidian University, China.Prior to joining Xidian University, she was a Post-Doctoral Research Fellow with the Departmentof Computer Science and Engineering, The Chi-nese University of Hong Kong. From 2013 to2014, she was a Post-Doctoral Research Fellowwith the Department of Systems Engineering

and Engineering Management, CUHK. Her current research interestsinclude machine learning, pattern recognition, and image processing.

Fanhua Shang (SM’20) received the Ph.D. de-gree in Circuits and Systems from Xidian Univer-sity, Xi’an, China, in 2012.

He is currently a professor with the Schoolof Artificial Intelligence, Xidian University, Chi-na. Prior to joining Xidian University, he wasa Research Associate with the Department ofComputer Science and Engineering, The Chi-nese University of Hong Kong. From 2013 to2015, he was a Post-Doctoral Research Fellowwith the Department of Computer Science and

Engineering, The Chinese University of Hong Kong. From 2012 to 2013,he was a Post-Doctoral Research Associate with the Department ofElectrical and Computer Engineering, Duke University, Durham, NC,USA. His current research interests include machine learning, datamining, pattern recognition, and computer vision.

Hongying Liu (M’10) received her B.E. and M.S.degrees in Computer Science and Technologyfrom Xi’An University of Technology, China, in2006 and 2009, respectively, and Ph.D. in Engi-neering from Waseda University, Japan in 2012.Currently, she is a faculty member at the Schoolof Artificial Intelligence, and also with the KeyLaboratory of Intelligent Perception and ImageUnderstanding of Ministry of Education, XidianUniversity, China. In addition, she is a memberof IEEE. Her major research interests include

image processing, intelligent signal processing, machine learning, etc.

Lin Kong received the BS degree in Statisticsfrom Xidian University in 2019. She is currentlyworking toward her Master degree in the Schoolof Artificial Intelligence, Xidian University, China.Her current research interests include stochas-tic optimization for machine learning, large-scalemachine learning, etc.

Licheng Jiao (F’18) received the B.S. degreefrom Shanghai Jiaotong University, Shanghai,China, in 1982, and the M.S. and Ph.D. degreesfrom Xi’an Jiaotong University, Xi’an, China, in1984 and 1990, respectively.

He was a Post-Doctoral Fellow with the Na-tional Key Laboratory for Radar Signal Process-ing, Xidian University, Xi’an, from 1990 to 1991,where he has been a Professor with the Schoolof Electronic Engineering, since 1992, and cur-rently the Director of the Key Laboratory of In-

telligent Perception and Image Understanding, Ministry of Educationof China. He has charged of about 40 important scientific researchprojects, and published over 20 monographs and a hundred papers ininternational journals and conferences. His current research interestsinclude image processing, natural computation, machine learning, andintelligent information processing.

Dr. Jiao is the Chairman of Awards and Recognition Committee, theVice Board Chairperson of the Chinese Association of Artificial Intelli-gence, a Councilor of the Chinese Institute of Electronics, a CommitteeMember of the Chinese Committee of Neural Networks, and an expertof the Academic Degrees Committee of the State Council. He is a fellowof the IEEE.

Zhouchen Lin (SM’08-F’17) received the PhDdegree in applied mathematics from Peking Uni-versity in 2000.

He is currently a professor with the KeyLaboratory of Machine Perception, School ofElectronics Engineering and Computer Science,Peking University. His research interests in-clude computer vision, image processing, ma-chine learning, pattern recognition, and numer-ical optimization. He is an area chair of CVPR2014/2016/2019/2020/2021, ICCV 2015, NIPS

2015/2018/2019, and AAAI 2019/2020, IJCAI 2020 and ICML 2020,and a senior program committee member of AAAI 2016/2017/2018 andIJCAI 2016/2018/2019. He is an associate editor of the IEEE Transac-tions on Pattern Analysis and Machine Intelligence and the InternationalJournal of Computer Vision. He is a fellow of the IAPR and the IEEE.