Asynchronous Parallel Computing in Signal Processing andMachine Learning
Wotao Yin (UCLA Math)
joint with Zhimin Peng (UCLA), Yangyang Xu (IMA), Ming Yan (MSU)
Optimization and Parsimonious Modeling – IMA, Jan 25, 2016
1 / 49
Do we need parallel computing?
2 / 49
Back in 1993
3 / 49
2006
4 / 49
2015
5 / 49
35 Years of CPU Trend
1995 2000 2005 2010 2015
Number
of CPUs
Performance
per core
Cores
per CPU
D. Henty. Emerging Architectures and Programming Models for Parallel Computing, 2012.
• In May 2014, Intel cancelled its Tejas project (single-core) and announced anew multi-core project.
6 / 49
Today: 4x ADM 16-core 3.5GHz CPUs (64 cores total)
7 / 49
Today: Tesla K80 GPU (2496 cores)
8 / 49
Today: Octa-Core Headsets
9 / 49
Free lunch was over
• before 2005: a single-threaded algorithm automatically gets faster
• now, new algorithms must be developed for faster speeds by
• exploiting problem structures
• taking advantages of dataset properties
• using all the cores available
10 / 49
How to use all the cores available?
11 / 49
Parallel computing
Agent
Agent
Agent
t1t2tN · · ·
Problem
12 / 49
Parallel speedup
• definition:speedup = serial time
parallel timetime is in the wall-clock sense
• Amdahl’s Law: N agents, no overhead, ρ = percentage of parallelcomputing
ideal speedup = 1(ρ/N) + (1− ρ)
100
102
104
106
108
0
5
10
15
20
Number of processors
Speedup
25%50%90%95%
13 / 49
Parallel speedup
• ε := parallel overhead (startup, synchronization, collection)• in the real world
actual speedup = 1(ρ/N) + (1− ρ) + ε
100
102
104
106
108
0
2
4
6
8
10
Number of processors
Speedup
25%50%90%95%
100
102
104
106
108
0
5
10
15
20
Number of processors
Speedup
25%50%90%95%
when ε = N when ε = log(N)
14 / 49
Sync-parallel versus async-parallel
Agent 1
Agent 2
Agent 3
idle idle
idle
idle
Synchronous(wait for the slowest)
Agent 1
Agent 2
Agent 3
Asynchronous(non-stop, no wait)
15 / 49
Async-parallel coordinate updates
16 / 49
Fixed point iteration and its parallel version
• H = H1 × · · · × Hm
• original iteration: xk+1 = Txk =: (I − ηS)xk
• all agents do in parallel:
agent 1: xk+11 ← T1(xk) = xk1 − ηS1(xk)
agent 2: xk+12 ← T2(xk) = xk2 − ηS2(xk)
...agent m: xk+1
m ← Tm(xk) = xkm − ηSm(xk)
• assumption:1. coordinate friendliness: cost of Six ∼ 1
mcost of Sx
2. synchronization after each iteration
17 / 49
Comparison
Synchronousnew iteration = all agents finish
Agent 1
Agent 2
Agent 3
t0 t1 t2 t3 t4 t5 t6 t7 t10t8 t9
Asynchronousnew iteration = any agent finishes
18 / 49
ARock1: Async-parallel coordinate update
• H = H1 × · · · × Hm• p agents, possibly p 6= m
• each agent randomly picks i ∈ {1, . . . ,m} and updates just xi:
xk+11 ← xk1
...xk+1i ← xki − ηkSixk−dk
...xk+1m ← xkm
• 0 ≤ dk ≤ τ , maximum delay
1Peng-Xu-Yan-Y.’1519 / 49
Two ways to model xk−dk
definitions: let x0, ..., xk, ... be the states of x in the memory
1. xk−dk is consistent if dk is a scalar2. xk−dk is possibly inconsistent if dk is a vector, different components are
delayed by different amounts
ARock allows both consistent and inconsistent read.
20 / 49
Memory lock illustration
Agent 1 read [0, 0, 0, 0]T = x0
consistent read
Agent 1 read [0, 0, 0, 2]T 6∈ {x0, x1, x2}
inconsistent read
21 / 49
History and recent literature
22 / 49
Brief history of async-parallel algorithms(mostly worst case analysis)
• 1969 – a linear equation solver by Chazan and Miranker;
• 1978 – extended to the fixed-point problem by Baudet under theabsolute-contraction2 type of assumption.
• For 20–30 years, mainly solve linear, nonlinear and differential equations bymany people
• 1989 – Parallel and Distributed Computation: Numerical Methods byBertsekas and Tsitsiklis. 2000 – Review by Frommer and Szyld.
• 1991 – gradient-projection itr assuming a local linear-error bound by Tseng
• 2001 – domain decomposition assuming strong convexity by Tai & Tseng
2An operator T : Rn → Rn is absolute-contractive if |T (x)− T (y)| ≤ P |x− y|, component-wise, where|x| denotes the vector with components |xi|, i = 1, ..., n, and P ∈ Rn×n
+ and ρ(P ) < 1.23 / 49
Absolute-contraction
• Absolute-contractive operator T : Rn → Rn:if |T (x)− T (y)| ≤ P |x− y|, component-wise, where |x| denotes thevector with components |xi|, i = 1, ..., n, and P ∈ Rn×n+ and ρ(P ) < 1.
• Interpretation: a series of nested rectangular boxes for xk+1 = Txk
• Applications:• diagonally dominated A for Ax = b
• diagonally dominated ∇2f for minx f(x) (just strong convexity is notenough)
• some network flow problems
24 / 49
Recent work(stochastic analysis)
• AsySCD for convex smooth and composite minimization by Liu et al’14and Liu-Wright’14. Async dual CD (regression problems) by Hsieh et al.’15
• Async randomized (splitting/distributed/incremental) methods:Wei-Ozdaglar’13, Iutzeler et al’13, Zhang-Kwok’14, Hong’14, Chang etal’15
• Async SGD: Hogwild!, Lian’15, etc.
• Async operator sample and CD: SMART Davis’15
25 / 49
Random coordinate selection
• select xi to update with probability pi, where mini pi > 0
• drawback:• agents cannot cache data• either global memory or communication is required• pseudo-random number generation takes time
• benefits:• often faster than the fixed cyclic order• automatic load balance• simplifies certain analysis
26 / 49
Convergence summary
27 / 49
Convergence guarantees
m is # coordinates, τ is the maximum delay, uniform selection pi ≡ 1m
Theorem (almost sure convergence)Assume that T is nonexpansive and has a fixed point. Use step sizesηk ∈ [ε, 1
2m−1/2τ+1 ), ∀k. Then, with probability one, xk ⇀ x∗ ∈ FixT .
In addition, rates can be derived.Consequence: step size is O(1) if τ ∼ √m. Under equal agents and updates,attaining linear speedup if using p = O(
√m) agents. p can be bigger if T is
sparse.
28 / 49
Sketch of proof
• typical inequality:
‖xk+1 − x∗‖2 ≤‖xk − x∗‖2 − c‖Txk − xk‖2
+ harmful terms(xk−1, . . . , xk−τ )
• Descent inequality under a new metric:
E(‖xk+1 − x∗‖2
M
∣∣X k) ≤ ‖xk − x∗‖2M − c ‖Txk − xk‖2
where• the history up to iteration k• xk = (xk, xk−1, . . . , xk−τ ) ∈ Hτ+1, k ≥ 0• any x∗ = (x∗, x∗, . . . , x∗) ∈ X∗ ⊆ Hτ+1
• M is a positive definite matrix.• c = c(ηk,m, τ)
29 / 49
• apply the Robbins-Siegmund theorem:
E(αk+1|Fk) + vk ≤ (1 + ξk)αk + ηk
where all are nonnegative, αk is random, and ξk, ηk are summable. Thenαk converges a.s.
• prove weakly convergent clustering points are fixed-points;
• assume H is separable and apply results [Combettes, Pesquet 2014].
30 / 49
Applications and numerical results
31 / 49
Linear equations (asynchronous Jacobi)
• require: invertible square matrix A with nonzero diagonal entries
• let D be the diagonal part of A; then
Ax = b ⇐⇒ (I −D−1A)x+D−1b︸ ︷︷ ︸Tx
= x.
• T is nonexpansive if ‖I −D−1A‖2 ≤ 1, i.e., A is diagonal dominating
• xk+1 = Txk recovers the Jacobi algorithm
32 / 49
Algorithm 1: ARock for linear equations
Input : shared variables x ∈ Rn, K > 0;set global iteration counter k = 0;while k < K, every agent asynchronously and continuously do
sample i ∈ {1, . . . ,m} uniformly at random;add − ηk
aii(∑
jaij x
kj − bi) to shared variable xi;
update the global counter k ← k + 1;
33 / 49
Sample code
loadData (A, data_file_name );loadData (b, label_file_name );# pragma omp parallel num_threads (p) shared (A,b,x,para){ // A, b, x, and para are passed by reference
call Jacobi (A,b,x,para) or ARock(A,b,x,para );}
• p: the number of threads• A,b,x: shared variable• para: other parameters
34 / 49
Jacobi worker function
for(int itr =0; itr < max_itr ; itr ++){
// compute the update for the assigned x[i]// ...
# pragma omp barrier{
// write x[i] in global memory}
# pragma omp barrier}
Jacobi needs the barrier directive for synchronization
35 / 49
ARock worker function
for(int itr =0; itr < max_itr ; itr ++){
// pick i at random// compute the update for x[i]// ...// write x(i) in global memory
}
ARock has no synchronization barrier directive
36 / 49
Minimizing smooth functions
• require: convex and Lipschitz differentiable function f
• if ∇f is L-Lipschitz, then
minimizex
f(x) ⇐⇒ x =(I − 2
L∇f)︸ ︷︷ ︸
T
x.
where T is nonexpansive
• ARock will be efficient when ∇xif(x) is easy to compute
37 / 49
Minimizing composite functions
• require: convex smooth g(·) and convex (possibly nonsmooth) f(·)
• proximal map: proxγf (y) = arg min f(x) + 12γ ‖x− y‖
2.
minimizex
f(x) + g(x) ⇐⇒ x = proxγf ◦ (I − γ∇g)︸ ︷︷ ︸T
x.
• ARock will be fast if• ∇xig(x) is easy to compute• f(·) is separable (e.g., `1 and `1,2)
38 / 49
Example: sparse logistic regression
• n features, N labeled samples
• each sample ai ∈ Rn has its label bi ∈ {1,−1}
• `1 regularized logistic regression:
minimizex∈Rn
λ‖x‖1 + 1N
N∑i=1
log(1 + exp(−bi · aTi x)
), (1)
• compare sync-parallel and ARock (async-parallel) on two datasets:
Name N (#samples) n (# features) # nonzeros in {a1, . . . , aN}rcv1 20,242 47,236 1,498,952
news20 19,996 1,355,191 9,097,916
39 / 49
Speedup tests
• implemented in C++ and OpenMP.• 32 cores shared memory machine.
#coresrcv1 news20
Time (s) Speedup Time (s) Speedupasync sync async sync async sync async sync
1 122.0 122.0 1.0 1.0 591.1 591.3 1.0 1.02 63.4 104.1 1.9 1.2 304.2 590.1 1.9 1.04 32.7 83.7 3.7 1.5 150.4 557.0 3.9 1.18 16.8 63.4 7.3 1.9 78.3 525.1 7.5 1.1
16 9.1 45.4 13.5 2.7 41.6 493.2 14.2 1.232 4.9 30.3 24.6 4.0 22.6 455.2 26.1 1.3
40 / 49
More applications
41 / 49
Minimizing composite functions
• require: both f and g are convex (possibly nonsmooth) functions
minimize f(x) + g(x) ⇐⇒ z = reflγf ◦ reflγg︸ ︷︷ ︸TPRS
(z).
recover x = proxγg(z)
• TPRS is known as the Peaceman-Rachford splitting operator3
• also, the Douglas-Rachford splitting operator: 12I + 1
2TPRS
• ARock runs fast when• reflγf is separable• (reflγg)i is easy-to-compute/maintain
3reflective proximal map: reflγf := 2proxγf − I. The maps reflγf , reflγg and thusreflγf ◦ reflγg are nonexpansive
42 / 49
Parallel/distributed ADMM
• require: m convex functions fi
• consensus problem:
minimizex
∑m
i=1 fi(x) + g(x)
⇐⇒ minimizexi,y
∑m
i=1 fi(xi) + g(y)
subject to
I
I
. . .I
x1
x2...xm
−I
I...I
y = 0
• Douglas-Rachford-ARock to the dual problem ⇒ async-parallel ADMM:
• m subproblems are solved in the async-parallel fashion• y and zi (dual variables) are updated in global memory (no lock)
43 / 49
Decentralized computing
• n agents in a connected network G = (V,E) with bi-directional links E
• each agent i has a private function fi
• problem: find a consensus solution x∗ to
minimizex∈Rp
f(x) :=n∑i=1
fi(Aixi) subject to xi = xj , ∀i, j.
• challenges: no center, only between-neighbor communication
• benefits: fault tolerance, no long-dist communication, privacy
44 / 49
Async-parallel decentralized ADMM
• a graph of connected agents: G = (V,E).
• decentralized consensus optimization problem:
minimizexi∈Rd,i∈V
f(x) :=∑
i∈V fi(xi)
subject to xi = xj , ∀(i, j) ∈ E
• ADMM reformulation: constraints xi = yij , xj = yij , ∀(i, j) ∈ E
• apply• ARock version 1: nodes asynchronously activate• ARock version 2: edges asynchronously activate
no global clock, no central controller, each agent keeps fi private andtalks to its neighbors
45 / 49
notation:
• N(i) all edges of agent i, N(i) = L(i) ∪R(i)
• L(i) neighbors j of agent i, j < i
• R(i) neighbors j of agent i, j > i
Algorithm 2: ARock for the decentralized consensus problem
Input : each agent i sets x0i ∈ Rd, dual variables z0
e,i for e ∈ E(i), K > 0.while k < K, any activated agent i do
receive zkli,l from neighbors l ∈ L(i) and zkir,r from neighbors r ∈ R(i);update local xki , zk+1
li,i and zk+1ir,i according to (2a)–(2c), respectively;
send zk+1li,i to neighbors l ∈ L(i) and zk+1
ir,i to neighbors r ∈ R(i).
xki ∈ arg minxi
fi(xi) +(∑
l∈L(i) zkli,l +
∑r∈R(i) z
kir,r
)xi + γ
2 |E(i)|‖xi‖2,
(2a)
zk+1ir,i = zkir,i − ηk((zkir,i + zir,r)/2 + γxki ), ∀r ∈ R(i), (2b)zk+1li,i = zkli,i − ηk((zkli,i + zli,l)/2 + γxki ), ∀l ∈ L(i). (2c)
46 / 49
Summary
47 / 49
Summary of async-parallel coordinate descent
benefits:
• eliminate idle time
• reduce communication / memory-access congestion
• random job selection: load balance
mathematics: analysis of disorderly (partial) updates
• asynchronous delay
• inconsistent read and write
48 / 49
Thank you!
Acknowledgements: NSF DMSReference: Zhimin Peng, Yangyang Xu, Ming Yan, Wotao Yin. UCLA CAM15-37.Website: http://www.math.ucla.edu/˜wotaoyin/arock
49 / 49
