Model-based Sketching and Recovery with Expanders · 2019-01-23 · Introduction Model expanders...

IntroductionModel expanders

Model expander constructionModel expander algorithm

Conclusion

Model-based Sketching and Recoverywith Expanders

Bubacarr Bah, Luca Baldassarre, and Volkan Cevher

Laboratory for Information and Inference Systems (LIONS), EPFL

Information Theory and ApplicationsSan Diego

Bubacarr Bah, Luca Baldassarre, and Volkan Cevher Model-based Sketching and Recovery with Expanders



Conclusion

Linear SketchingSparsityRecovery conditionsRecovery algorithms

Three key aspects of linear sketching

Sparse or compressible xnot sufficient alone

Projection Ainformation preserving(stable embedding)

Recovery algorithm ∆tractable & correct

Applications: Data streaming, compressive sensing (CS),graph sketching, machine learning, group testing, etc.




Conclusion


Sparsity and beyond

Generic sparsity (or compressibility) not always enough

Structured sparsity⇒ model-based CS [Baraniuk, Cevher, Duarte,

Hegde, IEEE Transactions on Information Theory 2010]:

Model-based CS exploits structure insparsity model improves interpretability reduces sketch length increases speed of recovery

tree-sparse

Block-sparse




Conclusion


Overlapping Group Models

A natural generalization of sparsity

Group models application examples:

Genetic Pathways in Microarray data analysis

Wavelet models in image processing

Brain regions in neuroimaging




Conclusion


Information preserving linear embeddings A

Definition (`p-norm Restricted Isometry Property (RIP-p))

A matrix A has RIP-p of order k , if for all k -sparse x, it satisfies

(1 − δk )‖x‖pp ≤ ‖Ax‖pp ≤ (1 + δk )‖x‖pp

Subgaussian A ∈ Rm×N (w.h.p) have RIP-2with m = O (k log(N/k)), but sparse binary Adoes not have RIP-2 unless m = Ω

(k 2

)Model sparsity requires fewer m for RIP-2

O(k) for tree structureO(k + log(M)) for block structure with Mblocks [Baraniuk et al. ’10]

Scaled adjacency mat. of lossless expandershave RIP-1 with m = O (k log(N/k))

A

A




Conclusion


Recovery algorithms

Tractable recovery algorithms (∆) with provable guarantees Convex `1-minimization approaches, and Discrete algorithms (OMP, IHT, CoSaMP, ALPS)

∆ returns approximations with `p/`q-approximation error:

Definition (`p/`q-approximation error - instance optimality)

A ∆ returns x = ∆(Ax + e) with `p/`q-approximation error if

‖x − x‖p ≤ C1σk (x)q + C2‖e‖p

for a noise vector e, C1,C2 > 0, 1 ≤ q ≤ p ≤ 2, σk (x)q := mink−sparse x′

‖x − x′‖q

The pair (A,∆) ⇒ two types of error guarantees for each - one pair (A,∆) for each given x for all - one pair (A,∆) for all x




Conclusion

MotivationContribution

Goal of this work

To combine benefits of sparsity in A and benefits of model-based CS

Prior work on model-based CS use dense A

Difficult to store, creates computational bottlenecks, and notpractical in real applications




Conclusion

MotivationContribution

Our results in perspective

Price 2011 I. & R. 2013 this work

Models (structures) standard tree1 tree2 & groups3

Error guarantees `2/`2 `1/`1 `1/`1

Guarantee types for each for all for allRecovery algorithm sublinear exponential polynomial

1binary trees, 2D-ary trees for D ≥ 2, 3Loopless overlapping groups

Contribution summary

Primary: “Tractable” algorithm with provable for all `1/`1 error

Secondary: Existence of model expander (model-RIP-1) A,consistent with known sampling bounds, for more general models




Conclusion

PreliminariesModel expandersProof sketch

PART I: Existence of Model Expanders




Conclusion


Definition (RIP-1 for (k , d, ε)-lossless expanders)

If A is an adjacency matrix of a (k , d, ε)-lossless expanders, thenΦ = A/d has RIP-1 of order k , if for all k -sparse x, it satisfies

(1 − 2ε)‖x‖1 ≤ ‖Φx‖1 ≤ ‖x‖1

Probabilistic constructions of expandersachieve optimal m = O (k log(N/k))

But their deterministic constructions aresub-optimal m = O

(k 1+α

)for α > 0

(S)

S U|S| k

U : |U| = N

V : |V| = m

| (S)| (1 )d|S|

d = 3, 2 (0, 1/2)

(k, d, )-lossless expanderG = (U , V, E)

Standard random construction of G = ([N], [m],E)

For every u ∈ [N], sample a subset of [m] of size d and connect u and allthe vertices from this subset




Conclusion


Models everywhere

Tk & Gk denotes D-ary tree & loopless overlapping groupsrespectively, which are jointly denoted byMk

Definition ((Nested) Model sparse vectors)

A vector x isMk -sparse if supp(x) ⊆ K for K ∈ Mk

Definition ((k , d, ε)-model expander graph)

Let K ∈ Mk , G is a model expander if forall S ⊆ K , we have

∣∣∣Γ(S)∣∣∣ ≥ (1 − ε)d|S|

(S)

S U

U : |U| = N

V : |V| = m

| (S)| (1 )d|S|

d = 3, 2 (0, 1/2)

G = (U , V, E)(k, d, )-model expander

S K 2 Mk

Definition (Model expander matrix)

A matrix A is a model expander if it is the adjacency matrix of a(k , d, ε)-model expander graph.




Conclusion


Randomized model RIP-1 constructions

Theorem ((k , d, ε)-model expanders for D-ary (D ≥ 2) tree models)

These exist with d = O(

log(N/k)ε log log(N/k)

)and m = O

(dkε

).

Note: D is subsumed (as log(D)) in the order constant for mThis matches bounds for binary tree models by [I. & R. ’13]

Theorem ((k , d, ε)-model expanders for overlapping group models)

For M > 2 number of groups of maximum size gmax = ω(log N) such thatN ≥ kgmax, these exist with d = O

(log(N)

ε log(kgmax)

)and m = O

(dkgmaxε

).

This matches bounds for block sparsity models by [I. & R. ’13]

Note: Block sparse models are a subset of the loopless overlappinggroup sparsity models




Conclusion


Our approach-I

Proof technique similar to those of [Indyk & Razenshteyn ’13]

Key ingredient of the proof is the standard tail inequality

Lemma (For G = ([N], [m],E), a variant proven in [Buhrman et al. 2002])

There exist C > 1 and µ > 0 such that, whenever m ≥ Cdt/ε, for any T ⊆ [N] with

|T | = t we have: Prob [|j ∈ [m] : ∃i ∈ T , eij ∈ E| < (1 − ε)dt] ≤(µεmdt

)−εdt

Then a union bound over allMk -sparse sets of sparsity tThe enumeration of the cardinality of these sets involves Pfaff-Fuss-Catalan or k -Raney numbers for Tk

a careful counting of such groups in Gk




Conclusion


Our approach-II

Lemma ([Bah, Baldassarre, and Cevher 2014])

Let Tk -sparse & Gk -sparse sets with sparsity t be Tk ,t & Gk ,t respectively & the

Catalan no. be Tk , then |Tk ,t | ≤ min[Tk

(kt

),

(Nt

)], |Gk ,t | ≤ min

[(Mk

)(kgmax

t

),

(Nt

)]

It suffice to show that the following holds∣∣∣Mk ,t

∣∣∣ · (µεmdt

)−εdt≤ f(N)

where f(N) = decaying function of N, we used f(N) = 1/N

First, bound∣∣∣Mk ,t

∣∣∣ using the fact that (ns) ≤

(ens

)s

Substitute for d & m as given with arbitrary order constants

Finally, show that for different values of t ∈ [1, k ] for Tk ,t or t ∈ [1, kgmax] forGk ,t , this bound holds




Conclusion

MEIHT algorithmProjectionsAlgorithm’s key featuresProof sketchExperimental results

PART II: Model Expander Algorithm




Conclusion


Model-Expander Iterative Hard Thresholding (MEIHT)

Initialize x0 = 0, iteratexn+1 = PMk [xn +M (y − Axn)]

M(·) is the median operator whichreturns a vectorM(u) ∈ RN for aninput u ∈ Rm; defined elementwise[M(u)]i := median[uj , j ∈ Γ(i)], i ∈ [N]

Note: M operates like an adjoint

(S)

S U

U : |U| = N

V : |V| = m

| (S)| (1 )d|S|

d = 3, 2 (0, 1/2)

G = (U , V, E)(k, d, )-model expander

S K 2 Mk

PMk (u) ∈ argminz∈Mk‖u − z‖1 is the projection of u ontoMk

MEIHT is a fusion of various works [Berinde & Indyk 2008; Foucart &

Rauhut 2013; Baldassare, Bhan, and Cevher 2013; Baraniuk, Cevher, Duarte, and

Hegde 2010].




Conclusion


Tractability of structured sparse models

minz:supp(z)∈M

‖z − u‖1 = maxsupp(z)⊆S∈M

‖uS‖1 ≡Weighted Max

Cover (WMC) for group-sparse problems

All WMC instances can be formulated as PM(·)

Caveat: WMC is NP-hard⇒ PM(·) is NP-hard too

But: for someM,Mk (i.e. Tk & Gk ) in particular, ∃ linear timealgorithms

These include dynamic programs that recursively compute theoptimal solution via the model graph [Baldassarre, Bhan, Cevher 2013]




Conclusion


Runtime: polynomial in N for all tractable models

Thanks to the sparsity of A, the model projections are thedominant operation in MEIHTThus, using projection complexity from [Baldassarre et al. 2013],for a fixed n MEIHT achieves linear runtime of:

O(knN) for the Tk modelO(M2kn + nN) for the Gk model; M groups

Error guarantees: `1/`1 in the for all case

‖x − x‖1 ≤ C1σMk (x)1 + C2‖e‖1where C1,C2 > 0 and σMk (x)1 := minx′∈Mk ‖x − x′‖1

Approximate solutions are in the model,Mk ; this is veryuseful for some applications




Conclusion


Lemma (Key ingredient of proof)

Let A ∈ 0, 1m×N be a (k , d, εMk )-model expander. If S ⊂ [N] isMk -sparse, thenfor all x ∈ RN and e ∈ Rm,

‖ [M (AxS + e) − x]S ‖1 ≤

4εMk

1 − 4εMk

‖xS‖1 +2

(1 − 4εMk ) d‖eΓ(S)‖1

For Qn+1 := S ∪ supp (xn) ∪ supp(xn+1

), the triangle inequality yields

‖xn+1 − xS‖1 ≤ 2‖ [xS − xn −M (A (xS − xn) + AxS + e)]Qn+1 ‖1

Using the nestedness property ofMk and the lemma gives:

‖xn+1 − xS‖1 ≤8εM3k

1 − 4εM3k

‖xS − xn‖1 +4(

1 − 4εM3k

)d‖AxS + e‖1

Taking limn→∞ xn = x, using the RIP-1 property of A and the triangleinequality with the condition εM3k < 1/12, we have:

‖x−x‖1 ≤ C1σMk (x)1+C2‖e‖1, C2 = β = 4((

1 − 12εM3k

)d)−1

, C1 = 1+βd




Conclusion


Simulations, with different N, on group and tree modelsThe median over different realizations of the minimum no. ofsamples for which ‖x−x‖1

‖x‖1≤ 10−5 is plotted for MEIHT & EIHT

Group sparse Tree sparse

7 8 9 10 11 12 13100

150

200

250

300

350

400

450

500

log2(N )

m∗

MEIHTEIHT

7 8 9 10 11 12 130

50

100

150

200

250

log2(N )

m∗

MEIHTEIHT

M = bN/ log2(N)c, g = bN/Mc, m ∈ [2k , 10 log2(N)], k = b2 log2(N)c,

k = 5, d = b2 log(N)/log(kg)c d = b5 log(N/k)/(2 log log(N/k))c

MEIHT requires fewer measurements than EIHT as expected




Conclusion


A surprising result

Constant node degree, d = 16

7 8 9 10 11 12 1340

60

80

100

120

140

160

log2(N )

m∗

MEIHTEIHT




Conclusion

Summary

Model expanders = model-based sketching + sparse matrices;⇒ improvement in sampling and recovery

Proposed an efficient algorithm with linear runtime for modelsconsidered & achieves `1/`1 guarantees in the for all case

Random construction of model expanders for more a general classof models provably possible

Extensions

Basis adaptivity for when the x is sparse in a basis not canonical

Explicit construction of model expanders

Application of model expanders to real-life sketching & compressedsensing applications




Conclusion

References[1] B. Bah, L. Baldassarre, & V. Cevher. Model-based sketching and recovery with

expanders. ACM-SIAM Symposium on Discrete Algorithms (SODA ’14)

[2] L. Baldassarre, N. Bhan, V. Cevher, & A. Kyrillidis, Group-sparse model selection:Hardness and relaxations, arXiv, (2013)

[3] R. Baraniuk, V. Cevher, M. Duarte, & C. Hegde, Model-based compressive sensing,IEEE IT. on, 56 (2010), pp. 1982-2001

[4] H. Buhrman, P. Miltersen, J. Radhakrishnan, & S. Venkatesh. Are BitvectorsOptimal? SIAM J. Comput., 31(6):1723ÂU1744, 2002. 9,13

[5] S. Foucart & H. Rauhut, A mathematical introduction to compressive sensing,Applied Numerical Harmonic Analysis Birkhäuser, Boston, (2013)

[6] P. Indyk & I. Razenshteyn, On model-based RIP-1 matrices, arXiv:1304.3604, (2013)

[7] E. Price, Efficient sketches for the set query problem, in Proceedings of the 22ndAnnual ACM-SIAM Symposium on Discrete Algorithms, SIAM, 2011, pp. 41-56


Date post:	27-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Model-based Sketching and Recovery with Expanders · 2019-01-23 · Introduction Model expanders...

Documents