Applied Harmonic Analysis meets Compressed Sensing · PDF fileGoalforToday Challenges for...

transcript

Applied Harmonic Analysis meets Compressed Sensing

Gitta Kutyniok

(Technische Universitat Berlin)

ICERM Program “Network Science and Graph Algorithms”February 4, 2014

Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 1 / 50

Outline

1 Imaging SciencesInspiring Empirical ResultsGoal for Today

2 Review of Compressed SensingSeparation via Compressed SensingInpainting via Compressed Sensing

3 Algorithmic AspectsShearlet SystemsNumerical Results

4 Theoretical AnalysisGeometrically Clustered SparsityAnalysis of Separation (joint with D. L. Donoho)Analysis of Inpainting (joint with E. King and X. Zhuang)

Ill-posed Inverse Problems in Data Analysis

Two Challenges:

Modern Data in general is often composed of two or moremorphologically distinct constituents. Task: Separation of components given the composed data.

Applications often cause loss of information or necessary informationcan not be collected. Task: Recovery of missing data given the observed data.

Novel Approach:

Applied Harmonic Analysis

Compressed Sensing

First empirical results by J. L. Starck, M. Elad, and D. L. Donoho.

Separating Artifacts in Images, I

(Source: Starck, Elad, and Donoho; 2006)

Separating Artifacts in Images, II

Neurobiological Imaging:

Detection of characteristics of Alzheimer.

Separation of spines and dendrites.

(Source: Brandt, K, Lim, and Sundermann; 2010)

Inpainting, I

(Source: Hennenfent and Herrmann; 2008)

Inpainting, II

(Source: King, K, Lim, Zhuang; 2012)

Applied Harmonic Analysis Approach

Methodology:Exploit a carefully designed representation system (ψλ)λ ⊆ H:

H ⊇ C ∋ f −→ (〈f , ψλ〉)λ −→∑

〈f , ψλ〉ψλ = f

Two Main Goals:

(1) Decomposition

(2) Efficient representations

Main Desiderata:

Multiscale representation system.

Partition of Fourier domain.

Fast decomposition and reconstruction algorithm.

Optimally sparse approximation of the considered class.

Goal for Today

Challenges for Today:

Methodology to derive the empirical results!◮ Applied Harmonic Analysis.◮ Compressed Sensing.

Improvement of the methodology!◮ Shearlets as sparsifying system.

Analysis of the methodology!◮ Continuum model.◮ Geometrically clustered sparsity.

Goal for Today

Challenges for Today:

Methodology to derive the empirical results!◮ Applied Harmonic Analysis.◮ Compressed Sensing.

Improvement of the methodology!◮ Shearlets as sparsifying system.

Analysis of the methodology!◮ Continuum model.◮ Geometrically clustered sparsity.

Another Path to Imaging Science are Variational Approaches:

Contributors: Bertozzi, Burger, Chan, Esedoglu, Kang, Osher, Sapiro,Setzer, Shen, Steidl, Vese, Weikert, ...

How does Compressed Sensing come into play?

Underdetermined Situations

Separation:

Observe a signal x composed of two subsignals x1 and x2:

x = x1 + x2.

Extract the two subsignals x1 and x2 from x , if only x is known.

Separation:

x = x1 + x2.

The two components are geometrically different.

Separation:

x = x1 + x2.

Inpainting:

Given a signalx = xK + xM ∈ HK ⊕HM .

Recover x , if only xK is known.

Separation:

x = x1 + x2.

Inpainting:

Given a signalx = xK + xM ∈ HK ⊕HM .

Recover x , if only xK is known.

The original signal is sparse within a frame.

Birth of Separation via Compressed Sensing

Composition of Sinusoids and Spikes sampled at n points:

x = x01 + x02 = Φ1c01 +Φ2c

02 = [ Φ1 | Φ2 ]

c01c02

x , c01 , and c02 are n × 1.

Φ1 is the n × n-Fourier matrix ((Φ1)t,k = e2πitk/n).

Φ2 is the n × n-Identity matrix.

0 50 100 150 200 250-1

Sparsity and ℓ1

Assumption: Letting A be an n × N-matrix, n << N, the seeked solutionc0 of x = Ac0 satisfies:

‖c0‖0 = #{i : c0i 6= 0} is ‘small’, i.e., c0 is sparse.

Sparsity and ℓ1

Ideal: Solve...(P0) min

c‖c‖0 subject to x = Ac

Sparsity and ℓ1

Basis Pursuit (Chen, Donoho, Saunders; 1998)

(P1) minc

‖c‖1 subject to x = Ac

Sparsity and ℓ1

Basis Pursuit (Chen, Donoho, Saunders; 1998)

(P1) minc

‖c‖1 subject to x = Ac

Meta-Result: If the solution c0 is sufficiently sparse, and A is sufficientlyincoherent, then c0 can be recovered from x via (P1).

First Results of Compressed Sensing

x = x01 + x02 = Φ1c01 +Φ2c

02 = [ Φ1 | Φ2 ]

c01c02

x = x01 + x02 = Φ1c01 +Φ2c

02 = [ Φ1 | Φ2 ]

c01c02

Theorem (Donoho, Huo; 2001)If #(Sinusoids) + #(Spikes) = ‖(c01 )‖0 + ‖(c02 )‖0 < (1 +

√n)/2, then

(c01 , c02 ) = argmin(‖c1‖1 + ‖c2‖1) subject to x = Φ1c1 +Φ2c2.

x = x01 + x02 = Φ1c01 +Φ2c

02 = [ Φ1 | Φ2 ]

c01c02

Theorem (Donoho, Huo; 2001)If #(Sinusoids) + #(Spikes) = ‖(c01 )‖0 + ‖(c02 )‖0 < (1 +

√n)/2, then

(c01 , c02 ) = argmin(‖c1‖1 + ‖c2‖1) subject to x = Φ1c1 +Φ2c2.

Theorem (Bruckstein, Elad; 2002)(Donoho, Elad; 2003)Let A = (ai)

Ni=1 be an n× N-matrix with normalized columns, n << N,

and let c0 satisfy

‖c0‖0 <1

1 + µ(A)−1)

with coherence µ(A) = maxi 6=j |〈ai , aj〉|. Thenc0 = argmin‖c‖1 subject to x = Ac .

Birth of Inpainting via Compressed Sensing

Main Idea:Let

x0 ∈ H be a signal.

Φ be an ONB (x0 = Φc0).

H = HM ⊕HK with orthogonal projections PM and PK .

ℓ1 Minimization Problem (Elad, Starck, Querre, Donoho; 2005):

c = argmin‖c‖1 subject to PKx0 = PKΦc x = Φc

Theorem (Donoho, Elad; 2003)

=⇒ If ‖c0‖0 < 12(1 + µ(PKΦ)

−1), then x0 = x .

Two Paths

Avalanche of Recent Work

Problem: Solve x = Ac0 with A an n × N-matrix (n < N).

Deterministic World:

Mutual coherence of A = (ak)k .

Bound ‖c0‖0 dependent on µ(A).

Efficiently solve the problem x = Ac0.

Contributors: Bruckstein, Cohen, Dahmen, DeVore, Donoho, Elad,Eldar, Fuchs, Gribonval, Huo, K, Rauhut, Temlyakov, Tropp, ...

Random World:

Restricted isometry constants of a random A = (ak)k .

Bound ‖c0‖0 by n/(2 log(N/n))(1 + o(1)).

Efficiently solve the problem x = Ac0 with high probability.

Contributors: Candes, Cohen, Dahmen, DeVore, Donoho, K,Krahmer, Rauhut, Romberg, Tanner, Tao, Tropp, Ward, ...

Novel Direction for Sparsity

Geometric Sparsity (Donoho, K; 2009):

y = Ax0 with A an n × N-matrix (n < N).

Nonzeros of x0 often◮ arise not in arbitrary patterns,◮ but are rather highly structured.

Interactions between columns of A inill-posed problems

◮ is not arbitrary,◮ but rather geometrically driven.

Novel Direction for Sparsity

Geometric Sparsity (Donoho, K; 2009):

y = Ax0 with A an n × N-matrix (n < N).

Nonzeros of x0 often◮ arise not in arbitrary patterns,◮ but are rather highly structured.

Interactions between columns of A inill-posed problems

◮ is not arbitrary,◮ but rather geometrically driven.

Other results on “structured sparsity”:

Joint sparsity (Fornasier, Rauhut; 2008)

Block sparsity (Eldar, Kuppinger, Bolcskei; 2010)

Fusion frame sparsity (Boufonous, K, Rauhut; 2011)

Which Sparsifying System to choose...?

First: Separation

Main Idea:

Two morphologically distinct components:◮ Points◮ Curves

Choose suitable representation systems which provide optimallysparse representations of

◮ pointlike structures −→ Wavelets◮ curvelike structures −→ ???

Minimize the ℓ1 norm of the coefficients.

This forces◮ the pointlike objects into the wavelets part of the expansion◮ the curvelike objects into the ??? part.

Review of 2-D Wavelets

Definition (1D): Let φ ∈ L2(R) be a scaling function and ψ ∈ L2(R) be awavelet. Then the associated wavelet system is defined by

{φ(x −m) : m ∈ Z} ∪ {2j/2 ψ(2jx −m) : j ≥ 0,m ∈ Z}.

Review of 2-D Wavelets

Definition (1D): Let φ ∈ L2(R) be a scaling function and ψ ∈ L2(R) be awavelet. Then the associated wavelet system is defined by

{φ(x −m) : m ∈ Z} ∪ {2j/2 ψ(2jx −m) : j ≥ 0,m ∈ Z}.

Definition (2D): A wavelet system is defined by

{φ(1)(x −m) : m ∈ Z2} ∪ {2jψ(i)(2jx −m) : j ≥ 0,m ∈ Z

2, i = 1, 2, 3},

where ψ(1)(x) = φ(x1)ψ(x2),

φ(1)(x) = φ(x1)φ(x2) and ψ(2)(x) = ψ(x1)φ(x2),

ψ(3)(x) = ψ(x1)ψ(x2).

Theorem: Discrete wavelets provide optimally sparse approximations forfunctions f ∈ L2(R2), which are C 2 apart from point singularities:

‖f − fN‖22 ≍ N−1, N → ∞.

First: Separation

Main Idea:

Two morphologically distinct components:◮ Points◮ Curves

Choose suitable representation systems which provide optimallysparse representations of

◮ pointlike structures −→ Wavelets◮ curvelike structures −→ ???

This forces◮ the pointlike objects into the wavelets part of the expansion◮ the curvelike objects into the ??? part.

Second: Inpainting

Main Idea:

Choose suitable representation system which provide optimallysparse representations of the original image.

This fills in the missing part automatically.

Question: What is a good model for an image?

Second: Inpainting

Main Idea:

Choose suitable representation system which provide optimallysparse representations of the original image.

This fills in the missing part automatically.

Question: What is a good model for an image?

Field et al., 1993

Require: System to sparsify curvelike structures −→ ???

Fitting Model for Anisotropic Structures

Definition (Donoho; 2001):The set of cartoon-like images E2(R2) is defined by

E2(R2) = {f ∈ L2(R2) : f = f0 + f1 · χB},

where B ⊂ [0, 1]2 with ∂B a closed C 2-curve, f0, f1 ∈ C 20 ([0, 1]

Fitting Model for Anisotropic Structures

Definition (Donoho; 2001):The set of cartoon-like images E2(R2) is defined by

E2(R2) = {f ∈ L2(R2) : f = f0 + f1 · χB},

where B ⊂ [0, 1]2 with ∂B a closed C 2-curve, f0, f1 ∈ C 20 ([0, 1]

Theorem (Donoho; 2001):Let (ψλ)λ ⊆ L2(R2). Allowing only polynomial depth search, the optimalasymptotic approximation error of f ∈ E2(R2) is

‖f − fN‖22 ≍ N−2, N → ∞, where fN =∑

λ∈IN

cλψλ.

Beyond Wavelets...

Observation:

Wavelets only achieve ‖f − fN‖22 ≍ N−1, N → ∞.

Wavelets can not approximate curvilinear singularities optimallysparse.

Reason: Isotropic structure of wavelets:

2j 00 2j

x −m)

Intuitive explanation:

Main Goal in Applied Harmonic Analysis

Design a representation system which...

...is generated by one ‘mother function’,

...provides optimally sparse approximation of cartoons,

...allows for compactly supported analyzing elements,

...is associated with fast decomposition algorithms,

...treats the continuum and digital ‘world’ uniformly.

Main Goal in Applied Harmonic Analysis

Design a representation system which...

...is generated by one ‘mother function’,

...provides optimally sparse approximation of cartoons,

...allows for compactly supported analyzing elements,

...is associated with fast decomposition algorithms,

...treats the continuum and digital ‘world’ uniformly.

Non-exhaustive list of approaches:

Ridgelets (Candes and Donoho; 1999)

Curvelets (Candes and Donoho; 2002)

Contourlets (Do and Vetterli; 2002)

Bandlets (LePennec and Mallat; 2003)

Shearlets (K and Labate; 2006)

Scaling and Orientation

Parabolic scaling:

0 2j/2

, j ∈ Z.

Historical remark:

1970’s: Fefferman und Seeger/Sogge/Stein.

Scaling and Orientation

Parabolic scaling:

0 2j/2

, j ∈ Z.

Historical remark:

1970’s: Fefferman und Seeger/Sogge/Stein.

Orientation via shearing:

1 k0 1

, k ∈ Z.

Advantage:

Shearing leaves the digital grid Z2 invariant.

Uniform theory for the continuum and digital situation.

(Cone-adapted) Discrete Shearlet Systems

Definition (K, Labate; 2006):The (cone-adapted) discrete shearlet system SH(φ,ψ, ψ) generated byφ ∈ L2(R2) and ψ, ψ ∈ L2(R2) is the set

{φ(· −m) : m ∈ Z2}

∪{23j/4ψ(SkAj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}

∪{23j/4ψ(Sk Aj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}.

(Cone-adapted) Discrete Shearlet Systems

Definition (K, Labate; 2006):The (cone-adapted) discrete shearlet system SH(φ,ψ, ψ) generated byφ ∈ L2(R2) and ψ, ψ ∈ L2(R2) is the set

{φ(· −m) : m ∈ Z2}

∪{23j/4ψ(SkAj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}

∪{23j/4ψ(Sk Aj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}.

General Framework:

Parabolic Molecules (Grohs, K; 2013)

α-Molecules (Grohs, Keiper, K, Schafer; 2014)

Compactly Supported Shearlets

Theorem (Kittipoom, K, Lim; 2012):

Let φ,ψ, ψ ∈ L2(R2) be compactly supported, and let ψ, ˆψ satisfy certaindecay condition. Then SH(φ,ψ, ψ) forms a shearlet frame withcontrollable frame bounds.

Remark: Exemplary class with B/A ≈ 4.

Compactly Supported Shearlets

Theorem (Kittipoom, K, Lim; 2012):

Let φ,ψ, ψ ∈ L2(R2) be compactly supported, and let ψ, ˆψ satisfy certaindecay condition. Then SH(φ,ψ, ψ) forms a shearlet frame withcontrollable frame bounds.

Remark: Exemplary class with B/A ≈ 4.

Theorem (K, Lim; 2011):

Let φ,ψ, ψ ∈ L2(R2) be compactly supported, and let ψ, ˆψ satisfy certaindecay condition. Then SH(φ,ψ, ψ) provides an optimally sparseapproximation of f ∈ E2(R2), i.e.,

‖f − fN‖22 ≤ C · N−2 · (logN)3, N → ∞.

Recent Approaches to Fast Shearlet Transforms

www.ShearLab.org:

Separable Shearlet Transform (Lim; 2009)

Digital Shearlet Transform (K, Shahram, Zhuang; 2011)

2D&3D (parallelized) Shearlet Transform (K, Lim, Reisenhofer; 2013)

Additional Code:

Filter-based implementation (Easley, Labate, Lim; 2009)

Theoretical Approaches:

Adaptive Directional Subdivision Schemes (K, Sauer; 2009)

Shearlet Unitary Extension Principle (Han, K, Shen; 2011)

Gabor Shearlets (Bodmann, K, Zhuang; 2013)

Image Separation: Points + Curves

MCALab 120 (52.74 sec) ShearLab (33.75 sec)

(Source: K, Lim; 2011)

Inpainting

Original Noisy Version (80% missing)

Curvelets (29.95dB, 182.15sec) Shearlets (31.04dB, 85.18sec)

(Source: Lim; 2012)

What is the Fundamental Mathematical Concept

behind the Empirical Success?

General Analysis of Separation

Signal Model:x = x01 + x02 ∈ H

Remarks:

Given two tight frames Φ1, Φ2 (Φi(ΦTi x) = x for all x).

Too many decompositions x = Φ1c1 +Φ2c2.

Use x = Φ1(ΦT1 x1) + Φ2(Φ

T2 x2), where x = x1 + x2.

Norm is placed on analysis rather than synthesis side.

Decomposition Technique:

(x⋆1 , x⋆2 ) = argminx1,x2‖Φ

T1 x1‖1 + ‖ΦT

2 x2‖1 subject to x = x1 + x2

Relative Sparsity and Cluster Coherence

Let Φ1 = (ϕ1,i )i∈I1 and Φ2 = (ϕ2,i )i∈I2.

Definition:

For each i = 1, 2, x0i is relatively sparse in Φi w.r.t. Λi , if

‖1Λc1ΦT1 x

01‖1 + ‖1Λc

2ΦT2 x

02‖1 ≤ δ.

We call Λ1 and Λ2 sets of significant coefficients.

We define cluster coherence for Λ1 by

µc(Λ1) = maxj∈I2

i∈Λ1

|〈ϕ1,i , ϕ2,j〉|.

Central Estimate

Theorem (Donoho, K; 2011):Suppose x01 and x02 are relatively sparse with Λ1 and Λ2 sets of significantcoefficients. Then

‖x⋆1 − x01‖2 + ‖x⋆2 − x02‖2 ≤2δ

1− 2µc,

whereµc = max(µc(Λ1), µc (Λ2)).

δ: Relative sparsity measure.

µc : Cluster coherence.

Continuum Model

Neurobiological Geometric Mixture in 2D:

Point Singularity:

P(x) =P∑

|x − xi |−3/2

Curvilinear Singularity:

δτ(t)dt, τ a closed C 2-curve.

Observed Signal:f = P + C

Scale-Dependent Decomposition

Observed Object:f = P + C.

Subband Decomposition:Wavelets and shearlets use the same scaling subbands!

fj = Pj + Cj , Pj = P ⋆ Fj and Cj = C ⋆ Fj .

ℓ1-Minimization:

(Wj ,Sj) = argmin‖(〈Wj , ψλ〉)λ‖1 + ‖(〈Sj , ση〉)η‖1 s.t. fj = Wj + Sj

Application of Previous Result

x : Filtered signal fj (= Pj + Cj).Φ1: Wavelets filtered with Fj .

Φ2: Shearlets filtered with Fj .

Λ1: Significant wavelet coefficients of 〈ψλ,Pj 〉.Λ2: Significant shearlet coefficients of 〈ση, Cj 〉.δ: Degree of approximation by significant coefficients.

µc(Λ1), µc(Λ2): Cluster coherence of wavelets-shearlets.

Estimate of error: 2δ1−2µc

Application of Previous Result

x : Filtered signal fj (= Pj + Cj).Φ1: Wavelets filtered with Fj .

Φ2: Shearlets filtered with Fj .

Λ1: Significant wavelet coefficients of 〈ψλ,Pj 〉?Λ2: Significant shearlet coefficients of 〈ση, Cj 〉?δ: Degree of approximation by significant coefficients.

µc(Λ1), µc(Λ2): Cluster coherence of wavelets-shearlets.

Estimate of error: 2δ1−2µc

= o(‖Pj‖2 + ‖Cj‖2) as j → ∞.

Microlocal Analysis Heuristics

Asymptotic Separation

Theorem (Donoho, K; 2011)

‖Wj − Pj‖2 + ‖Sj − Cj‖2‖Pj‖2 + ‖Cj‖2

→ 0, j → ∞.

At all sufficiently fine scales, nearly-perfect separation is achieved!

Asymptotic Separation

Theorem (Donoho, K; 2011)

‖Wj − Pj‖2 + ‖Sj − Cj‖2‖Pj‖2 + ‖Cj‖2

→ 0, j → ∞.

At all sufficiently fine scales, nearly-perfect separation is achieved!

Theorem (K; 2013)Using thresholding as separation strategy, we can even prove that

WF (∑

Fj ⋆Wj) = WF (P) and WF (∑

Fj ⋆ Sj) = WF (C).

Exact separation of the wavefront sets!

General Analysis of Inpainting

Signal Model:x0 = PKx

0 + PMx0 ∈ HK ⊕HM ,

where PK (PM) is the orthogonal projection onto HK (HM).

Remarks:

Given a tight frame Φ.

Redundancy Too many decompositions x = Φc .

Use x = Φ(ΦT x).

Norm is placed on analysis rather than synthesis side.

Inpainting Technique:

x⋆ = argminx‖ΦT x‖1 subject to PKx = PKx0.

Central Estimate

Theorem (King, K, Zhuang; 2013)Suppose x0 is δ-relatively sparse in Φ with Λ a set of significantcoefficients, and HM is the masked subspace. Then

‖x⋆ − x0‖2 ≤2δ

1− 2µc.

δ = δ(x0,Φ,Λ): Relative sparsity measure.

µc = µc(Φ,Λ,HM): Cluster coherence.

Continuum Model

Curvilinear Singularity:

∫ ρ

−ρw(t)δτ(t)dt,

τ : [−1, 1] → R2 a C 2-curve, ρ < 1, and w : [−ρ, ρ] → R

+0 ‘bump’.

Mask:Mh = {(x1, x2) ∈ R

2 : |x1| ≤ h}, h > 0.

Observed Signal:f = 1R2\Mh

· C.Subband Decomposition:

C 7→ Cj = C ⋆ Fj .

Asymptotic analysis:

Consider h = hj .

Set fj = 1R2\Mhj· Cj .

Algorithm:

Sj = argmin‖(〈Sj , ση〉)η‖1 s.t. fj = 1R2\Mhj· Sj

Asymptotic analysis:

Consider h = hj .

Set fj = 1R2\Mhj· Cj .

Algorithm:

Sj = argmin‖(〈Sj , ση〉)η‖1 s.t. fj = 1R2\Mhj· Sj

Microlocal Analysis Heuristics:

Asymptotic Inpainting

Theorem (King, K, Zhuang; 2013)For hj = o(2−j/2) as j → ∞, shearlet inpainting satisfies

‖Sj − Cj‖2‖Cj‖2

→ 0, j → ∞.

Theorem (King, K, Zhuang; 2013)For hj = o(2−j ) as j → ∞, wavelet inpainting satisfies

‖Wj − Cj‖2‖Cj‖2

→ 0, j → ∞.

Theorem (King, K, Zhuang; 2013)For hj = o(2−j ) as j → ∞, wavelet inpainting satisfies

‖Wj − Cj‖2‖Cj‖2

→ 0, j → ∞.

Theorem (King, K, Zhuang; 2013)“In case of thresholding, if hj = ω(2−j ), wavelets fail.”

Let’s conclude...

What to take Home...?

Compressed Sensing solves underdetermined linear systems ofequations exactly if the solution is sparse and the matrix is incoherent.

(Compactly supported) Shearlets provide optimally sparseapproximations of anisotropic features with a unified treatment of thecontinuum and digital world.

The Geometric Separation Problem and the Inpainting Problem canbe solved by these methodologies.

Asymptotically optimal performance can be theoretically proved.

Key features of our analysis:◮ Continuum model.◮ Geometrically clustered sparsity and cluster coherence.◮ Microlocal analysis viewpoint.

Technische Universität BerlinApplied Functional Analysis Group

THANK YOU!

References available at:

www.math.tu-berlin.de/∼kutyniokCode available at:

www.ShearLab.org

Related Books:Y. Eldar and G. KutyniokCompressed Sensing: Theory and ApplicationsCambridge University Press, 2012.

G. Kutyniok and D. LabateShearlets: Multiscale Analysis for Multivariate DataBirkhauser-Springer, 2012.

Applied Harmonic Analysis meets Compressed Sensing · PDF fileGoalforToday Challenges for...

Documents