+ All Categories
Home > Documents > The Iterative and Regularized Least Squares Algorithm for...

The Iterative and Regularized Least Squares Algorithm for...

Date post: 01-May-2018
Category:
Upload: doanh
View: 219 times
Download: 0 times
Share this document with a friend
46
The Iterative and Regularized Least Squares Algorithm for Phase Retrieval Radu Balan Naveed Haghani Department of Mathematics, AMSC, CSCAMM and NWC University of Maryland, College Park, MD April 17, 2016 Special Session ”Frames, Wavelets and Gabor Systems” AMS Regional Meeting, Fargo ND
Transcript
Page 1: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Iterative and Regularized Least Squares Algorithmfor Phase Retrieval

Radu Balan

Naveed Haghani

Department of Mathematics, AMSC, CSCAMM and NWCUniversity of Maryland, College Park, MD

April 17, 2016Special Session ”Frames, Wavelets and Gabor Systems”

AMS Regional Meeting, Fargo ND

Page 2: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

”This material is based upon work supported by the National ScienceFoundation under Grant No. DMS-1413249 and by ARO under ContractNo. W911NF-16-1-0008. Any opinions, findings, and conclusions orrecommendations expressed in this material are those of the author(s) anddo not necessarily reflect the views of the National Science Foundation.”

Page 3: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

Table of Contents:

1 The Phase Retrieval Problem

2 Existing Algorithms

3 The IRLS Algorithm

4 Numerical Results

Page 4: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Phase RetrievalThe phase retrieval problem

Hilbert space H = Cn, H = H/T 1, frame F = {f1, · · · , fm} ⊂ Cn andmeasurements

yk = |〈x , fk〉|2 + νk , 1 ≤ k ≤ m.

The frame is said phase retrievable (or that it gives phase retrieval) ifx 7→ (|〈x , fk〉|)1≤k≤m is injective.

The general phase retrieval problem a.k.a. phaseless reconstruction:Decide when a given frame is phase retrievable, and, if so, find analgorithm to recover x from y = (yk)k up to a global phase factor.

Our problem today: A reconstruction algorithm.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 5: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

General Purpose AlgorithmsUnstructured Frames. Unstructured Data

1 Iterative Algorithms:Gerchberg-Saxton [Gerchberg&all]Wirtinger flow - gradient descent [CLS14]IRLS [B13]

2 Rank 1 Tensor Recovery:PhaseLift; PhaseCut [Candes&all];[Waldspurger&all]Higher-Order Tensor Recovery [B.]

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 6: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Specialized AlgorithmsStructured Frames and/or Structured Data

1 Structured Frames:Fourier Frames: 4n-4 [BH13]; Masking DFT [CLS13];STFT/Spectograms [B.][Eldar&all][Hayes&all]; Alternating Projections[GriffinLim][Fannjiang]Polarization: 3-term [ABFM12], masking [BCM]Shift-Invariant Spaces: Bandlimited [Thakur]; Filterbanks/CirculantMatrices [IVW2]; Other spaces [Chen&all]X-Ray Crystallography – over 100 years old, lots of Nobel prizes ...

2 Special Signals:Sparse general case: GESPAR[SBE14];Specialized: sparse [IVW1]; speech [ARF03]

... and others – ”phase retrieval” in title: 2680 papers

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 7: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmFirst Motivation: Graduation Method. Homotopic Continuation

The IRLS algorithm belongs to the class of Graduation Methods , orHomotopic Continuations.Idea:

Our target is to optimize a complicated (possibly non-convex)optimization criterion J(x), argminx∈DJ(x).However we know how to optimize a closely related criterion J0(x),argminx∈D0J0(x).Then we introduce a monotonic sequence 0 ≤ tn ≤ 1 with t0 = 1 andtn → 0 and solve iteratively

xn+1 = argminx∈Dn F (tn, J(x), J0(x))

using xn as starting point. Here F is a continue function so thatF (1, J(x), J0(x)) = J0(x) and F (0, J(x), J0(x)) = J(x).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 8: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmFirst Motivation: Graduation Method. Homotopic Continuation

The IRLS algorithm belongs to the class of Graduation Methods , orHomotopic Continuations.Idea:Our target is to optimize a complicated (possibly non-convex)optimization criterion J(x), argminx∈DJ(x).

However we know how to optimize a closely related criterion J0(x),argminx∈D0J0(x).Then we introduce a monotonic sequence 0 ≤ tn ≤ 1 with t0 = 1 andtn → 0 and solve iteratively

xn+1 = argminx∈Dn F (tn, J(x), J0(x))

using xn as starting point. Here F is a continue function so thatF (1, J(x), J0(x)) = J0(x) and F (0, J(x), J0(x)) = J(x).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 9: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmFirst Motivation: Graduation Method. Homotopic Continuation

The IRLS algorithm belongs to the class of Graduation Methods , orHomotopic Continuations.Idea:Our target is to optimize a complicated (possibly non-convex)optimization criterion J(x), argminx∈DJ(x).However we know how to optimize a closely related criterion J0(x),argminx∈D0J0(x).

Then we introduce a monotonic sequence 0 ≤ tn ≤ 1 with t0 = 1 andtn → 0 and solve iteratively

xn+1 = argminx∈Dn F (tn, J(x), J0(x))

using xn as starting point. Here F is a continue function so thatF (1, J(x), J0(x)) = J0(x) and F (0, J(x), J0(x)) = J(x).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 10: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmFirst Motivation: Graduation Method. Homotopic Continuation

The IRLS algorithm belongs to the class of Graduation Methods , orHomotopic Continuations.Idea:Our target is to optimize a complicated (possibly non-convex)optimization criterion J(x), argminx∈DJ(x).However we know how to optimize a closely related criterion J0(x),argminx∈D0J0(x).Then we introduce a monotonic sequence 0 ≤ tn ≤ 1 with t0 = 1 andtn → 0 and solve iteratively

xn+1 = argminx∈Dn F (tn, J(x), J0(x))

using xn as starting point. Here F is a continue function so thatF (1, J(x), J0(x)) = J0(x) and F (0, J(x), J0(x)) = J(x).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 11: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmLARS Algorithm

Least Angle Regression (LARS) [EHJT04] designed to solve LASSO, orvariants:

argminx‖y − Ax‖22 + λ‖x‖1

It is proved the optimizer xopt = x(λ) is a continuous and piecewisedifferentiable function of λ (linear, in the case of LASSO).Method: Start with λ = λ0 = 2

‖AT y‖2and the optimal solution is x0 = 0.

Then LARS finds monotonicallydecreasing λ values where the slope(and support) of x(λ) changes. Thealgorithm ends at the desired valueof λ = λ∞ (see also HierarchicalDecompositions of Tadmor&all).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 12: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmLARS Algorithm

Least Angle Regression (LARS) [EHJT04] designed to solve LASSO, orvariants:

argminx‖y − Ax‖22 + λ‖x‖1It is proved the optimizer xopt = x(λ) is a continuous and piecewisedifferentiable function of λ (linear, in the case of LASSO).

Method: Start with λ = λ0 = 2‖AT y‖2

and the optimal solution is x0 = 0.

Then LARS finds monotonicallydecreasing λ values where the slope(and support) of x(λ) changes. Thealgorithm ends at the desired valueof λ = λ∞ (see also HierarchicalDecompositions of Tadmor&all).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 13: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmLARS Algorithm

Least Angle Regression (LARS) [EHJT04] designed to solve LASSO, orvariants:

argminx‖y − Ax‖22 + λ‖x‖1It is proved the optimizer xopt = x(λ) is a continuous and piecewisedifferentiable function of λ (linear, in the case of LASSO).Method: Start with λ = λ0 = 2

‖AT y‖2and the optimal solution is x0 = 0.

Then LARS finds monotonicallydecreasing λ values where the slope(and support) of x(λ) changes. Thealgorithm ends at the desired valueof λ = λ∞ (see also HierarchicalDecompositions of Tadmor&all).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 14: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmLARS Algorithm

Least Angle Regression (LARS) [EHJT04] designed to solve LASSO, orvariants:

argminx‖y − Ax‖22 + λ‖x‖1It is proved the optimizer xopt = x(λ) is a continuous and piecewisedifferentiable function of λ (linear, in the case of LASSO).Method: Start with λ = λ0 = 2

‖AT y‖2and the optimal solution is x0 = 0.

Then LARS finds monotonicallydecreasing λ values where the slope(and support) of x(λ) changes. Thealgorithm ends at the desired valueof λ = λ∞ (see also HierarchicalDecompositions of Tadmor&all).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 15: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmIRLS Algorithm

The Iterative Regularized Least-Squares Algorithm attempts to fnd theglobal minimum of the non-convex problem

argminx

m∑k=1|yk − |〈x , fk〉|2|2 + 2λ∞‖x‖22

using a sequence of iterative least-squares problems:

x (t+1) = argminx

m∑k=1|yk − |〈x , fk〉|2|2 + 2λt‖x‖22 + µt‖x − x (t)‖2

together with a polarization relaxation:

|〈x , fk〉|2 ≈12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 16: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmIRLS Algorithm

The Iterative Regularized Least-Squares Algorithm attempts to fnd theglobal minimum of the non-convex problem

argminx

m∑k=1|yk − |〈x , fk〉|2|2 + 2λ∞‖x‖22

using a sequence of iterative least-squares problems:

x (t+1) = argminx

m∑k=1|yk − |〈x , fk〉|2|2 + 2λt‖x‖22 + µt‖x − x (t)‖2

together with a polarization relaxation:

|〈x , fk〉|2 ≈12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 17: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmIRLS Algorithm

The Iterative Regularized Least-Squares Algorithm attempts to fnd theglobal minimum of the non-convex problem

argminx

m∑k=1|yk − |〈x , fk〉|2|2 + 2λ∞‖x‖22

using a sequence of iterative least-squares problems:

x (t+1) = argminx

m∑k=1|yk − |〈x , fk〉|2|2 + 2λt‖x‖22 + µt‖x − x (t)‖2

together with a polarization relaxation:

|〈x , fk〉|2 ≈12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 18: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmMain Optimization

The optimization problem:

x (t+1) = argminx

m∑k=1

∣∣∣∣yk −12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

∣∣∣∣2 +

+λt‖x‖22 + µt‖x − x (t)‖22 + λt‖x (t)‖22= argminx J(x , x (t);λ, µ)

Note:J(x , .; ., .) is quadratic in x ⇒ hence a least-squares problem!J(x , x ;λ, µ) =

∑mk=1 |yk − |〈x , fk〉|2|2 + 2λ‖x‖22 ⇒ Fixed points of

IRLS are local minima of the original problem.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 19: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmMain Optimization

The optimization problem:

x (t+1) = argminx

m∑k=1

∣∣∣∣yk −12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

∣∣∣∣2 +

+λt‖x‖22 + µt‖x − x (t)‖22 + λt‖x (t)‖22= argminx J(x , x (t);λ, µ)

Note:

J(x , .; ., .) is quadratic in x ⇒ hence a least-squares problem!J(x , x ;λ, µ) =

∑mk=1 |yk − |〈x , fk〉|2|2 + 2λ‖x‖22 ⇒ Fixed points of

IRLS are local minima of the original problem.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 20: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmMain Optimization

The optimization problem:

x (t+1) = argminx

m∑k=1

∣∣∣∣yk −12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

∣∣∣∣2 +

+λt‖x‖22 + µt‖x − x (t)‖22 + λt‖x (t)‖22= argminx J(x , x (t);λ, µ)

Note:J(x , .; ., .) is quadratic in x ⇒ hence a least-squares problem!

J(x , x ;λ, µ) =∑m

k=1 |yk − |〈x , fk〉|2|2 + 2λ‖x‖22 ⇒ Fixed points ofIRLS are local minima of the original problem.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 21: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmMain Optimization

The optimization problem:

x (t+1) = argminx

m∑k=1

∣∣∣∣yk −12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

∣∣∣∣2 +

+λt‖x‖22 + µt‖x − x (t)‖22 + λt‖x (t)‖22= argminx J(x , x (t);λ, µ)

Note:J(x , .; ., .) is quadratic in x ⇒ hence a least-squares problem!J(x , x ;λ, µ) =

∑mk=1 |yk − |〈x , fk〉|2|2 + 2λ‖x‖22 ⇒ Fixed points of

IRLS are local minima of the original problem.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 22: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmSecond Motivation: Relaxation of Constraints

Another motivation: seek X = xx∗ that solves

minX≥0,rank(X)=1

m∑k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2λtrace(X ).

PhaseLift algorithm removes the condition rank(X ) = 1 and shows (forlarge λ) this produces the desired result with high probability.Another way to relax the problem is to search for X in a larger space. TheIRLS is essentially equivalent to optimize a convex functional of X on thelarger space

S1,1 = {T = T ∗ ∈ Cn×n , T has at most one positive eigenvalueand at most one negative eigenvalue}.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 23: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmSecond Motivation: Relaxation of Constraints

Another motivation: seek X = xx∗ that solves

minX≥0,rank(X)=1

m∑k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2λtrace(X ).

PhaseLift algorithm removes the condition rank(X ) = 1 and shows (forlarge λ) this produces the desired result with high probability.

Another way to relax the problem is to search for X in a larger space. TheIRLS is essentially equivalent to optimize a convex functional of X on thelarger space

S1,1 = {T = T ∗ ∈ Cn×n , T has at most one positive eigenvalueand at most one negative eigenvalue}.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 24: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmSecond Motivation: Relaxation of Constraints

Another motivation: seek X = xx∗ that solves

minX≥0,rank(X)=1

m∑k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2λtrace(X ).

PhaseLift algorithm removes the condition rank(X ) = 1 and shows (forlarge λ) this produces the desired result with high probability.Another way to relax the problem is to search for X in a larger space. TheIRLS is essentially equivalent to optimize a convex functional of X on thelarger space

S1,1 = {T = T ∗ ∈ Cn×n , T has at most one positive eigenvalueand at most one negative eigenvalue}.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 25: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmSecond Formulation

Consider the following three convex criteria:

J1(X ;λ, µ) =m∑

k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2(λ+ µ)‖X‖1 − 2µtrace(X )

J2(X ;λ, µ) =m∑

k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2λeigmax (X )− (2λ+ 4µ)eigmin(X )

J3(X ;λ, µ) =m∑

k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2λ‖X‖1 − 4µeigmin(X )

which coincide on S1,1.

Consider the optimization problem

(Jopt,X ) = minX∈S1,1

Jk(X ;λ, µ) , 1 ≤ k ≤ 3

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 26: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmSecond Formulation

Consider the following three convex criteria:

J1(X ;λ, µ) =m∑

k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2(λ+ µ)‖X‖1 − 2µtrace(X )

J2(X ;λ, µ) =m∑

k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2λeigmax (X )− (2λ+ 4µ)eigmin(X )

J3(X ;λ, µ) =m∑

k=1|yk − 〈X , fk f ∗k 〉HS |

2 + 2λ‖X‖1 − 4µeigmin(X )

which coincide on S1,1. Consider the optimization problem

(Jopt,X ) = minX∈S1,1

Jk(X ;λ, µ) , 1 ≤ k ≤ 3

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 27: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmSecond Formulation -2

The following are true:1 Optimization in S1,1:

minX∈S1,1

Jk(X ;λ, µ) = minu,v∈Cn

J(u, v ;λ, µ)

If X and (u, v) denote optimizers so that imag(〈u, v〉) = 0, thenX = 1

2(uv∗ + v u∗).

2 Optimization in S1,0:

minX∈S1,0

Jk(X ;λ, µ) = minx∈Cn

J(x , x ;λ, µ)

If X and x denote optimizers, then X = x x∗. S1,0 = {xx∗}.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 28: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmSecond Formulation -2

The following are true:1 Optimization in S1,1:

minX∈S1,1

Jk(X ;λ, µ) = minu,v∈Cn

J(u, v ;λ, µ)

If X and (u, v) denote optimizers so that imag(〈u, v〉) = 0, thenX = 1

2(uv∗ + v u∗).2 Optimization in S1,0:

minX∈S1,0

Jk(X ;λ, µ) = minx∈Cn

J(x , x ;λ, µ)

If X and x denote optimizers, then X = x x∗. S1,0 = {xx∗}.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 29: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmInitialization

For λ ≥ eigmax (R(y)), where R(y) =∑m

k=1 yk fk f ∗k ,J(x ;λ) =

∑mk=1 |yk − |〈x , fk〉|2|2 + 2λ‖x‖22 is convex. The unique global

minimum is x0 = 0.

Initialization Procedure:

Solve the principal eigenpair (e, eigmax ) of matrix R(y) using e.g. thepower method;Set

λ0 = (1− ε)eigmax , x0 =√

(1− ε)eigmax∑mk=1 |〈e, fk〉|4

e.

Here ε > 0 is a parameter that depends on the frame set as well asthe spectral gap of R(y).Set µ0 = λ0 and t = 0.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 30: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmInitialization

For λ ≥ eigmax (R(y)), where R(y) =∑m

k=1 yk fk f ∗k ,J(x ;λ) =

∑mk=1 |yk − |〈x , fk〉|2|2 + 2λ‖x‖22 is convex. The unique global

minimum is x0 = 0.Initialization Procedure:

Solve the principal eigenpair (e, eigmax ) of matrix R(y) using e.g. thepower method;Set

λ0 = (1− ε)eigmax , x0 =√

(1− ε)eigmax∑mk=1 |〈e, fk〉|4

e.

Here ε > 0 is a parameter that depends on the frame set as well asthe spectral gap of R(y).Set µ0 = λ0 and t = 0.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 31: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmIterations

Repeat the following steps until stopping:Optimization: Solve the least-square problem:

x (t+1) = argminx

m∑k=1

∣∣∣∣yk −12(〈x , fk〉〈fk , x (t)〉+ 〈x (t), fk〉〈fk , x〉)

∣∣∣∣2 +

+λt‖x‖22 + µt‖x − x (t)‖22 + λt‖x (t)‖22= argminx J(x , x (t);λ, µ)

Update: λt+1 = γλt , µt+1 = max(γµt , µmin), t = t + 1. Here γ is

the learning rate, and µmin is related to performance.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 32: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmPerformance

Let yk = |〈x , fk〉|2 + νk . Assume the algorithm is stopped at some T sothat

J(x (T ), x (T−1);λ, µ) ≤ J(x , x ;λ, µ).

Denote X = 12(x (T )x (T−1)∗ + x (T−1)x (T )∗) and x x∗ = P+(X ).

Then the following hold true:1 Matrix norm error:

‖X − xx∗‖1 ≤λ

C0+√

C0‖ν‖

2 Natural distance:

D(x , x)2 = ‖X − xx∗‖1 +|eigmin(X )| ≤ λ

C0+√

C0‖ν‖+ ‖ν‖2

4µ + λ‖x‖2

where C0 is a frame dependent constant (lower Lipschitz constant in S1,1).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 33: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

The IRLS AlgorithmPerformance

Let yk = |〈x , fk〉|2 + νk . Assume the algorithm is stopped at some T sothat

J(x (T ), x (T−1);λ, µ) ≤ J(x , x ;λ, µ).

Denote X = 12(x (T )x (T−1)∗ + x (T−1)x (T )∗) and x x∗ = P+(X ).

Then the following hold true:1 Matrix norm error:

‖X − xx∗‖1 ≤λ

C0+√

C0‖ν‖

2 Natural distance:

D(x , x)2 = ‖X − xx∗‖1 +|eigmin(X )| ≤ λ

C0+√

C0‖ν‖+ ‖ν‖2

4µ + λ‖x‖2

where C0 is a frame dependent constant (lower Lipschitz constant in S1,1).Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 34: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Numerical SimulationsSetup

The algorithm requires O(m) memory. Simulations with m = Rn (complexcase) with n = 1000 and R ∈ {4, 6, 8, 12}. Frame vectors corresponding tomasked (windowed) DFT:

fjn+k = 1√8n

(w j

l e2πik(l−1)/n)

0≤l≤n−1, 1 ≤ j ≤ R, 1 ≤ k ≤ n

[f1 f2 · · · fm

]=[

Diag(w1) · · · Diag(wR)] DFTn 0 0

0 . . . 00 0 DFTn

Parameters: ε = 0.1, γ = 0.95, µmin = µ0

10 . Power method tolerance: 10−8

Conjugate gradient tolerance: 10−14.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 35: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Numerical SimulationsMSE Plots

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 36: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Numerical SimulationsMSE Plots

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 37: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Numerical SimulationsPerformance

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 38: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Numerical SimulationsPerformance

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 39: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Numerical SimulationsPerformance - 2

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 40: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Numerical SimulationsPerformance - 2

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 41: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

ReferencesK. Achan, S.T. Roweis, B.J. Frey, Probabilistic Inference of SpeechSignals from Phaseless Spectrograms, NIPS 2003.

B. Alexeev, A. S. Bandeira, M. Fickus, D. G. Mixon, Phase Retrievalwith Polarization, SIAM J. Imaging Sci., 7 (1) (2014), 35–66.

R. Balan, P. Casazza, D. Edidin, On signal reconstruction withoutphase, Appl.Comput.Harmon.Anal. 20 (2006), 345–356.

R. Balan, B. Bodmann, P. Casazza, D. Edidin, Painless reconstructionfrom Magnitudes of Frame Coefficients, J.Fourier Anal.Applic., 15 (4)(2009), 488–501.

R. Balan, Reconstruction of Signals from Magnitudes of FrameRepresentations, arXiv submission arXiv:1207.1134 (2012).

R. Balan, Reconstruction of Signals from Magnitudes of RedundantRepresentations: The Complex Case, available online

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 42: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

arXiv:1304.1839v1, Found.Comput.Math. 2015,http://dx.doi.org/10.1007/s10208-015-9261-0

R. Balan, The Fisher Information Matrix and the Cramer-Rao LowerBound in a Non-Additive White Gaussian Noise Model for the PhaseRetrieval Problem, proceedings of SampTA 2015.

A.S. Bandeira, Y. Chen, D.G. Mixon, Phase Retrieval from PowerSpectra of Masked Signals, arXiv:1303.4458v1 (2013).

B. G. Bodmann and N. Hammen, Stable Phase Retrieval withLow-Redundancy Frames, available online arXiv:1302.5487v1. Adv.Comput. Math., accepted 10 April 2014.

E. Candes, T. Strohmer, V. Voroninski, PhaseLift: Exact and StableSignal Recovery from Magnitude Measurements via ConvexProgramming, Communications in Pure and Applied Mathematics vol.66, 1241–1274 (2013).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 43: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

E. Candes, Y. Eldar, T. Strohmer, V. Voroninski, Phase Retrieval viaMatrix Completion Problem, SIAM J. Imaging Sci., 6(1) (2013),199–225.E. Candes, X. Li, Solving Quadratic Equations Via PhaseLift WhenThere Are As Many Equations As Unknowns, available onlinearXiv:1208.6247E. Candes, X. Li, M. Soltanolkotabi, Phase Retrieval from CodedDiffraction Patterns,E. Candes, X. Li and M. Soltanolkotabi, Phase retrieval via Wirtingerflow: theory and algorithms, IEEE Transactions on Information Theory61(4), (2014) 1985–2007.

Yang Chen, Cheng Cheng, Qiyu Sun and Haichao Wang, PhaseRetrieval of Real-Valued Signals in a Shift-Invariant Space,arXiv:1603.01592 (2016).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 44: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Y. C. Eldar, P. Sidorenko, D. G. Mixon, S. Barel and O. Cohen, Sparsephase retrieval from short-time Fourier measurements, IEEE SignalProcessing Letters 22, no. 5 (2015): 638-642.

B. Efron, T. Hastie, I. Johnstone, R. Tibshirani, Least AngleRegression, The Annals of Statistics, vol. 32(2), 407–499 (2004).

A. Fannjiang, W. Liao, Compressed Sensing Phase Retrieval, Asilomar2011.M. Fickus, D.G. Mixon, A.A. Nelson, Y. Wang, Phase retrieval fromvery few measurements, available online arXiv:1307.7176v1. LinearAlgebra and its Applications 449 (2014), 475–499

J.R. Fienup. Phase retrieval algorithms: A comparison, AppliedOptics, 21(15):2758–2768, 1982.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 45: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

R. W. Gerchberg and W. O. Saxton, A practical algorithm for thedetermination of the phase from image and diffraction plane pictures,Optik 35, 237 (1972).

D. Griffin and J.S. Lim, Signal Estimation from Modified Short-TimeFourier Transform, ICASSP 83, Boston, April 1983.

M. H. Hayes, J. S. Lim, and A. V. Oppenheim, Signal Reconstructionfrom Phase and Magnitude, IEEE Trans. ASSP 28, no.6 (1980),672–680.M. Iwen, A. Viswanathan, Y. Wang, Robust Sparse Phase RetrievalMade Easy, preprint.

M. Iwen, A. Viswanathan, Y. Wang, Fast Phase Retrieval forHigh-Dimensions, preprint.

[Overview2] K. Jaganathany Y.Eldar B.Hassibiy, Phase Retrieval: AnOverview of Recent Developments, arXiv:1510.07713 (2015).

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm

Page 46: The Iterative and Regularized Least Squares Algorithm for ...rvbalan/PRESENTATIONS/ams2016fargo.pdf · The Iterative and Regularized Least Squares Algorithm ... 2680 papers Radu Balan

The Phase Retrieval Problem Existing Algorithms The IRLS Algorithm Numerical Results

Y. Shechtman, A. Beck and Y. C. Eldar, GESPAR: Efficient phaseretrieval of sparse signals, IEEE Transactions on Signal Processing 62,no. 4 (2014): 928-938.

J. Sun, Q. Qu, J. Wright, A Geometric Analysis of Phase Retrieval,preprint 2016.

G. Thakur, Reconstruction of bandlimited functions from unsignedsamples, J. Fourier Anal. Appl., 17(2011), 720–732.

Radu Balan , Naveed Haghani (UMD) IRLS Algorithm


Recommended