Post on 21-Jun-2015
description
transcript
Inverse ProblemsRegularization
www.numerical-tours.com
Gabriel Peyré
Overview
• Variational Priors
• Gradient Descent and PDE’s
• Inverse Problems Regularization
J(f) = ||f ||2W 1,2 =
Z
R2
||rf(x)||dxSobolev semi-norm:
Smooth and Cartoon Priors
�|�f |2
J(f) = ||f ||2W 1,2 =
Z
R2
||rf(x)||dxSobolev semi-norm:
Total variation semi-norm:
J(f) = ||f ||TV =
Z
R2
||rf(x)||dx
Smooth and Cartoon Priors
�|�f |
�|�f |2
J(f) = ||f ||2W 1,2 =
Z
R2
||rf(x)||dxSobolev semi-norm:
Total variation semi-norm:
J(f) = ||f ||TV =
Z
R2
||rf(x)||dx
Smooth and Cartoon Priors
�|�f |
�|�f |2
Natural Image Priors
Discrete Priors
Discrete Priors
Discrete Differential Operators
Discrete Differential Operators
Laplacian Operator
Laplacian Operator
Function:
f̃ : x 2 R2 7! f(x) 2 R
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
Gradient: Images vs. Functionals
Function:
f̃ : x 2 R2 7! f(x) 2 R
Discrete image: f 2 RN , N = n2
f [i1, i2] = f̃(i1/n, i2/n) rf [i] ⇡ rf̃(i/n)
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
Gradient: Images vs. Functionals
Function:
f̃ : x 2 R2 7! f(x) 2 R
Discrete image: f 2 RN , N = n2
f [i1, i2] = f̃(i1/n, i2/n)
Functional:
J : f 2 RN 7! J(f) 2 RJ(f + ⌘) = J(f) + hrJ(f), ⌘iRN +O(||⌘||2RN )
rf [i] ⇡ rf̃(i/n)
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
rJ : RN 7! RN
Gradient: Images vs. Functionals
Function:
f̃ : x 2 R2 7! f(x) 2 R
Discrete image: f 2 RN , N = n2
f [i1, i2] = f̃(i1/n, i2/n)
Functional:
J : f 2 RN 7! J(f) 2 R
Sobolev:
rJ(f) = (r⇤ � r)f = ��f
J(f) =1
2||rf ||2
J(f + ⌘) = J(f) + hrJ(f), ⌘iRN +O(||⌘||2RN )
rf [i] ⇡ rf̃(i/n)
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
rJ : RN 7! RN
Gradient: Images vs. Functionals
rJ(f) = �div
✓rf
||rf ||
◆If 8n,rf [n] 6= 0,
If 9n,rf [n] = 0, J not di↵erentiable at f .
Total Variation Gradient||rf ||
rJ(f)
rJ(f) = �div
✓rf
||rf ||
◆If 8n,rf [n] 6= 0,
Sub-di↵erential:
If 9n,rf [n] = 0, J not di↵erentiable at f .
Cu =�↵ 2 R2⇥N \ (u[n] = 0) ) (↵[n] = u[n]/||u[n]||)
@J(f) = {� div(↵) ; ||↵[n]|| 6 1 and ↵ 2 Crf}
Total Variation Gradient||rf ||
rJ(f)
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
px
2 + "
2
|x|
Regularized Total Variation
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
rJ"(f) = �div
✓rf
||rf ||"
◆
px
2 + "
2
|x|
rJ" ⇠ ��/" when " ! +1
Regularized Total Variation
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
rJ"(f)
Overview
• Variational Priors
• Gradient Descent and PDE’s
• Inverse Problems Regularization
f (k+1) = f (k) � ⌧krJ(f (k)) f (0) is given.
Gradient Descent
and 0 < ⌧ < 2/L, then f (k) k!+1�! f?
a solution of min
fJ(f).
If f is convex, C1, rf is L-Lipschitz,Theorem:
f (k+1) = f (k) � ⌧krJ(f (k)) f (0) is given.
Gradient Descent
and 0 < ⌧ < 2/L, then f (k) k!+1�! f?
a solution of min
fJ(f).
If f is convex, C1, rf is L-Lipschitz,Theorem:
f (k+1) = f (k) � ⌧krJ(f (k)) f (0) is given.
Optimal step size: ⌧k = argmin⌧2R+
J(f (k) � ⌧rJ(f (k)))
Proposition: One has
hrJ(f (k+1)), rJ(f (k))i = 0
Gradient Descent
Gradient Flows and PDE’sf (k+1) � f (k)
⌧= �rJ(f (k))Fixed step size ⌧k = ⌧ :
Gradient Flows and PDE’sf (k+1) � f (k)
⌧= �rJ(f (k))Fixed step size ⌧k = ⌧ :
Denote ft = f (k)for t = k⌧ , one obtains formally as ⌧ ! 0:
8 t > 0,@ft@t
= �rJ(ft) and f0 = f (0)
J(f) =R||rf(x)||dx
Sobolev flow:
@ft@t
= �ftHeat equation:
Explicit solution:
Gradient Flows and PDE’sf (k+1) � f (k)
⌧= �rJ(f (k))Fixed step size ⌧k = ⌧ :
Denote ft = f (k)for t = k⌧ , one obtains formally as ⌧ ! 0:
8 t > 0,@ft@t
= �rJ(ft) and f0 = f (0)
Total Variation Flow
@ft@t
= �rJ(ft)
Noisy observations: y = f + w, w ⇠ N (0, IdN ).
and ft=0 = y
Application: Denoising
Optimal choice of t: minimize ||ft � f ||
�! not accessible in practice.
SNR(ft, f) = �20 log10
✓||f � ft||
||f ||
◆
Optimal Parameter Selection
t t
Overview
• Variational Priors
• Gradient Descent and PDE’s
• Inverse Problems Regularization
Inverse Problems
Inverse Problems
Inverse Problems
Inverse Problems
Inverse Problems
Inverse Problem Regularization
Inverse Problem Regularization
Inverse Problem Regularization
Sobolev prior: J(f) = 12 ||rf ||2
f? = argminf2RN
E(f) = ||y � �f ||2 + �||rf ||2
(assuming 1 /2 ker(�))
Sobolev Regularization
Sobolev prior: J(f) = 12 ||rf ||2
f? = argminf2RN
E(f) = ||y � �f ||2 + �||rf ||2
rE(f?) = 0 () (�⇤�� ��)f? = �⇤yProposition:
�! Large scale linear system.
(assuming 1 /2 ker(�))
Sobolev Regularization
Sobolev prior: J(f) = 12 ||rf ||2
f? = argminf2RN
E(f) = ||y � �f ||2 + �||rf ||2
rE(f?) = 0 () (�⇤�� ��)f? = �⇤yProposition:
�! Large scale linear system.
Gradient descent:
(assuming 1 /2 ker(�))
where ||A|| = �max
(A)
�! Slow convergence.
Sobolev Regularization
Convergence: ⇥ < 2/||⇥�⇥� ��||
Mask M , � = diagi(1i2M )
Example: Inpainting
since x � [0, 1]2. This distance corresponds to the Euclidean distance over thecube ��1(M), but since c̃f has a complex convoluted geometry, this distanceis not Euclidean when displayed as a 2D image.
Image f Surface c̃f Distance dMFig. 2. Manifold of smooth images.
4.3 Numerical Experiments
Figure 3 shows iterations of the algorithm 1 to solve the inpainting problemon a smooth image using a manifold prior with 2D linear patches, as defined in16. This manifold together with the overlapping of the patches allow a smoothinterpolation of the missing pixels.
Measurements y Iter. #1 Iter. #3 Iter. #50
Fig. 3. Iterations of the inpainting algorithm on an uniformly regular image.
5 Manifold of Step Discontinuities
In order to introduce some non-linearity in the manifoldM, one needs to gobeyond the Fourier world of uniformly regular functions and consider signalsand images with discontinuities.
13
log10(||f (k) � f (�)||/||f0||)
k k
E(f (k))
M
(�f)[i] =
⇢0 if i 2 M,f [i] otherwise.
Symmetric linear system:
Conjugate Gradient
Ax = b () minx2Rn
E(x) = 1
2hAx, xi � hx, bi
Symmetric linear system:
x
(k+1) = argmin E(x)s.t. x� x
(k) 2 span(rE(x(0)), . . . ,rE(x(k)))
Intuition:
Conjugate Gradient
Ax = b () minx2Rn
E(x) = 1
2hAx, xi � hx, bi
Proposition:
8 ` < k, hrE(xk), rE(x`)i = 0
Symmetric linear system:
Initialization:
x
(0) 2 RN, r
(0) = b�Ax
(0), p
(0) = r
(0)
r
(k) =hrE(x(k)), d(k)ihAd(k), d(k)i
d
(k) = rE(x(k)) +||v(k)||
||v(k�1)||d
(k�1)
v
(k) = rE(x(k)) = Ax
(k) � b
x
(k+1) = x
(k) � r
(k)d
(k)
Iterations:
x
(k+1) = argmin E(x)s.t. x� x
(k) 2 span(rE(x(0)), . . . ,rE(x(k)))
Intuition:
Conjugate Gradient
Ax = b () minx2Rn
E(x) = 1
2hAx, xi � hx, bi
Proposition:
8 ` < k, hrE(xk), rE(x`)i = 0
TV" regularization:
(assuming 1 /2 ker(�))
f? = argminf2RN
E(f) = 1
2||�f � y||+ �J"(f)
Total Variation Regularization
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
TV" regularization:
(assuming 1 /2 ker(�))
f (k+1) = f (k) � ⌧krE(f (k))
rE(f) = �⇤(�f � y) + �rJ"(f)
rJ"(f) = �div
✓rf
||rf ||"
◆
Convergence: requires ⌧ ⇠ ".
Gradient descent:
f? = argminf2RN
E(f) = 1
2||�f � y||+ �J"(f)
Total Variation Regularization
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
TV" regularization:
(assuming 1 /2 ker(�))
f (k+1) = f (k) � ⌧krE(f (k))
rE(f) = �⇤(�f � y) + �rJ"(f)
rJ"(f) = �div
✓rf
||rf ||"
◆
Convergence: requires ⌧ ⇠ ".
Gradient descent:
f? = argminf2RN
E(f) = 1
2||�f � y||+ �J"(f)
Newton descent:
f (k+1) = f (k) �H�1k rE(f (k)) where Hk = @2E"(f (k))
Total Variation Regularization
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
k
Large
"TV vs. Sobolev ConvergeSmall"
Observations y Sobolev
Total variation
Inpainting: Sobolev vs. TV
Noiseless problem: f? 2 argminf
J"(f) s.t. f 2 HContraint: H = {f ; �f = y}.
f (k+1)= ProjH
⇣f (k) � ⌧krJ"(f
(k))
⌘
ProjH(f) = argmin
�g=y||g � f ||2 = f + �⇤(�⇤�)�1(y � �f)
Inpainting:ProjH(f)[i] =
⇢f [i] if i 2 M,y[i] otherwise.
Projected gradient descent:
f (k) k!+1�! f?a solution of (?).
(?)
Projected Gradient Descent
Proposition: If rJ" is L-Lipschitz and 0 < ⌧k < 2/L,
TV
Priors:
Non-quadratic
better edge recovery.
=)
Conclusion
Sobolev
TV
Priors:
Non-quadratic
better edge recovery.
=)
Optimization
Variational regularization:()
– Gradient descent.– Newton.
– Projected gradient.
– Conjugate gradient.
Non-smooth optimization ?
Conclusion
Sobolev