Post on 15-May-2018
transcript
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Chapter 1Convex AnalysisNonlinear Multiscale Methods for Image and Signal AnalysisSS 2015
Michael MoellerComputer Vision
Department of Computer ScienceTU Munchen
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Example: Inpainting
u = arg minu∈Rn
∥∥∥∥√(Dxu)2 + (Dy u)2
∥∥∥∥1, such that ui = fi ∀i ∈ I
with index set I of uncorrupted pixels.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Example: Inpainting
u = arg minu∈Rn
∥∥∥∥√(Dxu)2 + (Dy u)2
∥∥∥∥1, such that ui = fi ∀i ∈ I
with index set I of uncorrupted pixels.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Convexity
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Let us repeat some basics things to talk about
u = arg minu∈Rn
E(u).
Definition
• For E : Rn → R ∪ {∞}, we call
dom(E) := {u ∈ Rn | E(u) <∞}
the domain of E .• We call E proper if dom(E) 6= ∅.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Definition: Convex Function
We call E : Rn → R ∪ {∞} a convex function if
1 dom(E) is a convex set, i.e. for all u, v ∈ dom(E) and allθ ∈ [0,1] it holds that θu + (1− θ)v ∈ dom(E).
2 For all u, v ∈ dom(E) and all θ ∈ [0,1] it holds that
E(θu + (1− θ)v) ≤ θE(u) + (1− θ)E(v)
We call E strictly convex, if the inequality in 2 is strict for allθ ∈]0,1[, and v 6= u.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Example: Inpainting
u = arg minu∈Rn
∥∥∥∥√(Dxu)2 + (Dy u)2
∥∥∥∥1, such that ui = fi ∀i ∈ I
with index set I of uncorrupted pixels.→ Discuss convexity.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Existence
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
When doesu = arg min
u∈RnE(u)
exist?
• E is lower semi-continuous, i.e. for all u
lim infv→u
E(v) ≥ E(u)
holds.• There exists an α such that
{u | E(u) ≤ α}
is non-empty and bounded.
Proof: Board.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Fundamental Theorem of Optimization
If E : Rn → R ∪ {∞} is lower semi-continuous and has anonempty bounded sublevelset, then there exists
u = arg minu∈Rn
E(u)
Remark: For a proper convex function, lower semi-continuity isthe same as the closedness of the sublevelsets.
Examples on the board:• A convex continuous function that does not have a
minimizer• A convex function with bounded sublevelsets that does not
have a minimizer
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Continuity of Convex Functions
If E : Rn → R ∪ {∞} is convex, then E is locally Lipschitz (andhence continuous) on int(dom(E)).
Proof: Exercise (in 1d)
Board: Considering the interior is important!
Conclusion
If E : Rn → R is convex, then E is continuous.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Definition
We call E : Rn → R ∪ {∞} coercive, if all sequences (un)n with‖un‖ → ∞ meet E(un)→∞.
Theorem
If E : Rn → R is convex and coercive, then there exists
u = arg minu∈Rn
E(u).
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
When isu = arg min
u∈RnE(u)
unique?
Theorem
If E : Rn → R ∪ {∞} is convex, then any local minimum is aglobal minimum. If E is strictly convex, the global minimum isunique.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Subdifferential Calculus
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational ProblemsWhat is an optimality condition for
u = arg minu∈Rn
E(u)?
Definition: Subdifferential
We call
∂E(u) = {p ∈ Rn | E(v)− E(u)− 〈p, v − u〉 ≥ 0, ∀v ∈ Rn}
the subdifferential of E at u.• Elements of ∂E(u) are called subgradients.• If ∂E(u) 6= ∅, we call E subdifferentiable at E .• By convention, ∂E(u) = ∅ for u 6= dom(E).
Theorem: Optimality condition
Let 0 ∈ ∂E(u). Then u ∈ arg minu E(u).
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Examples for non-differentiable functions:• The `1 norm.• Functional
E(u) ={
0 if u ≥ 0∞ else.
Subdifferential and derivatives
Let the convex function E : Rn → R ∪ {∞} be differentiable atx ∈ dom(E). Then
∂E(x) = {∇E(x)}.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Is any convex E subdifferentiable at x ∈ dom(E)?
Answer: Almost...
Definition: Relative Interior
The relative interior of a convex set M is defined as
ri(M) := {x ∈ M | ∀y ∈ M, ∃λ > 1, s.t. λx + (1− λ)y ∈ M}
Theorem: Subdifferentiability1
If E is a proper convex function and u ∈ ri(dom(E)), then∂E(u) is non-empty and bounded.
1Rockafellar, Convex Analysis, Theorem 23.4
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Theorem: Sum rule2
Let E1, E2 be convex functions such that
ri(dom(E1)) ∩ ri(dom(E2)) 6= ∅,
then it holds that
∂(E1 + E2)(u) = ∂E1(u) + ∂E2(u).
Example: Minimize (u − f )2 + ιu≥0(u).
Example: Minimize 0.5(u − f )2 + α|u|.
2Rockafellar, Convex Analysis, Theorem 23.8
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Theorem: Chain rule3
If A ∈ Rm×n, E : Rm → R ∪ {∞} is convex, andri(dom(E)) ∩ range(A) 6= ∅, then
∂(E ◦ A)(u) = A∗∂E(Au)
Example: Minimize ‖Au − f‖22.
3Rockafellar, Convex Analysis, Theorem 23.9
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Summary (without assumptions):
• ∂E(u) = {p ∈ Rn | E(v)− E(u)− 〈p, v − u〉 ≥ 0,∀v ∈ Rn}
• If E differentiable: ∂E(x) = {∇E(x)}
• Sum rule ∂(E1 + E2)(x) = ∂E1(x) + ∂E2(x)
• Cain rule ∂(E ◦ A)(u) = A∗∂E(Au)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV minimization
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV minimization
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV minimization
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
What is TV again?
For u ∈ Rm×n let us consider the anisotropic total variation
TVa(u) =m∑
i=2
n∑j=1
|ui,j − ui−1,j |+m∑
i=1
n∑j=2
|ui,j − ui,j−1|
For doing math, it is often easier to consider ~ui+m(j−1) = u(i , j)and write
TVa(u) = ‖K~u‖1
for a suitable matrix K that discretizes the gradient.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Our problem becomes
u(α) = arg minu∈Rnm
12‖u − f‖2
2 + α‖Ku‖1.
Let us try to apply all the learned theory. The minimizer isobtained at
0 ∈ u(α)− f + αK T q
with q ∈ ∂‖Ku(α)‖1, i.e.
qi
= 1 if (Ku(α))i > 0= −1 if (Ku(α))i < 0∈ [−1,1] if (Ku(α))i = 0
Seems extremely difficult to find...
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Crazy idea:
minu
12‖u − f‖2
2 + α‖Ku‖1 =minu
12‖u − f‖2
2 + α sup‖q‖∞≤1
〈Ku,q〉
=minu
sup‖q‖∞≤1
12‖u − f‖2
2 + α〈Ku,q〉
Can we exchange min and sup?
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Saddle point problems4
Let C and D be non-empty closed convex sets in Rn and Rm,respectively, and let S be a continuous finite concave-convexfunction on C × D. If either C or D is bounded, one has
infv∈D
supq∈C
S(v ,q) = supq∈C
infv∈D
S(v ,q).
We can therefore compute
minu
12‖u − f‖2
2 + α‖Ku‖1 =minu
sup‖q‖∞≤1
12‖u − f‖2
2 + α〈Ku,q〉
= sup‖q‖∞≤1
minu
12‖u − f‖2
2 + α〈Ku,q〉.
4Rockafellar, Convex Analysis, Corollary 37.3.2
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Now the inner minimization problem obtains its optimum at
0 = u − f + αK T q,
⇒u = f − αK T q.
The remaining problem in q becomes
sup‖q‖∞≤1
12‖f − αK T q − f‖2
2 + α〈K (f − αK T q),q〉
= sup‖q‖∞≤1
12‖αK T q‖2
2 + α〈Kf ,q〉 − ‖αK T q‖22
= sup‖q‖∞≤1
−12‖αK T q − f‖2
2
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Since we prefer minimizations over maximizations, we write
q =arg max‖q‖∞≤1
−12‖αK T q − f‖2
2
= arg min‖q‖∞≤1
12
∥∥∥∥K T q − fα
∥∥∥∥2
2
Idea: Gradient descent + project onto feasible set.
qk+1 = π‖q‖∞≤1
(qk − τK
(K T qk − f
α
))
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Gradient projection algorithm5
The algorithm
qk+1 = π‖·‖∞≤1
(qk − τK
(K T qk − f
α
))with uk = f − αqk , for TV minimization converges for τ < 1
4 .
Remark: The 1/4 is two over the Lipschitz constant of thegradient of the smooth objective.
5Levitin, Polyak, Constrained minimization problems, 1966. Goldstein,Convex programming in Hilbert space, 1964.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
1 img = im2double(imread('cameraman.tif'));2 [m,n] = size(img);3
4 e = ones(m,1);5 yDerMat = spdiags([-e e], 0:1, m, m);6 yDerMat(end) =0;7
8 e = ones(n,1);9 xDerMat = spdiags([-e e], 0:1, n, n)';
10 xDerMat(end)=0;11
12 yDer = yDerMat*img;13 xDer = img*xDerMat;14
15 gradientMatrixForVectorizedImage = ...[kron(xDerMat',speye(m,m)); kron(speye(m,m), ...yDerMat)];
16
17 imgGradient = gradientMatrixForVectorizedImage*img(:);18 figure, imagesc(reshape(imgGradient(1:n*m), ...
[n,m])), colorbar
Note that AXB = C ⇔ kron(B′,A)~X = ~C!
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Duality
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
We replaced ‖Ku‖1 (not differentiable) with sup‖q‖∞≤1〈q,Ku〉.
Numerics became easier.
Is there a systematic concept behind this idea?
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Very important concept: Duality!
Definition: Convex Conjugate
We define the convex conjugate of the functionE : Rn → R ∪ {∞} to be
E∗(p) = supu∈Rn
(〈u,p〉 − E(u)) .
Board: E∗ is convex.
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Examples:
• E(u) = u2 leads to E∗(p) = p2
• E(u) = ‖u‖2 leads to E∗(p) ={
0 if ‖p‖2 ≤ 1,∞ else.
• E(u) = ‖u‖1 leads to E∗(p) ={
0 if ‖p‖∞ ≤ 1,∞ else.
• E(u) = ‖u‖∞ leads to E∗(p) ={
0 if ‖p‖1 ≤ 1,∞ else.
• E(u) ={
0 if ‖u‖2 ≤ 1,∞ else. leads to E∗(p) = ‖p‖2.
• E(u) ={
0 if ‖u‖∞ ≤ 1,∞ else. leads to E∗(p) = ‖p‖1.
• E(u) ={
0 if ‖u‖1 ≤ 1,∞ else. leads to E∗(p) = ‖p‖∞.
Suspicion: E∗∗ = E?
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Fenchel-Young Inequality6
Let E be proper, convex and lower semi-continuous,u ∈ dom(E) ⊂ Rn, and p ∈ Rn, then
E(u) + E∗(p) ≥ 〈u,p〉.
Equality holds if and only if p ∈ ∂E(u).
Theorem: Biconjugate7
Let E be proper, convex and lower semi-continuous, thenE∗∗ = E .
Now we understand what we did for TV minimization...
6Borwein, Zhu Techniques of variational analysis, Proposition 4.4.17Rockafellar, Convex Analysis, Theorem 12.2
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Theorem: Subgradient of convex conjugate8
Let E be proper, convex and lower semi-continuous, then thefollowing two conditions are equivalent:
• p ∈ ∂E(u)• u ∈ ∂E∗(p)
Board: Example with proximity operator.
8Rockafellar, Convex Analysis, Theorem 23.5
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
What is this good for? Easier analysis and optimization!
Fenchel’s Duality Theorem9
The following problems yield the same result:
infu H(u) + R(Ku) ”Primal”
infu supq H(u) + 〈q,Ku〉 − R∗(q)”Saddle point”
supq infu H(u) + 〈q,Ku〉 − R∗(q)
supq −H∗(−K ∗q)− R∗(q) ”Dual”
Assuming that H and R are proper, lower semi-continuous,convex functions with ri(dom(H)) ∩ ri(dom(R ◦ K )) 6= ∅.
9C.f. Rockafellar, Convex Analysis, Section 31, or Borwein, Zhu Techniquesof variational analysis, Theorem 4.4.3
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational ProblemsConsider the isotropic total variation
TV (u) =m∑
i=2
n∑j=2
√(ui,j − ui−1,j)2 + (ui,j − ui,j−1)2 = ‖Ku‖2,1
How do we minimize
12‖u − f‖2
2 + α‖Ku‖2,1 ?
Dual problem:
12
∥∥∥∥K T q − fα
∥∥∥∥2
2+ (‖ · ‖2,1)
∗(q)
Similar to previous examples:
(‖ · ‖2,1)∗(q) =
{0 if ‖q‖2,∞ ≤ 1 ∀i ,∞ else.
→ Gradient projection algorithm!