Nonsmooth, Nonconvex Optimization€¦ · Convex and Nonsmooth Optimization Class, Spring 2018,...

1 / 71

Nonsmooth, Nonconvex OptimizationAlgorithms and Examples

Michael L. OvertonCourant Institute of Mathematical Sciences

New York University

Convex and Nonsmooth Optimization Class, Spring 2018, Final Lecture

Based on my research work with Jim Burke (Washington), Adrian Lewis (Cornell)

and others

Introduction

IntroductionNonsmooth,NonconvexOptimization

A Simple NonconvexExample

Failure of GradientDescent inNonsmooth CaseArmijo-Wolfe LineSearchFailure of GradientMethod: SimpleConvex Example

Illustration of Failureand SuccessMethods Suitable forNonsmoothFunctions

Gradient Sampling

Quasi-NewtonMethods

A DifficultNonconvex Problemfrom Nesterov

Limited MemoryMethods

Concluding Remarks

2 / 71

Nonsmooth, Nonconvex Optimization





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71

Problem: find x that locally minimizes f , where f : Rn → R is






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71


■ Continuous






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71


■ Continuous■ Not differentiable everywhere, in particular often not

differentiable at local minimizers






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71



differentiable at local minimizers■ Not convex






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71



differentiable at local minimizers■ Not convex■ Usually, but not always, locally Lipschitz: for all x there

exists Lx s.t. |f(y)− f(z)| ≤ Lx‖y − z‖ for all y, z near x






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71




exists Lx s.t. |f(y)− f(z)| ≤ Lx‖y − z‖ for all y, z near x■ Otherwise, no structure assumed






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71





Lots of interesting applications






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71






Any locally Lipschitz function is differentiable almost everywhereon its domain. So, whp, can evaluate gradient at any given point.






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

3 / 71






Any locally Lipschitz function is differentiable almost everywhereon its domain. So, whp, can evaluate gradient at any given point.

What happens if we simply use gradient descent (steepestdescent) with a standard line search?

A Simple Nonconvex Example





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

4 / 71

f(x)=10*|x2 − x

12| + (1−x

1)2

steepest descent iterates−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

A Simple Nonconvex Example





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

4 / 71

f(x)=10*|x2 − x

12| + (1−x

1)2

steepest descent iterates−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

On this example, iterates invariably converge to a nonstationary point

Failure of Gradient Descent in Nonsmooth Case





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

5 / 71

Known for decades that gradient descent may converge tononstationary points when f is nonsmooth, even if it is convex.






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

5 / 71


■ V.F. Dem’janov and V.N. Malozemov, 1970■ P. Wolfe, 1975■ J.-B. Hiriart-Urruty and C. Lemarechal, 1993

But these are all examples cooked up to defeat exact linesearches from a specific starting point.






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

5 / 71


■ V.F. Dem’janov and V.N. Malozemov, 1970■ P. Wolfe, 1975■ J.-B. Hiriart-Urruty and C. Lemarechal, 1993

But these are all examples cooked up to defeat exact linesearches from a specific starting point.

Failure can be avoided by using sufficiently short steplengths(N.Z. Shor, 1970s), but this is slow.

Armijo-Wolfe Line Search





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

6 / 71

Given x with f differentiable at x and d with ∇f(x)T d < 0, andparameters 0 < c1 < c2 < 1, find steplength t so that






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

6 / 71


■ sufficient decrease in function value:f(x+ td) < f(x) + c1t∇f(x)Td (L. Armijo, 1966)






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

6 / 71



■ sufficient increase in directional derivative: f is differentiable atx+ td and ∇f(x+ td)T d > c2∇f(x)Td (P. Wolfe, 1969)






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

6 / 71




Assuming inft f(x+ td) is bounded below,

■ the Armijo condition holds for sufficiently small t as long as f iscontinuous

■ the Wolfe condition holds for sufficiently large t as long as f isdifferentiable

■ the intervals where each holds overlap

so combining the two conditions leads to a convenient, convergentbracketing line search (M.J.D. Powell, 1976)






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

6 / 71









Extends to locally Lipschitz case (A.S. Lewis and M.L.O., 2013)






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

6 / 71









Extends to locally Lipschitz case (A.S. Lewis and M.L.O., 2013)

Searching for “Armijo-Wolfe” on the web, we foundMelissa Armijo-Wolfe’s LinkedIn page!

Failure of Gradient Method: Simple Convex Example





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

7 / 71

Let f(x) = a|x1|+ x2, with a ≥ 1. Although f is unboundedbelow, it is bounded below along any direction d = −∇f(x).

105

0

u

-5-10-10

-5

v

05

10

60

50

40

30

20

10

0

-10

5|u|

+v

Failure of Gradient Method: Simple Convex Example





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

7 / 71

Let f(x) = a|x1|+ x2, with a ≥ 1. Although f is unboundedbelow, it is bounded below along any direction d = −∇f(x).

105

0

u

-5-10-10

-5

v

05

10

60

50

40

30

20

10

0

-10

5|u|

+v

Theorem. Let x(0) satisfy x(0)1 6= 0 and define x(k) ∈ R

2 by

x(k+1) = x(k) + tkd(k) where d(k) = −∇f(x(k))

and tk is any steplength satisfying the Armijo and Wolfeconditions with Armijo parameter c1. If

c1(a2 + 1) > 1

then x(k) converges to x with x1 = 0, even though f isunbounded below.

Azam Asl and M.L.O., 2017

Illustration of Failure and Success





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

8 / 71

u-3 -2 -1 0 1 2 3

v

0

1

2

3

4

5

6f(u,v) = 5|u|+v. x_0 = (-2.264; 5), c_1=0.1, τ =0.064

u-3 -2 -1 0 1 2 3

v

0

1

2

3

4

5

6f(u,v) = 2|u|+v. x_0 = (-2.264; 5), c_1=0.1, τ =-0.125

a = 5, c1 = 0.1 a = 2, c1 = 0.1

x(k) → x f(x(k)) ↓ −∞

Methods Suitable for Nonsmooth Functions





Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

9 / 71

Exploit the gradient information obtained at several points, notjust at one point:






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

9 / 71


■ Bundle methods (C. Lemarechal, K.C. Kiwiel, 1980s –)extensive practical use and theoretical analysis, butcomplicated in nonconvex case






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

9 / 71



■ Gradient sampling: an easily stated method with niceconvergence theory (J.V. Burke, A.S. Lewis, M.L.O., 2005;K.C. Kiwiel, 2007), but computationally intensive






Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

9 / 71



■ Gradient sampling: an easily stated method with niceconvergence theory (J.V. Burke, A.S. Lewis, M.L.O., 2005;K.C. Kiwiel, 2007), but computationally intensive

■ BFGS: traditional workhorse for smooth optimization, worksamazingly well for nonsmooth optimization too, but verylimited convergence theory

Gradient Sampling

Introduction

Gradient Sampling

The GradientSampling Method

With First Phase ofGradient Sampling

With Second Phaseof GradientSampling

The ClarkeSubdifferential

Example

Note that0 ∈ ∂Cf(x) at

x = [1; 1]T

Grad. Samp.: AStabilized SteepestDescent MethodConvergence ofGradient SamplingMethod

ExtensionsSome GradientSampling SuccessStories

Quasi-NewtonMethods



10 / 71

The Gradient Sampling Method

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71

Fix sample size m ≥ n+ 1, line search parameter β ∈ (0, 1), reductionfactors µ ∈ (0, 1) and θ ∈ (0, 1).


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71


Initialize sampling radius ǫ > 0, tolerance τ > 0, iterate x.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)

■ Repeat (inner loop: gradient sampling with fixed ǫ):


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)


◆ Set G = {∇f(x),∇f(x+ ǫu1), . . . ,∇f(x+ ǫum)}, samplingu1, · · · , um uniformly from the unit ball


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)



◆ Set g = argmin{||g|| : g ∈ conv(G)}


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)



◆ Set g = argmin{||g|| : g ∈ conv(G)}◆ If ‖g ≤ τ , break out of loop.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)



◆ Set g = argmin{||g|| : g ∈ conv(G)}◆ If ‖g ≤ τ , break out of loop.◆ Backtracking line search: set d = −g and replace x by x+ td,

with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)




with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖

◆ If f is not differentiable at x+ td, replace x+ td by a nearbypoint where f is differentiable.1


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)




with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖


■ New phase: set ǫ = µǫ and τ = θτ .


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



11 / 71



Repeat (outer loop)




with t ∈ {1, 12 ,

14 , . . .} and f(x+ td) < f(x)− βt‖g‖


■ New phase: set ǫ = µǫ and τ = θτ .

J.V. Burke, A.S. Lewis and M.L.O., SIOPT, 2005.

1Needed in theory, but typically not in practice.

With First Phase of Gradient Sampling

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



12 / 71

f(x)=10*|x2 − x

12| + (1−x

1)2

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

With Second Phase of Gradient Sampling

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



13 / 71

f(x)=10*|x2 − x

12| + (1−x

1)2

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

The Clarke Subdifferential

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71

Assume f : Rn → R is locally Lipschitz, andlet D = {x ∈ R

n : f is differentiable at x}.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71


n : f is differentiable at x}.Rademacher’s Theorem: Rn\D has measure zero.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71



The Clarke subdifferential of f at x is

∂Cf(x) = conv

{lim

x→x,x∈D∇f(x)

}.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71




∂Cf(x) = conv

{lim

x→x,x∈D∇f(x)

}.

F.H. Clarke, 1973 (he used the name “generalized gradient”).


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71




∂Cf(x) = conv

{lim

x→x,x∈D∇f(x)

}.


If f is continuously differentiable at x, then ∂Cf(x) = {∇f(x)}.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71




∂Cf(x) = conv

{lim

x→x,x∈D∇f(x)

}.


If f is continuously differentiable at x, then ∂Cf(x) = {∇f(x)}.If f is convex, ∂Cf is the subdifferential of convex analysis.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71




∂Cf(x) = conv

{lim

x→x,x∈D∇f(x)

}.



We say x is Clarke stationary for f if 0 ∈ ∂Cf(x).


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



14 / 71




∂Cf(x) = conv

{lim

x→x,x∈D∇f(x)

}.



We say x is Clarke stationary for f if 0 ∈ ∂Cf(x).

Key point: the convex hull of the set G generated by GradientSampling is a surrogate for ∂Cf .

Example

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

2

Example

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

2

For x with x2 6= x21, f is differentiable with gradient

∇f(x) = 10 sgn{x2 − x21}[−2x11

]+

[−2(1− x1)

0

]

so ∂Cf(x) = {∇f(x)}.

Example

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

2


∇f(x) = 10 sgn{x2 − x21}[−2x11

]+

[−2(1− x1)

0

]

so ∂Cf(x) = {∇f(x)}. For x with x2 = x21, there are twolimiting gradients, namely

± 10

[−2x11

]+

[−2(1− x1)

0

]

so ∂Cf(x) consists of the convex hull of these two vectors.

Example

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



15 / 71

Letf(x) = 10|x2 − x21|+ (1− x1)

2


∇f(x) = 10 sgn{x2 − x21}[−2x11

]+

[−2(1− x1)

0

]

so ∂Cf(x) = {∇f(x)}. For x with x2 = x21, there are twolimiting gradients, namely

± 10

[−2x11

]+

[−2(1− x1)

0

]

so ∂Cf(x) consists of the convex hull of these two vectors.The unique x for which 0 ∈ ∂Cf(x) is x = [1; 1]T , so this is theunique Clarke stationary point of f (it follows that it is theglobal minimizer).

Note that 0 ∈ ∂Cf(x) at x = [1; 1]T

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



16 / 71

f(x)=10*|x2 − x

12| + (1−x

1)2

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

Grad. Samp.: A Stabilized Steepest Descent Method

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71

Lemma. Let C be a compact convex set and ‖ · ‖ = ‖ · ‖2. Then

−dist(0, C) = min‖d‖≤1

maxg∈C

gTd


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd

Proof.−dist(0, C) = −min

g∈C‖g‖


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd

= − max‖d‖≤1

ming∈C

gTd


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd

= − max‖d‖≤1

ming∈C

gTd

= − max‖d‖≤1

ming∈C

gT (−d)


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd

= − max‖d‖≤1

ming∈C

gTd

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

gT d.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd

= − max‖d‖≤1

ming∈C

gTd

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

gT d.

Note: the distance is nonnegative, and zero iff 0 ∈ C.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd

= − max‖d‖≤1

ming∈C

gTd

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

gT d.

Note: the distance is nonnegative, and zero iff 0 ∈ C.Otherwise, equality is attained by g = ΠC(0), d = −g/|g‖.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd

= − max‖d‖≤1

ming∈C

gTd

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

gT d.

Note: the distance is nonnegative, and zero iff 0 ∈ C.Otherwise, equality is attained by g = ΠC(0), d = −g/|g‖.Ordinary steepest descent: C = {∇f(x)}.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



17 / 71



maxg∈C

gTd


g∈C‖g‖

= −ming∈C

max‖d‖≤1

gTd

= − max‖d‖≤1

ming∈C

gTd

= − max‖d‖≤1

ming∈C

gT (−d)

= min‖d‖≤1

maxg∈C

gT d.

Note: the distance is nonnegative, and zero iff 0 ∈ C.Otherwise, equality is attained by g = ΠC(0), d = −g/|g‖.Ordinary steepest descent: C = {∇f(x)}.Gradient sampling: C = conv(G)

= conv({∇f(x),∇f(x+ ǫu1), . . . ,∇f(x+ ǫum)}) .

Convergence of Gradient Sampling Method

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71

Theorem. Suppose that f : Rn → R


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71


■ is locally Lipschitz


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71


■ is locally Lipschitz■ is cont. differentiable on an open full-measure subset of Rn


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71



■ has bounded level sets


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71




Then, with probability one, f is differentiable at the sampledpoints, the line search always terminates, and if the sequence ofiterates {x} converges to some point x, then, with probability 1


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71





■ the inner loop always terminates, so the sequences ofsampling radii {ǫ} and tolerances {τ} converge to zero, and


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71






■ x is Clarke stationary for f , i.e., 0 ∈ ∂Cf(x).


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71









Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71








Drop the assumption that f has bounded level sets. Then, wp 1,either the sequence {f(x)} → −∞, or every cluster point of thesequence of iterates {x} is Clarke stationary.


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



18 / 71








Drop the assumption that f has bounded level sets. Then, wp 1,either the sequence {f(x)} → −∞, or every cluster point of thesequence of iterates {x} is Clarke stationary.

K.C. Kiwiel, SIOPT, 2007.

Extensions

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



19 / 71

A more efficient version: Adaptive Gradient Sampling(F.E. Curtis and X. Que, 2013).

Extensions

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



19 / 71


Problems with Nonsmooth Constraints

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

Extensions

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



19 / 71



min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p

where f and c1, . . . , cp are locally Lipschitz but may not bedifferentiable at local minimizers.

Extensions

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



19 / 71



min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p


A successive quadratic programming gradient sampling methodwith convergence theory.

Extensions

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



19 / 71



min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p


A successive quadratic programming gradient sampling methodwith convergence theory.

F.E. Curtis and M.L.O., SIOPT, 2012.

Some Gradient Sampling Success Stories

Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



20 / 71

■ Non-Lipschitz eigenvalue optimization for non-normalmatrices (J.V. Burke, A.S. Lewis and M.L.O., 2002 – )


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



20 / 71


■ Design of fixed-order controllers for linear dynamical systemswith input and output (D. Henrion and M.L.O., 2006, andmany subsequent users of our HIFOO (H-Infinity Fixed OrderOptimization) toolbox)


Introduction

Gradient Sampling





Example


x = [1; 1]T



Quasi-NewtonMethods



20 / 71



■ Design of path planning for robots: avoids “chattering” thatotherwise arises from nonsmoothness (I. Mitchell et al, 2017)

Quasi-Newton Methods

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell

BFGSThe BFGS Method(“Full” Version)

BFGS forNonsmoothOptimization

With BFGSExample:Minimizing aProduct ofEigenvalues

BFGS from 10Randomly GeneratedStarting Points

Evolution ofEigenvalues ofA ◦ XEvolution ofEigenvalues of H

Regularity

Partly SmoothFunctionsPartly SmoothFunctions, continued

Same ExampleAgain

Relation of Partial

21 / 71

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

22 / 71

W. Davidon, a physicist at Argonne, had the breakthrough ideain 1959: since it’s too expensive to compute and factor theHessian ∇2f(x) at every iteration, update an approximation toits inverse using information from gradient differences, andmultiply this onto the negative gradient to approximateNewton’s method.

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

22 / 71


Each inverse Hessian approximation differs from the previous oneby a rank-two correction.

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

22 / 71



Ahead of its time: the paper was rejected by the physicsjournals, but published 30 years later in the first issue of SIAM J.Optimization.

Bill Davidon

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

22 / 71



Ahead of its time: the paper was rejected by the physicsjournals, but published 30 years later in the first issue of SIAM J.Optimization.

Davidon was a well known active anti-war protester during theVietnam War. In December 2013, it was revealed that he wasthe mastermind behind the break-in at the FBI office in Media,PA, on March 8, 1971, during the Muhammad Ali - Joe Frazierworld heavyweight boxing championship.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

23 / 71

In 1963, R. Fletcher and M.J.D. Powell improved Davidon’smethod and established convergence for convex quadraticfunctions.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

23 / 71


They applied it to solve problems in 100 variables: a lot at thetime.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

23 / 71



The method became known as the DFP method.

Fletcher and Powell

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

23 / 71



The method became known as the DFP method.

Davidon, Fletcher and Powell all died during 2013–2016.

BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

24 / 71

In 1970, C.G. Broyden, R. Fletcher, D. Goldfarb and D. Shannoall independently proposed the BFGS method, which is a kind ofdual of the DFP method. It was soon recognized that this was aremarkably effective method for smooth optimization.

BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

24 / 71


In 1973, C.G. Broyden, J.E. Dennis and J.J. More proved genericlocal superlinear convergence of BFGS and DFP and otherquasi-Newton methods.

BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

24 / 71



In 1976, M.J.D. Powell established convergence of BFGS with aninexact Armijo-Wolfe line search for a general class of smoothconvex functions for BFGS. In 1987, this was extended byR.H. Byrd, J. Nocedal and Y.-X. Yuan to include the whole“Broyden” class of methods interpolating BFGS and DFP:except for the DFP end point.

BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

24 / 71



In 1976, M.J.D. Powell established convergence of BFGS with aninexact Armijo-Wolfe line search for a general class of smoothconvex functions for BFGS. In 1987, this was extended byR.H. Byrd, J. Nocedal and Y.-X. Yuan to include the whole“Broyden” class of methods interpolating BFGS and DFP:except for the DFP end point.

Pathological counterexamples to convergence in the smooth,nonconvex case are known to exist (Y.-H. Dai, 2002, 2013;W. Mascarenhas 2004), but it is widely accepted that themethod works well in practice in the smooth, nonconvex case.

The BFGS Method (“Full” Version)

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71

Initialize iterate x and positive-definite symmetric matrix H(which is supposed to approximate the inverse Hessian of f)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat

■ Set d = −H∇f(x).


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search■ Set s = td, y = ∇f(x+ td)−∇f(x)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search■ Set s = td, y = ∇f(x+ td)−∇f(x)■ Replace x by x+ td


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat

■ Set d = −H∇f(x).■ Obtain t from Armijo-Wolfe line search■ Set s = td, y = ∇f(x+ td)−∇f(x)■ Replace x by x+ td■ Replace H by V HV T + 1

sT yssT , where V = I − 1

sT ysyT


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat



sT ysyT

Note that H can be computed in O(n2) operations since V is arank one perturbation of the identity


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

25 / 71


Repeat



sT ysyT

Note that H can be computed in O(n2) operations since V is arank one perturbation of the identityThe Wolfe condition guarantees that sT y > 0 and hence thatthe new H is positive definite.

BFGS for Nonsmooth Optimization

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

26 / 71

In 1982, C. Lemarechal observed that quasi-Newton methods can beeffective for nonsmooth optimization, but dismissed them as there wasno theory behind them and no good way to terminate them.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

26 / 71


Otherwise, there is not much in the literature on the subject untilA.S. Lewis and M.L.O. (Math. Prog., 2013): we address both issues indetail, but our convergence results are limited to special cases.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

26 / 71



Key point: use an Armijo-Wolfe line search. Do not insist on reducingthe magnitude of the directional derivative along the line!


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

26 / 71




In the nonsmooth case, BFGS builds a very ill-conditioned inverse“Hessian” approximation, with some tiny eigenvalues converging tozero, corresponding to “infinitely large” curvature in the directionsdefined by the associated eigenvectors.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

26 / 71





Remarkably, the condition number of the inverse Hessianapproximation typically reaches 1016 before the method breaks down.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

26 / 71






We have never seen convergence to non-stationary points that cannotbe explained by numerical difficulties.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

26 / 71






We have never seen convergence to non-stationary points that cannotbe explained by numerical difficulties.

Convergence rate of BFGS is typically linear (not superlinear) in thenonsmooth case.

With BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

27 / 71

f(x)=10*|x2 − x

12| + (1−x

1)2

steepest descent, grad sampling and BFGS iterates

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

contours of fstarting pointoptimal pointsteepest descentgrad samp (1st phase)grad samp (2nd phase)bfgs

Example: Minimizing a Product of Eigenvalues

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

28 / 71

Let SN denote the space of real symmetric N ×N matrices, and

λ1(X) ≥ λ2(X) ≥ · · ·λN (X)

denote the eigenvalues of X ∈ SN .


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

28 / 71


λ1(X) ≥ λ2(X) ≥ · · ·λN (X)

denote the eigenvalues of X ∈ SN . We wish to minimize

f(X) = log

N/2∏

i=1

λi(A ◦X)

where A ∈ SN is fixed and ◦ is the Hadamard (componentwise)matrix product, subject to the constraints that X is positivesemidefinite and has diagonal entries equal to 1.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

28 / 71


λ1(X) ≥ λ2(X) ≥ · · ·λN (X)


f(X) = log

N/2∏

i=1

λi(A ◦X)


If we replace∏

by∑

we would have a semidefinite program.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

28 / 71


λ1(X) ≥ λ2(X) ≥ · · ·λN (X)


f(X) = log

N/2∏

i=1

λi(A ◦X)


If we replace∏

by∑


Since f is not convex, may as well replace X by Y Y T whereY ∈ R

N×N : eliminates psd constraint, and then also easy toeliminate diagonal constraint.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

28 / 71


λ1(X) ≥ λ2(X) ≥ · · ·λN (X)


f(X) = log

N/2∏

i=1

λi(A ◦X)


If we replace∏

by∑


Since f is not convex, may as well replace X by Y Y T whereY ∈ R

N×N : eliminates psd constraint, and then also easy toeliminate diagonal constraint.

Application: entropy minimization in an environmentalapplication (K.M. Anstreicher and J. Lee, 2004)

BFGS from 10 Randomly Generated Starting Points

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

29 / 71

0 200 400 600 800 1000 1200 140010

−15

10−10

10−5

100

105

iteration

f − f op

t (di

ffere

nt s

tart

ing

poin

ts)

Log eigenvalue product, N=20, n=400, fopt

= −4.37938e+000

f − fopt, where fopt is least value of f found over all runs

Evolution of Eigenvalues of A ◦X

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

30 / 71

0 200 400 600 800 1000 1200 14000.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8


= −4.37938e+000

iteration

eige

nval

ues

of A

o X

Evolution of Eigenvalues of A ◦X

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

30 / 71

0 200 400 600 800 1000 1200 14000.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8


= −4.37938e+000

iteration

eige

nval

ues

of A

o X

Note that λ6(X), . . . , λ14(X) coalesce

Evolution of Eigenvalues of H

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

31 / 71

0 200 400 600 800 1000 1200 140010

−15

10−10

10−5

100

105


= −4.37938e+000

iteration

eige

nval

ues

of H

Evolution of Eigenvalues of H

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

31 / 71

0 200 400 600 800 1000 1200 140010

−15

10−10

10−5

100

105


= −4.37938e+000

iteration

eige

nval

ues

of H

44 eigenvalues of H converge to zero...why???

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

32 / 71

A locally Lipschitz, directionally differentiable function f isregular (Clarke 1970s) near a point x when its directionalderivative f ′( · ; d) is upper semicontinuous there for every fixeddirection d.

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

32 / 71


In this case 0 ∈ ∂Cf(x) is equivalent to the first-order optimalitycondition f ′(x, d) ≥ 0 for all directions d.

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

32 / 71



■ All convex functions are regular

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

32 / 71



■ All convex functions are regular■ All smooth functions are regular

Regularity

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

32 / 71



■ All convex functions are regular■ All smooth functions are regular■ Nonsmooth concave functions are not regular

Example: f(x) = −|x|

Note: this is a somewhat simpler definition of regularity than theone in Lecture 12, but it is less precise: it defines regularity in aneighborhood, not at a point.

Partly Smooth Functions

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

33 / 71

A regular function f is partly smooth at x relative to a manifoldM containing x (A.S. Lewis 2003) if


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

33 / 71


■ its restriction to M is twice continuously differentiable near x


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

33 / 71


■ its restriction to M is twice continuously differentiable near x■ the Clarke subdifferential ∂f is continuous on M near x


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

33 / 71


■ its restriction to M is twice continuously differentiable near x■ the Clarke subdifferential ∂f is continuous on M near x■ par ∂f(x), the subspace parallel to the affine hull of the

subdifferential of f at x, is exactly the subspace normal toM at x.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

33 / 71




We refer to par ∂f(x) as the V -space for f at x (with respect toM), and to its orthogonal complement, the subspace tangent toM at x, as the U -space for f at x.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

33 / 71





When we refer to the V -space and U -space without reference toa point x, we mean at a minimizer.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

33 / 71





When we refer to the V -space and U -space without reference toa point x, we mean at a minimizer.

For nonzero y in the V -space, the mapping t 7→ f(x+ ty) isnecessarily nonsmooth at t = 0, while for nonzero y in theU -space, t 7→ f(x+ ty) is differentiable at t = 0 as long as f islocally Lipschitz.

Partly Smooth Functions, continued

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

34 / 71

Example: f(x) = 10|x2 − x21|+ (1− x1)2.

Question: What is M and what are the U and V spaces at theminimizer?

Partly Smooth Functions, continued

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

34 / 71

Example: f(x) = 10|x2 − x21|+ (1− x1)2.

Question: What is M and what are the U and V spaces at theminimizer?

Example: f(x) = ‖x‖2.Question: What is M and what are the U and V spaces at theminimizer?

Same Example Again

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

35 / 71

f(x)=10*|x2 − x

12| + (1−x

1)2

steepest descent, grad sampling and BFGS iterates

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

contours of fstarting pointoptimal pointsteepest descentgrad samp (1st phase)grad samp (2nd phase)bfgs

Relation of Partial Smoothness to Earlier Work

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

36 / 71

Partial smoothness is closely related to earlier work of J.V. Burkeand J.J. More (1990,1994) and S.J. Wright (1993) onidentification of constraint structure by algorithms.

Relation of Partial Smoothness to Earlier Work

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

36 / 71

Partial smoothness is closely related to earlier work of J.V. Burkeand J.J. More (1990,1994) and S.J. Wright (1993) onidentification of constraint structure by algorithms.

When f is convex, the partly smooth nomenclature is consistentwith the usage of V -space and U -space by C. Lemarechal,F. Oustry and C. Sagastizabal (2000), but partial smoothnessdoes not imply convexity and convexity does not imply partialsmoothness.

Why Did 44 Eigenvalues of H Converge to Zero?

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

37 / 71

The eigenvalue product is regular and also partly smooth (in the senseof A.S. Lewis, 2003) with respect to the manifold of matrices with aneigenvalue with given multiplicity. This implies that tangent to thismanifold (preserving the multiplicity to first-order) the function issmooth (“U -shaped”) and normal to it, the function is nonsmooth

(“V -shaped”).


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

37 / 71


(“V -shaped”).

Recall that at the computed minimizer,

λ6(A ◦X) ≈ . . . ≈ λ14(A ◦X).

Matrix theory says that imposing multiplicity m on an eigenvalue a

matrix ∈ SN is m(m+1)2 − 1 conditions, or 44 when m = 9, so the

dimension of the V -space at this minimizer is 44.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

37 / 71


(“V -shaped”).


λ6(A ◦X) ≈ . . . ≈ λ14(A ◦X).




Tiny eigenvalues of H correspond to huge curvature, whichcorresponds to V -space directions.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

37 / 71


(“V -shaped”).


λ6(A ◦X) ≈ . . . ≈ λ14(A ◦X).




Tiny eigenvalues of H correspond to huge curvature, whichcorresponds to V -space directions.

Thus BFGS automatically detected the U and V space partitioningwithout knowing anything about the mathematical structure of f !

Variation of f from Minimizer, along EigVecs of H

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

38 / 71

−10 −5 0 5 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

f(x op

t + t

w)

− f op

t

t


= −4.37938e+000

w is eigvector for eigvalue 10 of final Hw is eigvector for eigvalue 20 of final Hw is eigvector for eigvalue 30 of final Hw is eigvector for eigvalue 40 of final Hw is eigvector for eigvalue 50 of final Hw is eigvector for eigvalue 60 of final H

Eigenvalues of H numbered smallest to largest

BFGS Theory for Special Nonsmooth Functions

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

39 / 71

Convergence results for BFGS with Armijo-Wolfe line searchwhen f is nonsmooth are limited to very special cases.

■ f(x) = |x| (one variable!): sequence generated converging to0 is related to a certain binary expansion of the startingpoint (A.S. Lewis and M.L.O., 2013)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

39 / 71



■ f(x) = |x1|+ x2: f(x) ↓ −∞ (A.S. Lewis and ShanshanZhang, 2015)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

39 / 71




■ f(x) = |x1|+∑n

i=2 xi: eventually a direction is identified onwhich f is unbounded below (Yuchen Xie and A. Waechter,2017)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

39 / 71




■ f(x) = |x1|+∑n


■ f(x) =√∑n

i=1 x2i : iterates converge to [0, . . . , 0] (Jiayi Guo

and A.S. Lewis, 2017) (proof based on Powell (1976))


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

39 / 71




■ f(x) = |x1|+∑n


■ f(x) =√∑n

i=1 x2i : iterates converge to [0, . . . , 0] (Jiayi Guo

and A.S. Lewis, 2017) (proof based on Powell (1976))

■ f(x) = |x1|+ x22: remains open!

Challenge: General Nonsmooth Case

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71

Assume f is locally Lipschitz with bounded level sets and issemi-algebraic (its graph is a finite union of sets each defined bya finite list of polynomial inequalities)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71


Assume the initial x and H are generated randomly (e.g. fromnormal and Wishart distributions)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71



Prove or disprove that the following hold with probability one:


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71




1. BFGS generates an infinite sequence {x} with fdifferentiable at all iterates


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71





2. Any cluster point x is Clarke stationary


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71





2. Any cluster point x is Clarke stationary3. The sequence of function values generated (including all of

the line search iterates) converges to f(x) R-linearly


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

40 / 71





2. Any cluster point x is Clarke stationary3. The sequence of function values generated (including all of

the line search iterates) converges to f(x) R-linearly4. If {x} converges to x where f is “partly smooth” w.r.t. a

manifold M then the subspace defined by the eigenvectorscorresponding to eigenvalues of H converging to zeroconverges to the “V -space” of f w.r.t. M at x

Some BFGS Nonsmooth Success Stories

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

41 / 71


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

41 / 71



Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

41 / 71


■ Shape optimization for spectral functions ofDirichlet-Laplacian operators (B. Osting, 2010)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

41 / 71



■ Condition metric optimization (P. Boito and J. Dedieu, 2010)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

41 / 71



■ Condition metric optimization (P. Boito and J. Dedieu, 2010)

Software is available: HANSO

Extensions of BFGS for Nonsmooth Optimization

Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

42 / 71

A combined BFGS-Gradient Sampling method with convergencetheory (F.E. Curtis and X. Que, 2015)


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

42 / 71


Constrained Problems

min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p



Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

42 / 71



min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p


A successive quadratic programming (SQP) BFGS methodapplied to challenging problems in static-output-feedback controldesign (F.E. Curtis, T. Mitchell and M.L.O., 2015).


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

42 / 71



min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p



Although there are no theoretical results, it is much moreefficient and effective than the SQP Gradient Sampling methodwhich does have convergence results.


Introduction

Gradient Sampling

Quasi-NewtonMethods

Bill Davidon

Fletcher and Powell






Regularity


Same ExampleAgain

Relation of Partial

42 / 71



min f(x)

subject to ci(x) ≤ 0, i = 1, . . . , p



Although there are no theoretical results, it is much moreefficient and effective than the SQP Gradient Sampling methodwhich does have convergence results.

Software is available: GRANSO

A Difficult Nonconvex Problem

from Nesterov

Introduction

Gradient Sampling

Quasi-NewtonMethods

A DifficultNonconvex Problemfrom NesterovAn Aside:ChebyshevPolynomials

Plots of ChebyshevPolynomials

Nesterov’sChebyshev-RosenbrockFunctionsWhy BFGS Takes SoMany Iterations toMinimize N2

First NonsmoothVariant ofNesterov’s FunctionSecond NonsmoothVariant ofNesterov’s FunctionContour Plots of theNonsmooth Variantsfor n = 2Properties of theSecond NonsmoothVariant N1

Behavior of BFGSon the SecondNonsmooth Variant

43 / 71

An Aside: Chebyshev Polynomials

Introduction

Gradient Sampling

Quasi-NewtonMethods






44 / 71

A sequence of orthogonal polynomials defined on [−1, 1] by

T0(x) = 1, T1(x) = x, Tn+1(x) = 2xTn(x)− Tn−1(x).


Introduction

Gradient Sampling

Quasi-NewtonMethods






44 / 71



So T2(x) = 2x2 − 1,


Introduction

Gradient Sampling

Quasi-NewtonMethods






44 / 71



So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.


Introduction

Gradient Sampling

Quasi-NewtonMethods






44 / 71



So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.

Important properties that can be proved easily include


Introduction

Gradient Sampling

Quasi-NewtonMethods






44 / 71



So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.


■ Tn(x) = cos(n cos−1(x))


Introduction

Gradient Sampling

Quasi-NewtonMethods






44 / 71



So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.



■ Tm(Tn(x)) = Tmn(x)


Introduction

Gradient Sampling

Quasi-NewtonMethods






44 / 71



So T2(x) = 2x2 − 1, T3(x) = 4x3 − 3, etc.



■ Tm(Tn(x)) = Tmn(x)

■

∫ 1−1

1√1−x2

Ti(x)Tj(x)dx = 0 if i 6= j

Plots of Chebyshev Polynomials

Introduction

Gradient Sampling

Quasi-NewtonMethods






45 / 71

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Left: Plots of T0(x), . . . , T4(x) Right: Plot of T8(x).

Plots of Chebyshev Polynomials

Introduction

Gradient Sampling

Quasi-NewtonMethods






45 / 71

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Left: Plots of T0(x), . . . , T4(x) Right: Plot of T8(x).Question: How many extrema does Tn(x) have in [−1, 1]?

Nesterov’s Chebyshev-Rosenbrock Functions

Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71

Consider the function

Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1


Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71


Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1

The unique minimizer is x∗ = [1, 1, . . . , 1]T with Np(x∗) = 0.


Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71


Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1


Define x = [−1, 1, 1, . . . , 1]T with Np(x) = 1 and the manifold

MN = {x : xi+1 = 2x2i − 1, i = 1, . . . , n− 1}


Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71


Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1



MN = {x : xi+1 = 2x2i − 1, i = 1, . . . , n− 1}For x ∈ MN , e.g. x = x∗ or x = x, the 2nd term of Np is zero.Starting at x, BFGS needs to approximately follow MN to reachx∗ (unless it “gets lucky”).


Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71


Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1




When p = 2: N2 is smooth but not convex. Starting at x:


Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71


Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1





■ n = 5: BFGS needs 370 iterations to reduce N2 below 10−15


Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71


Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1






■ n = 10: needs ∼ 50,000 iterations to reduce N2 below 10−15

even though N2 is smooth!


Introduction

Gradient Sampling

Quasi-NewtonMethods






46 / 71


Np(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|p, where p ≥ 1






■ n = 10: needs ∼ 50,000 iterations to reduce N2 below 10−15

even though N2 is smooth! . . . In the last few iterations, weobserve superlinear convergence!

Why BFGS Takes So Many Iterations to Minimize N2

Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71

Let Ti(x) denote the ith Chebyshev polynomial. For x ∈ MN ,

xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).

To move from x to x∗ along the manifold MN exactly requires


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).


■ x1 to change from −1 to 1


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).


■ x1 to change from −1 to 1■ x2 = 2x21 − 1 to trace the graph of T2(x1) on [−1, 1]


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).


■ x1 to change from −1 to 1■ x2 = 2x21 − 1 to trace the graph of T2(x1) on [−1, 1]■ x3 = T2(T2(x)) to trace the graph of T4(x1) on [−1, 1]


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).


■ x1 to change from −1 to 1■ x2 = 2x21 − 1 to trace the graph of T2(x1) on [−1, 1]■ x3 = T2(T2(x)) to trace the graph of T4(x1) on [−1, 1]■ xn = T2n−1(x) to trace the graph of T2n−1(x1) on [−1, 1]

which has 2n−1 − 1 extrema in (−1, 1).


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).



which has 2n−1 − 1 extrema in (−1, 1).Even though BFGS will not track the manifold MN exactly, itwill follow it approximately. So, since the manifold is highlyoscillatory, BFGS must take relatively short steps to obtainreduction in N2 in the line search, and hence many iterations!


Introduction

Gradient Sampling

Quasi-NewtonMethods






47 / 71


xi+1 = 2x2i − 1 = T2(xi) = T2(T2(xi−1))

= T2(T2(. . . T2(x1) . . .)) = T2i(x1).



which has 2n−1 − 1 extrema in (−1, 1).Even though BFGS will not track the manifold MN exactly, itwill follow it approximately. So, since the manifold is highlyoscillatory, BFGS must take relatively short steps to obtainreduction in N2 in the line search, and hence many iterations!

Newton’s method is not much faster, although it convergesquadratically at the end.

First Nonsmooth Variant of Nesterov’s Function

Introduction

Gradient Sampling

Quasi-NewtonMethods






48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|


Introduction

Gradient Sampling

Quasi-NewtonMethods






48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|

N1 is nonsmooth (though locally Lipschitz) as well asnonconvex. The second term is still zero on the manifold MN ,but N1 is not differentiable on MN .


Introduction

Gradient Sampling

Quasi-NewtonMethods






48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|


However, N1 is regular at x ∈ MN and partly smooth at x w.r.t.MN .


Introduction

Gradient Sampling

Quasi-NewtonMethods






48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|



We cannot initialize BFGS at x, so starting at normallydistributed random points:


Introduction

Gradient Sampling

Quasi-NewtonMethods






48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|




■ n = 5: BFGS reduces N1 only to about 10−2 in 10,000iterations


Introduction

Gradient Sampling

Quasi-NewtonMethods






48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|





■ n = 10: BFGS reduces N1 only to about 5× 10−2 in 10,000iterations


Introduction

Gradient Sampling

Quasi-NewtonMethods






48 / 71

N1(x) =1

4(x1 − 1)2 +

n−1∑

i=1

|xi+1 − 2x2i + 1|





■ n = 10: BFGS reduces N1 only to about 5× 10−2 in 10,000iterations

The method appears to be converging, very slowly, but may behaving numerical difficulties.

Second Nonsmooth Variant of Nesterov’s Function

Introduction

Gradient Sampling

Quasi-NewtonMethods






49 / 71

N1(x) =1

4|x1 − 1|+

n−1∑

i=1

|xi+1 − 2|xi|+ 1|.

Again, the unique global minimizer is x∗. The second term iszero on the set

S = {x : xi+1 = 2|xi| − 1, i = 1, . . . , n− 1}

but S is not a manifold: it has “corners”.

Contour Plots of the Nonsmooth Variants for n = 2

Introduction

Gradient Sampling

Quasi-NewtonMethods






50 / 71

Nesterov−Chebyshev−Rosenbrock, first variant

x1

x2

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2Nesterov−Chebyshev−Rosenbrock, second variant

x1

x2

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

Contour plots of nonsmooth Chebyshev-Rosenbrock functions N1

(left) and N1 (right), with n = 2, with iterates generated byBFGS initialized at 7 different randomly generated points.

Contour Plots of the Nonsmooth Variants for n = 2

Introduction

Gradient Sampling

Quasi-NewtonMethods






50 / 71

Nesterov−Chebyshev−Rosenbrock, first variant

x1

x2

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2Nesterov−Chebyshev−Rosenbrock, second variant

x1

x2

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

Contour plots of nonsmooth Chebyshev-Rosenbrock functions N1

(left) and N1 (right), with n = 2, with iterates generated byBFGS initialized at 7 different randomly generated points.On the left, always get convergence to x∗ = [1, 1]T . On theright, most runs converge to [1, 1] but some go to x = [0,−1]T .

Properties of the Second Nonsmooth Variant N1

Introduction

Gradient Sampling

Quasi-NewtonMethods






51 / 71

When n = 2, the point x = [0,−1]T is Clarke stationary for thesecond nonsmooth variant N1. We can see this because zero isin the convex hull of the gradient limits for N1 at the point x.


Introduction

Gradient Sampling

Quasi-NewtonMethods






51 / 71


However, x = [0,−1]T is not a local minimizer, becaused = [1, 2]T is a direction of linear descent: N ′

1(x, d) < 0.


Introduction

Gradient Sampling

Quasi-NewtonMethods






51 / 71



1(x, d) < 0.

These two properties mean that N1 is not regular at [0,−1]T .


Introduction

Gradient Sampling

Quasi-NewtonMethods






51 / 71



1(x, d) < 0.


In fact, for n ≥ 2:

■ N1 has 2n−1 Clarke stationary points


Introduction

Gradient Sampling

Quasi-NewtonMethods






51 / 71



1(x, d) < 0.



■ N1 has 2n−1 Clarke stationary points■ the only local minimizer is the global minimizer x∗


Introduction

Gradient Sampling

Quasi-NewtonMethods






51 / 71



1(x, d) < 0.




■ x∗ is the only stationary point in the sense of Mordukhovich(i.e., with 0 ∈ ∂N1(x) where we defined ∂ in Lecture 12)(see also Rockafellar and Wets, Variational Analysis, 1998).

(M. Gurbuzbalaban and M.L.O., 2012)


Introduction

Gradient Sampling

Quasi-NewtonMethods






51 / 71



1(x, d) < 0.




■ x∗ is the only stationary point in the sense of Mordukhovich(i.e., with 0 ∈ ∂N1(x) where we defined ∂ in Lecture 12)(see also Rockafellar and Wets, Variational Analysis, 1998).

(M. Gurbuzbalaban and M.L.O., 2012)

Furthermore, starting from enough randomly generated startingpoints, BFGS finds all 2n−1 Clarke stationary points!

Behavior of BFGS on the Second Nonsmooth Variant

Introduction

Gradient Sampling

Quasi-NewtonMethods






52 / 71

0 200 400 600 800 10000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5Nesterov−Chebyshev−Rosenbrock, n=5

different starting points

so

rte

d f

ina

l va

lue

of

f

0 200 400 600 800 10000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5Nesterov−Chebyshev−Rosenbrock, n=6

different starting points

so

rte

d f

ina

l va

lue

of

f

Left: sorted final values of N1 for 1000 randomly generatedstarting points, when n = 5: BFGS finds all 16 Clarke stationarypoints. Right: same with n = 6: BFGS finds all 32 Clarkestationary points.

Convergence to Non-Locally-Minimizing Points

Introduction

Gradient Sampling

Quasi-NewtonMethods






53 / 71

When f is smooth, convergence of methods such as BFGS tonon-locally-minimizing stationary points or local maxima ispossible but not likely, because of the line search, and suchconvergence will not be stable under perturbation.


Introduction

Gradient Sampling

Quasi-NewtonMethods






53 / 71


However, this kind of convergence is what we are seeing for thenon-regular, non-smooth Nesterov Chebyshev-Rosenbrockexample, and it is stable under perturbation. The same behavioroccurs for gradient sampling or bundle methods.


Introduction

Gradient Sampling

Quasi-NewtonMethods






53 / 71



Kiwiel (private communication): the Nesterov example is the firsthe had seen which causes his bundle code to have this behavior.


Introduction

Gradient Sampling

Quasi-NewtonMethods






53 / 71



Kiwiel (private communication): the Nesterov example is the firsthe had seen which causes his bundle code to have this behavior.

Nonetheless, we don’t know whether, in exact arithmetic, themethods would actually generate sequences converging to thenonminimizing Clarke stationary points. Experiments by Kaku(2011) suggest that the higher the precision used, the more likelyBFGS is to eventually move away from such a point.

Limited Memory Methods

Introduction

Gradient Sampling

Quasi-NewtonMethods


Limited MemoryMethodsLimited MemoryBFGSLimited MemoryBFGS on theEigenvalue Product

A More BasicExample

Smooth, Convex:nA = 200, nB =0, nR = 1

Nonsmooth, Convex:nA = 200, nB =10, nR = 1

Nonsmooth,Nonconvex:nA = 200, nB =10, nR = 5

A NonsmoothConvex Function,Unbounded BelowL-BFGS-1 vs.Gradient DescentConvergence of theL-BFGS-1 Search

54 / 71

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





55 / 71

“Full” BFGS requires storing an n× n matrix and doingmatrix-vector multiplies, which is not possible when n is large.

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





55 / 71


In the 1980s, J. Nocedal and others developed a “limitedmemory” version of BFGS, with O(n) space and timerequirements, which is very widely used for minimizing smoothfunctions in many variables. At the kth iteration, it applies onlythe most recent m rank-two updates, defined by

(sj , yj), j = k −m, . . . , k − 1

to an initial inverse Hessian approximation H(k)0 .

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





55 / 71



(sj , yj), j = k −m, . . . , k − 1


There are two variants: with “scaling” (H(k)0 =

sTk−1yk−1

yTk−1yk−1

I) and

without scaling (H(k)0 = I).

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





55 / 71



(sj , yj), j = k −m, . . . , k − 1



sTk−1yk−1

yTk−1yk−1

I) and


The convergence rate of limited memory BFGS is linear, notsuperlinear, on smooth problems.

Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





55 / 71



(sj , yj), j = k −m, . . . , k − 1



sTk−1yk−1

yTk−1yk−1

I) and


The convergence rate of limited memory BFGS is linear, notsuperlinear, on smooth problems.

Question: how effective is it on nonsmooth problems?

Limited Memory BFGS on the Eigenvalue Product

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





56 / 71

0 10 20 30 40 50

10−5

10−4

10−3

10−2

10−1

100

Log eig prod, N=20, r=20, K=10, n=400, maxit = 400

number of vectors k

med

ian

redu

ctio

n in

f−f op

t (ov

er 1

0 ru

ns)

Lim Mem BFGS (scaling)Lim Mem BFGS (no scaling)full BFGS (scaling once)full BFGS (no scaling)


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





56 / 71

0 10 20 30 40 50

10−5

10−4

10−3

10−2

10−1

100


number of vectors k

med

ian

redu

ctio

n in

f−f op

t (ov

er 1

0 ru

ns)


Limited Memory is not nearly as good as full BFGS.


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





56 / 71

0 10 20 30 40 50

10−5

10−4

10−3

10−2

10−1

100


number of vectors k

med

ian

redu

ctio

n in

f−f op

t (ov

er 1

0 ru

ns)


Limited Memory is not nearly as good as full BFGS.

No significant improvement when k reaches 44.

A More Basic Example

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





57 / 71

Let x = [y; z;w] ∈ RnA+nB+nR and consider the test function

f(x) = (y − e)TA(y − e) + {(z − e)TB(z − e)}1/2 +R1(w)

where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





57 / 71



where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].

The first term is quadratic, the second is nonsmooth but convex,and the third is a nonsmooth, nonconvex Rosenbrock function.


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





57 / 71



where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].


The optimal value is 0, with x = e. The function f is partlysmooth and the dimension of the V-space is nB + nR − 1.


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





57 / 71



where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].



Set A = XXT where xij are normally distributed, with conditionnumber about 106 when nA = 200. Similarly B with nB < nA.


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





57 / 71



where A = AT ≻ 0, B = BT ≻ 0, e = [1; 1; . . . ; 1].



Set A = XXT where xij are normally distributed, with conditionnumber about 106 when nA = 200. Similarly B with nB < nA.

Besides limited memory BFGS and full BFGS, we also comparelimited memory Gradient Sampling, where we sample k ≪ ngradients per iteration.

Smooth, Convex: nA = 200, nB = 0, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





58 / 71

0 5 10 15 2010

−7

10−6

10−5

nA=200, nB=0, nR=1, maxit = 201

number of vectors k

med

ian

redu

ctio

n in

f (o

ver

10 r

uns)

Grad Samp 1e−02, 1e−04Lim Mem BFGS (scaling)Lim Mem BFGS (no scaling)full BFGS (scaling once)full BFGS (no scaling)

Smooth, Convex: nA = 200, nB = 0, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





58 / 71

0 5 10 15 2010

−7

10−6

10−5

nA=200, nB=0, nR=1, maxit = 201

number of vectors k

med

ian

redu

ctio

n in

f (o

ver

10 r

uns)


LM-BFGS with scaling even better than full BFGS

Nonsmooth, Convex: nA = 200, nB = 10, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





59 / 71

0 5 10 15 2010

−6

10−5

10−4

10−3

10−2

nA=200, nB=10, nR=1, maxit = 211

number of vectors k

med

ian

redu

ctio

n in

f (o

ver

10 r

uns)


Nonsmooth, Convex: nA = 200, nB = 10, nR = 1

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





59 / 71

0 5 10 15 2010

−6

10−5

10−4

10−3

10−2

nA=200, nB=10, nR=1, maxit = 211

number of vectors k

med

ian

redu

ctio

n in

f (o

ver

10 r

uns)


LM-BFGS much worse than full BFGS

Nonsmooth, Nonconvex: nA = 200, nB = 10, nR = 5

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





60 / 71

0 5 10 15 2010

−4

10−3

10−2

10−1

nA=200, nB=10, nR=5, maxit = 215

number of vectors k

med

ian

redu

ctio

n in

f (o

ver

10 r

uns)


Nonsmooth, Nonconvex: nA = 200, nB = 10, nR = 5

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





60 / 71

0 5 10 15 2010

−4

10−3

10−2

10−1

nA=200, nB=10, nR=5, maxit = 215

number of vectors k

med

ian

redu

ctio

n in

f (o

ver

10 r

uns)


LM-BFGS with scaling even worse than LM-Grad-Samp

A Nonsmooth Convex Function, Unbounded Below

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





61 / 71

Let’s reconsiderf(x) = a|x1|+ x2

with a ≥ 1.

105

0

u

-5-10-10

-5

v

05

10

60

50

40

30

20

10

0

-10

5|u|

+v

Turns out that L-BFGS-1 (saving just one update) with scalingfails for smaller values of a than the critical value beyond whichGradient Descent fails!

L-BFGS-1 vs. Gradient Descent

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





62 / 71

Red: path of L-BFGS-1 with scaling, converges to non-stationarypoint.Blue: path of the gradient method with same Armijo-Wolfe linesearch, generates f(x) ↓ −∞.

u-8 -6 -4 -2 0 2 4 6 8 10

v

-12

-10

-8

-6

-4

-2

0

2

4

6

f(u, v) = 3|u|+ v. x0 = (8.284; 2.177), c1=0.05, τ =-0.056

LBFGS-1Gradient method

Convergence of the L-BFGS-1 Search Direction

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





63 / 71

Theorem. Let d(k) be the search direction generated byL-BFGS-1 with scaling applied to f(x) = a|x1|+

∑ni=2 xi using

an Armijo-Wolfe line search. If√

4(n− 1) ≤ a, then|d(k)|||d(k)||

converges to some constant direction d. Furthermore, if

a(a+√

a2 − 3(n− 1)) > (1

c1− 1)(n− 1),

where c1 is the Armijo parameter, then the iterates x(k) convergeto a non-stationary point.

Azam Asl, 2018.

Experiment, with n = 2 and a =√3

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





64 / 71

In practice we observe that√

3(n− 1) ≤ a suffices for themethod to fail, which is a weaker condition than the previousone. Below with n = 2 and a =

√3 the method fails:

u-8 -6 -4 -2 0 2 4 6 8 10

v

-26

-24

-22

-20

-18

-16

-14

-12

-10

-8

f(u, v) = 1.7321|u|+ v. x0 = (8.284; 2.177), c1=0.05, τ =-0.27


Experiment: slightly smaller a

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





65 / 71

But if we set a =√3− 0.001, it succeeds “at the last minute”.

u-8 -6 -4 -2 0 2 4 6 8 10

v

-30

-28

-26

-24

-22

-20

-18

-16

-14

-12

f(u, v) = 1.7311|u|+ v. x0 = (8.284; 2.177), c1=0.05, τ =-0.27


Experiments: Top, scaling on; Bottom, scaling off

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





66 / 71

n = 30,√

3(n− 1) = 9.327

N=30, f(X) = a|x(1)|+ Σi=2N x(i), c

1=0.05, (3(N-1))0.5 = 9.33, nrand = 5000

a9.315 9.32 9.325 9.33 9.335 9.34

Fai

lure

rat

e

0

0.2

0.4

0.6

0.8

1

a9.315 9.32 9.325 9.33 9.335 9.34

Fai

lure

rat

e

-1

-0.5

0

0.5

1

Limited Effectiveness of Limited Memory BFGS

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





67 / 71

We have observed that that addition of nonsmoothness to aproblem, convex or nonconvex, creates great difficulties forLimited Memory BFGS, with and without scaling, even when thedimension of the V -space is less than the size of the memory.


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





67 / 71


Azam Asl’s result establishes failure of L-BFGS-1 for a specific fwhen scaling is on; no such result is proved yet when scaling isoff.


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





67 / 71


Azam Asl’s result establishes failure of L-BFGS-1 for a specific fwhen scaling is on; no such result is proved yet when scaling isoff.

We have also investigated Limited Memory Gradient Samplingwhich does not work well either.

Other Ideas for Large Scale Nonsmooth Optimization

Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





68 / 71


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





68 / 71

■ Exploit structure! Lots of work on this has been done, e.g.using proximal point methods or ADMM (AlternatingDirection Method of Multipliers)


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





68 / 71


■ Smoothing! Lots of work on this has been done too, mostnotably by Yu. Nesterov


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





68 / 71



■ Automatic Differentiation (AD): (A. Griewank et. al.)


Introduction

Gradient Sampling

Quasi-NewtonMethods



A More BasicExample





68 / 71



■ Automatic Differentiation (AD): (A. Griewank et. al.)

■ Stochastic Subgradient Method (D. Davis andD. Drusvyatskiy, 2018)

Concluding Remarks

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

69 / 71

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

70 / 71

Gradient descent frequently fails on nonsmooth problems.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

70 / 71


Gradient Sampling is a simple method for nonsmooth, nonconvexoptimization for which a convergence theory is known, but it istoo expensive to use in most applications.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

70 / 71



BFGS — the full version — is remarkably effective onnonsmooth problems, but little theory is known.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

70 / 71




Limited Memory BFGS is not so effective on nonsmoothproblems.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

70 / 71





Diabolical nonconvex problems such as Nesterov’sChebyshev-Rosenbrock problems can be very difficult, especiallyin the nonsmooth case.

Summary

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

70 / 71





Diabolical nonconvex problems such as Nesterov’sChebyshev-Rosenbrock problems can be very difficult, especiallyin the nonsmooth case.

Our software, HANSO and GRANSO, is available (unconstrainedand constrained) along with HIFOO (H-infinity fixed orderoptimization) for controller design, which has been usedsuccessfully in many applications.

Our Papers

Introduction

Gradient Sampling

Quasi-NewtonMethods



Concluding Remarks

Summary

Our Papers

71 / 71

J. V. Burke, A. S. Lewis and M. L. Overton, A Robust GradientSampling Method for Nonsmooth, Nonconvex Optimization, SIAMJ. Optimization, 2005

M. Gurbuzbalaban and M. L. Overton, On Nesterov’s NonsmoothChebyshev-Rosenbrock Functions, SIAM J. Optimization, 2012

A. S. Lewis and M. L. Overton, Nonsmooth Optimization viaQuasi-Newton Methods, Math. Programming, 2013

F. E. Curtis, T. Mitchell and M.L. Overton, A BFGS-SQP Method forNonsmooth, Nonconvex, Constrained Optimization and its Evaluationusing Relative Minimization Profiles, Optimization Methods and

Software, 2016

A. Asl and M. L. Overton, Analysis of the Gradient Method with anArmijo-Wolfe Line Search on a Class of Nonsmooth Convex Functions,arXiv, 2017

J. V. Burke, F. E. Curtis, A. S. Lewis, M. L. Overton and L. Simoes,Gradient Sampling Methods for Nonsmooth Optimization, arXiv, 2018

Papers, software are available at www.cs.nyu.edu/overton.

Date post:	10-Oct-2020
Category:	Documents
Upload:	others
View:	7 times
Download:	0 times

Nonsmooth, Nonconvex Optimization€¦ · Convex and Nonsmooth Optimization Class, Spring 2018,...

Documents