Properties of functions Optimization definitions Directions References
Mathematical concepts
MTH8418
S. Le Digabel, Polytechnique Montreal
Winter 2020(v4)
MTH8418: Concepts 1/40
Properties of functions Optimization definitions Directions References
Plan
Properties of functions
Optimization definitions
Directions
References
MTH8418: Concepts 2/40
Properties of functions Optimization definitions Directions References
Properties of functions
Optimization definitions
Directions
References
MTH8418: Concepts 3/40
Properties of functions Optimization definitions Directions References
Properties of a function
I We consider a function f : Rn → R
I We define some properties of f at x, a point of its domain
I We say that these properties apply near x if the property issatisfied on some open neighborhood of x
I We can also consider some properties on a domain X ⊂ Rn
I f is continuous at x ∈ Rn if the limit limy→x
f(y) exists and is
equal to f(x)
MTH8418: Concepts 4/40
Properties of functions Optimization definitions Directions References
DifferentiabilityI We consider the function f : Rn → RI f is differentiable at x ∈ Rn if there exists g ∈ Rn such that
limy→x
f(y)− f(x)− g>(y − x)
‖y − x‖= 0
I If this g exists, it is unique and is called the gradient of f atx, denoted ∇f(x)
I If f is differentiable at x, then f is continuous at x
I Partial derivatives:∂f(x)∂xi
= limh→0
1h (f(x1, . . . , xi + h, . . . , xn)− f(x)) (if the limit exists)
I Gradient and partial derivatives:
∇f(x) =
(∂f(x)
∂x1,∂f(x)
∂x2, . . . ,
∂f(x)
∂xn
)MTH8418: Concepts 5/40
Properties of functions Optimization definitions Directions References
Example 1
Illustrate that all partial derivatives of a function may exist, even ifthe function is not differentiable.
MTH8418: Concepts 6/40
Properties of functions Optimization definitions Directions References
Directional derivatives
I If it exists, the directional derivative at x in the directionv ∈ Rn is defined by
f ′(x; v) = limt↓0
f(x+ tv)− f(x)
t
I If the ith partial derivative exists, then ∂f(x)∂xi
= f ′(x; ei) whereei is the ith unit vector
I If f is differentiable at x, then f ′(x; v) = v>∇f(x) for allv ∈ Rn, and f ′(x;−d) = −f ′(x; d)
I All directional derivatives of a function may exist, even if thefunction is not differentiable.
MTH8418: Concepts 7/40
Properties of functions Optimization definitions Directions References
Example 2
Consider f(x) = ‖x‖ =√x>x
I What is the directional derivative of f at the origin and in thedirection v ∈ Rn?
I What is the gradient of f at the origin?
MTH8418: Concepts 8/40
Properties of functions Optimization definitions Directions References
Differentiability classes, smoothness
I A function f : Rn → R is said of class Ck, denoted f ∈ Ck,with 0 ≤ k ≤ ∞, if all the possible partial derivatives of theform
∂kf
∂xi1∂xi2 · · · ∂xikexist and are continuous, where i` ∈ 1, 2, . . . , n for all` ∈ 1, 2, . . . , k
I C0: Continuous functions
I C1: Continuously differentiable functions
I C∞: Smooth functions
MTH8418: Concepts 9/40
Properties of functions Optimization definitions Directions References
Strict differentiability
Let f be differentiable at x ∈ Rn. f is said strictly differentiable atx if for all v ∈ Rn:
limy→x,t↓0
f(y + tv)− f(y)
t= lim
y→xf ′(y; v) = f ′(x; v) = v>∇f(x)
MTH8418: Concepts 10/40
Properties of functions Optimization definitions Directions References
Convex sets
I A set in a vector space is convex if for every pair of pointswithin the set, every point on the straight line segment thatjoins them is also within the object:
U convex ⇔ tx+ (1− t)y ∈ U for all t ∈ [0; 1] and (x, y) ∈ U
I If the set U is convex, any convex combination of elements ofU belongs to U :
U convex ⇔p∑i=1
λiui ∈ U with ui ∈ U , λi ≥ 0 for
i ∈ 1, 2, . . . , p, andp∑i=1
λi = 1
MTH8418: Concepts 11/40
Properties of functions Optimization definitions Directions References
Convex functions
I The epigraph of f : Rn → R is the set of points lying aboveits graph:
epi(f) = (x, z) : x ∈ Rn, z ∈ R : f(x) ≤ z ⊆ Rn+1
I f is convex if epi(f) is a convex set
I With n = 1, if f ′′ exists and f ′′(x) ≥ 0 for all x ∈ R, then fis convex
I With convex functions, local optimality = global optimality
MTH8418: Concepts 12/40
Properties of functions Optimization definitions Directions References
Lipschitz functions
I f is Lipschitz on the set X ⊂ Rn if there exists a scalarK > 0 such that
|f(x)− f(y)| ≤ K‖x− y‖ for all x, y ∈ X
I K is called the Lipschitz constant
I Example of non-Lipschitz functions: f(x) =√x for x ≥ 0,
any discontinuous function
MTH8418: Concepts 13/40
Properties of functions Optimization definitions Directions References
Generalized derivatives
I Let f be Lipschitz near x
I The Clarke generalized derivative [Clarke, 1983] of f at x inthe direction v ∈ Rn is
f(x; v) = lim supy→x, t↓0
f(y + tv)− f(y)
t
I If f is convex, the standard directional derivatives and theClarke generalized derivatives are identical
I f is regular at x if for every direction v ∈ Rn, f ′(x; v) existsand equals f(x; v)
MTH8418: Concepts 14/40
Properties of functions Optimization definitions Directions References
Generalized gradient
I The generalized gradient [Clarke, 1983] of f at x is the set
∂f(x) =ξ ∈ Rn : f(x; v) ≥ v>ξ for all v ∈ Rn
I f(x; v) = maxξ∈∂f(x)
v>ξ
I If ∂f(x) reduces to ξ, then f is strictly differentiable at xand ∇f(x) = ξ
MTH8418: Concepts 15/40
Properties of functions Optimization definitions Directions References
Example 3
Consider f(x) = |x|. We have:
I f(0; d) = |d|
I f is regular
I ∂f(0) = [−1; 1]
Note that f(x) = −|x| is not regular: At x = 0, for every direction,the Clarke derivative is > 0 while the function is decreasing
MTH8418: Concepts 16/40
Properties of functions Optimization definitions Directions References
Example 4 (1/2)
f(x) =
x2(2 + sin(πx )
)if x 6= 0
0 if x = 0
−1 −0.5 0 0.5 10
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
y = x2(2+sin(π/x))
x
y
MTH8418: Concepts 17/40
Properties of functions Optimization definitions Directions References
Example 4 (2/2)I Single global optimizer at x = 0I Infinitely many local optima in any open neighborhood of 0I f is differentiable everywhere as
∇f(x) = f ′(x) =
2x(2 + sin(πx )
)− π cos(πx ) if x 6= 0
0 if x = 0
I f is Lipschitz near every x ∈ RI The derivative is not continuous at 0 since ∇f(0) = 0 and∇f(1/2k) = 2/k − π for every integer k 6= 0
I f is not strictly differentiable at x = 0 because the generalizedgradient is ∂f(0) = [−π;π]
I f is not regular at x = 0 since the Clarke generalizedderivatives f(0;±1) = π differ from the standard directionalderivatives f ′(0;±1) = 0
MTH8418: Concepts 18/40
Properties of functions Optimization definitions Directions References
Types of functions: Summaryf : X ⊆ Rn → R is...
Continuous near x Lipschitz near x
Regular at x
Differentiable at x
Continuously differentiable at xConvex on X
Strictly differentiable at x
MTH8418: Concepts 19/40
Properties of functions Optimization definitions Directions References
Properties of functions
Optimization definitions
Directions
References
MTH8418: Concepts 20/40
Properties of functions Optimization definitions Directions References
Optimization problem
We consider the optimization problem:
minx∈Ω
f(x)
I Ω =x ∈ X ⊆ Rn : cj(x) ≤ 0, j ∈ J = 1, 2, . . . ,m
I n variables, m general constraints
I Typically X contains bounds, a-priori constraints,nonquantifiable constraints (defined in Lecture #9)
MTH8418: Concepts 21/40
Properties of functions Optimization definitions Directions References
Some types of problems
I Linear Optimization: f linear, Ω = X , and X contains onlybounds and linear constraints
I Nonlinear Optimization: Functions f and cj , j ∈ J , arenonlinear
I Derivative-Free Optimization: Derivatives are unavailable
I Discrete or combinatorial Optimization: X * Rn and some orall the variables are integers or booleans. Metaheuristicsexploiting the structure of the problem are best fitted
MTH8418: Concepts 22/40
Properties of functions Optimization definitions Directions References
Types of optimality
I x is a feasible solution if x ∈ Ω
I x /∈ Ω⇔ x is infeasible
I x∗ ∈ Ω is a global optimum if f(x∗) ≤ f(x) for all x ∈ Ω.Also noted x∗ ∈ arg min
x∈Ωf(x)
I x∗ ∈ Ω is a local optimum if there exists ε > 0 such thatf(x∗) ≤ f(x) for all x ∈ Ω and ‖x− x∗‖ ≤ ε
I Global optimum = Optimum = Optimizer.In the minimization context: replace “opt” with “min”
I With convexity, local optimality = global optimality
MTH8418: Concepts 23/40
Properties of functions Optimization definitions Directions References
Convergence analysis (1/2)
I An optimization algorithm is not considered as a heuristicwhen it is backed by a convergence analysis which ensuressome properties at the resulting solution x
I This analysis typically depends on some assumptions madeabout the nature of the problem. For example:differentiability of f , convexity of Ω, etc.
I Usually, these properties are given as necessary or sufficientoptimality conditions
I There is global convergence when the properties of the resultare independent of the starting solution(s)
MTH8418: Concepts 24/40
Properties of functions Optimization definitions Directions References
Convergence analysis (2/2)
I In DFO, we expect global convergence to solutions satisfyingsome local and necessary optimality conditions, when thefunction is supposed Lipschitz
I However, a blackbox has no exploitable property and cannotbe proven Lipschitz
I But consider the following choice between two algorithms toapply to such a problem:I Algorithm A is a heuristic. It can give you ∇f(x) 6= 0 when f
is differentiableI Algorithm B guarantees ∇f(x) = 0 when f is differentiable
The choice is obvious
MTH8418: Concepts 25/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Unconstrained case
If all the directional derivatives at x exist, then:
I x is a local minimum ⇒ all the directional derivatives arenon-negative
I If in addition f is differentiable, then f ′(x; d) = 0 for alld ∈ Rn and ∇f(x) = 0 since f ′(x;−d) = −f ′(x; d) for alld ∈ Rn
MTH8418: Concepts 26/40
Properties of functions Optimization definitions Directions References
Active set
I Feasible set:
Ω =x ∈ Rn : cj(x) ≤ 0, j ∈ J
(with X = Rn)
I For any x ∈ Rn, the active set A(x) is the set of indices ofthe constraints satisfied to equality at x:
A(x) = j ∈ J : cj(x) = 0
MTH8418: Concepts 27/40
Properties of functions Optimization definitions Directions References
Cones
I Cones are used to state the properties of the feasible region
I K ⊆ Rn is a cone if λd ∈ K for all scalar λ > 0 and all d ∈ K
I The polar of cone K ⊆ Rn isK∗ = d ∈ Rn : d>v ≤ 0 for all v ∈ K
I If the cj functions, j ∈ J , are differentiable, then the normal
cone is NΩ(x) =
∑j∈A(x)
λj∇cj(x) : λj ≥ 0, j ∈ A(x)
I Polar of normal cone = tangent cone: N∗Ω(x) = TΩ(x)
I Tangent cone ' cone of feasible directions
MTH8418: Concepts 28/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
&%'$.
...............................
............................
.........................
......................
....................
..................
................
..............
........
....
........
...
......................
............. ................ .................. ..................... .......................
.............................
.........................
...........................................
................. .............. ............. ............ ........................
...............
........
........
.
........
........
...
. ........................... .................................................
...........................................
....................
..................
....................
.....................
......................
........................
.........................
...........................
. ...................................... ........................................................................
.................................................................
..............................
.............................
...........................
..........................
..........................
............................
.............................
..............................
................................
.................................
..................................
....................................
. ..............................................................................
......................................
......................................
.....................................
.....................................
....................................
....................................
...................................
.....................................
........................................
..........................................
............................................
..............................................
.................................................
..........................................
..........................................
...........................................
............................................
.............................................
..............................................
...............................................
................................................
.................................................
....................................................
.......................................................
..........................................................
.............................................................
.
..................................................
................................................
..............................................
............................................
..........................................
........................................
......................................
...................................
................................................................
............................. ........................... . ..................... ........................................
....................
....................
.....................
.......................
..........................
............................
...............................
.................................
....................................
......................................
.
.................................................
...............................................
.............................................
...........................................
.........................................
.......................................
........................................................................
................................. ............................... ............................... ..........................................................
..... ............... .............. .............. ..............
...............
...............
..................
.....................
........................
...........................
..............................
.................................
....................................
.......................................
.........................................
............................................
.............................
..............................
..................................................................
.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................
...................
...................
....................
......................
.........................
............................
...............................
.................................
....................................
.......................................
..........................................
............................................
...............................................
5
6
76
5 43
2
1c2(x) = 0
c1(x) = 0
Ω
..........................................
........................................
...............................................................................
............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................
..........................................
............................................
.............................................
..............................................
............................................
.....
......................................
....................................
.................................. ................................. ............................... ............................. ............................ ..........................................................
............................................................
.
....................................
.................................
...............................
.............................
..........................
.............
...........
.............
.........
.....................
.....................
....................
....................
...................
.....................
........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................
............................................................................
....................................................................
...............................
.............................
............................
...........................
MTH8418: Concepts 29/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
&%'$.
...............................
............................
.........................
......................
....................
..................
................
..............
........
....
........
...
......................
............. ................ .................. ..................... .......................
.............................
.........................
...........................................
................. .............. ............. ............ ........................
...............
........
........
.
........
........
...
. ........................... .................................................
...........................................
....................
..................
....................
.....................
......................
........................
.........................
...........................
. ...................................... ........................................................................
.................................................................
..............................
.............................
...........................
..........................
..........................
............................
.............................
..............................
................................
.................................
..................................
....................................
. ..............................................................................
......................................
......................................
.....................................
.....................................
....................................
....................................
...................................
.....................................
........................................
..........................................
............................................
..............................................
.................................................
..........................................
..........................................
...........................................
............................................
.............................................
..............................................
...............................................
................................................
.................................................
....................................................
.......................................................
..........................................................
.............................................................
.
..................................................
................................................
..............................................
............................................
..........................................
........................................
......................................
...................................
................................................................
............................. ........................... . ..................... ........................................
....................
....................
.....................
.......................
..........................
............................
...............................
.................................
....................................
......................................
.
.................................................
...............................................
.............................................
...........................................
.........................................
.......................................
........................................................................
................................. ............................... ............................... ..........................................................
..... ............... .............. .............. ..............
...............
...............
..................
.....................
........................
...........................
..............................
.................................
....................................
.......................................
.........................................
............................................
.............................
..............................
..................................................................
.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................
...................
...................
....................
......................
.........................
............................
...............................
.................................
....................................
.......................................
..........................................
............................................
...............................................
5
6
76
5 43
2
1c2(x) = 0
c1(x) = 0
Ω
..........................................
........................................
...............................................................................
............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................
..........................................
............................................
.............................................
..............................................
............................................
.....
......................................
....................................
.................................. ................................. ............................... ............................. ............................ ..........................................................
............................................................
.
....................................
.................................
...............................
.............................
..........................
.............
...........
.............
.........
.....................
.....................
....................
....................
...................
.....................
........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................
............................................................................
....................................................................
...............................
.............................
............................
...........................
• xa
MTH8418: Concepts 29/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
&%'$.
...............................
............................
.........................
......................
....................
..................
................
..............
........
....
........
...
......................
............. ................ .................. ..................... .......................
.............................
.........................
...........................................
................. .............. ............. ............ ........................
...............
........
........
.
........
........
...
. ........................... .................................................
...........................................
....................
..................
....................
.....................
......................
........................
.........................
...........................
. ...................................... ........................................................................
.................................................................
..............................
.............................
...........................
..........................
..........................
............................
.............................
..............................
................................
.................................
..................................
....................................
. ..............................................................................
......................................
......................................
.....................................
.....................................
....................................
....................................
...................................
.....................................
........................................
..........................................
............................................
..............................................
.................................................
..........................................
..........................................
...........................................
............................................
.............................................
..............................................
...............................................
................................................
.................................................
....................................................
.......................................................
..........................................................
.............................................................
.
..................................................
................................................
..............................................
............................................
..........................................
........................................
......................................
...................................
................................................................
............................. ........................... . ..................... ........................................
....................
....................
.....................
.......................
..........................
............................
...............................
.................................
....................................
......................................
.
.................................................
...............................................
.............................................
...........................................
.........................................
.......................................
........................................................................
................................. ............................... ............................... ..........................................................
..... ............... .............. .............. ..............
...............
...............
..................
.....................
........................
...........................
..............................
.................................
....................................
.......................................
.........................................
............................................
.............................
..............................
..................................................................
.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................
...................
...................
....................
......................
.........................
............................
...............................
.................................
....................................
.......................................
..........................................
............................................
...............................................
5
6
76
5 43
2
1c2(x) = 0
c1(x) = 0
Ω
..........................................
........................................
...............................................................................
............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................
..........................................
............................................
.............................................
..............................................
............................................
.....
......................................
....................................
.................................. ................................. ............................... ............................. ............................ ..........................................................
............................................................
.
....................................
.................................
...............................
.............................
..........................
.............
...........
.............
.........
.....................
.....................
....................
....................
...................
.....................
........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................
............................................................................
....................................................................
...............................
.............................
............................
...........................
• xa∇f(xa)=0
MTH8418: Concepts 29/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
&%'$.
...............................
............................
.........................
......................
....................
..................
................
..............
........
....
........
...
......................
............. ................ .................. ..................... .......................
.............................
.........................
...........................................
................. .............. ............. ............ ........................
...............
........
........
.
........
........
...
. ........................... .................................................
...........................................
....................
..................
....................
.....................
......................
........................
.........................
...........................
. ...................................... ........................................................................
.................................................................
..............................
.............................
...........................
..........................
..........................
............................
.............................
..............................
................................
.................................
..................................
....................................
. ..............................................................................
......................................
......................................
.....................................
.....................................
....................................
....................................
...................................
.....................................
........................................
..........................................
............................................
..............................................
.................................................
..........................................
..........................................
...........................................
............................................
.............................................
..............................................
...............................................
................................................
.................................................
....................................................
.......................................................
..........................................................
.............................................................
.
..................................................
................................................
..............................................
............................................
..........................................
........................................
......................................
...................................
................................................................
............................. ........................... . ..................... ........................................
....................
....................
.....................
.......................
..........................
............................
...............................
.................................
....................................
......................................
.
.................................................
...............................................
.............................................
...........................................
.........................................
.......................................
........................................................................
................................. ............................... ............................... ..........................................................
..... ............... .............. .............. ..............
...............
...............
..................
.....................
........................
...........................
..............................
.................................
....................................
.......................................
.........................................
............................................
.............................
..............................
..................................................................
.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................
...................
...................
....................
......................
.........................
............................
...............................
.................................
....................................
.......................................
..........................................
............................................
...............................................
5
6
76
5 43
2
1c2(x) = 0
c1(x) = 0
Ω
..........................................
........................................
...............................................................................
............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................
..........................................
............................................
.............................................
..............................................
............................................
.....
......................................
....................................
.................................. ................................. ............................... ............................. ............................ ..........................................................
............................................................
.
....................................
.................................
...............................
.............................
..........................
.............
...........
.............
.........
.....................
.....................
....................
....................
...................
.....................
........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................
............................................................................
....................................................................
...............................
.............................
............................
...........................
•xb
MTH8418: Concepts 29/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
&%'$.
...............................
............................
.........................
......................
....................
..................
................
..............
........
....
........
...
......................
............. ................ .................. ..................... .......................
.............................
.........................
...........................................
................. .............. ............. ............ ........................
...............
........
........
.
........
........
...
. ........................... .................................................
...........................................
....................
..................
....................
.....................
......................
........................
.........................
...........................
. ...................................... ........................................................................
.................................................................
..............................
.............................
...........................
..........................
..........................
............................
.............................
..............................
................................
.................................
..................................
....................................
. ..............................................................................
......................................
......................................
.....................................
.....................................
....................................
....................................
...................................
.....................................
........................................
..........................................
............................................
..............................................
.................................................
..........................................
..........................................
...........................................
............................................
.............................................
..............................................
...............................................
................................................
.................................................
....................................................
.......................................................
..........................................................
.............................................................
.
..................................................
................................................
..............................................
............................................
..........................................
........................................
......................................
...................................
................................................................
............................. ........................... . ..................... ........................................
....................
....................
.....................
.......................
..........................
............................
...............................
.................................
....................................
......................................
.
.................................................
...............................................
.............................................
...........................................
.........................................
.......................................
........................................................................
................................. ............................... ............................... ..........................................................
..... ............... .............. .............. ..............
...............
...............
..................
.....................
........................
...........................
..............................
.................................
....................................
.......................................
.........................................
............................................
.............................
..............................
..................................................................
.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................
...................
...................
....................
......................
.........................
............................
...............................
.................................
....................................
.......................................
..........................................
............................................
...............................................
5
6
76
5 43
2
1c2(x) = 0
c1(x) = 0
Ω
..........................................
........................................
...............................................................................
............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................
..........................................
............................................
.............................................
..............................................
............................................
.....
......................................
....................................
.................................. ................................. ............................... ............................. ............................ ..........................................................
............................................................
.
....................................
.................................
...............................
.............................
..........................
.............
...........
.............
.........
.....................
.....................
....................
....................
...................
.....................
........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................
............................................................................
....................................................................
...............................
.............................
............................
...........................
•xb
∇f(xb)=−λ∇c1(xb)
∇c1(xb)
MTH8418: Concepts 29/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
&%'$.
...............................
............................
.........................
......................
....................
..................
................
..............
........
....
........
...
......................
............. ................ .................. ..................... .......................
.............................
.........................
...........................................
................. .............. ............. ............ ........................
...............
........
........
.
........
........
...
. ........................... .................................................
...........................................
....................
..................
....................
.....................
......................
........................
.........................
...........................
. ...................................... ........................................................................
.................................................................
..............................
.............................
...........................
..........................
..........................
............................
.............................
..............................
................................
.................................
..................................
....................................
. ..............................................................................
......................................
......................................
.....................................
.....................................
....................................
....................................
...................................
.....................................
........................................
..........................................
............................................
..............................................
.................................................
..........................................
..........................................
...........................................
............................................
.............................................
..............................................
...............................................
................................................
.................................................
....................................................
.......................................................
..........................................................
.............................................................
.
..................................................
................................................
..............................................
............................................
..........................................
........................................
......................................
...................................
................................................................
............................. ........................... . ..................... ........................................
....................
....................
.....................
.......................
..........................
............................
...............................
.................................
....................................
......................................
.
.................................................
...............................................
.............................................
...........................................
.........................................
.......................................
........................................................................
................................. ............................... ............................... ..........................................................
..... ............... .............. .............. ..............
...............
...............
..................
.....................
........................
...........................
..............................
.................................
....................................
.......................................
.........................................
............................................
.............................
..............................
..................................................................
.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................
...................
...................
....................
......................
.........................
............................
...............................
.................................
....................................
.......................................
..........................................
............................................
...............................................
5
6
76
5 43
2
1c2(x) = 0
c1(x) = 0
Ω
..........................................
........................................
...............................................................................
............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................
..........................................
............................................
.............................................
..............................................
............................................
.....
......................................
....................................
.................................. ................................. ............................... ............................. ............................ ..........................................................
............................................................
.
....................................
.................................
...............................
.............................
..........................
.............
...........
.............
.........
.....................
.....................
....................
....................
...................
.....................
........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................
............................................................................
....................................................................
...............................
.............................
............................
...........................•xc
MTH8418: Concepts 29/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
&%'$.
...............................
............................
.........................
......................
....................
..................
................
..............
........
....
........
...
......................
............. ................ .................. ..................... .......................
.............................
.........................
...........................................
................. .............. ............. ............ ........................
...............
........
........
.
........
........
...
. ........................... .................................................
...........................................
....................
..................
....................
.....................
......................
........................
.........................
...........................
. ...................................... ........................................................................
.................................................................
..............................
.............................
...........................
..........................
..........................
............................
.............................
..............................
................................
.................................
..................................
....................................
. ..............................................................................
......................................
......................................
.....................................
.....................................
....................................
....................................
...................................
.....................................
........................................
..........................................
............................................
..............................................
.................................................
..........................................
..........................................
...........................................
............................................
.............................................
..............................................
...............................................
................................................
.................................................
....................................................
.......................................................
..........................................................
.............................................................
.
..................................................
................................................
..............................................
............................................
..........................................
........................................
......................................
...................................
................................................................
............................. ........................... . ..................... ........................................
....................
....................
.....................
.......................
..........................
............................
...............................
.................................
....................................
......................................
.
.................................................
...............................................
.............................................
...........................................
.........................................
.......................................
........................................................................
................................. ............................... ............................... ..........................................................
..... ............... .............. .............. ..............
...............
...............
..................
.....................
........................
...........................
..............................
.................................
....................................
.......................................
.........................................
............................................
.............................
..............................
..................................................................
.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................
...................
...................
....................
......................
.........................
............................
...............................
.................................
....................................
.......................................
..........................................
............................................
...............................................
5
6
76
5 43
2
1c2(x) = 0
c1(x) = 0
Ω
..........................................
........................................
...............................................................................
............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................
..........................................
............................................
.............................................
..............................................
............................................
.....
......................................
....................................
.................................. ................................. ............................... ............................. ............................ ..........................................................
............................................................
.
....................................
.................................
...............................
.............................
..........................
.............
...........
.............
.........
.....................
.....................
....................
....................
...................
.....................
........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................
............................................................................
....................................................................
...............................
.............................
............................
...........................•xc∇c2(xc)
∇c1(xc)
∇f(xc)= −λ1∇c1(xc)−λ2∇c2(xc)
MTH8418: Concepts 29/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Graphical interpretation
I ∇f(xa) = 0, A(xa) = ∅, NΩ(xa) = ∅, TΩ(xa) = R2
I A(xb) = 1, NΩ(xb) reduces to the half-line in the direction∇c1(xb), TΩ(xb) union of 0 and the open half-spaceorthogonal to ∇c1(xb)
I A(xc) = 1, 2,NΩ(xc) = λ1∇c1(xc) + λ2∇c2(xc) : λ1, λ2 ≥ 0
MTH8418: Concepts 30/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Differentiable case
If x ∈ Ω is a local minimizer of a differentiable function f subjectto differentiable constraints, then
−∇f(x) ∈ NΩ(x)
andf ′(x; d) ≥ 0 ∀ d ∈ TΩ(x)
where
f ′(x; d) = limt→0
f(x+ td)− f(x)
t= d>∇f(x)
MTH8418: Concepts 31/40
Properties of functions Optimization definitions Directions References
Optimality conditions: Generalization
If x ∈ Ω is a local minimizer of a Lipschitz function f over a setΩ ⊂ Rn, then
f(x; d) ≥ 0 ∀ d ∈ THΩ (x)
where
I f(x; d) is the generalized directional derivative
I THΩ (x) is the hypertangent cone, a generalization of thetangent cone
MTH8418: Concepts 32/40
Properties of functions Optimization definitions Directions References
Properties of functions
Optimization definitions
Directions
References
MTH8418: Concepts 33/40
Properties of functions Optimization definitions Directions References
Descent directions
Let f : Rn → R be a differentiable function at x
I A descent direction from x is a nonzero vector v ∈ Rn suchthat f ′(x; v) ≤ 0
I A strict descent direction from x is a nonzero vector v ∈ Rnsuch that f ′(x; v) < 0
I If ∇f(x) 6= 0:
I There is a closed half space H ⊆ Rn such that f ′(x; v) ≥ 0 ifand only if v ∈ H
I For any direction v ∈ Rn, either v or −v is a descent direction
MTH8418: Concepts 34/40
Properties of functions Optimization definitions Directions References
Positive spanning sets and bases
I A positive spanning set (or generating set) for Rn is a finiteset of vectors whose nonnegative linear combinations span Rn
I A positive basis is a positive spanning set such that no propersubset is a positive spanning set
I A basis of Rn is not a positive basis for Rn
I Positive spanning sets contain at least one element in everyopen half-space
I If f is differentiable at a non-stationary point x, there is atleast one element of any positive spanning set that is a strictdescent direction
MTH8418: Concepts 35/40
Properties of functions Optimization definitions Directions References
Example 5
6
HHHj
(a)
6
HHHj
(b)
6
HHHj
?
HHHY
(c)
6
HHHj
?
HHHY
(d)
A basis, two positive bases and a positive spanning set of R2
MTH8418: Concepts 36/40
Properties of functions Optimization definitions Directions References
Cosine measure
I D: set of directions in Rn
I The cosine measure [Kolda et al., 2003] is defined by
κ(D) = minv∈Rn
maxd∈D
v>d
‖v‖‖d‖= min
v∈Rnmaxd∈D
cos〈v, d〉
I It is the cosine of the largest angle between an arbitrary vectorv and the closest direction in D
I It identifies the “largest hole” in the directions
I If D is a positive spanning set, κ(D) > 0
I Sets with higher κ span the space better
MTH8418: Concepts 37/40
Properties of functions Optimization definitions Directions References
Example 6 (cosine measure)
With ei the ith coordinate vector for i ∈ 1, 2, . . . , n, give thevalue of κ(D) for the following sets of directions with n = 2:
1. D = e1
2. D = e1, e2
3. D = e1, e2,−e2
4. D = e1, e2,−e1,−e2
5. D = e1, e2,−e1 − e2
MTH8418: Concepts 38/40
Properties of functions Optimization definitions Directions References
Properties of functions
Optimization definitions
Directions
References
MTH8418: Concepts 39/40
Properties of functions Optimization definitions Directions References
References I
Audet, C. and Hare, W. (2017).
Derivative-Free and Blackbox Optimization.Springer Series in Operations Research and Financial Engineering. Springer International Publishing, Berlin.
Clarke, F. (1983).
Optimization and Nonsmooth Analysis.John Wiley & Sons, New York.Reissued in 1990 by SIAM Publications, Philadelphia, as Vol. 5 in the series Classics in Applied Mathematics.
Kolda, T., Lewis, R., and Torczon, V. (2003).
Optimization by direct search: New perspectives on some classical and modern methods.SIAM Review, 45(3):385–482.
Nocedal, J. and Wright, S. (2006).
Numerical Optimization.Springer Series in Operations Research and Financial Engineering. Springer, Berlin, second edition.
MTH8418: Concepts 40/40