Page 1
Optimization Theory MMC 52212 / MME 52106
by
Dr. Shibayan Sarkar Department of Mechanical Engg. Indian School of Mines Dhanbad
Multivariable Optimization
[Unconstrained]
Page 2
When? Multiple decision variable Gradient of a function is not a scalar quantity. It is a vector. Optimality criteria can be derived by using the definition of a local optimal point
and by using Taylor’s series expansion of a multivariable function. unidirectional method Direct search Gradient based method
The objective function of a N variable is represented as x1, x2 ….xN. The gradient vector is represented as
( )
( )
1 2
( ) , ......t
Tt
Nx
f f ff xx x x
∂ ∂ ∂∇ = ∂ ∂ ∂
( )
2 2 2
21 1 2 12 2 2
2 ( ) 21 2 2 2
2 2 2
21 2
......
......( )
...........
......t
T
N
tN
N N Nx
f f fx x x x x
f f ff x x x x x x
f f fx x x x x
∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂
∇ = ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂
Hessian Matrix A point is a stationary point if .
Furthermore the point is minimum , maximum and inflation point if is positive-definite, negative-definite or otherwise.
( ) 0f x∇ =x
2 ( ) 0f x∇ =
Page 3
Page 4
Page 5
Unidirectional Method Find minimum point at a particular direction One dimensional search.
Page 6
x1 x2 f consider 0 0 200 200 0 1 181 180 0 2 164 160 0 3 149 150 0 4 136 140 0 5 125 130 1 0 181 180 1 1 162 160 1 2 145 150 1 3 130 130 1 4 117 120 1 5 106 110 2 0 164 160 2 1 145 150 2 2 128 130 2 3 113 110 2 4 100 100 2 5 89 90 3 0 149 150 3 1 130 130 3 2 113 110 3 3 98 100 3 4 85 90 3 5 74 70 4 0 136 140 4 1 117 120 4 2 100 100 4 3 85 90 4 4 72 70 4 5 61 60 5 0 125 130 5 1 106 110 5 2 89 90 5 3 74 70 5 4 61 60 5 5 50 50
Contour line for 90 Contour line for 130
Page 7
Unidirectional Method: Example 1
(1) ( )
( )
( ) t
tx x
sα α−
=
1 2
1 2
( ) (2,1) 0.5(2,5)
( ) (2) ( ) (1)0.5 0.5(2) (5)
( ) 3.0 ( ) 0.5( ) (3.0,3.5)
x
x xand
x and xx
α
α α
α αα
−=
− −= =
= ==
a
b
2
5
Graphical method
Page 8
Unidirectional Method: Example 1
(1)
1 2
1 2
( ) (2,1) 0.5(2,5)
( ) (2) ( ) (1)0.5 0.5(2) (5)
( ) 3.0 ( ) 0.5( ) (3.0,3.5)
x
x xand
x and xx
α
α α
α αα
−=
− −= =
= ==
a
b
2
5
1 2
( ) (2,1) (2,5)( ) 2 2 ( ) 1 5
xx and xα αα α α α= += + = +
(1) (2)
From equitation (2) ….
(3)
Substitute the value of eq(1) by eq(3) …. 2 2 2( ) (2 2 10) (1 5 10) 29 122 145Min f α α α α α= + − + + − = − +
2( ) 0 58 122 2.1034dfdα α αα
= = − => =
Page 9
Gradient Search Method: Box Evolutionary 2 it is a geometrical figure in four or more dimensions which is analogous to a cube in three dimensions.
In geometry, a hypercube is an n-dimensional analogue of a square (n = 2) and a cube (n = 3). It is a closed, compact, convex figure whose 1-skeleton consists of groups of opposite parallel line segments aligned in each of the space's dimensions, perpendicular to each other and of the same length. A unit hypercube's longest diagonal in n-dimensions is equal to √n.
2 2i i∆ = ∆ + ∆
Page 10
Box Evolutionary : Example 1
Page 11
Box Evolutionary : Example 1
Corresponding point is
2 22 2 2.828∆ = + =
Next fig shows
Page 12
Box Evolutionary : Example 1
2 22 2 2.828∆ = + =
Page 13
Box Evolutionary : Example 1
2 21 1 1.414∆ = + =
( )0x x=
Page 14
Box Evolutionary : Example 1
2 21 1 1.414∆ = + =
Page 15
Simplex Search Method In the simplex search method the number of points in the initial simplex is much less compared to Box Evolutionary method. With N variable (N+1) number of variables are used in the initial simplex. In each iteration worst point (xh) in the simplex is found first. Then a new simplex with xnew is formed. Four different situation may arise depend on function value.
First, the centroid (xc) of all but worst point is determined.
Thereafter, the worst point in the simplex is reflected about the xc and a new point xr is found.
if
Page 16
Simplex Search Method Usually γ ≈ 2.0, β = 0.5
If function value at reflected point is worse than the worst point in the simplex, a contraction is made with β is negative
If function value at reflected point is better than the worst and worse than the next to worst point in the simplex, a contraction is made with β is positive
centroid (xc) of all but worst point .
f (xg) f (xh) f (xl)
f (x)
β +ve β -ve γ +ve
Even though the reflected point (xr) is better than the new point (xnew), the basic simplex search algorithm does not allow this point in the new simplex.
Page 17
Simplex Search Method : Example 1
Page 18
Simplex Search Method : Example 1 1 221
1
( ( ) ( ))1
Ni c
i
f x f xQN
+
=
−= + ∑
1 2(1) 2
(2) 2 (3) 2
( ( ) ( ))3
( ( ) ( )) ( ( ) ( ))3 3
c
c c
f x f x
f x f x f x f x
− =
− − + +
f(x(1))=21.4 ; f(x(2))=74; f(x(3))=106, f(xc)=95.6 =>Q=45.01
18.395
40.40
Page 19
Gradient Based Method: Cauchy’s steepest method 1
2
f xf
f x∂ ∂
∇ = ∂ ∂ Step 1:
Step 2:
Step 3:
Step 4:
Step 5:
1 1( )f f X∇ =∇
11
2
00
xX
x
= =
1 1( )S f X= −∇
To find X2, optimal step length α1* is required to be find out, therefore minimize f(X1+α1S1) with respect to α1. As df1/dα1=0, α1=α1* and we get
*2 1 1 1X X Sα= +
If , Terminate or proceed to further iteration to get X3 and corresponding value of In new iteration ........
2 2( ) 0f f X∇ =∇ =
3f∇
2 2( )S f X= −∇
To find X2, optimal step length α2* is required to be find out, therefore minimize f(X2+α2S2) with respect to α2. As df2/dα2=0, α2=α2* and we get
*3 2 2 2X X Sα= +
Convergence limit:
11
( ) ( )( )
i i
i
f X f Xf X
ε+ −≤ 2
i
fx
ε∂≤
∂ 1 3i iX X ε+ − ≤
Page 20
Gradient Based Method: Cauchy’s steepest method
1 1
1( )
1S f X
− = −∇ =
Page 21
Gradient Based Method: Cauchy’s steepest method
Page 22
Gradient Based Method: Cauchy’s steepest method
Page 23
Gradient Based Method: Cauchy’s steepest method
Page 24
Alternative steps for Cauchy’s Steepest descent
Cauchy’s method works well when x(0)
is far away from x*. When the current
point is very close to the minimum, the change in the gradient vector is small. Thus the new point
created by the unidirectional search is also close to the current point.
( ) ( )( )( )
( ) ( ) ( ) ( ) ( )( ) 2t
t t t t ti i i i i
i x
f x f x x f x x xx
∂= + ∆ − −∆ ∆
∂
( )
( )
1 2
( ) , ......t
Tt
Nx
f f ff xx x x
∂ ∂ ∂∇ = ∂ ∂ ∂
Analytically Numerically by central difference scheme
Page 25
Cauchy’s Steepest descent: Example 1 ∂f/∂x1 = 2*x1 + 4*x1*(x1^2 + x2 - 11) + 2*x2^2 - 14 ∂f/∂x2= 2*x2 + 4*x2*(x2^2 + x1 - 7) + 2*x1^2 - 22
( )
( )
( ) t
tx x
sα α−
=
(1.788,2.840) (0,0) 0.127(2,5)
T T
T α−= =
Slope 14:22
3,2
1.788, 2.810
By geometry
Page 26
Cauchy’s Steepest descent: Example 1
For golden search method please see next slide ............
Page 27
SVO: Golden Search Method
Page 28
SVO: Golden Search Method : Example 1
Page 29
SVO: Golden Search Method : Example 1
Page 30
SVO: Golden Search Method : Example 1