Date post: | 18-Dec-2015 |
Category: |
Documents |
Upload: | kimberly-tyler |
View: | 232 times |
Download: | 5 times |
Chapter 3 UNCONSTRAINED
OPTIMIZATION
Prof. S.S. Jang
Department of Chemical Engineering
National Tsing-Hua Univeristy
Contents
Optimality Criteria Direct Search Methods
• Nelder and Mead (Simplex Search)• Hook and Jeeves(Pattern Search)• Powell’s Method (The Conjugated Direction Search)
Gradient Based Methods• Cauchy’s method (the Steepest Decent) • Newton’s Method • Conjugate Gradient Method (The Fletcher and
Reese Method)• Quasi-Newton Method – Variable Metric Method
3-1 The Optimality Criteria
Given a function (objective function) f(x), where and let
Stationary Condition:
Sufficient minimum criteria positive definite
Sufficient maximum criteria negative definite
Nx
x
x
2
1
x
xxxxxxxxxxx 32 ***
2
1*** ffff TT
0* xf
*2 xf
*2 xf
Definiteness
A real quadratic for Q=xTCx and its symmetric matrix C=[cjk] are said to be positive definite (p.d.) if Q>0 for all x≠0. A necessary and sufficient condition for p.d. is that all the determinants:
11 12 1311 12
1 11 2 3 21 22 2321 22
31 32 33
, , ..., detn
c c cc c
C c C C c c c C Cc c
c c c
are positive, C is negative definitive (n.d.) if –C is p.d.
Example :
x=linspace(-3,3);y=x;>> for i=1:100for j=1:100z(i,j)=2*x(i)*x(i)+4*x(i)*y(j)-10*x(i)*y(j)^3+y(j)^2;endendmesh(x,y,z)contour(x,y,z,100)
22
32121
21 1042 xxxxxxxf
Example-Continued
0
0*x
At
01044 22
211
xxxx
f*xx
101012
2
4
22
21
2
22
2
21
2
xxx
f
x
f
x
f
*xx
*xx
*xx
210
104H Indefinite, x* is a non-optimum point.
Mesh and Contour – Himmelblau’s function : f(x)=(x1
2+x2-11)2+(x1+x22-7)2
Optimality Criteria may find a local minimum in this casex=linspace(-5,5);y=x; for i=1:100for j=1:100z(i,j)=(x(i)^2+y(j)-11)^2+(x(i)+y(j)^2-7)^2;endendmesh(x,y,z)contour(x,y,z,80)
Mesh and Contour - Continued
-5
0
5
-5
0
50
200
400
600
800
1000
-5 -4 -3 -2 -1 0 1 2 3 4 5-5
-4
-3
-2
-1
0
1
2
3
4
5
Example: f(x)=2x13+4x1x2
3-10x1x2+x23
Steepest decent search from the initial point at (2,2); we have d=[6x1
2+4x23-10x2, 12x1x2
2-10x1+3x22]’=
Example: Curve Fitting
From the theoretical considerations, it is believe that the dependent variable y is related to variable x via a two-parameter function:
xk
xky
2
1
1
The parameters k1 and k2 are to be determined by a least square of the following experimental data:x y1.0 1.052.0 1.253.0 1.554.0 1.59
Problem Formulation and mesh, contour
2
2
1
2
2
1
2
2
1
2
2
1
, 41
459.1
31
355.1
21
225.1
105.1min
21
k
k
k
k
k
k
k
kkk
The Importance of One-dimensional Problem - The Iterative Optimization Procedure
Consider a objectivefunction Min f(X)=x1
2+x22, with an initial
point X0=(-4,-1) anda direction (1,0), whatis the optimum at thisdirection, i.e. X1=X0
+*(1,0). This is a one-dimensional search for.
3-2 Steepest Decent Approach – Cauchy’s Method
A direction d=[x1,x2,..,xn]’ is a n-dimensional vector, a line search is an approach such that from a based point x0, find *, for all , x0+ *d is an optimum.
Theorem: the direction d=-[f/x1,f/x2, .., f/xn]’ is the steepest decent direction.
Proof: Consider a two dimensional system: f(x1,x2), the movementds2=dx1
2+dx22, and f= (f/x1)dx1+ (f/x2)dx2=
(f/x1)dx1+ (f/x2)(ds2-dx12)1/2. Then to maximize the change f,
we have d(f)/dx1=0. It can be shown that dx1/ds= (f/x1)/((f/x1)2
+ (f/x2)2)1/2, dx2/ds= (f/x2)/((f/x1)2+ (f/x2)2)1/2
Example: f(x)=2x13+4x1x2
3-10x1x2+x23
Steepest decent search from the initial point at (2,2); we have d=[6x1
2+4x23-10x2, 12x1x2
2-10x1+3x22]’=
Quadratic Function PropertiesProperty #1: Optimal linear search for a quadratic function:
kTk
kkT
kTkkk
T
kk
kT
kkkT
k
Hss
sf
Hsssfd
df
sxx
xxHxxxxfxfxf
0
Let 2
1)()(
Quadratic Function Properties- Continued
Property #2 fk+1 is orthogonal to sk for a quadratic function
0
0
0
)( 0 Since
or 2
1)(Let
1
1
1
1
kTk
kkTkk
Tk
kkTkk
Tk
kkkTT
kk
TT
fs
bfbfsfs
xxHsfs
xxssHsfs
bfHxHxbf
Hxxbxaxf
3-2 Newton’s Method – Best Convergence approaching the optimum
kkk
k
kkk
kkT
kk
kk
kT
kkkT
k
fHxx
fHd
fHxx
xxHfxfxfx
d
xxΔx
xxHxxxxfxfxf
11
1
11
*
Define
0Want
;Let 2
1)(
3-2 Newton’s Method – Best Convergence approaching the optimum
1 12 2 31 1 2 1 2 2
2 2 22 1 2 2 1 2 1 2
24 38 36 0.473912 12 10 6x +4x -10x
38 108 88 0.648112 10 24 6 12x x -10x +3x
x xd H f
x x x x
Remarks
Newton’s method is much more efficient than Steep Decent approach. Especially, the starting point is getting close to the optimum.
There is a requirement for the positive definite of the hessian in implementing Newton’s method. Otherwise, the search will be diverse.
The evaluation of hessian requires second derivative of the objective function, thus, the number of function evaluation will be drastically increased if a numerical derivative is used.
Steepest decent is very inefficient around the optimum, but very efficient at the very beginning of the search.
Conclusion
Cauchy’s method is very seldom to use because of its inefficiency around the optimum.Newton’s method is also very seldom to
use because of the requirement of second derivative.A useful approach should be somewhere
in between.
3-3 Method of Powell’s Conjugate Direction
Definition: Let u and v denote two vectors in En. They are said to be orthogonal, if their scalar product equals to zero, i.e. uv=0. Similarly, u and v are said to be mutually conjugative with respect to a matrix A, if u and Av are mutually orthogonal, i.e. uAv=0. Theorem: If conjugate directions to the hessian
are used to any quadratic function of n variables that has a minimum, can be minimized in n steps.
Powell’s Conjugated Direction Method-Continued
Corollary: Generation of conjugated vectors-Extended Parallel Space PropertyGiven a direction d and two initial point x(1), x(2), we
assume:
Hdyy
Hdyy
dHybd
dq
yx
dHybd
dqy
dHxbd
dx
dx
dq
d
dq
dxx
xqHxxbxaxf T
w.r.t. toconjugated is ''
0 '' yields (2)-(1)
(1) )''(0
such that get fromstart approach, Same
(1) )''(0at Assume,
)''(
then
)(2
1)(
21
21
2
2)2(
11
)1(
Powell’s Algorithm
Step 1: Define x0, and set N linearly indep. Directions say:s(i)=e(i)Step 2: Perfrom N+1 one directional searches
along with s(N), s(1),,s(N)Step 3: Form the conjugate direction d using the
extended parallel space property.Step 4: Set Go to Step 2
d )()()1()3()2()2()1( ,,,, NNN sssssss
Powell’s Algorithm (MATLAB) function opt=powell_n_dim(x0,n,eps,tol,opt_func) xx=x0;obj_old=feval(opt_func,x0);s=zeros(n,n);obj=obj_old;df=100;absdx=100; for i=1:n;s(i,i)=1;end while df>eps/absdx>tol x_old=xx; for i=1:n+1 if(i==n+1) j=1; else j=i; end ss=s(:,j); alopt=one_dim_pw(xx,ss',opt_func); xx=xx+alopt*ss'; if(i==1) y1=xx; end if(i==n+1) y2=xx; end end d=y2-y1; nn=norm(d,'fro'); for i=1:n d(i)=d(i)/nn; end dd=d'; alopt=one_dim_pw(xx,dd',opt_func); xx=xx+alopt*d dx=xx-x_old; plot(xx(1),xx(2),'ro') absdx=norm(dx,'fro'); obj=feval(opt_func,xx); df=abs(obj-obj_old); obj_old=obj; for i=1:n-1 s(:,i)=s(:,i+1); end s(:,n)=dd; end opt=xx
Function Fminsearch >> help fminsearch
FMINSEARCH Multidimensional unconstrained nonlinear minimization (Nelder-Mead). X = FMINSEARCH(FUN,X0) starts at X0 and finds a local minimizer X of the function FUN. FUN accepts input X and returns a scalar function value F evaluated at X. X0 can be a scalar, vector or matrix. X = FMINSEARCH(FUN,X0,OPTIONS) minimizes with the default optimization parameters replaced by values in the structure OPTIONS, created with the OPTIMSET function. See OPTIMSET for details. FMINSEARCH uses these options: Display, TolX, TolFun, MaxFunEvals, and MaxIter. X = FMINSEARCH(FUN,X0,OPTIONS,P1,P2,...) provides for additional arguments which are passed to the objective function, F=feval(FUN,X,P1,P2,...). Pass an empty matrix for OPTIONS to use the default values. (Use OPTIONS = [] as a place holder if no options are set.) [X,FVAL]= FMINSEARCH(...) returns the value of the objective function, described in FUN, at X. [X,FVAL,EXITFLAG] = FMINSEARCH(...) returns a string EXITFLAG that describes the exit condition of FMINSEARCH. If EXITFLAG is: 1 then FMINSEARCH converged with a solution X. 0 then the maximum number of iterations was reached. [X,FVAL,EXITFLAG,OUTPUT] = FMINSEARCH(...) returns a structure OUTPUT with the number of iterations taken in OUTPUT.iterations. Examples FUN can be specified using @: X = fminsearch(@sin,3) finds a minimum of the SIN function near 3. In this case, SIN is a function that returns a scalar function value SIN evaluated at X. FUN can also be an inline object: X = fminsearch(inline('norm(x)'),[1;2;3]) returns a minimum near [0;0;0]. FMINSEARCH uses the Nelder-Mead simplex (direct search) method. See also OPTIMSET, FMINBND, @, INLINE.
Pipeline design
)/()(
39.3
)/($)1()10)(722()10)(57.6(
3690045000084.7),,,(
2/1522
2
219.066
12
1
1 hrSCFLf
DPPQ
yrrLL
DPDrLPDC
…
L
P1P2
D
Q
3-4 Conjugated Gradient Method Fletcher-Reeves’s Method (FRP)
Given s1=-f1, s2=-f2+s1
It can be shown that if
where, s2 is started at the optimum of the line search along the direction s1. And fa+bTxk+1/2xk
THxk, then s2 is conjugated to s1 with respect to HMost general iteration:
11
22
ff
ffT
T
)1(2)1(
2)(
)()(
k
k
k
kk sg
ggs
3-5 Quasi-Newton’s Method (Variable Matrix Method)
The variable matric approach is using an approximate Hessian matrix without taking second derivative of the objective function, the Newton’s direction can be approximated by: xk+1=xk-kAkfk
As Ak=I Cauchy’s directionAs Ak=H-1 Newton’s directionIn the variable matric approach, Ak can be
updated to approximate the inverse of the Hessian by iterative approaches
3-5 Quasi-Newton’s Method (Variable Matrix Method)- Continued
Given the objective function f(x)a+bx+ 1/2xTHx, then f b+Hx, and
fk+1- f kH(xk+1- xk ), or xk+1- xk= H-1gk
where gk = fk+1- f k
Let Ak+1=H-1 = Ak+ Ak xk+1- xk= (Ak+ Ak )gk
Ak gk = xk- Ak gk
Finally, k
T
Tkk
kT
Tk
k gz
zgA
gy
yxA
Quasi-Newton’s Method – Continued
Choose y= xk, z= Ak gk (The Davidon-Flecher-Powell’s method
Broyden-Flecher-Shanno’s Method
Summary:In the first run, given initial conditions and convergence criteria, and
evaluate the gradientFirst direction is on the negative of the gradient, Ak =I
In the iteration phase, evaluate the new gradient and find gk, xk, and solve Ak using the equation in the previous slide
The new direction is on sk+1=Ak+1f k+1
Check convergence, and continue the iteration phase
)()(
)()(
)()(
)()(
)()(
)()()1(
kTk
Tkk
kTk
Tkkk
kTk
Tkkk
gx
xx
gx
gxIA
gx
gxIA
Heuristic Approaches (1) s-2 Simplex ApproachMain Idea
Rules1: Straddled: Selected the “worse” vertex generated in the
previous iteration, then choose instead the vertex with the next highest function value2. Cycling: In a given vertex unchanged more than M
iterations, reduce the size by some factor3. Termination: The search is terminated when the simlex
gets small enough.
Base Point
Trial Point
Simplex Approach: Nelder and Mead
xnew
xc
xnew
X(h) xc
xnew
X(h)
z
xc
xnew
X(h) z
xnew
xcX(h)
z
)()( )(
)1(ReflectionNormal)(g
newl fxff
a
)()(
)1(E)(l
new fxf
xpansionb
)(
)(
)(
)(
)(C)(
hnew
gnew
fxf
andfxf
Oontractionc
)()( )(
)(C)(h
newg fxff
Oontractiond
Comparisons of Different Approach for Solving Rosenbrock functionMethod subprogram Optimum found Number of
iterationsCost function value
Number of function called
Cauchy’s Steepest_decent_rosenbrock
0.9809 0.9622 537 3.6379e-004 11021
Newton’s Newton_method_rosenbrock
0.9993 0.9985 9 1.8443e-006 N/A
Powell’s Pw_rosenbrock_main 1.0000 0.9999 10 3.7942e-009 799
Fletcher-Reeves
Frp_rosenbrock_main 0.9906 0.9813 8 8.7997e-005 155
Broyden_Fletcher_Shanno
Bfs_rosenbrock_main 1.0000 1.0001 17 1.0128e-004 388
Nelder/Mead fminsearch 1.0000 1.0000 N/A 3.7222e-010 103
Heuristic Approach (Reduction Sampling with Random Search)
Step 1: Given initial sampling radius and initial center of the generate a random N points within the region.Step 2: Get the best of the N sampling as
the new center of the circle.Step 3: Reduce the radius of the search
circle, if the radius is not small enough, go back to step 2, otherwise stop.