Date post: | 22-Feb-2018 |
Category: |
Documents |
Upload: | carolinarvsocn |
View: | 216 times |
Download: | 0 times |
of 78
7/24/2019 OD Nonlinear Programming LARGE 2010
1/78
NONLINEARPROGRAMMING
7/24/2019 OD Nonlinear Programming LARGE 2010
2/78
Nonlinear Programming
Linear programming has a fundamental role in OR.
In linear programmingall it s funct ions(objective
function and constraint functions) are linear.
This assumption frequently does not hold, and
nonlinear programming problems are formulated:
Findx = (x1,x2,...,xn) to
Maximizef(x)
subject to
gi(x) bi , for i = 1, 2, ...,mandx0
331
7/24/2019 OD Nonlinear Programming LARGE 2010
3/78
Nonlinear Programming
There are many types of nonlinear programming
problems, depending onf(x) andgi(x). Different algorithms are used for different types.
Some problems can be solved very efficiently, whilst
others, even small, can be very difficult.
Nonlinear programming is a particularly large subject
(all the animals that are not elephants).
Only some important types will be dealt with here.
332
7/24/2019 OD Nonlinear Programming LARGE 2010
4/78
Application: product-mix problem
Inproduct-mixproblems (as Wyndor Glass Co.) the
goal is to determine optimal mix of production levels. Sometimesprice elast icit yis present: the amount of
sold product has an inverse relation to price charged:
333
7/24/2019 OD Nonlinear Programming LARGE 2010
5/78
Price elasticity
p(x) is the price required to sellx units.
c is the unit cost for producing and distributingproduct.
Profit from producing and sellingx is:
P(x) =xp(x) cx
334
7/24/2019 OD Nonlinear Programming LARGE 2010
6/78
Product-mix problem
If each product has a similar profit function, overall
objective function is
Other nonlinearity:marginal costvary with productionlevel.
It may decrease when production level is increased due
to thelearning-curve effect.
It may increase due to overtime or more expensiveproduction facilities when production increase.
335
1
( ) ( )n
j j
j
f P x
x
7/24/2019 OD Nonlinear Programming LARGE 2010
7/78
Application: transportation problem
Determine optimal plan for shipping goods from
various sources to various destinations (see P&TCompany problem).
Cost per unit shippedmay not be fixed.Volume
discountsare sometimes available for large shipments.
Marginal costcan have a pattern like in the figure.
Cost of shippingx units is a piecewise linear function
C(x), with slope equal to the marginal cost.
336
7/24/2019 OD Nonlinear Programming LARGE 2010
8/78
Volume discounts on shipping costs
Marginal cost Cost of shipping
337
7/24/2019 OD Nonlinear Programming LARGE 2010
9/78
Transportation problem
If each combination of source and destination has a
similar shipping cost function, so that cost of shippingxij units from sourcei (i = 1, 2, ...,m)
to destinationj (j = 1, 2, ...,n) is given by a nonlinear
functionCij(xij).
The overall objective function is
338
1 1
Minimize ( ) ( )m n
ij ij
i j
f C x
x
7/24/2019 OD Nonlinear Programming LARGE 2010
10/78
Graphical illustration
Example: Wyndor Glass Co. problem with constraint
339
2 2
1 29 5 216x x
7/24/2019 OD Nonlinear Programming LARGE 2010
11/78
Graphical illustration
Example: Wyndor Glass Co. problem with objective
function
340
2 21 1 2 2126 9 128 13Z x x x x
7/24/2019 OD Nonlinear Programming LARGE 2010
12/78
Example: Wyndor Glass Co. (3)
341
7/24/2019 OD Nonlinear Programming LARGE 2010
13/78
Global and local optimum
Example:f(x) with three local maxima (x = 0, 2,4), and
three local minima (x = 1, 3,5). Global?
342
7/24/2019 OD Nonlinear Programming LARGE 2010
14/78
Guaranteed local maximum
Global maximum when:
Function always curving downward is a concavefunction (concave downward).
Function always curving upward is a convex function
(concave upward).
343
2
2
( )0, for all
f x
x
7/24/2019 OD Nonlinear Programming LARGE 2010
15/78
Guaranteed local optimum
Nonlinear programming with no constraints and
concaveobjective function, a local maximum is theglobal maximum.
Nonlinear programming with no constraints and
convexobjective function, a local minimum is the
global minimum.
With constraints, this guarantee still holds if the
feasible regionis a convex set.
344
7/24/2019 OD Nonlinear Programming LARGE 2010
16/78
Ex: Wyndor Glass with one concavegi(x)
345
7/24/2019 OD Nonlinear Programming LARGE 2010
17/78
Types of NP problems
Unconstrained Optimization: no constraints
necessarycondition for a solutionx* =x to be optimal:
whenf(x) is aconcavefunction this condition is
sufficient.
whenxj has a constraintxj0, sufficient condition
changes to:
346
Maximize ( )f x
*( ) 0 at , for 1, 2, ,j
fj n
x
xx x
* *
* *
0 at , if 0( )
0 at , if 0
j
jj
xf
xx
x xx
x x
7/24/2019 OD Nonlinear Programming LARGE 2010
18/78
Example: nonnegative constraint
347
7/24/2019 OD Nonlinear Programming LARGE 2010
19/78
Types of NP problems
Linearly Constrained Optimization
All constraints are linear and objective function is nonlinear.
Special case: Quadratic Programming
Objective function isquadratic.
Many applications, e.g. portfolio selection, predictive control
Convex Programming assumptions for maximization
f(x) is aconcavefunction.
Eachgi(x) is aconvexfunction.
For a minimization problem,f(x) must be a convex function.
348
7/24/2019 OD Nonlinear Programming LARGE 2010
20/78
Types of NP problems
Separable Programming convex programming where
f(x) andgi(x) are separable functions.
A separable function is a function where each term
involves only a single variable:
Nonconvex Programming: local optimum is not
assured to be a global optimum.
349
1( ) ( )
n
j j
jf f x
x
7/24/2019 OD Nonlinear Programming LARGE 2010
21/78
Types of NP problems
Geometric Programming is applied to engineering
design as well as economics and statistics problems. Objective function and constraint functions are of the
form:
ci andaij are typically physical constraints.
When they are all strictly positive, functions are
generalized positive polynomials (posynomials), and a
convex programming algorithm can be applied.
350
1 2
1 21( ) ( ), where ( )
i i in
Na a a
i i i nig c P P x x x x x x
7/24/2019 OD Nonlinear Programming LARGE 2010
22/78
Types of NP problems
Fractional Programming
maximizesf1(x) and minimizesf2(x).
whenf1(x) andf2(x) are linear:
can be transformed into a linear programming problem.
351
0
0
( ) c
fd
cxx
dx
1
2
( )Maximize ( )
( )
ff
f xx
x
7/24/2019 OD Nonlinear Programming LARGE 2010
23/78
One-variable unconstrained optimization
Methods for solving unconstrained optimization with
only one variable (n = 1), where the differentiablefunctionf(x) isconcave.
Necessary and sufficientcondition for optimum:
352
*( ) 0 at .f x
x xx
l i h i i i bl
7/24/2019 OD Nonlinear Programming LARGE 2010
24/78
Solving the optimization problem
Iff(x) is not simple, problem cannot be solved
analytically. If not, search procedures can solve the problem
numerically.
We will describe two methods:
Bisection method
Newtons method
353
i i h d
7/24/2019 OD Nonlinear Programming LARGE 2010
25/78
Bisection method
We know that:
If derivative ofxisposit ive,x is alower boundofx*.
If derivative ofxisnegative,x is anupper boundofx*
.
354
*( ) 0 if ,df x
x xdx
*( ) 0 if ,df x
x xdx
*( ) 0 if .df x
x xdx
Bi i h d
7/24/2019 OD Nonlinear Programming LARGE 2010
26/78
Bisection method
Having:
In the bisection method, new trial solution is the
midpont between the two current bounds.
355
*
*
*
= current trial solution,
= current lower bound on ,
= current upper bound on ,
= error tolerance for .
x
x x
x x
x
Al i h f h Bi i M h d
7/24/2019 OD Nonlinear Programming LARGE 2010
27/78
Algorithm of the Bisection Method
Initialization:Select. Find initial upper and lower
bounds. Select initial trial as: Iteration:
1. Evaluate
2. If
3. If
4. Select a new
Stopping rule:If stop. Otherwise, go to 1.
356
2x xx
*( ) at ,df x
x xdx
( )
0, reset ,df x
x xdx
( ) 0, reset ,
df xx x
dx
2
x xx
2x x
E l
7/24/2019 OD Nonlinear Programming LARGE 2010
28/78
Example
Maximize
357
4 6 ( ) 12 3 2f x x x x
S l ti
7/24/2019 OD Nonlinear Programming LARGE 2010
29/78
Solution
First two derivatives:
358
3 5( ) 12(1 )df x
x x
dx
22 4
2
( ) 12(3 5 )
d f xx x
dx
Iteration x )/ x x x Newx x)
0 0 2 1 7.0000
1 12 0 1 0.5 5.7812
2 +10.12 0.5 1 0.75 7.6948
3 +4.09 0.75 1 0.875 7.8439
4 2.19 0.75 0.875 0.8125 7.8672
5 +1.31 0.8125 0.875 0.84375 7.8829
6 0.34 0.8125 0.84375 0.828125 7.8815
7 +0.51 0.828125 0.84375 0.8359375 7.8839
S l ti
7/24/2019 OD Nonlinear Programming LARGE 2010
30/78
Solution
x* 0.836
0.828125
7/24/2019 OD Nonlinear Programming LARGE 2010
31/78
Newtons method
This method approximatef(x) in neighborhood of
current trial solution by a quadratic function. This quadratic approximation uses Taylor series
truncated after second derivative term:
Maximized by settingf (xi+1) equal to zero (xi,f(xi) and
f (xi) are constants):
360
2
1 1 1
( )( ) ( ) ( )( ) ( )
2
i
i i i i i i i
f xf x f x f x x x x x
1 1( ) ( ) ( )( ) 0i i i i if x f x f x x x
1
( )
( )i
i i
i
f xx x
f x
Al ith f N t M th d
7/24/2019 OD Nonlinear Programming LARGE 2010
32/78
Algorithm of Newtons Method
Initialization:Select. Find initial trial solutionxi by
inspection. Seti = 1. Iterationi:
1.
2. Set
Stopping rule:If stop;xi+1 is optimal.
Otherwise,i =i + 1 (another iteration).
361
Calculate ( ) and ( ).i if x f x
1
( ) .
( )i
i i
i
f xx x
f x
1 ,i ix x
E l
7/24/2019 OD Nonlinear Programming LARGE 2010
33/78
Example
Maximize again
New solution is given by:
Selectingx1 = 1, and= 0.00001,
362
4 6 ( ) 12 3 2f x x x x
3 5 3 5
1 2 4 2 4
( ) 12(1 ) 1
( ) 12(3 5 ) 3 5i
i i i i
i
f x x x x xx x x x
f x x x x x
Iteration i x
i
x
i
) x
i
) x
i
) x
i
+1
1 1 7 12 96 0.875
2 0.875 7.8439 2.1940 62.733 0.84003
3 0.84003 7.8838 0.1325 55.279 0.83763
4 0.83763 7.8839 0.0006 54.790 0.83762
M lti i bl t i d ti i ti
7/24/2019 OD Nonlinear Programming LARGE 2010
34/78
Multivariable unconstrained optimization
Problem: maximizing a concavefunctionf(x) of
multiple variables x = (x1,x2,...,xn) with no constraints. Necessary and sufficient condition for optimality:
partial derivatives equal to zero.
No analytical solution numerical searchprocedure
must be used.
One of these is thegradient search procedure:
It identifies the direction that maximizes the rate at
whichf(x) is increased.
363
Gradient search procedure
7/24/2019 OD Nonlinear Programming LARGE 2010
35/78
Gradient search procedure
Use values of partial derivatives to select the specific
direction to move, using the gradient. Gradient of a pointx =x is thevectorwithpartial
derivativesevaluated atx =x:
Moves in the direction of this gradient untilf(x) stops
increasing. Each iteration changes the trial solutionx:
364
1 2( ) , , , at
n
f f f
f x x x
x x x
*Next ( )t f x x x
Gradient search procedure
7/24/2019 OD Nonlinear Programming LARGE 2010
36/78
Gradient search procedure
wheret* is the value oftthatmaximizesf(x+tf(x)):
The functionf(x +tf(x)) is simplyf(x) where:
Iterations continue untilf(x) = 0 withtolerance:
365
, for 1,2, , .j
f j nx
, for 1, 2, ,j jj
fx x t j nx
x x
*
0 ( ( )) max ( ( ))
tf t f f t f
x x x x
Summary of gradient search procedure
7/24/2019 OD Nonlinear Programming LARGE 2010
37/78
Summary of gradient search procedure
Init ializat ion:Selectand any initial trial solutionx.
Go to stopping rule.Iterat ion:
1. Expressf(x +tf(x)) as a function oftby setting
and substitute these expressions intof(x).
366
, for 1,2, ,j j
j
fx x t j n
x
x x
Summary of gradient search procedure
7/24/2019 OD Nonlinear Programming LARGE 2010
38/78
Summary of gradient search procedure
Iterat ion (concl.):
2. Use search procedure to findt=t* that maximizesf(x +tf(x)) overt0.
3. Resetx =x +t* f(x). Go to stopping rule.
Stopping rule:Evaluatef(x) atx =x. Check if:
If so, stop with currentx as the approximation ofx*.
Otherwise, perform another iteration.
367
, for 1,2, , .j
fj n
x
Example
7/24/2019 OD Nonlinear Programming LARGE 2010
39/78
Example
Maximize
Thus,
Verify thatf(x) is concave (see Appendix 2 of Hilliers
book).
Suppose thatx = (0, 0) is initial trial solution. Thus,
368
2 2
1 2 2 1 2( ) 2 2 2 .f x x x x x x
2 1
1
2 2f
x xx
1 2
2
2 2 4f
x xx
(0, 0) (0, 2)f
Example (2)
7/24/2019 OD Nonlinear Programming LARGE 2010
40/78
Example (2)
Iterat ion 1:Step 1 sets
by substituting these expressions intof(x):
Because
369
1
2
0 (0) 00 (2) 2
x t
x t t
2 2
2
( ( )) (0,2 )2(0)(2 ) 2(2 ) 0 2(2 )
4 8
f t f f tt t t
t t
x x
* 2
0 0 (0, 2 ) max (0, 2 ) max {4 8 }
t tf t f t f t t
Example (3)
7/24/2019 OD Nonlinear Programming LARGE 2010
41/78
Example (3)
and
it follows that
so
This completes first iteration. For new trial gradient is:
370
24 8 4 16 0d
t t tdt
* 1
4t
1 1
Reset (0,0) (0, 2) 0,4 2
x
10, (1,0)
2
f
Example (4)
7/24/2019 OD Nonlinear Programming LARGE 2010
42/78
Example (4)
As < 1,Iterat ion 2:
so
371
1 10, (1,0) ,2 2
t t
x
2
2
2
1 1 ( ( )) 0 , 0 ,
2 21 1 1
(2 ) 2 22 2 2
1
2
f t f f t t f t
t t
t t
x x
* 2
0 0 (0,2 ) max (0,2 ) max {4 8 }
t tf t f t f t t
Example (5)
7/24/2019 OD Nonlinear Programming LARGE 2010
43/78
Example (5)
Because
and
then
so
This completes second iteration. See figure.
372
* 2
0 0
1 1 1 , max , max
2 2 2t t
f t f t t t
2 1 1 2 0
2
dt t t
dt
* 1
2t
1 1 1 1Reset 0, (1,0) ,
2 2 2 2
x
Illustration of example
7/24/2019 OD Nonlinear Programming LARGE 2010
44/78
Illustration of example
Optimal solution is (1, 1), as f(1, 1) = (0, 0)
373
Newtons method
7/24/2019 OD Nonlinear Programming LARGE 2010
45/78
Newton s method
It is aquadratic approximat ionof objective function
f(x). When objective function isconcaveand its gradient
f(x) are written ascolumn vectors,
The solutionx that maximizes the approximating
quadratic function is:
where2f(x) is thennHessian matrix.
374
2 1[ ( )] ( ),f f x x x x
Newtons method
7/24/2019 OD Nonlinear Programming LARGE 2010
46/78
Newton s method
Theinverseof the Hessian matrix is commonly
approximated in various ways. These approximations are referred to as quasi-Newton
methods (orvariable metr ic methods).
Recall that this topic was mentioned in the discipline
Intelligent Systems, e.g. in neural network learning.
375
Conditions for optimality
7/24/2019 OD Nonlinear Programming LARGE 2010
47/78
Conditions for optimality
376
Problem
Necessary conditions
for optimality
Also sufficient if:
One-variableunconstrained
f(x) concave
Multivariableunconstrained f(x) concave
Constrained, nonnegativeconstraints only
f(x) concave
General constrainedproblem Karush-Kuhn-Tuckerconditions
f(x) concave and gi(x)
convex (j = 1, 2,...,n)
Karush-Kuhn-Tucker conditions
7/24/2019 OD Nonlinear Programming LARGE 2010
48/78
Karush-Kuhn-Tucker conditions
Theorem: Assume thatf(x),g1(x),g2(x),...,gm(x) are
differentiablefunctions satisfying regularityconditions. Then
x = (x1*,x1
*,...,xn*)
can be anopt imal solutionfor the NP problem if
there arem numbersu1,u2,...,um such thatalltheKKT condit ionsare satisfied:
1.
2.
377
1*
*
1
0
at , for 1, 2 , .
0
mi
i
ij j
mi
j i
ij j
gfu
x xj n
gfx ux x
x x
Karush-Kuhn-Tucker conditions
7/24/2019 OD Nonlinear Programming LARGE 2010
49/78
Karush-Kuhn-Tucker conditions
3.
4.
5.
6.
Conditions 2. and 4. require that one of the twoquantities must be zero.
Thus, conditions 3. and 4. can be combined:
(3,4)
378
*
*
( ) 0for 1,2, , .
[ ( ) ] 0
i i
i i i
g bj m
u g b
x
x
* 0, for 1, 2, , .j
x j n
0, for 1,2, , .iu j m
*( ) 0
(or 0, if 0), for 1, 2, , .
i i
i
g b
u j m
x
Karush-Kuhn-Tucker conditions
7/24/2019 OD Nonlinear Programming LARGE 2010
50/78
Karush-Kuhn-Tucker conditions
Similarly, conditions 1. and 2. can be combined:
(1,2)
Corollary: assume thatf(x) isconcaveandg1(x),g2(x),...,gm(x) areconvexfunctions, where all
functions satisfy the regularity conditions. Then,
x = (x1*,x1
*,...,xn*) is anopt imal solutionif and only if
all the conditions of the theorem are satisfied.
379
1*
0
(or 0 if 0), for 1,2 , .
m ii
ij j
j
gf ux x
x j n
Example
7/24/2019 OD Nonlinear Programming LARGE 2010
51/78
Example
Thus,m = 1, andg1(x) = 2x1 +x2 is convex.
Further,f(x) is concave (check it!).
Thus, the KKT conditions gives conditions to find anoptimal solution.
380
1 2Maximize ( ) ln( 1)f x x x
1 20, 0x x
subject to
1 22 3x x and
Example: KKT conditions
7/24/2019 OD Nonlinear Programming LARGE 2010
52/78
Example: KKT conditions
1. (j= 1)
2. (j= 1)
1. (j= 2)
2. (j= 2)3.
4.
5.
6.
381
1
1
12 0
1u
x
1 1
1
12 0
1x u
x
11 0u
2 11 0x u 1 22 3 0x x
1 1 2(2 3) 0u x x
1 2
0, 0x x
1 0u
Example: solving KKT conditions
7/24/2019 OD Nonlinear Programming LARGE 2010
53/78
Example: solving KKT conditions
From condition 1 (j= 2)u11.x10 from condition 5.
Therefore,
Therefore,x1 = 0, from condition 2 (j= 1).
u10 implies that 2x1 +x2 3 = 0 from condition 4.
Two previous steps implies thatx2 = 3.
x20 implies thatu1 = 1 from condition 2 (j= 2).
No conditions are violated forx1 = 0,x2 = 3,u1 = 1.
Consequentlyx*
= (0,3).
382
1
1
1 2 0.1
ux
Quadratic Programming
7/24/2019 OD Nonlinear Programming LARGE 2010
54/78
Quadratic Programming
Objective function can be expressed as:
383
1Maximize ( ) 2
Tf x cx x Qx
subject to
, and Ax b x 0
1 1 1
1 1( )
2 2
n n nT
j j ij i j
j i j
f c x q x x
x cx x Qx
Example
7/24/2019 OD Nonlinear Programming LARGE 2010
55/78
Example
In this case,
384
2 2
1 2 1 2 1 2Maximize ( ) 15 30 4 2 4f x x x x x x xsubject to
1 2 1 22 30, and 0, 0x x x x
[15 30]c
[1 2]A
1
2
x
x
x
[30]b
4 4
4 8
Q
Solving QP problems
7/24/2019 OD Nonlinear Programming LARGE 2010
56/78
Solving QP problems
Some KKT conditions for quadratic programming
problems can be transformed in equality constraintsby introducing slack variables.
KKT conditions can be condensed due to the
complementary variables, introducing
complementary constraints. As this, except for the complementary constraints, all
KKT conditions are linear programming constraints.
385
Solving QP problems
7/24/2019 OD Nonlinear Programming LARGE 2010
57/78
Solving QP problems
Using the previous properties, QP problems can be
solved using a modified simplex method. See example of a QP problem in Hilliers book (pages
580-581).
Excel, LINGO, LINDO, Matlab and MPL/CPLEX all can
solve quadratic programming problems.
386
Separable Programming
7/24/2019 OD Nonlinear Programming LARGE 2010
58/78
Separable Programming
Assumed thatf(x) is concave andgi(x) are convex.
f(x) is a (concave)piecewise linear function (see
example).
Ifgi(x) are linear, this problem can be reformulated as
an LP problem by using a separate variable for each
line segment.
The same technique can be used for nonlineargi(x).
387
1
( ) ( )n
j j
j
f f x
x
Example
7/24/2019 OD Nonlinear Programming LARGE 2010
59/78
Example
388
Example
7/24/2019 OD Nonlinear Programming LARGE 2010
60/78
Example
389
Convex Programming
7/24/2019 OD Nonlinear Programming LARGE 2010
61/78
Convex Programming
Many algorithms can be used, falling into 3 categories:
1. Gradient algorithms, where the gradient searchprocedure is modified to avoid violating a constraint.
Example: generalized reduced gradient(GRG).
2. Sequential unconstrained algorithms, includepenalty
function and barrier function methods. Example: sequential unconstrained minimization
technique (SUMT).
3. Sequential approximation algorithms, include linearand quadratic approximation methods.
Example: Frank-Wolfe algorithm.
390
Frank-Wolfe algorithm
7/24/2019 OD Nonlinear Programming LARGE 2010
62/78
Frank Wolfe algorithm
It is a sequential linear approximation algorithm.
It replaces the objective functionf(x) by the first-orderTaylor expansion off(x) aroundx =x, namely:
Asf(x) and f(x)x have fixed values, they can be
dropped to give a linear objective function:
391
1
( )( ) ( ) ( ) ( ) ( )( )
n
j j
j j
ff f x x f f
x
x
x x x x x x
1
( )
( ) ( ) , where at .
n
j j jj j
f
g f c x c x
x
x x x x x
Frank-Wolfe algorithm
7/24/2019 OD Nonlinear Programming LARGE 2010
63/78
g
Simplex method is applied to find a solutionxLP.
Then, chose the point that maximizes the nonlinearobjective function along the line segment.
This can be done using an one-variable unconstrained
optimization algorithm.
The algorithm continues the iterations until the stop
condition is satisfied.
392
Summary of Frank-Wolfe algorithm
7/24/2019 OD Nonlinear Programming LARGE 2010
64/78
y g
Init ializat ion:Find feasible initial trial solutionx(0), e.g.
using LP to find initial BF solution, Setk= 1.Iterat ion k:
1. Forj = 1, 2, ...,n, evaluate
and setcj equal to this value.
2. Find optimal solution by solving LP problem:
393
( )LP
kx
( 1)( ) at .k
j
f
x
xx x
1
Maximize ( ) ,n
j j
j
g c x
x
subject toand Ax b x 0
Summary of Frank-Wolfe algorithm
7/24/2019 OD Nonlinear Programming LARGE 2010
65/78
y g
3. For the variablet [0,1], set
so thath(t) gives value off(x) on line segment
between (wheret= 0) and (wheret= 1).
Use one-variable unconstrained optimization to
maximizeh(t) to findx(k).
Stopping rule:Ifx(k1) andx(k) are sufficiently close
stop.x(k) is the estimate of optimal solution.
Otherwise, resetk=k+ 1.
394
( 1) ( ) ( 1)
LP LP( ) ( ) for ( ),k k k
h t f t
x x x x x
( 1)kx
( )
LP
kx
Example
7/24/2019 OD Nonlinear Programming LARGE 2010
66/78
p
As
the unconstrained maximumx = (2.5, 2) violates the
functional constraint.
395
2 2
1 1 2 2Maximize ( ) 5 8 2f x x x x x
subject to
1 2 1 23 2 6, and 0, 0x x x x
1 2
1 2
5 2 , 8 4f f
x xx x
Example (2)
7/24/2019 OD Nonlinear Programming LARGE 2010
67/78
p ( )
Iterat ion 1:x = (0, 0) is feasible (initial trialx(0)).
Step 1 givesc1 = 5 andc2 = 8, sog(x) = 5x1 + 8x2. Step 2: solving graphically yields = (0, 3).
Step 3: points between (0, 0) and (0, 3) are:
This expression gives
396
(1)
LPx
1 2( , ) (0,0) [(0,3) (0,0)] for [0,1](0,3 )
x x t tt
2
2
( ) (0,3 ) 8(3 ) 2(3 )
24 18
h t f t t t
t t
Example (3)
7/24/2019 OD Nonlinear Programming LARGE 2010
68/78
p ( )
the valuet=t* that maximizesh(t) is given by
sot* = 2/3. This results leads to the next trial solution,
(see figure):
Iterat ion 2:following the same procedure leads to the
next trial solutionx(2)
=(5/6, 7/6).
397
( ) 24 36 0dh t tdt
(1) 2(0,0) [(0,3) (0,0)]3
(0,2)
x
Example (4)
7/24/2019 OD Nonlinear Programming LARGE 2010
69/78
p ( )
398
Example (5)
7/24/2019 OD Nonlinear Programming LARGE 2010
70/78
p ( )
Figure shows next iterations.
Note that trial solutions alternate between twotrajectories that intersect at pointx = (1, 1.5).
This is the optimal solution (satisfy KKT conditions).
Usingquadraticinstead oflinearapproximations lead
to a much faster convergence.
399
Sequential unconstrained minimization
7/24/2019 OD Nonlinear Programming LARGE 2010
71/78
Main versions ofSUMT:
exterior-pointalgorithm: deals withinfeasiblesolutionsand apenaltyfunction,
interior-pointalgorithm: deals withfeasiblesolutions
and abarrierfunction.
Uses the advantage of solving unconstrainedproblems, which are much easier to solve.
Each unconstrained problem in the sequence chooses
a smaller and smaller value ofr, and solves forx to
400
Maximize ( ; ) ( ) ( )P r f rB x x x
SUMT
7/24/2019 OD Nonlinear Programming LARGE 2010
72/78
B(x) is a barrier function with following properties:
1.B(x) issmallwhenx isfarfrom boundary of feasibleregion.
2.B(x) islargewhenx isclosefrom boundary of feasible
region.
3.B(x) as distance from boundary of feasible region
0.
Most common choice ofB(x):
401
1 1
1 1( )( )
m n
i ji i j
Bb g x
xx
Summary of SUMT
7/24/2019 OD Nonlinear Programming LARGE 2010
73/78
Init ializat ion:Find feasible initial trial solutionx(0), not
on the boundary of feasible region. Setk= 1. Choosevalue forrand< 1 (e.g.r= 1 and= 0.01).
Iterat ion k:starting fromx(k1), apply a multivariable
unconstrained optimization procedure (e.g. gradient
search procedure) to find local maximumx(k) of
402
1 1
1 1( ; ) ( )
( )
m n
i ji i j
P r f r b g x
x x
x
Summary of SUMT
7/24/2019 OD Nonlinear Programming LARGE 2010
74/78
Stopping rule:If change fromx(k1) tox(k) is very small
stop and usesx(k)
aslocal maximum. Otherwise, setk=k+ 1 andr= rfor other iteration.
SUMT can be extended for equality constraints.
Note that SUMT is quite sensitive tonumerical
instability, so it should be applied cautiously.
403
Example
7/24/2019 OD Nonlinear Programming LARGE 2010
75/78
is convex, but is not concave. Initialization: (x1,x2) =x
(0) = (1, 1),r= 1 and= 0.01.
For each iteration:
404
1 2
Maximize ( )f x xx
subject to2
1 2 1 23, and 0, 0x x x x
2
1 1 2( )g x x x 1 2( )f x xx
1 2 2
1 2 1 2
1 1 1( ; ) 3
P r x x r x x x x
x
Example (2)
7/24/2019 OD Nonlinear Programming LARGE 2010
76/78
Forr= 1, maximization leads tox(1) = (0.90, 1.36).
Table below shows convergence to (1, 2).
405
k r x
1
k
)
x
2
k
)
0 1 1
1 1 0.90 1.36
2 102 0.987 1.925
3 104 0.998 1.993
1 2
Nonconvex Programming
7/24/2019 OD Nonlinear Programming LARGE 2010
77/78
Assumptions of convex programming often fail.
Nonconvex programming problems can be much moredifficult to solve.
Dealing with non differentiable and non continuous
objective functions is usually very complicated.
LINDO, LINGO and MPL have efficient algorithms to
deal with these problems.
Simple problems can be solved using hill-climbing to
find alocal maximumseveral times.
406
Nonconvex Programming
7/24/2019 OD Nonlinear Programming LARGE 2010
78/78
An example is given in Hilliers book using Excel Solver
to solve simple problems. More difficult problems can use Evolutionary Solver.
It uses metaheuristics based on genetics, evolution
and survival of the fittest: a genetic algorithm.
We presented several well known metaheuristics to
solve this type of problems.