L8 Optimal Design concepts pt D• Homework• Review• Inequality constraints• General LaGrange Function• Necessary Conditions

for general Lagrange Multiplier Method

• Example• Summary

1

MV Optimization- E. CONSTRAINED

2

For x* to be a local minimum:

n1=i

p1= 0)(

: ToSubject) (

x x x

j= h

f :MINIMIZE

(L )ii

(U )i

j

x

x

LaGrange Function

3

If we let x* be the minimum f(x*) in the feasible region:All x* satisfy the equality constraints (i.e. hj =0) 

)()( vectorsand using

)()(

notationsummation in or )(...)()()(

1

2211

xhνxνx,

xxνx,

xxxxνx,

T

p

iii

pp

fL

hυfL

hυhυhυfL

vectornuscalarupsilonυ

""""ν

Note: when x is not feasible, hj is not equal to 0 and by minimizing hj we are pushing x towards feasibility!

Necessary Condition

4

Necessary condition for a stationary pointGiven f(x), one equality constraint, and n=2

0)(

0)(

0)(

)}()({

11

2

11

22

1

11

11

11

x

x

x0xx

0νx,

hυ

Lx

hυ

x

f

x

Lx

hυ

x

f

x

Lhυf

L

0)(

0)(

0)(

1

2

11

2

1

11

1

x

x

x

0νx,

hx

hυ

x

fx

hυ

x

f

L

Example

5

02)(

0)5.1(2)(

0)5.1(2)(

211

222

111

xxh

υxx

hυ

x

f

υxx

hυ

x

f

L

x

x

x

0νx,

)2()5.1()5.1()()()()(

212

22

1

xxυxxLυhfL

υx,xxυx,

1

11

:givessolution ussimultaneo02

0)5.1(20)5.1(23unknsand eqns 3 ofsystem

*2

*1

21

2

1

x

xυ

xxυxυx

Example cont’d

6

11

*

11

)5.1(2)5.1(2

*2

1

x

x

h

xx

f5.0)5.11(1.5)-(1(1,1)

valueoptimal11 1

solution optimal

22

*2

*1

f

xxυ

**

**11

*

11

*

xx0xx

x

x

hυfhυf

h

f

Lagrange Multiplier Method

7

1. Both f(x) and all hj(x) are differentiable2. x* must be a regular point:

2.1 x* is feasible2.2 Gradient vectors of hj(x) are linearly independent

3. LaGrange multipliers can be +, - or 0.

MV OptimizationInequality Constrained

8

nkx x x

mjg

pi= h

f :MINIMIZE

(L )kk

(U )k

j

i

1

...10)(

...10)(: ToSubject

) (

x

x

x

To Use LaGrange ApproachConvert Inequalities to Equalties?

9

0)( xjg

Add a variable sj to take up the slack

0)( jj sg x

Given an inequality

No longer an inequality

Can now use Lagrange Multiplier Approach

MV OptimizationActive or Inactive Inequalities?

10

nk=x x x

...mjg

pi== h

f :MINIMIZE

(L )kk

(U )k

i

1

10)(

10)(: ToSubject

) (

j

x

x

x

KKT Necessary Conditions for Min

11

1. Lagrange Function (in standard form)

2. Gradient Conditions

))(()()(

))(( )()(

2

2

11

sxguxhνx

xxxsu,v,x,

TT

i

m

iii

p

iii

f

sguhυfL

nkx

gu

x

hυ

x

f

x

L m

j k

jj

p

i k

ii

kk

to1for 011

mjsgu

L

phυ

L

jjj

ii

to1for 0*)(

to1ifor 0*)(

2

x

x

KKT Conditions Cont’d

12

3. Feasibility Check for Inequalities

4. Switching Conditions, e.g.

5. Non-negative LaGrange Multipliers for inequalities

6. Regularity check gradients of active inequality constraints are linearly independent

mjs j to1for 02

mjsus

Ljj

j

to1for 02 *

mju j to1for 0*

111 off"turns"0,0 gus

KKT Necessary Conditions for Min

13

))(( )()( 2

11i

m

iii

p

iii sguhυfL

xxxsu,v,x,

nkx

gu

x

hυ

x

f m

j k

jj

p

i k

ii

k

to1for 011

mjsg

ph

jj

i

to1for 0*)(

to1ifor 0*)(2

x

x

Regularity check - gradients of active inequality constraints are linearly independent

mjs j to1for 02 mjsu jj to1for 0*

mju j to1for 0*

Ex 4.32 pg 150

14

042

042..

222),(

212

211

2122

2121

xxg

xxgTS

xxxxxxfMin

LaGrange Function

15

]42[]42[

...222

)()(),(

22212

21211

2122

21

2222

211121

sxxusxxu

xxxxL

sgusguxxfL

0,0

0,0

0,0

042

042

0222

0222

2211

222

121

22212

21211

2122

2111

susu

us

us

sxxg

sxxg

uuxx

L

uuxx

L

4 equations and 4 unknowns

16

042

042

02220222

2221

2121

212

211

sxx

sxx

uuxuux

Non-linear system of equations

Use Switching Conditions to Simplify

17

0,4 Case0,3 Case0,2 Case0,1 Case

21

21

21

21

21211

21211

22

ssussuuu

ssuss

suuuu

su

Case 1

18

Non-linear system of equations

0, 21 uu

1

1

0421

0412

042

042

22

21

22

21

2221

2121

s

s

s

s

sxx

sxx

11

022022

02220222

2

1

2

1

212

211

xx

xx

uuxuux

Both inequalities are VIOLATEDTherefore, x is INFEASIBLE

Case 2

19

020220022

02220222

22

21

212

211

uxux

uuxuux

0, 21 su

0042

042

042

042

21

2121

2221

2121

xx

sxx

sxx

sxx

0042

020220022

21

22

21

xx

uxux

System of 3 linear equations in 3 unknowns

4021

0220

4102

221

221

221

uxx

uxx

uxxRewrite

Gaussian Elimination

20

Gaussian Elimination cont’d

21

Ex 4.32 cont’d

22

2.0

044.14.2

04)4.1()2.1(2

042

4.04.12.1

21

21

21

2121

2

2

1

s

s

s

sxx

uxx

Check last eqn (for s1 feasiblity)

Nope! The point is not feasible.

Case 4

23

We can use Gaussian elimination again

Where arecases 1-4?

0, 21 ss

9/200

09/209/2

3/43/4

2

1

2

1

2

1

fssuuxx

Check if regular point

24

042

042

212

211

xxg

xxgrecall

21

12

21

12

2

1

A

g

gRank= order of largest non-singular matrix in A

since det (A) is non-singular,the A matrix is full rank

We can also see that the vectors are not parallel in figure.

All Equations must be satisfied

25

0,0

0,0

0,0

042

042

0222

0222

2211

222

121

22212

21211

2122

2111

susu

us

us

sxxg

sxxg

uuxx

L

uuxx

L

Summary

• General LaGrange Function L(x,v,u,s)• Necessary Conditions for Min• Use switching conditions • Check results

26