Post on 31-Dec-2015
description
transcript
L11 Optimal Design L.Multipliers• Homework• Review• Convex sets, functions• Convex Programming Problem• Summary
1
Constrained OptimizationLaGrange Multiplier Method
2
nkx x x
mjg
pi= h
f :MINIMIZE
(L )kk
(U )k
j
i
1
...10)(
...10)(: ToSubject
) (
x
x
x
Remember:1.Standard form2.Max problems f(x) = - F(x)
KKT Necessary Conditions for Min
3
))(( )()( 2
11i
m
iii
p
iii sguhυfL
xxxsu,v,x,
nkx
gu
x
hυ
x
f m
j k
jj
p
i k
ii
k
to1for 011
mjsg
ph
jj
i
to1for 0*)(
to1ifor 0*)(2
x
x
Regularity check - gradients of active inequality constraints are linearly independent
mjs j to1for 02 mjsu jj to1for 0*
mju j to1for 0*
Prob 4.120
4
02
04subject to
)1()1(),(
12
211
22
2121
xg
xxg
xxxxfMin
]201[]4[
...)1()1(
)()(),(
22212
21211
22
21
2222
211121
sxxusxxu
xxL
sgusguxxfL
KKT Necessary Conditions
5
int?Regular po00
0,0
0,0
0201
0411
00)1(2
0)1(2
:ConditionsNecessary KKT
21
2211
22
21
22212
21211
122
2111
uu
susu
ss
sxxg
sxxg
uxx
L
uuxx
L
Case 1
6
!012)1(0)1(1
02...04)1(1)1(1
0201
0411
1,1000)1(2000)1(2
0,01Case
22
21
21
22212
21211
21
2
1
21
BADs
ss
sxxg
sxxg
xxxx
uu
0,4 Case0,3 Case0,2 Case0,1 Case
21
21
21
21
21211
21211
22
ssussuuu
ssuss
suuuu
su
Case 2
7
1
04)1(1)2(1
2,1000)1(2
00201
0411
000)1(200)1(2
0,0 2. Case
21
21
12
2
21
2121
2
21
21
s
s
xxx
xx
sxx
xux
su
1,2,0),1,2( :pointKKT
0,0,0,0:
200)12(2
21
2122
21
2
2
fuux
uusscheck
uu
Regularity: 1. pt is feasible2. only one active constraintPoint is KKT pt, OK!
Case 3
8
4011
21202102
00)1(200)1(20,0Case3.
121
121
121
12
11
21
uxx
uxxuxx
uxux
us
2
!,242
6120
21202102
2,1
1
1
121
121
121
x
BADuu
uxx
uxxuxx
Case 4
9
!0,2
2,2
00201
00411
2020202
00)1(20)1(2
0,s 4. Case
21
21
21
21
2121
2121
12
211
21
BADuu
xx
xx
xx
uuxxuuxx
uxuux
s
Sensitivity or Case 2
10
12,0
)1,2(:pointKKT2
21
fuu
xCase
0)1,0(1)1(21)0.1,0(%50
68.0)2.0,0(6.0)2.0(21)2.0,0(%10
)0,0(*),()0,0(*),(
2211
factualffactual
feueufebfeubυfebf
ji
jjiiji
Constraint Variation Sensitivity
From convexity theorems:1. Constraints linear2. Hf is PDTherefore KKT Pt is global Min!
Graphical Solution
11
3 4
21
00
02
00
02
00
02
00
)11(2)12(2
00
01
211
0)1(2)1(2
0
2
1
2211
xx
guguf
LaGrange Multiplier Method• May produce a KKT point• A KKT point is a CANDIDATE minimum
It may not be a local Min
• If a point fails KKT conditions, we cannot guarantee anything….The point may still be a minimum.
• We need a SUFFICIENT condition12
Recall Unconstrained MVO
13
0x *)( Tf
For x* to be a local minimum:
2
1*)( dHddx TTff
0 *)()( xx fff
1rst orderNecessaryCondition
0 dHdT
2nd orderSufficientCondition
i.e. H(x*) must be positive definite
Considerationsfor Constrained Min?
14
Objective functionDifferentiable, continuous i.e. smooth?Hf(x) Positive definite (i.e. convexity of f(x) )
Weierstrass theorem hints:x closed and boundedx contiguous or separated, pockets of points?
Constraints h(x) & g(x) Define the constraint set, i.e. feasible regionx contiguous or separated, pockets of points?
Convex: sets, functions, constraint set and Programming problem
Punchline (Theorem 4.10, pg 165)
15
The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if:
1. f(x) is convexHf(x) Positive definite
2. x is defined as a convex feasible set.Equality constraints must be linearInequality constraints must be convex
HINT: linear functions are convex!
16
Convex set:All pts in feasible region on a straight line(s).
Convex sets
Non-convex setPts on line are not in feasible region
18
10);(10;)1(
)1()2()2(
)2()2(
ααααα
xxxxxxx
Multiple variables Fig 4.21
0122
21 xx
What if it were an equality constraint?
misprint
19
.
Figure 4.22 Convex function f(x)=x2
Bowl that holds water.
10)]()([)())1((10))()(()()(
10;)()1()()(
12112
121
12
αxfxfαxfxααxfαxfxfαxfxf
αxfαxαfxf
2010)]()([)())1((
10))()(()()(10;)()1()()(
12112
121
12
αxfxfαxfxααxfαxfxfαxfxf
αxfαxαfxf
Fig 4.23 Characterization of a convex function.
Test for Convex Function
21
10))()(()()(
10;)()1()()()1()2()1(
)1()2(
αffαff
αfααff
xxxx
xxx
Difficult to use above definition!
However, Thm 4.8 pg 163:If the Hessian matrix of the function is PD ro PSD at all points in the set S, then it is convex.
PD… “strictly” convex, otherwisePSD… “convex”
Theorem 4.9
22
}to1,0)(g; to1for,0)(|{
SetConstraint
j mjpihS i
xxx
Given:
S is convex if:1. hi are linear2. gj are convex i.e. Hg PD or PSD
When f(x) and S are convex= “convex programming problem”
“Sufficient” Theorem 4.10, pg 165
23
The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if:
1. f(x) is convexHf(x) Positive definite
2. x is defined as a convex feasible set SEquality constraints must be linearInequality constraints must be convex
HINT: linear functions are convex!
Summary• LaGrange multipliers are the
instantaneous rate of change in f(x) w.r.t. relaxing a constraint.
• KKT point is a CANDIDATE min!(need sufficient conditions for proof)
• Convex sets assure contiguity and or the smoothness of f(x)
• KKT pt of a convex progamming problem is a GLOBAL MINIMUM!
24