Date post: | 24-Dec-2015 |
Category: |
Documents |
Upload: | kerry-flynn |
View: | 222 times |
Download: | 0 times |
Unconstrained
First Order Necessary Condition
Second Order Necessary
Second Order Sufficient
2* min ( *) . . .If x is then FONChold and f x is p s d
* min ( *) 0If x is then f x
2( *) 0 ( *) . .
* min .
If f x and f x is p d
then x is strict local
KKT Conditions
Note for equality – multipliers are unconstrained
Complementarity not an issue
: ( , ) ( ) '( )
:
( , ) ( ) ' 0x
Lagrangian L x f x Ax b
KKT Ax b
L x u f x A
u unconstrained
Null Space Representation
Let x* be a feasible point, Ax*=b.
Any other feasible point can be written as x=x*+p where Ap=0
The feasible region
{x : x*+p pN(A)}
where N(A) is null space of A
See Section 3.2 of Nash and Sofer for example
mxnA
mAppAN 0|)(
A ofcolumn th theis A where,
somefor |
A of columns by the spanned vectorsofset the)(
i iAq
Aqq
AR
ii
mT
T
Null and Range Spaces
)R(Aq and N(A)p somefor q p x x,
0 Ap because , somefor 0,Appq
)R(Aq and N(A)p
subspaces orthogonal are )R(A and N(A)
T
TT
T
T
Orthogonality
Null Space Review
is the null space matrix of A:
, 0
, 0 ,
Z
p v Zp Av and if
v Av p v Zp
1 2 1 2[11] 0
1, ( )
1
A v v v v
Z p R Zp Null A
Constrained to Unconstrained
You can convert any linear equality constrained optimization problem to an equivalent unconstrained problem
Method 1 substitution
Method 2 using Null space representation and a feasible point.
Example
Solve by substitution
becomes
1 2 2 31 2 22
1 2 3
min
. . 3 4 4
x x x
s t x x x
1 2 2 31 2 22
1 2 3
min
. . 4 3 4
x x x
s t x x x
3
21 2 32 3 2 22
min 4 3 4x x x x
Null Space Method
x*= [4 0 0]’
x=x*+Zv
becomes
1 2 2 31 2 22
1 2 3
min
. . 3 4 4
x x x
s t x x x
3
21 2 31 2 1 22
min 4 3 4v v v v
3 4
1 0
0 1
Z
1 21
12
2
4 3 4 4 3 4
0 1 0
0 0 1
v vv
vv
v
General Method
There exists a Null Space Matrix
The feasible region is:
Equivalent “Reduced” Problem
n rZ R r n m
| *x x Zv
min ( * )v f x Zv
Optimality Conditions
Assume feasible point and convert to null space formulation
2 2 2
( ) ( * )
( ) ' ( * ) ' ( ) 0 *
( ) ' ( * ) ' ( )
k v f x Zv
k v Z f x Zv Z f y where y x Zv
k v Z f x Zv Z Z f y Z
Where is KKT?
KKT implies null space
Null Space implies KKT
Gradient is not in Null(A), thus it must be in Range(A’)
( *) ' 0
'( ( *) ' ) ' ( *) 0 ' ' 0
f x A
Z f x A Z f x because Z A
Lemma 14.1 Necessary Conditions
If x* is a local min of f over {x|Ax=b}, and Z is a null matrix
Or equivalently use KKT Conditions
2
' ( *) 0
' ( *) . . .
Z f x
and Z f x Z is p s d
2
( *) ' 0
*
' ( *) . . .
f x Ahas a solution
Ax b
Z f x Z is p s d
Lemma 14.2 Sufficient Conditions
If x* satisfies (where Z is a basis matrix for Null(A))
then x* is a strict local minimizer
2
*
' ( *) 0
' ( *) . .
Ax b
Z f x
Z f x Z is p d
Lemma 14.2 Sufficient Conditions (KKT form)
If (x*,*) satisfies (where Z is a basis matrix for Null(A))
then x* is a strict local minimizer
2
*
( *) ' 0
' ( *) . .
Ax b
f x A
Z f x Z is p d
Lagrangian Multiplier
* is called the Lagrangian Multiplier
It represents the sensitivity of solution to small perturbations of constraints
*
1
ˆ ˆ( ) ( *) ( *) ' ( *)
ˆ( *) ( *) ' ' *
ˆ
( *) ' * ( *)m
i ii
f x f x x x f x
f x x x A by KKT OC
Now let Ax b
f x f x
Optimality conditions
Consider min (x2+4y2)/2 s.t. x-y=10( ) ' 0
1
4 1
10
* 8, * 8, * 2,
f x A
Ax b
x
y
x y
x y
Optimality conditions
Find KKT point Check SOSC( ) ' 0
1
4 1
10
* 8, * 8, * 2,
f x A
Ax b
x
y
x y
x y
2
2
' [1 1]
1 0( )
0 4
' ( ) . .
So SOSC satisfied
Or we could just observe that
it is a convex program so FONC
are sufficient
Z
f x
Z f x Z is p d
2 21 2
1 2
1
2
1
2
1 2
1min 4
2s.t. 10
( ) Ax b
xf(x) A 1 1
4x
1
4 1
10
T
x x
x x
f x A
x
x
x x
Linear Equality Constraints - I
Linear Equality Constraints - II
1 2 1 1 2
2 2
2
1
2
Solve:
x 4 4
4 10
5 10
8
2
8
8* , * 8 KKT point
2
x x x x
x x
x
x
x
x
01
1
40
0111)(
40
01)(
1
1 Z1-1A SOSC
2
2
ZxfZ
xf
T
so SOSC satisfied, and x* is a strict local minimum
Objective is convex, so KKT conditions are sufficient.
Linear Equality Constraints - III
Next Easiest Problem
Linear equality constraints
Constraints form a polyhedron
min ( )
. . ,
n
m n m
f x f R
s t Ax b A R b R
Polyhedron Ax>=b
( *) *
*
Ti i
i
f x A a
Ax b
unconstrained minimum
contour set of function
a1x = b
*)(xf
-a1
Close to Equality Case
a4x = b
a3x = b
a2x = b
a2x = b
-a2
Equality FONC:
Which i are 0? What is the sign of I?
x*
Polyhedron Ax>=b
( *) *
*
Ti i
i
f x A a
Ax b
a1x = b
*)(xf
-a1
Close to Equality Case
a4x = b
a3x = b
a2x = b
a2x = b
-a2
Equality FONC:
Which i are 0? What is the sign of I?
x*
Polyhedron Ax>=b
( *) *
*
( * ) 0
0
Ti i
i
i i i
f x A a
Ax b
A x b
a1x = b
*)(xf
-a1
Inequality Case
a4x = b
a3x = b
a2x = b
a2x = b
-a2
Inequality FONC:
Nonnegative Multipliers imply gradient points to the less thanSide of the constraint.
x*
i
i
0
represents sensitivity
ˆIf 0 then increasing causes the objective to increase.
If b is changed to make feasible region bigger then
the objective will decrease.
i
i i
i
i i
Ax b
if
then A x b
A x b
Lagrangian Multipliers
Lemma 14.3 Necessary Conditions
If x* is a local min of f over {x|Ax≤b}, and Z is a null-space matrix for active constraints then for some vector *
2
( *) ' * 0 or equivalently ' ( *) 0
*
* 0
*'( * ) 0
' ( *) . . .
f x A Z f x
Ax bKKT
Ax b
and Z f x Z is p s d
Lemma 14.5 Sufficient Conditions (KKT form)
If (x*,*) satisfies
2
* Primal feasibility
( *) ' * Dual feasibility
* 0
*'( * ) 0 Complementarity
and SOSC ' ( *) . .
Then x* is a strict local minimizer
Ax b
f x A
Ax b
Z f x Z is p d
Lemma 14.5 Sufficient Conditions (KKT form)
where Z+ is a basis matrix for Null(A
+) and A + corresponds to nondegenerate active constraints)
i.e.+
*j
For the jth row of A
* Active Constraint
0 Nondegenerate
j jA x b
Sufficient Example
Find solution and verify SOSC
2 21 2
1
min 1/ 2( 1) 1/ 2
. . 0 1
x x
s t x
* [1,0]' * [2,0]x
1
2
2
1
2* 1( *)
0
1 0( *)
0 1
Active constraint is 1
A 1 0
xf x
x
f x
x
Linear Inequality Constraints - I
21 22
1
21 22
1
1
1 1min 1
2 2s.t. 0 1
Put in standard form:
1 1min 1
2 2s.t. 1
-x 0
1x* by inspection
0
x x
x
x x
x
T
T1
KKT Conditions
Ax* b
f(x*) A
* 0
*( * ) 0
? since second constraint is inactive
0
2 1f(x*) A 2 0
0 0
1 2KKT Point: x* *
0 0
Ax b
Linear Inequality Constraints - III
2
' 2
Now look at SOSC
0[ 1 0]
1
2*
0
1 0f(x*)
0 1
f(x*) 1 a p.d. matrix
Therefore SOSC are satistified.
X* is a strict local minima
A Z
Z Z
Linear Inequality Constraints - IV
Example
Problem 1 2 2 3
1 2 22
1 2 3
2 3
min
. . 3 4 4
0
x x x
s t x x x
x x
* [8 / 51,28 / 51,28 / 51], * [ 8 / 51, 4 / 51]x
You Try
Solve the problem using above theorems:
2 21 2
1 2
1 2
1
min 1/ 2
. . 2 2
1
0
x x
s t x x
x x
x
If you guess first two constraints active, what happens?
If you guess just first constraint active, what happens?
Why Necessary and Sufficient?
Sufficient conditions are good for? Way to confirm that a candidate point is a minimum (local) But…not every min satisifies any given SC
Necessary tells you: If necessary conditions don’t hold then you know you
don’t have a minimum. Under appropriate assumptions, every point that is a min
satisfies the necessary cond. Good stopping criteria Algorithms look for points that satisfy Necessary
conditions
Lagrangian Function
Optimality conditions expressed using
Lagrangian function
and Jacobian matrix
were each row is a gradient of a constraint
1
( , ) ( ) ( ) ( ) ' ( )m
i ii
L x f x g x f x g x
( ) 'g x
Theorem 14.2 Sufficient Conditions Equality (KKT
form)If (x*,*) satisfies
2
( *) 0 Primal feasibility
( *) ( *) ' *
( L(x*, *)=0) Dual feasibility
and SOSC ' ( *) . .
Then x* is a strict local minimizer
xx
x
g x
f x g x
equivalently
Z L x Z is p d
Theorem 14.4 Sufficient Conditions Inequality (KKT)If (x*,*) satisfies
2
( *) 0 Primal feasibility
( *) ( *) ' *
( L(x*, *)=0) Dual feasibility
* 0 (for inequalities only)
* ' ( *) 0 Complementarity
and SOSC ' ( *) . .
Then
xx
x
g x
f x g x
equivalently
g x
Z L x Z is p d
x* is a strict local minimizer
Lemma 14.4 Sufficient Conditions (KKT form)
where Z+ is a basis matrix for Null(A
+) and A + corresponds to Jacobian of nondegenerate active constraints)
i.e.
*j
For the jth row of Jacobian
( *) 0 Active Constraint
0 Nondegenerate
jg x
Sufficient Example
Find solution and verify SOSC
2 21 2
2 21 1
min 1/ 2( 1) 1/ 2
. . 1/ 2 1/ 2 1/ 2
x x
s t x x
* [1,0]' * 2x
10
01*)(
0
2)1()(
0
1 x*Guess
2
1
2
1
2
1 s.t.
2
1)1(
2
1- min
2
2
1
22
21
22
21
xf
x
xxf
xx
xx
Nonlinear Inequality Constraints - I
Nonlinear Inequality Constraints - II
2 21 2
T1 2
1 1
1
1 1 1Has one active constraint:
2 2 2
Jacobian: g(x) x 1 0
( , ) ( ) ' ( )
( , ) ( ) ( )
-2 1
0 0
x
x x
x
L x f x g x
L x f x g x
2 2 2
T
2
( , ) ( ) ' ( )
( , ) ( ) ( )
-1 0 1 0 2
0 1 0 1
1 0 positive definite
0 1
0Z , since g(x) 1 0
1
1 0( , ) 0 1
0 1
xx i ii
Txx
L x f x g x
L x f x g x
Z L x Z
01 positive definite
1
so SOSC satisifed, x* is a strict local minimum
Nonlinear Inequality Constraints - III
2
21 2
22 1 2
'
1
min x
s.t. x - x 0
0x* by observation
0
L(x, ) ( ) ' ( )
x (x -x )
L(x, ) ( ) ( )
0 2
1 1
0 0( *, *) 0 * 0 * 1
1 1
K
x
x
f x g x
f x g x
x
L x
0KT Point: x* * 1
0
Nonlinear Inequality Constraints - V
2 2 2
2
1( ) 0 1 Z
0
( , ) ( ) ( )
0 0 2 0 ( 1)
0 0 0 0
2 0
0 0
2 0 1( , ) 1 0 2 positive definite
0 0 0
So SOSC
T
xx i ii
Txx
g x
L x f x g x
Z L x Z
are satisfied, and x* is a strict local minimum.
Nonlinear Inequality Constraints - VI
Theorem 14.1 Necessary Conditions- Equality
If x* is a local min of f over {x|g(x)=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then
2
there exists *
( *, *) 0 or equivalently ' ( *) 0
( *) 0
' ( *) . . .
x
xx
L x Z f x
g x
and Z L x Z is p s d
Theorem 14.3 Necessary Conditions
If x* is a local min of f over {x|g(x)>=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then
'
2
there exists *
( *, *) 0 or equivalently ' ( *) 0
( *) 0
* ( *) 0
' ( *) . . .
x
xx
L x Z f x
g x
g x
and Z L x Z is p s d
Regular point
If x* is a regular point with respect to the constraints g(x*) if the gradient of the active constraints are linearly independent.For equality constraints, all constraints are active so
should have linearly independent rows.
( *) 'g x
Necessary Example
Show optimal solution x*=[1,0]’
is regular and find KKT point 1
2 21 2
31 2
max
. . 1
( 1) 0
x
s t x x
x x
1
2 21 2
31 2
min
. . ( ) 1 0
( 1) 0
x
s t x x
x x
KKT point
* [1, 0] '
* [1 / 2, 0] '
x
Constraint Qualifications
Regularity is an example of a constraint qualification CQ.The KKT conditions are based on linearizations of the constraints. CQ guarantees that this linearization is not getting us into trouble. Problem is
KKT point might not exist.There are many other CQ,e.g., for inequalities Slater is there exists g(x)<0.Note CQ not needed for linear constraints.