Process Optimization

Tier I: Mathematical Methods of Optimization

Section 2:

Linear Programming

Linear Programming (LP)

• Linear programming (linear optimization) is the area of optimization problems with linear objective functions and constraints

Example:

minimize: f(x) = 6x1 + 5x2 + 2x3 + 7x4

subject to: 2x1 + 8x3 + x4 ≥ 20

x1 – 5x2 – 2x3 + 3x4 = -5

Linear Programming con’t

• None of the variables are multiplied by another variable, raised to a power, or used in a nonlinear function

• Because the objective function and constraints are linear, they are convex. Thus, if an optimal solution to an LP problem is found, it is the global optimum

LP Standard Form

• LP Standard form:

minimize: f = cx

subject to: Ax = b

xi ≥ 0; i = 1, …, n

where c is called the cost vector (1 by n), x is the vector of variables (n by 1), A is the coefficient matrix (m by n), and b is a m by 1 vector of given constants.

Standard Form Basics

• For a maximization problem, we can transform using:

max(f(x)) min(-f(x))

• For inequality constraints, use “slack” variables:

2x1 + 3x2 ≤ 5 2x1 + 3x2 + s1 = 5where s1 ≥ 0

Using Slack Variables

When we transform the equation

2x1 + 3x2 ≤ 5 to 2x1 + 3x2 + s1 = 5

If the left-hand side (LHS) (2x1 + 3x2) is less than the right-hand side (RHS) (5), then s1 will take a positive value to make the equality true. The nearer the value of the LHS is to the RHS, the smaller the value of s1 is. If the LHS is equal to the RHS, s1 = 0. s1 cannot be negative because the LHS cannot be greater than the RHS.

Standard Form Example

Example:

Write in Standard Form:

maximize: f = x1 + x2

subject to: 2x1 + 3x2 ≤ 6

x1 + 7x2 ≥ 4

x1 + x2 = 3

x1 ≥ 0, x2 ≥ 0

Define slack variables x3 ≥ 0 & x4 ≥ 0

Example Problem Rewritten

The problem now can be written:

minimize: g = –x1 – x2

subject to: 2x1 + 3x2 + x3 = 6

x1 + 7x2 – x4 = 4

x1 + x2 = 3

x1 ≥ 0, x2 ≥ 0, x3 ≥ 0, x4 ≥ 0

0011 c

3

4

6

b

0011

1071

0132

A

Linear Algebra Review

• The next few slides review several concepts from linear algebra that are the basis of the methods used to solve linear optimization problems

Vectors & Linear Independence

• Vectors– A k-vector is a row or column array of k

numbers. It has a dimension of k.

• Linear Independence (LI)– A collection of vectors a1, a2, …, ak, each of

dimension n, is called linearly independent if the equation

means that for j=1, 2, …, k

k

jjj

1

0a

0j

Linear Independence con’t

• In other words, a set of vectors is linearly independent if one vector cannot be written as a combination of any of the other vectors.

• The maximum number of LI vectors in a n-dimensional space is n.

Linear Independence con’t

For example, in a 2-dimension space:

The vectors & are not

linearly independent because x2 = 5x1.

are LI because there is

no constant you can multiply one by to get the other.

5

41x

25

202x

2

01x

1

32x&

Spanning Sets

k

jjj

1

ab

• A set of vectors a1, a2, …, ak in a n-dimensional space is said to span the space if any other vector in the space can be written as a linear combination of the vectors

• In other words, for any vector b, there must exist scalars 1, 2, …, k such that

Bases

• A set of vectors is said to be a basis for a n-dimensional space if:

1. The vectors span the space

2. If any of the vectors are removed, the set will no longer span the space

• A basis of a n-dimensional space must have exactly n vectors

• There may be many different bases for a given space

Bases con’t

• An example of a basis is the coordinate axis of a graph. For a 2-D graph, you cannot remove one of the axes and still form any line with just the remaining axis.

• Or, you cannot have three axes in a 2-D plot because you can always represent the third using the other two.

Systems of Equations

• Linear Algebra can be used to solve a system of equations

Example:

2x1 + 4x2 = 8& 3x1 – 2x2 = 11

This can be written as an augmented matrix:

1123

842],[ bA

Systems of Equations con’t

• Row operations may be performed on the matrix without changing the result

• Valid row operations include the following:– Multiplying a row by a constant– Interchanging two rows– Adding one row to another

Solving SOE’s

• In the previous example, we want to change the A matrix to be upper triangular

multiply top

row by ½

add -3 times the

top row to the

bottom row

1123

842

1123

421

180

421

Solving SOE’s con’t

multiply bottom row by -1/8

• From the upper triangular augmented matrix, we can easily see that x2 = 1/8 and use this to get x1

x1 = 4 – 2 . 1/8 = 15/4

180

421

8110

421

421 21 xx

81

21 10 xx

Matrix Inversion

• The inverse of a matrix can be found by using row operations

Example:

Form the augmented matrix (A, I)

Transform to (I, A-1)

using row operations

211

121

112

A

100211

010121

001112

12

5

12

3

12

112

3

12

3

12

312

1

12

3

12

5

100

010

001

Optimization Equations

• We have seen that the constraints can be written in the form .

• We should have more variables than equations so that we have some degrees of freedom to optimize.– If the number of equations are more than or

equal to the number of variables, the values of the variables are already specified.

bAx

General Solution to SOE’s

• Given a system of equations in the form

– Assume m (number of equations) < n (number of variables) underspecified system

• We can split the system into (n-m) independent variables and (m) dependent variables. The values of the dependent variables will depend on the values we choose for the independent variables.

bAx

General Solution con’t

• We call the dependent variables the basic variables because their A-matrix coefficients will form a basis. The independent variables will be called the nonbasic variables.

• By changing the variables in the basis, we can change bases. It will be shown that this allows examining different possible optimum points.

General Solution con’t

Separate the A matrix in the following way:

Or,

baaa nn

mm xxx ......1

1

baa

n

mjj

jm

ii

i xx11

General Solution con’t

Define matrices B & N as the following:

where B is a m by m matrix, N is a m by (n-m) matrix, & aj is the jth column of the A matrix

• B is called the “basic matrix” and N is called the “nonbasic matrix”

maaaB ...21 nmm aaaN ...21

General Solution con’t

• The B matrix contains the columns of the A-matrix that correspond to the x-variables that are in the basis. Order must be maintained.– So, if x4 is the second variable of the basis, a4

must be the second column of the B-matrix

• The N matrix is just the columns of the A-matrix that are left over.

General Solution con’t

Similarly, define

&

We will see later how to determine which variables to put into the basis. This is an important step to examine all possible optimal solutions.

TmB xxx ]...[ 21x T

nmmN xxx ]...[ 21 x

General Solution con’t

Now, we have

Multiply both sides by B-1:

So,

bNxBx NB

)()( 11NNNB xNxBbBxxx

bBNxBx 11 NB

Basic Solution

• We can choose any values for (n-m) variables (the ones in xN) and then solve for the remaining m variables in xB

• If we choose xN = 0, thenThis is called a “basic solution” to the system

Basic Solution:

bBx 1B

)()( 1 0bBxxx NB

Basic Feasible Solutions

Now we have a solution to Ax = b. But that was just one of two sets of constraints for the optimization problem. The other was: xi ≥ 0, i = 1, …, n (non-negativity)

• A basic feasible solution (BFS) is a basic solution where every x is non-negative

A BFS satisfies all of the constraints of the optimization problem

Extreme Points

• A point is called an extreme point (EP) if it cannot be represented as a strict (0 < < 1) convex combination of two other feasible points.

• Remember: a convex combination of two points is a line between them.

• So, an EP cannot be on a line of two other feasible points.

Extreme Points (Graphical)

• Given a feasible region, an extreme point cannot lie on a line between two other feasible points (it must be on a corner)

• In a n-dimensional space, an extreme point is located at the intersection of n constraints

FeasibleRegion

Extreme Point

Not Extreme Points

Optimum & Extreme Points

• We have a maximization problem, so we want to go as far in the direction of the c (objective function) vector as we can

• Can we determine anything about the location of the optimum point?

Starting Point

c

Optimum & Extreme Points

• If we start on a line, we can move along that line in the direction of the objective function until we get to a corner

• In fact, for any c vector, the optimum point will always be on a corner

c

Basic Feasible Solutions

• In a n-dimensional space, a BFS is formed by the intersection of n equations.

• In 2-D:

Constraint 1

Constraint 2

Basic Feasible Solution

• But, we just saw that an extreme point is also a corner point. So, a BFS corresponds to an EP.

Tying it Together

• We just saw that a basic feasible solution corresponds to an extreme point.

• This is very important because for LP problems, the optimum point is always at an extreme point.

• Thus, if we can solve for all of the BFS’s (EP’s), we can compare them to find the optimum.

Unfortunately, this takes too much time.

Simplex Method Introduction

• The simplex method is the most common method for solving LP problems.

• It works by finding a BFS; determining whether it is optimal; and if it isn’t, it moves to a “better” BFS until the optimal is reached.

• This way, we don’t have to calculate every solution.

Simplex Method Algebra

Recall:

NNBBf xcxccx

bBNxBx 11 NBbBaBx 11

Nj

jj

B x

Sum over all non-basic variables

Nj Nj

jjjj

B xcxf )( 11 aBbBc

substitute

Nj

jj

B xaBbBx 11

Objective Function:

into above equation:

Simplex Method Algebra

Nj

jjjB xzcf )(1bBc

jBjz aBc 1where

Nj

jj

BjB xcf )( 11 aBcbBc

Multiply through and collect xj terms:

Simplex Method Equations

So, minimize

If (cj – zj) ≥ 0 for all j N, then the current BFS is optimal for a minimization problem.

Because, if it were < 0 for some j, that nonbasic variable, xj, could enter the basis and reduce the objective function.

Nj

jjjB xzcf )(1bBc

bBaBx 11

Nj

jj

B xSubject to:

Entering Variables

• A nonbasic variable may enter the basis and replace one of the basic variables

• Since xN = 0, and we have non-negativity constraints, the entering variable must increase in value.

• The entering variable’s value will increase, reducing the objective function, until a constraint is reached.

Entering Variable Equation

• The equation to determine which variable enters is: . Calculate for all nonbasic indices j

• For a minimization problem, choose the index j for which cj - zj is the most negative

– If cj - zj ≥ 0 for all j, the solution is optimal

• For a maximization problem, choose the index j for which cj - zj is the most positive

– If cj - zj ≤ 0 for all j, the solution is optimal

jjjT

Bj zcc aBc 1

Leaving Variables

• As the value of the entering variable increases, the value of at least one basic variable will usually decrease – If not, the problem is called “unbounded” and

the value of the minimum objective function is -

• The variable whose value reaches zero first will be the variable that leaves the basis

Entering & Leaving Variables

• Example: x1 is entering the basis while x2, x3 & x4 are the current basic variables

As soon as x2 reaches zero, we must stop because of the non-negativity constraints. But, now x2 = 0, so it is a nonbasic variable and x1 > 0, so it is a basic variable. So, x2 leaves the basis & x1 enters the basis.

x1

x2

x3

x4

Leaving Variable Equation

• Let j be the index of the variable that is entering the basis and i* be the index of the variable that is leaving the basis

Meaning, for every index i that is in the basis and has , calculate . The index of the value that is the minimum is the index of the leaving variable.

0)()(

)(argmin 1

1

1*

ij

ij

i aBaB

bBi

ij

i

)(

)(1

1

aB

bB

0)( 1 i

jaB

Leaving Variable Equation

The previous expression is obtained from the equation:

which applies when a constraint is reached

0aBbBx j

jB x11

The Example Revisited

• x2, x3, & x4 start out at (B-1b)i ; (i=2, 3, 4) and have

slopes of (–B-1aj)i ; (i=2, 3, 4) where j=1 because 1 is the index of the entering variable (x1)

• Thus, the distance we can go before a basic variable reaches zero is for B-1a1 > 0. But, if (B-1a1)i < 0 (like x3), it won’t ever reach zero.

x1

x2

x3

x4

i

i

)()(

11

1

aBbB

The Example Revisited

• We can also see how if none of the variables decreased, we could keep increasing x1 and improving the objective function without ever reaching a constraint –This gives an unbounded solution

x1

x2

x3

x4

Example Problem

Minimize f = -x1 – x2

Subject to: x1 + x2 ≤ 5

2x1 – x2 ≤ 4

x1 ≤ 3 ; x1, x2 ≥ 0

Given: The starting basis is x1, x2, & x3.

Insert slack variables x3, x4, & x5.

Example

Minimize f = -x1 – x2

Subject to: x1 + x2 + x3 = 5

2x1 – x2 + x4 = 4

x1 x5 = 3

x1, x2, x3, x4, x5 ≥ 0

10001

01012

00111

A

3

4

5

b 00011 c

Example

1st Iteration:

001

012

111321 aaaB

0

2

3

3

4

5

311

210

1001bBxB

311

210

100

001

012

1111

1B

5

0

2

3

011

BBf xc

Example

Now, check optimality

x4:

x5:

110

0

1

0

311

210

100

0110414

aBcBc

3)3(0

1

0

0

311

210

100

0110515

aBcBc

< 0

> 0

Example

So, x4 enters the basis since its optimality indicator is < 0.

0

2

3

)( 1bB 3)( 11 bB 2)( 2

1 bB 0)(& 31 bB

1

1

0

0

1

0

311

210

100

4 411 aBaB jj

Example

So, x3 is the leaving variable

310//argmin0)(

)(

)(argmin 1

1

1*

ANANaBaB

bBi i

j

ij

i

3)( 11 bB 2)( 2

1 bB 0)( 31 bB

0)( 141 aB 1)( 2

41 aB 1)( 341 aB

0 0 0

Example

2nd Iteration:

001

112

011

B

0

2

3

3

4

5

311

101

1001bBxB

311

101

1001B

5

0

2

3

011

BBf xc

a4 has been substituted for a3

Example

Optimality Indicators:

x3:

x5:

01)1(0

0

0

1

311

101

100

0110313

aBcBc

0000

1

0

0

311

101

100

0110515

aBcBc

Example Solution

All of the optimality indicators are ≥ 0, so this is the optimal solution.

So, 5* f

0

2

3

4

2

1*

x

x

x

Bx

0

0

5

3*

x

xNx

Simplex Algorithm Steps

1. With the chosen basis, get B and solve xB = B-1b and f = cBxB.

2. Calculate cj – zj for all nonbasic variables, j.

– For a min. problem, if all cj – zj’s are ≥ 0, the current solution is optimal. If not, choose the index with the most negative cj – zj.

– For a max. problem, if all cj – zj’s are ≤ 0, the current solution is optimal. If not, choose the index with the most positive cj – zj.

Simplex Algorithm Steps

3. Using the equation choose the leaving variable.

– If all (B-1aj)i’s are ≤ 0, then the solution is unbounded

4. Let xj enter the basis and xi* leave the basis. Obtain the new B matrix and start again with step 1.

0)()(

)(argmin 1

1

1*

ij

ij

i aBaB

bBi

Choosing A Starting Basis

• In the example, we were given a starting basis. How to come up with one on our own?

• Case #1: max (or min) problem with 1. Ax ≤ b (all ≤ inequalities) and

2. all entries of the b vector ≥ 0.

Insert slack variables into constraint equations and use the resulting identity

matrix as the starting basis

Choosing A Starting Basis

Let s = vector of slack variables

The problem will become

0scx

fmin

max

bIsAx ,0x 0s

Subject to

Where I = The Identity Matrix

10...00

01...00

00...10

00...01

. . .

. . .

. . .

Choosing A Starting Basis

Choose the slack variables to be the starting basis

The starting basis matrix (B) is the coefficients of the slack variables. This happens to be the identity matrix.

We can see that the starting basis is feasible

(xB ≥ 0):IB

0bIbbIbBx 11B

Example Problem #2

Minimize -x1 – 3x2

Subject to 2x1 + 3x2 ≤ 6

-x1 + x2 ≤ 1 x1, x2 ≥ 0

Insert slack variables:

2x1 + 3x2 + x3 = 6

-x1 + x2 + x4 = 1

x1, x2, x3, x4 ≥ 0

1011

0132A

1

6b 0031 c

Identity Matrix

0

Example #2

Use the slacks as the starting basis:

10

0143 aaB

10

01

10

011

1B

1

6

1

6

10

011bBxB 01

600

BBf xc

&

Example #2

Optimality Indicators:

j=1:

j=2:

1011

2

10

0100111

111

aBcczc B

3031

3

10

0100321

222

aBcczc B

c2 - z2 is the most negative, so x2 enters the basis

Example #2

1

6)( 1bB 6)( 3

1 bB 1)(& 41 bB

1

3

1

3

10

012 211 aBaB jj

x2 is entering the basis

Example #2

412argmin11

36argmin0)(

)(

)(argmin 1

1

1*

ij

ij

i aBaB

bBi

6)( 31 bB 1)( 4

1 bB

So, x4 is the leaving variable.

3)( 321 aB 1)( 4

21 aB

Example #2

2nd Iteration:

10

3123 aaB

10

31

10

311

1B

1

3

1

6

10

311bBxB 31

330

BBf xc

x2 replaced x4

Example #2

Optimality Indicators:

j=1:

j=4:

So, x1 enters the basis

4311

2

10

3130111

111

aBcczc B

3)3(01

0

10

3130041

444

aBcczc B

Example #2

Leaving Variable:

So, x3 leaves the basis and x1 replaces it.

1

3)( 1bB 3)( 3

1 bB 1)(& 21 bB

1

5

1

2

10

311 1 jj aB

3/53argmin0)(

)(

)(argmin 1

1

1*

ANaBaB

bBi i

j

ij

i

0

Example #2

3rd Iteration:

11

3221 aaB

4.02.0

6.02.0

11

321

1B

6.1

6.0

1

6

4.02.0

6.02.01bBxB

4.56.1

6.031

BBf xc

Example #2

Optimality Indicators:

j=3:

j=4:

Both cj-zj’s are ≥ 0, so the current solution is optimal

8.0)8.0(00

1

4.02.0

6.02.031031

333

aBcczc B

6.0)6.0(01

0

4.02.0

6.02.031041

444

aBcczc B

006.16.0* x 4.5* f

Example #2

This graph shows the path taken.

The dashed lines are perpendicular to the cost vector, c.

0

0

1

0

6.1

6.0

x1

x2

Increasing c

Example #2

• Since we were minimizing, we went in the opposite direction as the cost vector

0

0

1

0

6.1

6.0

x1

x2

Increasing c

More on Starting Bases

• Case #2: max (or min) problem with:1. Ax ≥ b (at least some ≥ constraints) &

2. All entries of the b vector ≥ 0

Add slacks to make the problem become

Ax – Is = b x, s ≥ 0.

We cannot do the same trick as before because now we would have a negative identity matrix as our B matrix.

Case #2 con’t

• 2-Phase Method:

Introduce “artificial variables” (y) where needed to get an identity matrix. If all constraints were ≥, the problem will become:

Ax – Is + Iy = b x, s, y ≥ 0

Artificial Variables

• Artificial variables are not real variables.

• We use them only to get a starting basis, so we must get rid of them.

• To get rid of them, we solve an extra optimization problem before we start solving the regular problem.

2-Phase Method

Phase 1:

Solve the following LP starting with B = I and xB = y = b:

Minimize y

Subject to: Ax – Is + Iy = b x, s, y ≥ 0

If y 0 at the optimum, stop – the problem is infeasible. If y = 0, then use the current basis and continue on to phase 2.

2-Phase Method con’t

Phase 2:

Using the objective function from the original problem, change the c vector and continue solving using the current basis.

Minimize (or Maximize) cx

Subject to: Ax – Is = b x, s ≥ 0

Artificial vs. Slack Variables

• Slack variables are real variables that may be positive in an optimum solution, meaning that their constraint is a strict (< or >) inequality at the optimum.

• Artificial variables are not real variables. They are only inserted to give us a starting basis to begin the simplex method. They must become zero to have a feasible solution to the original problem.

Artificial Variable Example 1

• Consider the constraints: x1 + 2x2 ≥ 4

-3x1 + 4x2 ≥ 5

2x1 + x2 ≤ 6 x1, x2 ≥ 0

• Introduce slack variables: x1 + 2x2 – x3 = 4

-3x1 + 4x2 – x4 = 5

2x1 + x2 + x5 = 6

AV Example 1

We can see that we cannot get an identity matrix in the coefficients and positive numbers on the right-hand side. We need to add artificial variables:

x1 + 2x2 – x3 + y1 = 4

-3x1 + 4x2 – x4 + y2 = 5

2x1 + x2 + x5 = 6

AV Example 1

Now we have an identity matrix, made up of the coefficient columns of y1, y2, & x5.

We would solve the problem with the objective of minimizing y1 + y2 to get rid of the artificial variables, then use whatever basis we had and continue solving, using the original objective function.

Artificial Variable Example 2

• Consider the constraints:x1 + 2x2 – 5x3 ≥ -4

3x1 – x2 + 3x3 ≤ 2

-x1 + x2 + x3 = -1 x1, x2, x3 ≥ 0

• Introduce slack variables:x1 + 2x2 – 5x3 – x4 = -4

3x1 – x2 + 3x3 + x5 = 2

-x1 + x2 + x3 = -1

AV Example 2

We don’t have to add an artificial variable to the 1st constraint if we multiply by -1.

When we multiply the last constraint by -1 and add an artificial variable, we have:

-x1 – 2x2 + 5x3 + x4 = 4

3x1 – x2 + 3x3 + x5 = 2

x1 – x2 – x3 + y1 = 1

x1, x2, x3, x4, x5, y1 ≥ 0

Constraint Manipulation

So, after adding slacks, we must make the right-hand side numbers positive. Then we add artificial variables if we need to.

Artificial Variable Example 3

• Consider the problem:

Maximize -x1 + 8x2

Subject to: x1 + x2 ≥ 1

-x1 + 6x2 ≤ 3

x2 ≤ 2 x1, x2 ≥ 0

AV Example 3

Insert slacks:

x1 + x2 – x3 = 1

-x1 + 6x2 + x4 = 3

x2 + x5 = 2

So, we need an artificial variable in the 1st constraint.

AV Example 3

Insert artificial variable:

x1 + x2 – x3 + y1 = 1

-x1 + 6x2 + x4 = 3

x2 + x5 = 2

AV Example 3

So, Phase 1 is:Minimize y1

Subject to: x1 + x2 – x3 + y1 = 1

-x1 + 6x2 + x4 = 3

x2 + x5 = 2

Our starting basis is: y1, x4, & x5.

AV Example 3

100

010

001546 aaaB

100

010

001

100

010

0011

1B

2

3

1

2

3

1

100

010

0011bBxB 1

2

3

1

001

BBf xc

&

100000c

2

3

1

b

AV Example 3

Optimality Indicators:

j=1:

j=2:

j=3:

110

0

1

1

100

010

001

001011111

aBcczc B

110

1

6

1

100

010

001

001021222

aBcczc B

110

0

0

1

100

010

001

001031333

aBcczc B

AV Example 3

It’s a tie between x1 & x2 – pick x1 to enter the basis

2

3

1

)( 1bB1)( 6

1 bB3)( 4

1 bB

0

1

1

0

1

1

100

010

001

1 111 aBaB jj

x1 is entering the basis

2)(& 51 bB

AV Example 3

6//1

1argmin0)(

)(

)(argmin 1

1

1*

ANANaBaB

bBi i

j

ij

i

1)( 61 bB 3)( 4

1 bB

So, x1 replaces y1 in the basis

1)( 621 aB 1)( 4

21 aB

2)( 51 bB

0)( 521 aB

0 00

AV Example 3

100

011

001541 aaaB

100

011

001

100

011

0011

1B

2

4

1

2

3

1

100

011

0011bBxB 0

2

4

1

000

BBf xc

AV Example 3

Optimality Indicators:

j=2:

j=3:

j=6: 000

0

1

1

100

011

001

000011111

aBcczc B

000

1

6

1

100

011

001

000021222

aBcczc B

000

0

0

1

100

011

001

000031333

aBcczc B

AV Example 3

All of the optimality indicators are ≥ 0, so this is an optimum solution.

So, we keep this basis and change the objective function to the original one:

Maximize –x1 + 8x2

Our basis is still x1, x4, & x5.

AV Example 3

Back to original problem:

100

011

0011B

2

4

11bBxB

1

2

4

1

001

BBf xc

00081c

The basis remains the same

AV Example 3

Optimality Indicators:

j=2:

j=3:

718

1

6

1

100

011

001

001821222

aBcczc B

110

0

0

1

100

011

001

001031333

aBcczc B

Since we are maximizing now, we want the most positive. So, x2 enters the basis.

AV Example 3

2

4

1

)( 1bB1)( 1

1 bB4)( 4

1 bB

1

7

1

1

6

1

100

011

001

2 211 aBaB jj

x2 is entering the basis

2)(& 51 bB

AV Example 3

41

2

7

4

1

1argmin0)(

)(

)(argmin 1

1

1*

ij

ij

i aBaB

bBi

1)( 51 bB 4)( 4

1 bB

1)( 121 aB 7)( 4

21 aB

2)( 51 bB

1)( 521 aB

MinimumSo x4 leaves the basis

AV Example 3

110

061

011521 aaaB

1143.0143.0

0143.0143.0

0143.0856.01B

429.1

571.0

429.0

2

3

1

1143.0143.0

0143.0143.0

0143.0856.01bBxB

143.4

429.1

571.0

429.0

081

BBf xc

AV Example 3

Optimality Indicators:

j=3:

j=4: 000

0

1

0

1143.0143.0

0143.0143.0

0143.0857.0

001041444

aBcczc B

857.0

0

0

1

1143.0143.0

0143.0143.0

0143.0857.0

001031333

aBcczc B

Artificial Variable Example 3

All of the optimality indicators are ≤ 0, so this is the optimum solution:

429.1

0

0

571.0

429.0

*x 143.4* f

KKT Conditions

• The Karush-Kuhn-Tucker (KKT) Conditions can be used to see optimality graphically

• We will use them more in nonlinear programming later, but we can use a simplified version here

KKT Conditions for LP

• Change the constraints so that they are all ≥ constraints.

• The optimum point is the point where the gradient of the objective function lies within the cone formed by the vectors normal to the intersecting constraints.

KKT Conditions

• Reminder: – The gradient () of a function f with n

variables is calculated:

nx

f

x

f

x

ff ...,,

21

Example:32

21 5)(3 xxxf

231 5,5,6 xxxf

KKT Constraints Example

• Example: In the example problem #2, we had the problem:

Minimize f = -x1 – 3x2

Subject to: 2x1 + 3x2 ≤ 6

-x1 + x2 ≤ 1

x1, x2 ≥ 0

The gradient of the cost function, -x1 – 3x2 is: 3,1 cf

KKT Example

Previously, we saw that this problem looks like:

Constraint 1

Constraint 2

x1

x2

Extreme Points

(0, 0)

(3/5, 8/5)

(0, 1)

(3, 0)

KKT Example

Change to constraints to all ≥:

g1: -2x1 – 3x2 ≥ -1

g2: x1 – x2 ≥ -1

g3: x1 ≥ 0

g4: x2 ≥ 0

KKT Example

The gradients of the four constraints (counting the non-negativity constraints), g1, …, g4 are:

3,21 g

1,12 g 1,04 g

0,13 g

KKT Example

The graph of the problem with the normals of the constraints becomes:

x1

x2

(0, 0)

(3/5, 8/5)

(0, 1)

(3, 0)

g1g2

g3

g2

g3

g4

g1

g4

Constraint 1

Constraint 2 The gradient corresponding to each constraint (gi) is perpendicular to the constraint i.

KKT Example

c = (-1, -3) looks like this:

So, whichever cone this vector fits into corresponds to the optimal extreme point.

KKT Example

x1

x2

(0, 0)

(3/5, 8/5)

(0, 1)

(3, 0)

g1g2

g3

g2

g3

g4

g1

g4

Doesn’t FitDoesn’t Fit

Doesn’t Fit

It Fits!

KKT Example

• So, we get the same optimal point as when we used the simplex method

• This can also be done for problems with three variables in a 3-D space

• With four or more variables, visualization is not possible and it is necessary to use the mathematical definition

Mathematical Definition of KKT Conditions for LP

Given a LP minimization problem:

Modify the constraints so that we have:

,0)( xig mi 1

Where gi(x) is the linear constraint equation i. The bi that was on the right side of the inequality sign is moved to the left side and included in gi.

Mathematical Definition of KKT Conditions for LP

If there exists a solution for x* & the i’s for the conditions below, then x* is the global optimum

)()( **11 xx mm ggcf

,0)( * xii g

,0)( * xig

,0i

mi 1

mi 1

mi 1

Equation 1

Equation 2

Equation 3

Equation 4

Explanation of Equation 1

• Equation 1 mathematically states that the objective function vector must lie inside the cone formed by the vectors normal to the active constraints at the optimal point

Explanation of Equation 2

• Equation 2 forces i to be zero for all of the inactive constraints – called the “complementary slackness” condition– If the constraint is active, gi(x*) = 0, so i may

be positive and gi will be part of the cone in Equation 1.

– If the constraint is inactive, gi(x*) 0, so i must be zero. gi will not be included in the cone in Equation 1 because it will be multiplied by zero.

Explanation of Equations 3 & 4

• Equation 3 ensures that x* is feasible

• Equation 4 ensures that the direction of the cone is correct.– If the i’s were negative, the cone would be in

the opposite direction. So, this equation prevents that from happening.

KKT Conditions Summary

• The KKT Conditions are not very useful in solving for optimal points, but they can be used to check for optimality and they help us visualize optimality

• We will use them frequently when we deal with nonlinear optimization problems in the next section

Automated LP Solvers

• There are many software programs available that will solve LP problems numerically

• Microsoft Excel is one program that solves LP problems– To see the default Excel examples for

optimization problems, search for and open the file “solvsamp.xls” (it should be included in a standard installation of Microsoft Office)

Excel LP Example #1

Let’s solve the first example problem in this chapter with Excel

The problem was:

Minimize f = -x1 – x2

Subject to: x1 + x2 ≤ 5

2x1 – x2 ≤ 4

x1 ≤ 3 ; x1, x2 ≥ 0

Excel LP Example #1

Here is the Excel spreadsheet with the necessary data:

x1 x20 0

value limitObjective Function: =-A2-B2

Constraint 1: =A2+B2 5Constraint 2: =2*A2-B2 4Constraint 3: =A2 3

In the spreadsheet, A2 is the cell reference for x1 & B2 is the reference for x2

Excel LP Example #1

You can see that under the “value” heading for the constraints & objective function, we simply use the given functions to calculate the value of the function

x1 x20 0

value limitObjective Function: =-A2-B2

Constraint 1: =A2+B2 5Constraint 2: =2*A2-B2 4Constraint 3: =A2 3

Excel LP Example #1

On the right side of the constraints, in the “limit” column, we write the value of the “bi” for that constraint

Obviously, the objective function doesn’t have a limit

x1 x20 0

value limitObjective Function: =-A2-B2

Constraint 1: =A2+B2 5Constraint 2: =2*A2-B2 4Constraint 3: =A2 3

Excel LP Example #1

So, the spreadsheet looks like this:

Excel LP Example #1

• Now, we need to use the Excel solver feature

• Look under the Tools menu for “solver”– If it is not there, go to “Add-Ins” under Tools

and select the Solver Add-In

Excel LP Example #1

The Solver toolbox should look something like this:

Excel LP Example #1

• This is a minimization problem, so select “Min” and set the target cell as the objective function value

• The variables are x1 & x2, so in the “By Changing Cells” box, select A2 & B2

Excel LP Example #1

• Now add the constraints: – For the “Cell Reference,” use the value of the

constraint function and for the “Constraint,” use the number in the Limit column

– The constraints are all ≤, so make sure that “<=“ is showing between the Cell Reference and Constraint boxes

Excel LP Example #1

• Now, the Solver window should look like this:

Excel LP Example #1

• Finally, click the Options button

• All of the variables are specified as being positive, so check the “Assume Non-Negative” box

Excel LP Example #1

• Since this is an LP problem, check the “Assume Linear Model” box

• Finally, the default tolerance of 5% is usually much too large. Unless the problem is very difficult, a tolerance of 1% or even 0.1% is usually fine

Excel LP Example #1

• Click “Solve” and the Solver Results box should appear

• Under “Reports,” select the Answer Report and click OK

• A new worksheet that contains the Answer Report is added to the file

Excel LP Example #1

• The spreadsheet with the optimum values should look like this:

Excel LP Example #1

• The values for x1 & x2 are the same as when we solved the problem using the simplex method

• Also, if you look under the Answer Report, you can see that all of the slack variables are equal to zero, which is also what we obtained with the simplex method

Excel Example #2

Let’s solve one more LP problem with Excel:

Maximize 5x1 – 2x2 + x3

Subject to: 2x1 + 4x2 + x3 ≤ 6

2x1 + x2 + 3x3 ≥ 2

x1, x2 ≥ 0

x3 unrestricted in sign

Excel Example #2

Entering the equations into the spreadsheet should give:

x1 x2 x30 0 0

Value: Limit:Objective Function: =5*A2-2*B2+C2

Constraint 1: =2*A2+4*B2+C2 6Constraint 2: =2*A2+B2+3*C2 2

Excel Example #2

Unlike last time, all the variables are not specified as being positive, so we cannot use the “Assume Non-negative” option for all of the variables.

So we have to manually specify x1 & x2 to be non-negative by adding two more constraints

Excel Example #2

Now, the formulas in the spreadsheet should look like this:

x1 x2 x30 0 0

Value: Limit:Objective Function: =5*A2-2*B2+C2

Constraint 1: =2*A2+4*B2+C2 6Constraint 2: =2*A2+B2+3*C2 2Constraint 3: =A2 0Constraint 4: =B2 0

Excel Example #2

Now, open the solver toolbox and specify:

• The Target Cell,

• The range of variable cells,

• Maximization problem

• The constraints – The first is ≤ and the rest are ≥ constraints.

Excel Example #2

Click the Options button and check the “Assume Linear Model” box.

Remember, since x3 is unrestricted in sign, do not check the “Assume Non-negative” box

You can reduce the tolerance if you would like

Excel Example #2

The Solver window should look like this:

Excel Example #2

After solving, the spreadsheet should look like this:

Excel Example #2

• Notice that because x3 was unrestricted in sign, it was able to have a negative value and this improved the solution

• To see how much of a difference this made in the solution, re-solve the problem with the “Assume Non-negative” option selected

Solving LP Problems With Excel

• From these examples, you can see that Excel can be an efficient tool to use for solving LP optimization problems

• The method for solving the problems that was outlined here is obviously just one way and the user should feel free to experiment to find their own style

References

• Linear Programming and Network Flows; Bazaraa, Mokhtar; John Jarvis; & Hanif Sherali.

• Optimization of Chemical Processes 2nd Ed.; Edgar, Thomas; David Himmelblau; & Leon Lasdon.

• Pollution Prevention Through Process Integration; El-Halwagi, Mahmoud