Optimization (168)

Lecture 3-4-5

Jesus De Loera

UC Davis, Mathematics

Wednesday, April 2, 2012

1

THE SIMPLEX METHOD

2

LINEAR PROGRAMS (LPs)

We wish to optimize subject to linear constraints (= or ≥ is special case)

maximize (minimize) C1x1 + C2x2 + · · ·+ Cd xd

among all x1,x2, . . . ,xd ,satisfying:

a1,1x1 + a1,2x2 + · · ·+ a1,d xd ≤ b1

a2,1x1 + a2,2x2 + · · ·+ a2,d xd ≤ b2

...

ak ,1x1 + ak ,2x2 + · · ·+ ak ,d xd ≤ bk

3

A vector satisfying all constraints is called a feasible solutionA feasible solution is Optimal if it attains a maximum or minimum.Some problems have no solutions, so they are called infeasible for example

max 5x1 + 4x2

s.t. x1 + x2 ≤ 2

−2x1−2x2 ≤−9

x1,x2 ≥ 0

An LP can be unbounded when there are arbitrarily large feasible solutions (sayin Euclidean norm).

max x1−4x2

s.t. −2x1 + x2 ≤−1

− x1−2x2 ≤−2

x1,x2 ≥ 0

GOAL: Find optimal solution OR detect when LP infeasible or unbounded!4

The simplex method

George Dantzig invented the SIMPLEX METHOD in the 1940’s.Lemma The set of all feasible solutions is a convex polyhedron. Let us use geometry!

5

LINEAR PROGRAMS (LPs)

The general LP problem can be reduced to the case

maximize (minimize) C1x1 + C2x2 + · · ·+ Cd xd

subject to x1,x2, . . . ,xd ,satisfying:

a1,1x1 + a1,2x2 + · · ·+ a1,d xd ≤ b1

a2,1x1 + a2,2x2 + · · ·+ a2,d xd ≤ b2

......

ak ,1x1 + ak ,2x2 + · · ·+ ak ,d xd ≤ bk

with the condition x1,x2, . . . ,xd ≥ 0.

WHY??Replace a non-restricted xi by xi = x+

i − x−i , with x+i ≥ 0,x−i ≥ 0.

6

Next we can rewrite this as

maximize (minimize) C1x1 + C2x2 + · · ·+ Cd xd

subject to x1,x2, . . . ,xd ,satisfying:

a1,1x1 + a1,2x2 + · · ·+ a1,d xd + s1 = b1

a2,1x1 + a2,2x2 + · · ·+ a2,d xd + s2 = b2

......

ak ,1x1 + ak ,2x2 + · · ·+ ak ,d xd + sk = bk

with the condition x1,x2, . . . ,xd ,s1,s2, . . .sk ≥ 0.

WHY??The variables si are capture the slack, slack variables. Original variables decisionvariables. Looks more like linear algebra.

From this form we follow an iterative procedure!7

How the simplex Method works

max 5x1 + 4x2 + 3x3

s.t. 2x1 + 3x2 + x3 ≤ 5

4x1 + x2 + 2x3 ≤ 11

3x1 + 4x2 + 2x3 ≤ 8

x1,x2,x3 ≥ 0

Rewrite problem using the slack variables (we call them s1,s2,s3

z = 5x1 + 4x2 + 3x3

s1 = 5−2x1−3x2− x3

s2 = 11−4x1− x2−2x3

s3 = 8−3x1−4x2−2x3

x1,x2,x3 ≥ 0

The problem is now the same as maxz subject to x1,x2,x3,s1,s2,s3 ≥ 0.8

Key idea 1 There is a one-to-one correspondence of the feasible solutions of theoriginal LP to the feasible solutions of the LP with slacks (canonical form)!

Key idea 2 Carry on successive improvements, starting with a feasible of LP incanonical form proceed to a better feasible solution, one with better value of z.

Given

z = 5x1 + 4x2 + 3x3

s1 = 5−2x1−3x2− x3

s2 = 11−4x1− x2−2x3

s3 = 8−3x1−4x2−2x3

x1,x2,x3,s1,s2,s3 ≥ 0

Set x1 = x2 = x3 so we have a solution but z = 0.

NOTE: if we keep x2 = x3 = 0 but increase x1 we increase z.

Just how much do we increase x1 (while keeping x2 = x3 = 0) and still maintainfeasibility?

9

Look at first equation! s1 = 5−2x1−3x2− x3, s1 must be non-negative! Thusx1 ≤ 5/2.

From second equation s2 = 11−4x1− x2−2x3 ≥ 0 Thus x1 ≤ 11/4.

From third equation s3 ≥ 0 implies x1 ≤ 8/3.

Of these 3 bounds which is more restrictive? The first is the most restrictive!!

Keep x2 = x3 = 0 but increase x1 we increase to 52 , thus now

s1 = 0,s2 = 1,s3 = 12 . Now z = 25

2

Need to rewrite the system in such a way that variables that are positive are givenin term of those that are zero, rewrite

x1 = 52 −

32 x2− 1

2 x3− 12 s1, use it to rewrite the system!!! Replace x1 everywhere in

z = 5x1 + 4x2 + 3x3

s1 = 5−2x1−3x2− x3

s2 = 11−4x1− x2−2x3

s3 = 8−3x1−4x2−2x3

x1,x2,x3,s1,s2,s3 ≥ 0

10

From

z = 5x1 + 4x2 + 3x3 → z = 5(52− 3

2x2−

12

x3−12

s1) + 4x2 + 3x3

s1 = 5−2x1−3x2− x3 → x1 =52− 3

2x2−

12

x3−12

s1

s2 = 11−4x1− x2−2x3 → s2 = 11−4(52− 3

2x2−

12

x3−12

s1)− x2−2x3

s3 = 8−3x1−4x2−2x3 → s3 = 8−3(52− 3

2x2−

12

x3−12

s1)−4x2−2x3

x1,x2,x3,s1,s2,s3 ≥ 0

We get a new system is

z =252− 7

2x2 +

12

x3−52

s1

x1 =52− 3

2x2−

12

x3−12

s1

s2 = 1 + 5x2 + 2s1

s3 =12

+12

x2−12

x3 +32

s1

x1,x2,x3,s1,s2,s3 ≥ 011

z =252− 7

2x2 +

12

x3−52

s1

x1 =52− 3

2x2−

12

x3−12

s1

s2 = 1 + 5x2 + 2s1

s3 =12

+12

x2−12

x3 +32

s1

x1,x2,x3,s1,s2,s3 ≥ 0

We shall again increase value of z by increasing the value of a variable on theright side, while keeping at zero the others.NOTE: increasing x2 or s1 would give a decrease!. Thus we must increase x3!How much?Read answer from new system: While x2 = s1 = 0 we have x3 ≤ 1 is the most wecan increase!!! (WHY?? Which equation says that?).Rewrite the system so that the positive value variables are on the left ( x1,x3,s2)and the right-side has only zero-valued variables (x2,s1,s3).From third equation we have x3 = 1 + x2 + 3s1−2s3. Substitute in the systemagain. We get new system!!!!

12

After substitution

z =252− 7

2x2 +

12

x3−52

s1

x1 =52− 3

2x2−

12

x3−12

s1

s2 = 1 + 5x2 + 2s1

s3 =12

+12

x2−12

x3 +32

s1

x1,x2,x3,s1,s2,s3 ≥ 0

turns into

z = 13−3x2− s1− s3

x1 = 2−2x2−2s1 + s3

s2 = 1 + 5x2 + 2s1

x3 = 1 + x2 + 3s1−2s3

x1,x2,x3,s1,s2,s3 ≥ 0

13

z = 13−3x2− s1− s3

x1 = 2−2x2−2s1 + s3

s2 = 1 + 5x2 + 2s1

x3 = 1 + x2 + 3s1−2s3

x1,x2,x3,s1,s2,s3 ≥ 0

To repeat the process we need to find a variable from the right side to go to theleft!!! One that when we increase it will increase z!!!

BUT if we increase either x2,s1,s3 we will decrease z!!!!

We have reached a stand-still. Claim: the answer we have now is OPTIMAL!!

Why Look at the first row. Setting x2 = s1 = s3 = 0, yields z = 13 , but since allfeasible solutions MUST satisfy x2,s1,s3 ≥ 0 this must be the best possible value!!

14

General Description

We start with a problem in the form

maxn

∑j=1

cjxj

s.t.n

∑j=1

aijxj ≤ bi i = 1 . . .m

xj ≥ 0

Turn it into a DICTIONARY by adding slack variables (for each inequality).

z =n

∑j=1

cjxj

si = bi −n

∑j=1

aijxj for i = 1 . . .m

xj ,si ≥ 0

The simplex method moves from one dictionary to the next. 15

We call the variables in the left BASIC variables and the variables in the rightNON-BASIC variables.If N denotes the NON-BASIC variables, B denotes the basic variables

z = z + ∑j∈N

cjxj

xi = bi −∑j∈N

aijxj for i ∈ B

xj ,si ≥ 0

The current solution when we set the non-basic variables to zero is a BASICFEASIBLE SOLUTION.Exactly one variable goes from NON-BASIC to become BASIC (moves fromRIGHT to LEFT) and vice versa.Pick a non-basic variable whose coefficient in the objective function is > 0.Candidates are {j ∈ N, cj > 0}.If Candidate set is empty, the we have an OPTIMAL SOLUTION!!!! WHY? Currentbasic feasible solution is best possible.Otherwise if Candidates has more than one element, choose one. HOW? Manyoptions!!!

16

Say xk is chosen to leave (currently non-basic, turned into basic). We requirebi − aik xk ≥ 0. This implies

xk ≤bi

aik

Pick index l where blalk

is smallest possible (NOTE: book talks about maximumreciprocal).

Next, pivot, using the equation

xl = bl −∑j∈N

aljxj

we rewrite, put xk in the left, xl in the right.

REPEAT!!!

BUT there are several questions? What if the initial bi are not all positive? How torecognize unboundedness? How do I know that the process will terminate?

17

Initialization: PHASE I

Suppose the original bi are not all non-negative. Then we cannot right away find afeasible solution!!!!

Consider instead of the original problem:

max− x0

s.t.n

∑j=1

aijxj − x0 ≤ bi i = 1 . . .m

xj ≥ 0

Lemma Easy to find a feasible solution: Set xj = 0 for j = 1, . . . ,n and set x0 largeenough positive.

Lemma Original problem has a feasible solution if and only if an optimal solutionof the auxiliar problem has objective value ZERO.

18

max x1− x2 + x3

s.t. 2x1− x2 + 2x3 ≤ 4

2x1−3x2 + x3 ≤−5

− x1 + x2−2x3 ≤−1

x1,x2,x3 ≥ 0

We need to put it in the auxiliary form, where a feasible solution is easy to find:

max − x0

s.t. 2x1− x2 + 2x3− x0 ≤ 4

2x1−3x2 + x3− x0 ≤−5

− x1 + x2−2x3− x0 ≤−1

x1,x2,x3,x0 ≥ 0

19

We put it in dictionary form by adding slacks.

z =− x0

x4 =4−2x1 + x2−2x3 + x0

x5 =−5−2x1 + 3x2− x3 + x0

x6 =−1 + x1− x2 + 2x3 + x0

x1,x2,x3,x4,x5,x6,x0 ≥ 0

With a single pivot, x0 enters basics, x5 leaves basics, we get

z =−5−2x1 + 3x2− x3− x5

x4 = 9 −2x2− x3 + x5

x0 = 5 + 2x1−3x2 + x3 + x5

x6 = 4 + 3x1−4x2 + 3x3 + x5

x1,x2,x3,x4,x5,x6,x0 ≥ 0

20

Starts normal pivoting!!!

z =−2 + 0.25x1 + 1.25x3−0.25x5−0.75x6

x2 =1 + 0.75x1 + 0.75x3 + 0.25x5−0.25x6

x0 =2−0.25x1−1.25x3 + 0.25x5 + 0.75x6

x4 =7−1.5x1−2.5x3 + 0.5x5 + 0.5x6

x1,x2,x3,x4,x5,x6,x0 ≥ 0

Pivot again!? x3 enters basics, x0 leaves.

z = − x0

x3 =1.6−0.2x1 + 0.2x5 + 0.6x6−0.8x0

x2 =2.2 + 0.6x1 + 0.4x5 + 0.2x6−0.6x0

x4 =3 − x1 − x6 + 2x0

x1,x2,x3,x4,x5,x6,x0 ≥ 0

WE have an optimal dictionary!! and x0 = 0.We are (almost) ready to solve the original problem!!!

21

z =x1− x2 + x3

x3 =1.6−0.2x1 + 0.2x5 + 0.6x6

x2 =2.2 + 0.6x1 + 0.4x5 + 0.2x6

x4 =3 − x1 − x6

x1,x2,x3,x4,x5,x6,x0 ≥ 0

WHAT IS THE PROBLEM NOW? WHY CAN’T WE START?Need to rewrite the objective function in terms of the non-basic variables x1,x5,x6. Thecorrect z is

z =−0.6 + 0.2x1−0.2x5 + 0.4x6

x3 =1.6−0.2x1 + 0.2x5 + 0.6x6

x2 =2.2 + 0.6x1 + 0.4x5 + 0.2x6

x4 =3− x1 − x6

x1,x2,x3,x4,x5,x6,x0 ≥ 0

PHASE I is finished!! Now we have a feasible solution for original problem!Starting from this feasible solution we find the optimum (PHASE II) 22

UNBOUNDEDNESS

What if all the ratios we test are negative or there is division by zero?

Sometimes the non-basic variable can be increased indefinitely!!! Producing anarbitrarily large objective value.

EXAMPLE

z = 5 + x3− x1

x2 = 5 + 2x3−3x1

x4 = 7 −4x1

x5 = x1

23

SUMMARY

Given a feasible dictionary we have to select an entering variable, find a leavingvariable and to construct the new dictionary by pivoting!!!

Choosing an entering variable: The entering variable is a non-basic variable xj

with a positive coefficient in the objective function row.

NOTE: The rule is ambiguous, we may have more than one candidate!!! (SO FARwe choose xj with largest cofficient!).

Finding the leaving variable: The leaving variable is that basic variable whosenon-negativity imposes the most stringent upper bound on the increase of theentering variable.

NOTE: Again the rule is ambiguous, we may have more than one candidate!!!This has VERY important consequences!!!

If there is no candidate for leaving the basis, then we can make the value of theentering variable as large as we wish!! UNBOUNDED!

24

DEGENERACY

z =2x1− x2 + 8x3

x4 =1 −2x3

x5 =3−2x1 + 4x2−6x3

x6 =2 + x1−3x2−4x3

x1,x2,x3,x4,x5,x6,x0 ≥ 0

Clearly x3 enters the basis, but who leaves? All variables x4,x5,x6 give the sameincrease! Choose any!! Say x4 pivot.

z =4 + 2x1− x2−4x4

x3 =0.5 −0.5x4

x5 =−2x1 + 4x2 + 3x4

x6 =x1−3x2−2x4

x1,x2,x3,x4,x5,x6,x0 ≥ 0

NOTE: x5,x6 are basic, but they are also equal to ZERO! DEGENERATE PROBLEM!25

This has some annoying consequences. For example if we pivot again, x1 enters thebasis and x5 leaves (limit of increment is zero!).

z =4 + 3x2− x4− x5

x1 =2x2 + 1.5x4−0.5x5

x3 =0.5 −0.5x4

x6 =− x2 + 3.5x4−0.5x5

x1,x2,x3,x4,x5,x6,x0 ≥ 0

This does not change the solution at all!!!Sometimes the simplex method goes through a few degenerate iterations one after theother, sometimes CYCLING CAN HAPPEN!!

26