Single Source Shortest Path and Linear Programming

Dr. Mustafa Sakalli

Marmara Univ.

Mid May of 2009, CS246

Reminding single source shortest path• Given: (directed or undirected) graph G = (V, E, w) and source node s, for each t, a

path in G from s to t with minimum weight. Negative edges not included

• Variants: b) Single-destination shortest-paths problem, c) Single-pair shortest-path problem, d) All-pairs shortest-paths problem: Floyd Warshall algorithm. Find a shortest path from u to v for every pair of vertices u and v

• Prim's MST algorithm, start with source node s and iteratively construct a tree rooted at s, at each progress keep the track of tree node that provides cheapest path from s, at each iteration, include the lightest path

• Dijkstra: Keep an estimate, d[u], of shortest path distance from s to t. (Using d as a key in a priority queue)

• When u is added to the tree, check if u is reached with a more expensive pat earlier to swap. Comparing d[v] to d[u] + w(u,v). Triangle inequality.

• Dijkstra cannot handle negative weight edges.. A very similar algorithm is Bellman-Ford SSSP algorithm.

5

4

16

Prim's MST

ss

5

4

16

Dijkstra's SSSP

ss

t’t’t’t’

tt ss

3

1

16

Dijkstra's SSSP

t’t’

tt

• Building heap takes O(V) + Operation over the heap O(logV) = O(VlogV).

• Going over the adjacent list O(E) + Relaxation is an operation over the heap O(logV) = O(ElogV).• +• -------------------

• O(ElogV + VlogV) = O(ElogV)

Single Source Shortest-Path PropertiesSingle Source Shortest-Path Properties• Problem:Problem: given a weighted directed graph G, find the minimum-weight path given a weighted directed graph G, find the minimum-weight path

from a given source vertex s to another vertex vfrom a given source vertex s to another vertex v– ““Shortest-path” = minimum weight Shortest-path” = minimum weight jj=1:n=1:n

aaijijxxj j

– E.g., a road map: what is the shortest path from Istanbul to Flint and Steel Point?E.g., a road map: what is the shortest path from Istanbul to Flint and Steel Point?• Optimal Substructure: TheoremTheorem: subpaths of shortest paths are shortest paths: subpaths of shortest paths are shortest paths

Proof (”Proof (”cut and pastecut and paste”): if some subpath were not the shortest path, one could ”): if some subpath were not the shortest path, one could substitute the shorter subpath and create a shorter total pathsubstitute the shorter subpath and create a shorter total pathContradictionContradiction: Suppose : Suppose some subpath is not a shortest path

• There must then exist a shorter subpath • Could substitute the shorter subpath for a shorter path• But then overall path is not shortest path. Contradiction

• Define Define (u,v) to be the (u,v) to be the weight of the shortest pathweight of the shortest path from u to v from u to v• Shortest paths satisfy the Shortest paths satisfy the triangle inequalitytriangle inequality: : (u,v) (u,v) (u,x) + (u,x) + (x,v)(x,v)• ““Proof”:Proof”:

• In graphs with negative weight cycles, some shortest paths will not exist

x

u v

This path is no longer than any other path

< 0

Bellman-FordBellman-Ford

• Applies Applies the same relaxationthe same relaxation principle applied in Dijkstra. principle applied in Dijkstra. • Consider every edge (u,v) and see if u + {u, v} offers a better path. Compare Consider every edge (u,v) and see if u + {u, v} offers a better path. Compare d[v] d[v]

with d[u] + w(u,v)with d[u] + w(u,v). . No matter what order the edgesNo matter what order the edges are considered in. are considered in.• Repeat this process |V| - 1 for all v Repeat this process |V| - 1 for all v {V-\s} times to ensure that accurate {V-\s} times to ensure that accurate

information propagates from s. information propagates from s. • In the In the presence of negative weight cyclespresence of negative weight cycles, algorithm doesn’t converge to a , algorithm doesn’t converge to a

solution. Therefore finally check if there is any negative weight cycle. solution. Therefore finally check if there is any negative weight cycle.

• AlgorithmAlgorithm • input: directed or undirected graph G = (V,E,w)input: directed or undirected graph G = (V,E,w)• d[s] d[s] 0 0• for each vfor each vV-{s} do V-{s} do //initialization//initialization

d[v] d[v] ∞ for i for i 1 to |V| - 1 do 1 to |V| - 1 do // main body// main body for each (u,v) for each (u,v) E(G) do E(G) do // consider in arbitrary order// consider in arbitrary order if d[u] + w(u,v) < d[v] then update if d[u] + w(u,v) < d[v] then update //relaxation //relaxation d[v]d[v] min(d[v], d[u] + w(u,v)) min(d[v], d[u] + w(u,v))

d[v] d[v] d[u] + w(u,v) d[u] + w(u,v) p[v] p[v] u u

• for each (u,v) for each (u,v) E(G) do E(G) do //check if any negative weight cycle.//check if any negative weight cycle. if d[u] + w(u,v) < d[v] then if d[u] + w(u,v) < d[v] then

return (false) return (false) return d[v]return d[v]

Time O(VE) slower but it is going to handle negative edges..Time O(VE) slower but it is going to handle negative edges..

Bellman-Ford Example

u

w

v5

-6

3 2 u

w

v5

6

-3 2u

w

v5

-9

3 2

s a b c d e

0 ∞ ∞ ∞ ∞ ∞

0 -2 8 4 ∞ ∞0 -2 5 -1 9 6

0 -2 5 -1 4 3/1

-2-2

48

71

-2 5

-2

3 9

00

88 44

∞∞

aa∞

48

71

-2 5

-2

3 9

s0s0

bb∞ cc∞

dd∞ee∞

-2-2

48

71

-2 5

-2

3 9

00

8/58/5 4/-14/-1

9966

-2-2

48

71

-2 5

-2

3 9

00

55 -1-1

4411

0 -2 5 -1 4 1

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

Example of Bellman-Ford

Note: Values decreasemonotonically.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

Correctness of Bellman Ford:Correctness of Bellman Ford:Theorem. Theorem. If If G G = (= (VV, , EE) ) contains no negative-weight cycles, then after the Bellman-contains no negative-weight cycles, then after the Bellman-Ford algorithm executes, Ford algorithm executes, dd[[vv] = ] = ((ss, , vv) ) for all for all v v VV..Proof. Proof. Let Let v v V V be any vertex, and consider a shortest path be any vertex, and consider a shortest path p p from from s s to to v v with the with the minimum number of edges.minimum number of edges.

δ(s, s) =0; δ(s, v1)δ(s, v1)δ(s, v2) ……δ(s, vk-1)δ(s, vk)

Since Since p p is a shortest path, we have is a shortest path, we have ((ss, , vvii) = ) = ((ss, , vvi-1i-1) + ) + ww((vvi-1 i-1 , , vvii))..Initially, Initially, dd[[vv00] = 0 = ] = 0 = ( (ss, , vv00)), and , and dd[[ss] ] is unchanged by subsequent relaxations is unchanged by subsequent relaxations (because of the lemma from Lemma** that (because of the lemma from Lemma** that dd[[vv] ≥ ] ≥ ( (ss, , vv))).).

•After After 1 1 pass through pass through EE, we have , we have dd[[vv11] = ] = ((ss, , vv11))..•After After 2 2 passes through passes through EE, we have , we have dd[[vv22] = ] = ((ss, , vv22))..

……....•After After k k passes through passes through EE, we have , we have dd[[vvkk] = ] = ((ss, , vvkk))..

Since Since G G contains no negative-weight cycles, contains no negative-weight cycles, p p is simple.is simple.Longest simple path has Longest simple path has ≤≤||VV| | – 1 – 1 edges.edges.

Corollary. Corollary. If a value If a value dd[[vv] ] fails to converge after fails to converge after ||VV| | – 1 – 1 passes, there exists a passes, there exists a negative-weight cycle in negative-weight cycle in G G reachable from reachable from ss..

** Lemma 25.2: Relaxation: ** Lemma 25.2: Relaxation:

Let Let GG = ( = (V, EV, E) be a weighted, directed graph with weight function ) be a weighted, directed graph with weight function w w : : EE RR. . Suppose that Suppose that a shortest path a shortest path pp from a source s to a vertex from a source s to a vertex vv can be decomposed into can be decomposed into s through another path s through another path p'p'. leading to . leading to uu and and vv, for some vertex , for some vertex uu and path and path p'p'. Then, . Then, the weight of a shortest path from the weight of a shortest path from ss to to vv is is ((s, vs, v) = ) = ((s, us, u) + ) + ww((u, vu, v).).

Proof: 25.1 defines subpath p’ is a shortest path from source s to vertex v, thenProof: 25.1 defines subpath p’ is a shortest path from source s to vertex v, then

(s,v) = w(p) (s,v) = w(p) = w(p’) + w(u,v) = w(p’) + w(u,v) = = (s,v) + w(u,v).. (s,v) + w(u,v)..

** Lemma 25.3 Let ** Lemma 25.3 Let GG = ( = (V, EV, E) be a weighted, directed graph ) be a weighted, directed graph GG = ( = (V, EV, E) with ) with weight function weight function ww: : EERR and source vertex and source vertex s s = 0. Then, for all edges (= 0. Then, for all edges (u, vu, v))EE, we , we have have

((s, vs, v) ) << ((s, us, u) + ) + ww((u, vu, v).).

Proof: A shortest path Proof: A shortest path pp from source from source ss to vertex to vertex vv has no more weight than any has no more weight than any other path from other path from ss to to vv. Specifically, path . Specifically, path pp has no more weight than the particular has no more weight than the particular path that takes a shortest path from source path that takes a shortest path from source ss to vertex to vertex u u and then takes edge (and then takes edge (u, vu, v). ).

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

Topological Sorting – Topological Sorting – Book The Algorithm Design ManualBook The Algorithm Design Manualhttp://www.cs.sunysb.edu/~algorith/files/topological-sorting.shtmlhttp://www.cs.sunysb.edu/~algorith/files/topological-sorting.shtml

• Input Description:Input Description: A directed, acyclic graph A directed, acyclic graph G=(V,E)G=(V,E) (also known as a partial order or (also known as a partial order or poset).poset).

• Problem:Problem: A linear ordering of the vertices of A linear ordering of the vertices of VV such that for each edge such that for each edge (i,j)(i,j)EE, vertex , vertex ii is is to the left of vertex to the left of vertex jj..

• Topological sorting arises as a natural subproblem in most algorithms on directed acyclic Topological sorting arises as a natural subproblem in most algorithms on directed acyclic graphs. Topological sorting orders the vertices and edges of a DAG in a simple and graphs. Topological sorting orders the vertices and edges of a DAG in a simple and consistent way, therefore plays the same role of DAGs for general graphs.consistent way, therefore plays the same role of DAGs for general graphs.

• Topological sorting used to scheduling tasks under Topological sorting used to scheduling tasks under precedence constraintsprecedence constraints. . • Suppose we have a set of tasks to do, some of which have to be performed before. Suppose we have a set of tasks to do, some of which have to be performed before. • These These precedence constraints precedence constraints form a directed acyclic graph, and any topological sort (also form a directed acyclic graph, and any topological sort (also

known as a known as a linear extensionlinear extension) defines an order to do these tasks such that each is ) defines an order to do these tasks such that each is performed only after all of its constraints are satisfied.performed only after all of its constraints are satisfied.

DAG shortest path algoDAG shortest path algo• http://homepages.ius.edu/rwisman/C455/html/notes/Chapter24/SingleSource.htm

• If the graph is a DAG (no cycles), • and if the edges topologically sorted

– (a linear ordering), the order such that any directed path in DAG  of G’ traverses vertices in increasing order.

– http://www.cs.fsu.edu/~cop4531/slideshow/chapter23/23-4.html– http://www.personal.kent.edu/~rmuhamma/Algorithms/MyAlgorithms/GraphAlgor/topoSort.htm

• move outward from s• input: directed graph G = (V,E,w) and source node s in V• topologically sort G• d[s] 0• d[v] ∞ for each veach vV-{s} V-{s} • for each u in V in topological sort order do

for each neighbor v of u dod[v] min{d[v], d[u] + w(u,v)}

• Time O(V + E).

13

Linear Programming (LP)• Linear Program: An optimization problem whose constraints

and cost function are linear functions• Find a set of solutions optimizing the cost.• Applications

– Political impacts: Evaluating the preferential voting – Industrial impacts: Building roads, gun control, farm subsidies, and

gasoline tax. – Scientific applications from space to our genes.– And in particular in source distribution to maximize public health with

minimum possible cost– Oil well location decision with maximum of oil output:

• A location is associated a cost and payoff of barrels of oil.• Limited budget.

– Flight crew schedule, minimize the number of crews:• Limitation on number of consecutive hours, • Limited to one model each month,…

An Introduction To Linear Programming and the Simplex Algorithm, Reveliotis (http://www2.isye.gatech.edu/~spyros/LP/LP.html)

14

Linear Programming • Suppose all the numbers below are real numbers.• Given a linear function (called objective function)

– f(x1, x2,…, xn) = c1x1+ c2x2+…+ cnxn = j=1:n cjxj.

• With constraints: j=1:n

aij xj bi for i=1,2,…,m and– xj0 for j=1,2,…,n. (nonnegativity)

• Question: find values for x1, x2,…, xn, which maximizes f(x1, x2,…, xn).

• Or change the direction of the constraints from i to , and then minimize f(x1, x2,…, xn).

• Algorithms for the general problem• Simplex methods — practical, but worst-case exponential time.• Ellipsoid algorithm — polynomial time, but very slow in practice.• Interior-point methods — polynomial time and competes with

simplex.

15

Linear program in slack form• Except nonnegativity constraints, all other constraints are

equalities.• Change standard form to slack form:

– If j=1n aijxj bi, then introduce new variable s, and set:

– si= bi - j=1n aijxj and si0. (i=1,2,…,m).

– If j=1n aijxj bi, then introduce new variable s, and set:

– si= j=1n aijxj -bi and si0.

• All the left-hand side variables are called basic variables, whereas all the right-hand side variables are called nonbasic variables.

• Initially, s1, s1,…, sm basic variables, x1, x1,…, xn non-basic variables.

16

An example of Simplex algorithm• Maximize 3x1+x2+2x3

• Subject to: – x1+x2+3x3 30 – 2x1+2x2+5x3 24 – 4x1+x2+2x3 36 – x1, x2, x30

• Change to slack form:– z= 3x1+x2+2x3

– x4=30- x1-x2-3x3

– x5=24- 2x1-2x2-5x3 – x6=36- 4x1-x2-2x3

– x1, x2, x3, x4, x5, x6 0

17

Simplex algorithm steps• Feasible solutions (infinite number of points) • basic solution:

– set all nonbasic variables to 0 and compute all basic variables

• Iteratively rewrite the set of equations such that– No change to the underlying LP problem.– The feasible solutions keep the same.– However the basic solution changes, resulting in a greater

objective value each time:• Select a nonbasic variable xe whose coefficient in objective function is

positive, • increase value of xe as much as possible without violating any of

constraints, • xe is changed to basic and some other variable to nonbasic.

18

Simplex algorithm example • Basic solution: (x1,x2,x3,x4,x5,x6) =(0,0,0,30,24,36).

– The result is z=3 0+0+2 0=0. Not maximum.

• Try to increase the value of x1:

– z= 3x1+x2+2x3

– x4=30- x1-x2-3x3

– x5=24- 2x1-2x2-5x3

– x6=36- 4x1-x2-2x3

• 30: x4 will be OK; 12: x5; 9: x6. So only to 9.

– Change x1to basic variable by rewriting x6 to:

• x1=9-x2/4 –x3/2 –x6/4

– Note: x6 becomes nonbasic.

– Replace x1 with above formula in all equations to get:

19

• zz=27+=27+xx22/4 +/4 +xx33/2 –3/2 –3xx66/4 /4 • xx11=9-=9-xx22/4 –/4 –xx33/2 –/2 –xx66/4 /4 • xx44=21-3=21-3xx22/4 –5/4 –5xx33/2 +/2 +xx66/4 /4 • xx55=6-3=6-3xx22/2 –4/2 –4xx33 + +xx66/2 /2 • This operation is called This operation is called pivotpivot. .

– A pivot chooses a nonbasic variable, called entering variable, and a basic variable, A pivot chooses a nonbasic variable, called entering variable, and a basic variable, called leaving variable, and changes their roles.called leaving variable, and changes their roles.

– Original solutionOriginal solution (0,0,0,30,24,36) (0,0,0,30,24,36) will always satisfy the new equationswill always satisfy the new equations. .

• In the example,In the example,– xx11 is entering variable, and is entering variable, and xx66 is leaving variable. is leaving variable.– xx22, , xx33,, xx66 are nonbasic, and are nonbasic, and xx11, , xx44,, xx55 becomes basic. becomes basic.– The basic solution for this is (9,0,0,21,6,0), with The basic solution for this is (9,0,0,21,6,0), with zz=27=27..

• A new variable whose value will contribute to the objective function. A new variable whose value will contribute to the objective function. – xx66 will not work, since will not work, since zz will decrease. will decrease.– xx22 and and xx33 are only possibility. Suppose are only possibility. Suppose xx33. is chosen. is chosen

• How far can we increase How far can we increase xx33::

xx11 limits it to 18, limits it to 18, xx44 to 42/5, (and to 42/5, (and xx55=6-3=6-3xx22/2 –4/2 –4xx33 + +xx66/2) to 3/2. So rewrite /2) to 3/2. So rewrite xx55 to: to:• x3=3/2-3x2/8 –x5/4+x6/8x3=3/2-3x2/8 –x5/4+x6/8

Replace x3 with this in all the equations to get:Replace x3 with this in all the equations to get:

20

• The LP equations:The LP equations:– zz=111/4+=111/4+xx22/16 –/16 –xx55/8 - 11/8 - 11xx66/16 /16 – xx11=33/2- =33/2- xx22/16 +/16 +xx55/8 - 5/8 - 5xx66/16/16– xx33=3/2-3=3/2-3xx22/8 –/8 –xx55/4+/4+xx66/8/8– xx44=69/4+3=69/4+3xx22/16 +5/16 +5xx55/8-/8-xx66/16/16

• The basic solution is (33/4,0,3/2,69/4,0,0) with The basic solution is (33/4,0,3/2,69/4,0,0) with zz=111/4.=111/4.• Now increasing Now increasing xx2,2, is limited by is limited by xx11, , xx33, and , and xx44 to 132, 4, and to 132, 4, and respectively. So respectively. So

rewrite rewrite xx33 to to xx22=4-8=4-8xx33/3 –2/3 –2xx55/3+/3+xx66/3/3• Replace in all equations to get:Replace in all equations to get:• LP equations:LP equations:

– zz=28-=28-xx33/6 –/6 –xx55/6-2/6-2xx66/3 /3 all coefficients are negative, state with optimal solutionall coefficients are negative, state with optimal solution– xx11=8+=8+xx33/6 +/6 +xx55/6-/6-xx66/3/3– xx22=4-8=4-8xx33/3 –2/3 –2xx55/3+/3+xx66/3/3– xx44=18-=18-xx33/2 +/2 +xx55/2./2.

• At this point, all coefficients in objective functions are At this point, all coefficients in objective functions are negativenegative. So no further . So no further rewrite can be done. rewrite can be done. This is the state with optimal solution.This is the state with optimal solution.

• And the final And the final basic solutionbasic solution is is (8,4,0,18,0,0)(8,4,0,18,0,0) with objective value with objective value zz=28.=28.• The original variables are The original variables are xx11, , xx22, , xx3 3 , with values (8,4,0), , with values (8,4,0),

the objective value is 3 the objective value is 3 8+4+2 8+4+2 0=28. 0=28.

Simplex algorithm --Pivot

NN: indices set of nonbasic variables: indices set of nonbasic variablesBB: indices set of basic variables: indices set of basic variablesAA: : aaijij

bb: : bbii

cc: : ccii

vv: constant coefficient.: constant coefficient.ee: index of entering variable: index of entering variablell: index of leaving variable: index of leaving variable

13:13: z z = = v v + + jjN N ccjjxxj j

14: x14: xi i = = bbi i - - jjNNaaijijxxjj for for iiBB

22

Formal Simplex algorithm

+m

Running time of Simplex

• Lemma:– Assuming that INITIALIZE-SIMPLEX returns a

slack form for which the basic solution is feasible, SIMPLEX either reports that a linear program is unbounded, or it terminates with a feasible solution in at most ( ) iterations.

• Feasible solution: a set of values for xi’s which satisfy all constraints.

• Unbound: has feasible solutions but does not have a finite optimal objective value.

m+nm

(0,0)(0,0)

C1C1C2C2

yy

xx

From From Romil Jain’s notes

STEP 1: An initial point x=0, y=0STEP 1: An initial point x=0, y=0 Feasible solutionFeasible solution x=0, y=0, P = 0x=0, y=0, P = 0

STEP 2: Next point, increasing x can exceed 30 due to the negativity of sSTEP 2: Next point, increasing x can exceed 30 due to the negativity of s22 which which

suggests considering (2) to obtain x , (Pivoting) suggests considering (2) to obtain x , (Pivoting)

x = 30 – y – sx = 30 – y – s22/3 /3 and inserting x in (1), and inserting x in (1), ss11 = 40 + 2/3s = 40 + 2/3s22 – 2y – 2y

P = 150 – 5/3sP = 150 – 5/3s22 + 2y + 2y Now put y, sNow put y, s2 2 = 0= 0 Feasible solution Feasible solution x=30, y=0, P = 150x=30, y=0, P = 150

x = 30 – y – sx = 30 – y – s22/3 s/3 s11 = 40 + 2/3s = 40 + 2/3s22 – 2y s – 2y s1, 1, , s, s2 2 , x , y , x , y 0 0

STEP 3: The next point , increasing y has a maximum to 20 (limited by the sSTEP 3: The next point , increasing y has a maximum to 20 (limited by the s11))

(Pivoting) y = 20 + 1/3s(Pivoting) y = 20 + 1/3s2 2 – 1/2s– 1/2s11 x = 10 – 1/2sx = 10 – 1/2s11 - 2/3s - 2/3s22

P = 190 - sP = 190 - s11 – s – s22 Feasible solution Feasible solution x=10, y=20 P = 190x=10, y=20 P = 190

Cost Function maximize P => 5x + 7y

Constraint Functions Using slack variables: s1,, s2

C1: 2x + 4y 100 s1 = 100 - 2x - 4y (1)

C2: 3x + 3y 90 s2 = 90 - 3x - 3y (2)

Non-Negativity:

x,y 0 x, y 0, s1,, s2

25

Find maximum via graph

26

Two variable LP problems

• Example:– Maximize x1+x2

– Subject to: – 4x1- x2 8 – 2x1+x2 10 – 5x1- 2x2 -2 – x1, x2 0– Graphically – By prune-and-search approach

27

A matrix view of Linear program in slack formLet A be an [m, n] matrix, [b] be an m-vector, and [c] be an n-

vector. Find an n-vector [x] that maximizes cTx subject to Ax ≤ b, or determine that no such solution exists.

Feasibility problem: No optimization criterion.Just find x such that Ax ≤ b.• In general, just as hard as ordinary LP.

Primal LP: maximize CT.X subject to A.X B. Canonical formmax(Σcj

. xj) subject to Aj,i . Xj Bj

Dual LP: minimize BT.y such that AT.B Y. Canonical formmin(Σbi .yi) s. t. Aj,i.Bj Cj

Maximize Function : 21x1 - 6x2 – 100x3 - 100x4

Subject to:5x1 + 2x2 +31x3 - 20x4 21x1 - 4x2 +3x3 + 10x4 566x1 + 60x2 - 31x3 - 15x4 200

Three types of Linear Programs:• Has an optimal solution with a finite cost value: e.g. nutrition problem• Unbounded: e.g maximize x, x x • Infeasible: e.g maximize x, x x x

Each linear programming problem (the primal problem) has an associated dual problem.• THEOREM: the maximum value of the primal (profit max problem) equals the

minimum value of the dual (cost minimization) problem. • The resource constraints of the primal problem appear in the objective function of the

dual problem

Primal LP: minimize C.x such that Q.x N. min(CiXi) subject to Mi,j Xi Ni Remember the only thing you can change is the set of xi

Dual LP: maximize NT.y such that QT.y CT. max(NjT.Yj) subject to Mj,i YjCj

T

Romil Jain’s notes

Primal:Primal:MaximizeMaximize= P= P11·Q·Q11 + P + P22·Q·Q22 subject to:subject to:

c·Qc·Q1 1 + d·Q+ d·Q22 <<RR11 The budget constraint, for example.The budget constraint, for example.

e·Qe·Q11 + f·Q + f·Q22 << R R22 The machine scheduling time The machine scheduling time constraint.constraint.

where Qwhere Q11 and Q and Q2 2 >> 0 0 Nonnegativity constraint.Nonnegativity constraint.

maximize maximize 33x1 + x1 + 55x2x2 minimize minimize 44y1 +y1 + 6 6y2 + y2 + 1818y3y3s.t. s.t. s.t. s.t. x1 x1 44 y1 + y1 + 33y3 y3 33 x2 x2 66 y2 + y2 + 22y3 y3 55 33·x1 + ·x1 + 22·x2 ·x2 1818

x1, x2 x1, x2 0 0 y1, y2, y3 y1, y2, y3 0 0

Dual:Dual:MinimizeMinimizeCC= R= R11·w·w11 + R + R22·w·w22 subject to:subject to:

c·Wc·W1 1 + e·W+ e·W22 >>PP11 Profit Contribution of Product 1Profit Contribution of Product 1

d·Wd·W11 + f·W + f·W22 >> P P22 Profit Contribution of Product 2Profit Contribution of Product 2

where Wwhere W11 and W and W2 2 >> 0 0 Nonnegativity constraint.Nonnegativity constraint.

• Each constraint has an implicit price, the shadow price of the constraint. If a constraint is slack, its shadow price is zero.

• Each shadow price has much the same meaning as a Lagrangian multiplier.Cost Minimization Problem Using Linear Programming• Multi-plant firms want to produce with the lowest cost across their disparate

facilities. Sometimes, the relative efficiencies of the different plants can be exploited to reduce costs.

• A firm may have two mines that produces different qualities of ore. The firm has output requirements in each ore quality.

• Scheduling of hours per week in each mine has the objective of minimizing cost, but achieving the required outputs.

• If one mine is more efficient in all categories of ore, and is less costly to operate, the optimal solution may involve shutting one mine down.

• The dual of this problem involves the shadow prices of the ore constraints. It tells the implicit value of each quality of ore.

Capital Rationing Problem: Financial decisions sometimes may be viewed as a linear programming problem.

• A financial officer may want to maximize the return on investments available, given a limited amount of money to invest.

• The usual problem in finance is to accept all projects with positive net present values, but sometimes the capital budgets are fixed or limited to create "capital rationing" among projects.

• The solution involves determining what fraction of money allotted should be invested in each of the possible projects or investments.

• In some problems, projects cannot be broken into small parts. • When this is the case, integer programming can be added to the problem.

31

• A linear program problem with additional constraint that all variables must take integer!!! values.– Given an integer mxn matrix A and an integer m-vector b,

whether there is an integer n-vector x such that Ax<=b.

– this problem is NP-complete.

• However the general linear program problem is poly time solvable.

Complexity of simplex and LP complete• Usually runs quickly in practice. However, note in the worst-case

the simplex method takes exponential time. The Klee-Minty Cube is an example LP problem the simplex taking exponential running time. This is a perturbation of an n-dimensional cube having about 2n vertices. For most pivot choices the simplex method will visit all of these vertices in 2n-1 pivot steps to find the optimal solution.

• Some linear programming algorithms that do have polynomial bounds in the worst-case will also be mentioned.

• The LP-Complete class of problems is the set of problems X such that: – the problem can be formulated as an instance of a LP, this means that there is

a polynomial-time deterministic algorithm that transforms instances of the problem X into instances of linear programs. So, first converting it into an LP problem in polynomial time and then solving LP problem.

– the problem is LP-hard. This is analogous to the definition for NP-Complete problems. Means that there is a polynomial-time deterministic algorithm converting any instance of an LP problem into an instance of problem type X. The LP-complete class of problems is polynomial-time solvable, i.e. P = LP.

Simplex method traverses the boundary of the polyhedron along its Simplex method traverses the boundary of the polyhedron along its edges, so its running time depends on the complexity of the edges, so its running time depends on the complexity of the polyhedron, even though we don’t really care about the polyhedron, even though we don’t really care about the polyhedron but are only interested in one of its vertices. polyhedron but are only interested in one of its vertices.

The ellipsoid methodThe ellipsoid method always remains outside of the polyhedron, and always remains outside of the polyhedron, and keeps on shrinking the ellipsoid until it finds a point inside the keeps on shrinking the ellipsoid until it finds a point inside the polyhedron. So one could think of the ellipsoid method as an polyhedron. So one could think of the ellipsoid method as an exterior point methodexterior point method. .

Instead, Instead, interior point methodsinterior point methods always remaining always remaining inside the inside the polyhedronpolyhedron, by starting inside and following a path to the , by starting inside and following a path to the boundary. boundary. This means the complexity of these methods does not This means the complexity of these methods does not really depend on the complexity of the polyhedron.really depend on the complexity of the polyhedron.

Among many of the Among many of the interior point methodsinterior point methods, , log barrier methodlog barrier method..Simulating a force at the boundary repelling the path in order to Simulating a force at the boundary repelling the path in order to keep solution sought inside the polyhedron at all times. But keep solution sought inside the polyhedron at all times. But initially the force is very strong, which restricts the search to a initially the force is very strong, which restricts the search to a small area in the center of the polyhedron, and the force weakens small area in the center of the polyhedron, and the force weakens with time to allow the path to eventually reach the boundary and with time to allow the path to eventually reach the boundary and find the optimal vertex of the polyhedron.find the optimal vertex of the polyhedron.

34

Interior Point MethodsInterior Point Methods• Logarithmic Barrier MethodLogarithmic Barrier Method

to max cto max cTTx, xx, xRRnn s.t. s.t. AAxxb, xb, x 0, 0,

Initially assuming a Initially assuming a strictly feasiblestrictly feasible point of point of xx00 satisfying satisfying hh((xx00)>0, )>0, and the boundary of the feasible set is not crossed. and the boundary of the feasible set is not crossed. A natural strategy is to decrease A natural strategy is to decrease ff.. .. One way to prevent an optimization algorithm from crossing the One way to prevent an optimization algorithm from crossing the boundary is to assign a penalty to prevent approaching to boundary is to assign a penalty to prevent approaching to boundaries. The most popular way of doing this is to augment the boundaries. The most popular way of doing this is to augment the objective function by a objective function by a logarithmic barrierlogarithmic barrier term: term:

B(x, µ) = f - µ B(x, µ) = f - µ ΣΣi:1.P i:1.P ln(hln(hii((xx00)))) = max c= max cTTx - x - µ µ ΣΣi:1.m i:1.m lnln((aaiiTTx - bx - bii)), ,

wwhere here µµ is the penalty function, the strength of repelling forces is the penalty function, the strength of repelling forces from boundaries towards inside.from boundaries towards inside.

35

Interior Point MethodsInterior Point MethodsB(x, µ) ``blows up'' at the boundary since ln(0) B(x, µ) ``blows up'' at the boundary since ln(0) inf, and inf, and therefore presents an optimization algorithm with ``barrier'' to therefore presents an optimization algorithm with ``barrier'' to crossing the boundary, where barrier is gradually relaxed, by crossing the boundary, where barrier is gradually relaxed, by reducing µ. reducing µ.

• www.math.mtu.edu/~msgocken/ma5630spring2003/lectures/bar/bar/node2.htmlwww.math.mtu.edu/~msgocken/ma5630spring2003/lectures/bar/bar/node2.html

• Prune-and-search approach (Meggido’s Algorithm) is equivalent Prune-and-search approach (Meggido’s Algorithm) is equivalent to calculating convex hull of n points in O(n log(n)), for where to calculating convex hull of n points in O(n log(n)), for where the intersection points of constrain functions are resumed for the intersection points of constrain functions are resumed for minimum and maximum. Therefore some constraint functions are minimum and maximum. Therefore some constraint functions are pruned. pruned.

Solving a system of differenceconstraints

Linear programming where each row of A containsexactly one 1, one –1, and the rest 0’s.

Example: Solution:

Constraint graph: (The “A”matrix hasdimensions|E| × |V|.)

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

x1 – x2 ≤ 3x2 – x3 ≤ –2x1 – x3 ≤ 2

xj – xi ≤ wij

x1 = 3x2 = 0x3 = 2

xj – xi ≤ wij

Unsatisfiable constraintsTheorem. If the constraint graph containsa negative-weight cycle, then the system ofdifferences is unsatisfiable.Proof. Suppose that the negative-weight cycle isv1 → v2 → → vk → v1. Then, we have

Therefore, novalues for the xi

can satisfy theconstraints.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

x2 – x1 ≤ w12

x3 – x2 ≤ w23

xk – xk-1 ≤ wk–1, k

x1 – xk ≤ wk1

0 ≤ weight of cycle < 0

Satisfying the constraints

Theorem. Suppose no negative-weight cycleexists in the constraint graph. Then, theconstraints are satisfiable.Proof. Add a new vertex s to V with a 0-weight edgeto each vertex vi V.

Note:No negative-weightcycles introduced shortest paths exist.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

Proof (continued)Claim: The assignment xi = (s, vi) solves the constraints.Consider any constraint xj – xi ≤ wij, and consider theshortest paths from s to vj and vi:

The triangle inequality gives us (s,vj) ≤ (s, vi) + wij.Since xi = (s, vi) and xj = (s, vj), the constraint xj – xi

≤ wij is satisfied.© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

Bellman-Ford and linear programming

Corollary. The Bellman-Ford algorithm cansolve a system of m difference constraints on nvariables in O(mn) time.

Single-source shortest paths is a simple LP problem.

In fact, Bellman-Ford maximizes x1 + x2 + + xn

subject to the constraints xj – xi ≤ wij and xi ≤ 0.

Bellman-Ford also minimizes maxi{xi} – mini{xi}.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.

…

41

Systems of Difference Systems of Difference ConstraintsConstraints

0 0 1- 0 1

0 0 1 1- 0 A

• Each row is a constraint equation of LP matrix A contains zeros and only one 1 and one -1, m difference equations with n unknowns, of the form,

Ax b, xj –xi bk where 1i, j n and 1km

• For example: xi is the time at which event i occurs and event j occurs at least bk hours later..

Example

1 1- 0 0 0

1 0 1- 0 0

0 1 1- 0 0

0 1 0 0 1-

0 0 1 0 1-

1- 0 0 1 0

1- 0 0 0 1

0 0 0 1- 1

xxxxx

5

4

3

2

1

3-

3-

1-

4

5

1

1-

0

1 2

1 5

2 5

3 1

4 1

4 3

5 3

5 4

0

1

1

5

4

1

3

3

x x

x x

x x

x x

x x

x x

x x

x x

Find a vector: x = <x1,x2 x3,x4,x5> that:

equivalent to solvingthe difference constraints:

One solution is x = (-5, -3, 0, -1, -4)

Another solution is x = (0, 2, 5, 4, 1)

In fact, for any d, (d-5, d-3, d, d-1, d-4) is a solution!

Lemma:

Let x = (x1,x2,…,xn) be a solution to a system Ax b of difference constraints.

Let d be any constant.

Then x+d = (x1+d,x2+d,…,xn+d) is also a solution to Ax b

Proof:

For each xi and xj, we have (xj+d) - (xi+d) = xj - xi

Thus, if x satisfies Ax b, so does x+d

44

Constraint Graphs• A system of difference constraints Ax b with

n variables and m constraints can be represented as a directed weighted graph G = (V, E)

• Each variable xi corresponds to a vertex viV

In addition, there is a special vertex v0

V = {v0, v1, …, vn}

• Each constraint xj-xi bk corresponds to an edge (vi, vj) E

In addition, there is an edge from v0 to every other node

E = {(vi, vj,): xj-xi bk is a constraint …, vn}{(v0, v1), (v0, v2), …, (v0, vn)}

to guarantee that every vertex is reachable from v0.

• If xj-xi bk is a difference constraint, then the weight of edge (vi, vj) is w(vi, vj) = bk.

The weight of each edge leaving v0 is 0

45

Example

v0

v1

v3

v5 v2

v4

-1

-3 -3

-1

45

1

000

0

0

0

1 2

1 5

2 5

3 1

4 1

4 3

5 3

5 4

0

1

1

5

4

1

3

3

x x

x x

x x

x x

x x

x x

x x

x x

It is possible to find a feasible solution to a system of difference constraints by finding shortest-path weights (from v0) in the corresponding constraint graph

46

Finding a Feasible SolutionTheorem - part I:Given a system Ax b of difference constraints Let G = (V,E) be the corresponding constraint graph

If G contains no negative-weight cycles,then x = ((v0,v1), (v0,v2),…, (v0,vn))is a feasible solution

Proof:Consider any edge (vi, vj)E

By the triangle inequality: (v0,vj) (v0,vi) + w(vi, vj)

(v0,vj) - (v0,vi) w(vi, vj)

Thus, xj-xi w(vi, vj) = bk that corresponds to the edge (vi, vj)

47

Theorem - part II:Theorem - part II:If G contains a negative-weight cycle,If G contains a negative-weight cycle,then there is no feasible solution for the systemthen there is no feasible solution for the systemProof:Proof:Suppose that there is such a negative weight cycle.Suppose that there is such a negative weight cycle.

Wlog let this cycle be c = <vWlog let this cycle be c = <v11, v, v22, ..., v, ..., vkk>, where v>, where v11 = v = vkk

(v(v00 cannot be in c, because it has no incoming edges) cannot be in c, because it has no incoming edges)Cycle c corresponds to the following constraints:Cycle c corresponds to the following constraints:xx2 2 - - xx11 ww((vv11, , vv22),),xx3 3 - - xx22 ww((vv22, , vv33),),…………

xxk k – – xxk-1k-1 ww((vvk-k-11, , vvkk),),xx11 - - xxkk ww((vvkk, , vv11).).------------------------------------------------------

0 0 w(c) < 0 w(c) < 0 Suppose that there is a solution x satisfying these k inequalities. Suppose that there is a solution x satisfying these k inequalities.

Then x also satisfies the inequality that results when we sum Then x also satisfies the inequality that results when we sum the k inequalities together. the k inequalities together. Contradiction to c being a negative Contradiction to c being a negative weight cycle, i.e., weight cycle, i.e.,

48

Example: Solving using Bellman-Ford

0

-5

-4 -3

-1

-3 -3

-1

45

1

0

0

0

0

0

0

v1

v2

v3v4

v5

v0 v1 v2 v3 v4 v5

0 0 ∞ ∞ ∞ ∞ ∞

1 0 -5 -3 0 -1 -4

v0

Edge order:

0 1 0 5

1 3 1 4 2 1

3 4 3 5 4 5

5 1 5 2

( , ), , ( , ),

( , ), ( , ), ( , ),

( , ), ( , ), ( , ),

( , ), ( , )

v v v v

v v v v v v

v v v v v v

v v v v

0

0

0

0

0-1

-3

x = (-5, -3, 0, -1, -4) is a feasible solution

so is (d-5, d-3, d, d-1, d-4), for any d

Time complexity:|V| = n+1; |E| = m

+ nBellman-Ford: O(VE) = O((n+1)(m+n))

= O(n2+nm)How can we reduce time complexity to

O(nm)?

49

Formatting problems as LPs

• (Single pair) Shortest path :– A weighted direct graph G=<V,E> with weighted

function w: ER, a source s and a destination t, compute d which is the weight of the shortest path from s to t.

– Change to LP:• For each vertex v, introduce a variable xv: the weight of the

shortest path from s to v.

• Maximize xt with the constraints:

• xv xu+w(u,v) for each edge (u,v)E, and xs =0.

50

Formatting Max-flow problem as LPs• Max-flow problem:

– A directed graph G=<V,E>, a capacity function on each edge c(u,v) 0 and a source s and a sink t. A flow is a function f : VVR that satisfies:

• Capacity constraints: for all Capacity constraints: for all uu,,vvV, V, ff((uu,,vv)) cc((uu,,vv).).• Skew symmetry: for all Skew symmetry: for all uu,,vvV, V, ff((uu,,vv)= -)= -ff((vv,,uu).).• Flow conservation: for all Flow conservation: for all uuV-{V-{ss,,tt}, }, vvVV ff((uu,,vv)=0, or to say, total )=0, or to say, total

flow out of a vertex other flow out of a vertex other ss or or tt is 0, or to say, how much comes in, is 0, or to say, how much comes in, also that much comes out. also that much comes out.

– Find a maximum flow from s to t.

• Maximize vV f(s,v) • Subject to:

• for all u,vV, f(u,v) c(u,v).• for all u,vV, f(u,v)= -f(v,u).• for all uV-{s,t}, vV f(u,v)=0.

51

Example of max-flow problemExample of max-flow problem

52

Example: Max-flow (borrowed from lecture-notes (Alex Shraer of Technion EE )- (Prof. Vazirani)

s t

a

b

3 1

1

2 3

0 , , , ,

0

0

3

1

1

2

3

btatabsbsa

btabsb

atabsa

bt

at

ab

sb

sa

fffff

fff

fff

f

f

f

f

f

sbffMaximize sa

s.t.

53

The Dual Program

btatabsb yyyyyMin 323 sa

0 , , , ,

0

0

3

1

1

2

3

btatabsbsa

btabsb

atabsa

bt

at

ab

sb

sa

fffff

fff

fff

f

f

f

f

f

sbffMax sa

s.t.

0 , , , ,

0

0

0

1

1

btatabsbsa

bbt

aat

baab

bsb

asa

yyyyy

uy

uy

uuy

uy

uy

Each uv abbreviates uv’ – uv’’ where uv’, uv’’ 0

54

• The dual program corresponds to the Min-cut problem! The dual program corresponds to the Min-cut problem!

• Suppose that the cut is (S, T)Suppose that the cut is (S, T)

• u variables correspond to nodes. uu variables correspond to nodes. uvv will be 1 if v will be 1 if vS and 0 S and 0

otherwiseotherwise

• y variables correspond to edges. yy variables correspond to edges. ykvkv will be 1 if k will be 1 if kS and vS and vT, T,

i.e., the edge contributes to the cut and 0 otherwisei.e., the edge contributes to the cut and 0 otherwise

• For example:For example:

– yysa sa + u+ ua a 1 states that if a 1 states that if aS, then (s,a) must be in the cutS, then (s,a) must be in the cut

– yyab ab – u– ua a + u+ ub b 0 states that if a 0 states that if aS and bS and bS then (a,b) must be S then (a,b) must be

in the cutin the cut

• Although the variables can be greater than 1, the minimization Although the variables can be greater than 1, the minimization of the objective function will make them be 0 or 1.of the objective function will make them be 0 or 1.

• The constraints define a legal cutThe constraints define a legal cut

• The objective function looks for a cut with minimal capacityThe objective function looks for a cut with minimal capacity

55

• By the Max-flow/Min-cut theoremBy the Max-flow/Min-cut theorem, the two linear programs , the two linear programs you must see have the same optimal objective value.you must see have the same optimal objective value.

• Which is true for a general LP!Which is true for a general LP!

• Strong Duality Theorem: Strong Duality Theorem: The The primal program has a finite primal program has a finite optimumoptimum if and only if the dual program has a finite optimumif and only if the dual program has a finite optimum. . Moreover, if xMoreover, if x** is an optimal solution of the primal problem is an optimal solution of the primal problem and yand y* * is an optimal solution to the dual problem, then is an optimal solution to the dual problem, then

• ccTTx* = bx* = bTTy* y*

• The best proof comes from the simplex algorithm, very much The best proof comes from the simplex algorithm, very much as the max-flow min-cut theorem comes from the max-flow as the max-flow min-cut theorem comes from the max-flow algorithmalgorithm