Integer and Combinatorial Optimization:Algorithm Complexity
John E. Mitchell
Department of Mathematical SciencesRPI, Troy, NY 12180 USA
January 2019
Mitchell Algorithm Complexity 1 / 23
How run-time grows as problem size grows
Outline
1 How run-time grows as problem size grows
2 Polynomial time algorithms and the class PExamples of problems in P
Mitchell Algorithm Complexity 2 / 23
How run-time grows as problem size grows
Big-O notation
DefinitionLet f and g be functions from the set of integers or the set of reals to
the set of real numbers. The function f (k) is O(g(k)) whenever there
exist positive constants C and k 0 such that
|f (k)| C|g(k)| 8 k � k0.
Mitchell Algorithm Complexity 3 / 23
106×4 i s 01×4) 106×3 e 10-6×4 i s
01×4)
How run-time grows as problem size grows
Size of an instance
The size of an instance is the number of binary characters required tostore the problem.For example, given a positive integer x , we set p = blog2(x)c. Then
2p x < 2p+1 and x =pX
i=0
�i2i for some binary �i .
For example, 42 = 0⇥ 20 + 1⇥ 21 + 0⇥ 22 + 1⇥ 23 + 0⇥ 24 + 1⇥ 25.So we need O(log(x)) bits to store x .
We don’t need to specify the base for the logarithm here, because
if x > 0 and y = loga(x), z = logb(x) then y = z loga(b).
Mitchell Algorithm Complexity 4 / 23
4 2 = 2 + 8 + 3 2
How run-time grows as problem size grows
Data representation
The major classification of algorithm efficiency is insensitive to thechoice of data representation, for the most part.
There are two restrictions:1 The alphabet used to represent the data must contain at least two
symbols. So the space to store a positive integer x is O(log(x)),instead of x with a unary alphabet.
2 The “data” can’t contain information that requires a lot ofcalculation. For example, can’t store the length of all tours as partof the representation of a traveling salesman problem.
Mitchell Algorithm Complexity 5 / 23
4 1 - # # # # # # # # 1 1
How run-time grows as problem size grows
Data representation
The major classification of algorithm efficiency is insensitive to thechoice of data representation, for the most part.
There are two restrictions:1 The alphabet used to represent the data must contain at least two
symbols. So the space to store a positive integer x is O(log(x)),instead of x with a unary alphabet.
2 The “data” can’t contain information that requires a lot ofcalculation. For example, can’t store the length of all tours as partof the representation of a traveling salesman problem.
Mitchell Algorithm Complexity 5 / 23
How run-time grows as problem size grows
Computation time
The time required for an elementary operation (addition, subtraction,multiplication, division, comparison) is counted as unit time.
(Need to be a little careful if multiply very large numbers together.)
Mitchell Algorithm Complexity 6 / 23
How run-time grows as problem size grows
Instances
Perfect matching, linear programming, and node packing are allexamples of optimization problems.
An optimization problem X consists of an infinite number of instancesd1, d2, . . . where the data for instance di is given by a binary string oflength l(di).
For example, an instance of a perfect matching problem is given bylists of the vertices and edges of the graph G = (V ,E).
Mitchell Algorithm Complexity 7 / 23
How run-time grows as problem size grows
Worst case running time
Let A be an algorithm the solves every instance of X in finite time.
Let gA(di) be the running time of A on instance di .
Measure performance of A on X using the worst-case running time, foreach problem size k :
fA(k) := max{gA(di) : l(di) = k}.
Advantages of using worst-case performance measure:Gives a guarantee of performance of the algorithmIndependent of the probability distribution of the instances.Appears to be the easiest to analyze.
Mitchell Algorithm Complexity 8 / 23
How run-time grows as problem size grows
Worst case running time
Let A be an algorithm the solves every instance of X in finite time.
Let gA(di) be the running time of A on instance di .
Measure performance of A on X using the worst-case running time, foreach problem size k :
fA(k) := max{gA(di) : l(di) = k}.
Advantages of using worst-case performance measure:Gives a guarantee of performance of the algorithmIndependent of the probability distribution of the instances.Appears to be the easiest to analyze.
Mitchell Algorithm Complexity 8 / 23
How run-time grows as problem size grows
Worst case running time
Let A be an algorithm the solves every instance of X in finite time.
Let gA(di) be the running time of A on instance di .
Measure performance of A on X using the worst-case running time, foreach problem size k :
fA(k) := max{gA(di) : l(di) = k}.
Advantages of using worst-case performance measure:Gives a guarantee of performance of the algorithmIndependent of the probability distribution of the instances.Appears to be the easiest to analyze.
Mitchell Algorithm Complexity 8 / 23
How run-time grows as problem size grows
Klee-MintyFor example, the simplex algorithm solves any linear optimizationproblem in finite time (provided we use an anti-cycling rule).
In the worst case, it may require O(2n) iterations to solve a problem inn nonnegative variables with n inequality constraints (namely, theKlee-Minty cube).
x1
x2
x3
(0, 0, 0)(0, 100, 0)
(0, 0, 10000)
(1, 0, 0)
(1, 80, 0)
(0, 100, 8000)
(1, 80, 8200)(1, 0, 9800)
Mitchell Algorithm Complexity 9 / 23
Polynomial time algorithms and the class P
Outline
1 How run-time grows as problem size grows
2 Polynomial time algorithms and the class PExamples of problems in P
Mitchell Algorithm Complexity 10 / 23
Polynomial time algorithms and the class P
Feasibility problems
For example, an instance of the perfect matching feasibility problem
consists of a graph G = (V ,E) and asks the question “Does there exista perfect matching on this graph”? The answer is either Yes or No.
DefinitionA feasibility problem X consists of a set of instances D and a set of
feasible instances F ✓ D. Given an instance d 2 D, determine
whether d 2 F. The answer is either Yes or No.
Mitchell Algorithm Complexity 11 / 23
÷⇒ .
Polynomial time algorithms and the class P
Binary searchMany instances of optimization problems can be solved using binarysearch on a sequence of instances of feasibility problems.
For example, can solve the 0� 1 linear optimization problem
minx
{cT
x : Ax � b, x 2 Bn}
by repeatedly solving the 0� 1 linear feasibility problem
Does there exist x 2 Bn with Ax � b, cT
x z?
At each iteration, we have upper and lower bounds on the optimalvalue z⇤. The initial bounds on z⇤ can be taken to be
zUB =
X
j:cj>0
cj , zLB =
X
j:cj<0
cj
since x is binary. We solve the feasibility problem withz = 1
2(zLB + zUB). We assume all the data is integral, so the optimal
value is integral, so we can maintain integral values for zLB, zUB, and z.Mitchell Algorithm Complexity 12 / 23
m m Tx , - T x , t 7x , -6×4 = : z
s t . A x > b
x binary.
Then 2 3 - 9 (assuming feasible)
a n d z e 1 2 (achievedi f onlyfealibl. solutioni s
x = (491,0).)
- Doe,Ixe lB"Initial guns: 2 = 1 f L'¥9))). w i n Axes,I f N O : love bound→ 2 . exez?i f Y E S : upperbound→ I .
Polynomial time algorithms and the class P
Binary search algorithm
1 Initialize zUB, zLB.2 If zLB = zUB, STOP and return the optimal value
z⇤ = zLB = zUB.3 Let z =
⌅12(z
LB + zUB)⇧.
4 Solve the 0� 1 linear feasibility problem
Does there exist x 2 Bn with Ax � b, cT
x z?
5 If answer is YES: update zUB z, return to Step 2.6 If answer is NO: update zLB z + 1, return to Step 2.
Let z0 be the initial value of zUB � zLB. The number of iterations of thealgorithm is O(log2(z
0)), since the difference between the bounds ishalved at each iteration.
Mitchell Algorithm Complexity 13 / 23
Polynomial time algorithms and the class P
Binary search algorithm
1 Initialize zUB, zLB.2 If zLB = zUB, STOP and return the optimal value
z⇤ = zLB = zUB.3 Let z =
⌅12(z
LB + zUB)⇧.
4 Solve the 0� 1 linear feasibility problem
Does there exist x 2 Bn with Ax � b, cT
x z?
5 If answer is YES: update zUB z, return to Step 2.6 If answer is NO: update zLB z + 1, return to Step 2.
Let z0 be the initial value of zUB � zLB. The number of iterations of thealgorithm is O(log2(z
0)), since the difference between the bounds ishalved at each iteration.
Mitchell Algorithm Complexity 13 / 23
Polynomial time algorithms and the class P
Binary search algorithm
1 Initialize zUB, zLB.2 If zLB = zUB, STOP and return the optimal value
z⇤ = zLB = zUB.3 Let z =
⌅12(z
LB + zUB)⇧.
4 Solve the 0� 1 linear feasibility problem
Does there exist x 2 Bn with Ax � b, cT
x z?
5 If answer is YES: update zUB z, return to Step 2.6 If answer is NO: update zLB z + 1, return to Step 2.
Let z0 be the initial value of zUB � zLB. The number of iterations of thealgorithm is O(log2(z
0)), since the difference between the bounds ishalved at each iteration.
Mitchell Algorithm Complexity 13 / 23
Polynomial time algorithms and the class P
Binary search algorithm
1 Initialize zUB, zLB.2 If zLB = zUB, STOP and return the optimal value
z⇤ = zLB = zUB.3 Let z =
⌅12(z
LB + zUB)⇧.
4 Solve the 0� 1 linear feasibility problem
Does there exist x 2 Bn with Ax � b, cT
x z?
5 If answer is YES: update zUB z, return to Step 2.6 If answer is NO: update zLB z + 1, return to Step 2.
Let z0 be the initial value of zUB � zLB. The number of iterations of thealgorithm is O(log2(z
0)), since the difference between the bounds ishalved at each iteration.
Mitchell Algorithm Complexity 13 / 23
Polynomial time algorithms and the class P
Binary search algorithm
1 Initialize zUB, zLB.2 If zLB = zUB, STOP and return the optimal value
z⇤ = zLB = zUB.3 Let z =
⌅12(z
LB + zUB)⇧.
4 Solve the 0� 1 linear feasibility problem
Does there exist x 2 Bn with Ax � b, cT
x z?
5 If answer is YES: update zUB z, return to Step 2.6 If answer is NO: update zLB z + 1, return to Step 2.
Let z0 be the initial value of zUB � zLB. The number of iterations of thealgorithm is O(log2(z
0)), since the difference between the bounds ishalved at each iteration.
Mitchell Algorithm Complexity 13 / 23
Polynomial time algorithms and the class P
Binary search algorithm
1 Initialize zUB, zLB.2 If zLB = zUB, STOP and return the optimal value
z⇤ = zLB = zUB.3 Let z =
⌅12(z
LB + zUB)⇧.
4 Solve the 0� 1 linear feasibility problem
Does there exist x 2 Bn with Ax � b, cT
x z?
5 If answer is YES: update zUB z, return to Step 2.6 If answer is NO: update zLB z + 1, return to Step 2.
Let z0 be the initial value of zUB � zLB. The number of iterations of thealgorithm is O(log2(z
0)), since the difference between the bounds ishalved at each iteration.
Mitchell Algorithm Complexity 13 / 23
I f c a n solve fearibility problem i npolynomial t ime f(di) forinsta.. d i :
Then time t o solve ophinication problem
i s f (di) log(initialgap)
i n i npolynomial polynomial
i n storage i n storage
i npolynomial i n storage.
Polynomial time algorithms and the class P
The Class P
DefinitionAlgorithm A is a polynomial time algorithm for the feasibility problem
X if fA(k) is O(kp) for some fixed p.
P denotes the class of feasibility problems that can be solved in
polynomial time.
Problem X 2 P if and only if there is a polynomial time algorithm for X.
Mitchell Algorithm Complexity 14 / 23
µk : storage.
Polynomial time algorithms and the class P Examples of problems in P
Outline
1 How run-time grows as problem size grows
2 Polynomial time algorithms and the class PExamples of problems in P
Mitchell Algorithm Complexity 15 / 23
Polynomial time algorithms and the class P Examples of problems in P
Minimum spanning tree with upper boundGiven a graph G = (V ,E), edge weights we for e 2 E , and an integerz, does there exist a spanning tree with total weight no greater than z?
Use a greedy algorithm.
1
2
3
4
5
6
7
8
2
351 5
4
67
4
6
7
64
8
3 5
Weight 25?
Solution:
1
2
3
4
5
6
7
8
2
31
4
6
4
3
YES (weight 23)
Mitchell Algorithm Complexity 16 / 23
Polynomial time algorithms and the class P Examples of problems in P
Minimum spanning tree with upper boundGiven a graph G = (V ,E), edge weights we for e 2 E , and an integerz, does there exist a spanning tree with total weight no greater than z?
Use a greedy algorithm.
1
2
3
4
5
6
7
8
2
351 5
4
67
4
6
7
64
8
3 5
Weight 25?
Solution:
1
2
3
4
5
6
7
8
2
31
4
6
4
3
YES (weight 23)
Mitchell Algorithm Complexity 16 / 23
Polynomial time algorithms and the class P Examples of problems in P
Shortest path with upper boundGiven a graph G = (V ,E), edge weights we for e 2 E , source node s
and sink node t , and an integer z, does there exist a path from s to t
with total length no greater than z?Use Dijkstra’s algorithm if edge weights are nonnegative.
s
2
3
4
tcs2 = 2, xs2 = 1
cs3 = 6
c24 = 1, x24 = 1
c32 =5
c3t = 3, x3t = 1
c43 = 2, x43 = 1
c4t= 6
Path 6? NO
Mitchell Algorithm Complexity 17 / 23
Polynomial time algorithms and the class P Examples of problems in P
Perfect matching feasibility problemGiven a graph G = (V ,E), does there exist a perfect matching?Use Edmonds blossom (alternating path) algorithm.
1
2
3
4
5
6
No alternating path between unmatched vertices 2 and 6
Mitchell Algorithm Complexity 18 / 23
Polynomial time algorithms and the class P Examples of problems in P
Linear programming with bound
Given m ⇥ n matrix A, vectors b 2 IRm and c 2 IRn, and a scalar z,does there exist a nonnegative x 2 IRn satisfying Ax = b and cT x z?
Use the ellipsoid algorithm or some interior point algorithms.
Mitchell Algorithm Complexity 19 / 23
Polynomial time algorithms and the class P Examples of problems in P
Exponential functions
DefinitionThe function f (k) is exponential if there exist positive constants
c1, c2 > 0, constants d1, d2 > 1, and a positive constant k 0 such that
c1dk
1 f (k) c2dk
2 8 k � k0.
For example, let b be the largest integer for an instance of anoptimization problem X , so this requires storage O(log2(b)).
Assume the total storage is also O(log2(b)).
If algorithm A runs in time O(b) then it requires exponential time forthis instance, since
b = 2log2(b).
Mitchell Algorithm Complexity 20 / 23
Polynomial time algorithms and the class P Examples of problems in P
The knapsack problem
The knapsack problem max{cT x : aT x b, x binary} can be solvedin O(b) iterations using dynamic programming.
However, the storage requirement is only O(log(b)), assuming b is thelargest integer in the problem definition.
So the runtime is exponential in the storage requirement.
Mitchell Algorithm Complexity 21 / 23
b - 24¥....,..f:::::"?"")
runtime
Polynomial time algorithms and the class P Examples of problems in P
Max Flow
Consider a max flow problem on the following graph. We seek tomaximize flow from node 1 to node 4. The arc capacities are indicated.
1
2
3
4M
M
1M
M
Mitchell Algorithm Complexity 22 / 23
Polynomial time algorithms and the class P Examples of problems in P
Solve with augmenting path method
1
2
3
4t t
t
1
2
3
4
(M, 1)
(M, 0)
(1, 1)
(M, 0)
(M, 1)
take t = 1
(ue, xe)
Next iteration:
1
2
3
4t
�t
t
1
2
3
4
(M, 1)
(M, 1)
(1, 0)
(M, 1)
(M, 1)
take t = 1
(ue, xe)
Need 2M iterations to get to optimal solution, which is exponential inthe storage requirement. To get polynomial algorithm, need to selectaugmenting path with fewest edges.
Mitchell Algorithm Complexity 23 / 23
Polynomial time algorithms and the class P Examples of problems in P
Solve with augmenting path method
1
2
3
4t t
t
1
2
3
4
(M, 1)
(M, 0)
(1, 1)
(M, 0)
(M, 1)
take t = 1
(ue, xe)
Next iteration:
1
2
3
4t
�t
t
1
2
3
4
(M, 1)
(M, 1)
(1, 0)
(M, 1)
(M, 1)
take t = 1
(ue, xe)
Need 2M iterations to get to optimal solution, which is exponential inthe storage requirement. To get polynomial algorithm, need to selectaugmenting path with fewest edges.
Mitchell Algorithm Complexity 23 / 23
Polynomial time algorithms and the class P Examples of problems in P
Solve with augmenting path method
1
2
3
4t t
t
1
2
3
4
(M, 1)
(M, 0)
(1, 1)
(M, 0)
(M, 1)
take t = 1
(ue, xe)
Next iteration:
1
2
3
4t
�t
t
1
2
3
4
(M, 1)
(M, 1)
(1, 0)
(M, 1)
(M, 1)
take t = 1
(ue, xe)
Need 2M iterations to get to optimal solution, which is exponential inthe storage requirement. To get polynomial algorithm, need to selectaugmenting path with fewest edges.
Mitchell Algorithm Complexity 23 / 23
Polynomial time algorithms and the class P Examples of problems in P
Solve with augmenting path method
1
2
3
4t t
t
1
2
3
4
(M, 1)
(M, 0)
(1, 1)
(M, 0)
(M, 1)
take t = 1
(ue, xe)
Next iteration:
1
2
3
4t
�t
t
1
2
3
4
(M, 1)
(M, 1)
(1, 0)
(M, 1)
(M, 1)
take t = 1
(ue, xe)
Need 2M iterations to get to optimal solution, which is exponential inthe storage requirement. To get polynomial algorithm, need to selectaugmenting path with fewest edges.
Mitchell Algorithm Complexity 23 / 23