Counting the bitsAnalysis of Algorithms

Will it run on a larger problem?When will it fail?

Software Concerns

• Correctness– test– understand– prove

• Maintainability– modify– enhance

• Decomposability– cohesion and coupling– multi-programmer, multi-year

• Efficiency– memory– cycles

Sample Questions

• Task: sorting n elements• How fast can you do it?

– in worst case– on average

• How much more memory is needed?• Data assumptions?

– in memory or on disk?– nearly sorted?– duplicates retained?– ....

• Different algorithms for different conditions

How can we tell how well an algorithm works?

1. Experimentally, I.E. Run algorithm on data, report cpu time

– depends on coding– depends on machine– depends on compiler– little generality+ can do it+ expensive

• run multiple times, show average performance• guess relevant variables and vary them• What’s relevant for sorting?

2. Theoretical Analysis

• Give upper and lower bounds for complexity+ demonstrates an understanding of algorithm+ shows relevant variables+ shows importance of variables+ independent of machine, language, coding- analysis is sometimes difficult- usually only works for simple algorithms- often doesn’t include important constants

• Same questions for time can be done for memory usage

Language of Analysis

• big O notation• A function f(n) is O(g(n)) iff there are positive

constants c and N such than f(n) < c*g(n) when n>N.• Intuitively this says that f is eventually less than a

constant times g.• Examples

– 17n2 +1000n+15 is O(n2). generalize– 10n+nlog10n is O(nlg(n)) where lg = log2

• It turns out that sorting is O(n log(n)).• What about finding the maximum in an array?• What about finding the k-th largest in an array?

Squeezing the complexity: lower bounds

• When upper bounds equal lower bounds, then you know you have the best possible algorithm.

• A function f(n) is big-omega(g(n)) iff there are positive constants c and N such that f(n)> c*g(n) for n > N.

• Intuitively, this says f is eventually larger than g.• example: n3 -1000n2 is big-omega of n3.• Theoreticians work on closing the gap between the upper

and lower bounds.• It turns out that sorting is big-omega(n*log(n)), hence

sorting is solved. – caveat: this assumes no special properties of the data.– sometimes constants count– sometimes eventually is not good enough

Useful Theorems

• Let f and g be function of n• If f <= g, then O(f+g) = O(g).• O(constant*f) = O(f).• O(f*f) = [O(f)]^2.• O(Polynomial(n) ) = O(nmax degree)• O(f/g) .. Can be anything.• Style: write the simplest and smallest O-notation.

– eg. do not write O(n2 + 3n), but O(n^2).– Do not write O(3*n^3), but O(n^3).

Proving O theorems

• To prove: if f1 and f2 are O(g(n)) then f1*f2 is O(g(n)*g(n)).

• Proof: by your nose.– Ask: what do I need to do?

• By definition find a constant c and integer N such that f1(n)*f2(n) < c* g(n)*g(n) for n>N

– Ask: what do I know?• There are constants c1,c2, N1,N2 such that• f1(n) < c1*g(n) for n >N1• f2(n) <c2*g(n) for n >N2

– Aren’t c and N obvious now?• Can you do this?

Another Proof

• To prove: if f1 and f2 are O(g(n)) then f1+f2 is O(g(n)).• By your nose• Since f1 is O(g(n)), f1(n) <C1*g(n) for n> N1.• Since f2 is O(g(n)), f2(n) < C2*g(n) for n> N2.• Then f1(n)+f2(n) < (C1+C2) for n> max(N1,N2).• Hint: go back to these proofs in a few days and see if you

can prove them without looking.

More notation

• A function f(n) is big-theta of g(n) iff it is both big-O and big-omega of g(n).

• Intuitively, they have same growth rate.• A function f(n) is little-o of g(n) iff it is big-O of g(n),

but not big-omega of g(n). • TSP is big-O(2n). Why?

– No one knows if it is big-omega(2n).• Intuitively this means there is some doubt.• These results require proofs. One can also get estimates

by running careful experiments. • But experiments can’t prove anything.

Traveling Salesman Problem

• Given: n cities and distances between them• Find: tour of minimum length.

– A tour is path that visits every city• How many tours are possible?

– n*(n-1)...*1 = n!– n! > 2*2……*1 = 2^(n-1) – So n! is omega of 2^n. ( lower bound)

• No one knows if there is a polynomial time algorithm to find minimum length tour.

• There are many very good heuristic algorithms.

Boolean Satisfiability• Suppose we have a boolean formula on n-variables

– e.g. (x1 or not x2 or x3)&(x2 or not x5 or x7)&(...)• This is a conjunct of disjuncts. If restricted to at most 3

literals per disjunct then 3-sat.• Theorem: 3-Sat and general Boolean satisfiability are

equivalently difficult.• Does this have a solution?

– i.e. assignment of true or false to the variables so that the entire formula is true.

• When the formula has at most 3 literals (variable or its negation) in each clause, the problem is called 3-sat.

• If we just checked every possibility, how many cases?– 2n , so exhaustive search is O(2n)

Satisfiability, the value

• No one knows if this can be done in polynomial time.• Nobel prize level problem, but none given in CS.• One Million $ prize• Application: Given 2 circuits, are they equivalent?

– Suppose they each have n inputs.– Let the two circuits, or formulas, be f1 and f2.– Consider: XOR(f1, f2).– This is a boolean formula on n-variables. – If it has a solution S, then f1(S) != f2(S) and the two

circuits are different.

A problem we can do:Linear Lookup/Search

• Task: determine if value is in array (n items)• Algorithm

For i = 1 to nif (a[i] = element) return trueelse return false

• In worst case, time O(n). • In best case, O(1).• What about space? O(1)

Space Analysis

• If program has a bad time complexity, you can wait.• If program has a bad space complexity, the program

bombs.• SPACE is critical.• Space utilization occurs in two places:

– 1. Every time you call new. • Analysis is like time complexity

– 2. Hidden usage: function calls• Depth of function calls is critical here.• Not a problem except when using recursion.

Find most similar student records

• Assume we have n records Record[I]• Each record has k grades• Algorithm Sketch

– for j = 1…n– for h = j+1…n

– compute similarity of records j and h – if similarity better than best, save j,h and score

• end h– end j– return j and h.

• Time complexity: O(n^2)• Space complexity: O(1).

Divide and Conquer

• General Problem Solving method• Top-down division of labor• General Algorithm: Solution = Solve(Problem) Solve(Problem) = if Problem easy, return basicSolve(Problem) else Pieces = split(Problem) PartSolutions =Solve(Pieces) return merge(PartSolutions)• Examples: mergesort, quicksort, binary search

General analysis of Divide and Conquer

• Suppose splitting yields a subproblems• Suppose size of subproblems is N/b• Suppose merging cost is O(n^k)• Then T(N) = if a>b^k O(N^(log b a)) if a = b^k O(N^k * log(N) ) if a < b^k O(N^k)• For example, in mergesort where N = b^m. A=2, b=2, k = 1 Therefore: O(N*log(N))• You should know how to use this theorem (you don’t

need to memorize result).

Binary Search

• Binary search (assumes ordered)• BinarySearch(a, l, u)

– mid = (l+u)/2– if ( l==u ) return a[mid] == element;– if (element = a[mid]) return mid;– else if (element <a[mid]) return BinarySearch(a,l,mid);– else if (element >a[mid]) return BinarySearch(a,mid,u);

• Time Analysis– f(n) = c+f(n/2)– f(1) = c.– Hence: f(n) = c*lg(n).– Expand out to see this, with n a power of 2.

Binary Search

• Space Analysis– Only new memory counts– New memory occurs when

• constructors are called• recursion occurs

– Back to code:– What is max depth of stack?

• O(log n)

Maximum contiguous subsequence problem• Given a[1]…a[n] are integers, find i, j such that sum of

a[k] from i to j is maximum.• Algorithmic Approaches

– Exhaustive, dumb but a start• optimize via problem constraints

– Divide and Conquer• requires breaking problem in pieces whose solutions

can be merged– Dynamic Programming

• requires merging subsolutions of smaller problems into full solution

– Be Clever: this is for the theoreticians– Transform a known problem solution

Exhaustive

Best = 0, begin = 0, end= 0For i = 1…n for j = i+1 … n temp = 0 for k = i …j temp += a[k] if (temp > best) best = temp update begin, endTime complexity: O(N^3)Space complexity: O(1)

see text for Java code

Partition trick

For i = 1…n/3 > n/3 times for j = 2n/3 … n > n/3 times temp = 0 for k = i …j > n/3 temp += a[k] if (temp > best) best = temp update begin, endTime complexity: O(N^3) N^3/9

More painful analysis in the text.

Optimize (exhaustive)

Best = 0, begin = 0, end= 0For i = 1…n temp = 0 for j = i+1 … n temp += a[k] if (temp > best) best = temp update begin, endTime: O(n^2)Space: O(1)

Dynamic Programming

• General Idea: use results of smaller problem• Decomposition: finding relationship between problem and

smaller ones• Idea Compute S[i][j] = sum a[i]+… a[j] Keep track of bestNote: S[i][i] = a[i], boundary condition S[i][k] = S[i][k-1] + a[k]See MatrixTime: O(n^2)Space: O(n^2)

Fibonacci via DP

• Definition of f(k)– if (k <2) return 1 else return f(k-1)+f(k-2

• Expand: f(4) = f(3) + f(2) = f(2)+f(1) + f(1)+f(0) = f(1)+f(0)+ f(1)+f(1)+f(0)• Redundant computation: exponential! (see text)• Dynnamic programming: bottom-up instead of top-

down.• StraightForward: use array: f(k) depends on two

previous values.• Optimise: don’t really need array, just last 2 values.• T(N) = T(N-1)+T(N-2) +2

Dynamic Programming (example 2)

• Define Comb(k, n) = n! / (k!*(n-k)!)• Why doesn’t this compute well?• Goal: find definition in terms of smaller k,n• Note: Comb(k,n) = Comb(k-1, n-1)+ Comb(k,n-1)• Ok, so dynamic programming will work where S[k][n] is determine by element above and to left• time analysis: O(n^2)• space analysis: O(n^2)• Why is this useful?

Divide and Conquer• Previous examples

– MergeSort, QuickSort, BinarySearch• Divide array in half, say left and right• Suppose solutions to left and right• How to combine?

– From each half need two solutions• interior solution (away from boundary)• boundary solution

– Solution is• best of:• interior left, interior right, sum of (boundary solutions)• See text for code.

Clever

• Look at properties of answer• Note: if a[i]+…+a[j] is a solution then a[i]+…+a[p] must be positive.Best = 0, temp = 0For j = 1…n temp += a[j] if (temp > best) best = temp else if (temp <0) temp = 0 Time: O(n)

Transform Problem

• Problem: what is minimum subsequence sum?– Replace each a[i] by -a[i]. Done.

• Problem. Assume each ai is positive. What is maximum product?– Replace each a[i] by log(a[i]).– Note xy > uv iff log(xy) >log(uv)– But log(xy) = log(x)+ log(y)

• Transforms not always easy to find.

Concrete classes in Collections

add remove contains

ArrayList O(n) O(n) O(n)

LinkedList (ordered)

O(1)O(log n)

O(1)O(log n)

O(n)O(log n)

HashSet O(1) O(1) O(1)

TreeSet(ordered)

O(log n) O(log n) O(log n)

HashMap O(1) O(1) O(1)

TreeMap O(log n) O(log n) O(log n)

Summary

• Memory and time complexity measures• Critical for some applications• Sometimes the constants count• Be aware of your computational expenses• Document non-O(1) memory or time costs• Consider problem decomposition, either dynamic

programming or divide and conquer