The Design & Analysis of The Design & Analysis of the algorithmsthe algorithms

Lecture 1. 2010.. by me M. Sakalli Lecture 1. 2010.. by me M. Sakalli

1-2M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

You must have learnt these today.. You must have learnt these today..

Why analysis?..Why analysis?..

What parameters to observe? Slight # 6..What parameters to observe? Slight # 6..

Methodological differences to achieve better outcomes, Methodological differences to achieve better outcomes, there Euclid’s algorithm with three methods. there Euclid’s algorithm with three methods.

Asymptotic order of growth.Asymptotic order of growth.

Examples with insertion and selection sortingExamples with insertion and selection sorting

1-3M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Finite number of steps. Finite number of steps.

An An algorithmalgorithm: A sequence of : A sequence of unambiguous – well defined unambiguous – well defined procedures, instructions for solving a problem. procedures, instructions for solving a problem.

For a desired output.For a desired output.

For given range of input values For given range of input values

Execution must be completed in a finite amount of time.Execution must be completed in a finite amount of time.

“computer”

Problem, relating input parameters with certain state parameters, analytic rendering..

algorithm

input output

1-4M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Important points to keep in mindImportant points to keep in mind

Reduce ambiguity Reduce ambiguity • Keep the code simple, clean with well-defined and Keep the code simple, clean with well-defined and

correct steps . correct steps . • Specify the range of inputs applicable.Specify the range of inputs applicable.

Investigate other approaches solving the problem Investigate other approaches solving the problem leading to determine if other efficient algorithms leading to determine if other efficient algorithms possible. possible.

Theoretically: Theoretically: • Prove its correctness.. Prove its correctness.. • Efficiency: Theoretical and Empirical analysisEfficiency: Theoretical and Empirical analysis• Its optimality Its optimality

1-5M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Efficiency. Efficiency.

Complexity Complexity • Time efficiency: Estimation in the asymptotic sense Time efficiency: Estimation in the asymptotic sense Big OBig O, , omegaomega, ,

thetatheta. In comparison two also you have a hidden constant. . In comparison two also you have a hidden constant.

• Space efficiency.Space efficiency.

• The methods applied, recursive, parallel. The methods applied, recursive, parallel.

• Desired scalability: Various range of inputs and the size and Desired scalability: Various range of inputs and the size and dimension of the problem under consideration. Examples: Video dimension of the problem under consideration. Examples: Video sequences. ROI.sequences. ROI.

Computational Model in terms of an abstract computer: A Computational Model in terms of an abstract computer: A Turing machine. Turing machine.

http://en.wikipedia.org/wiki/Analysis_of_algorithms. http://en.wikipedia.org/wiki/Analysis_of_algorithms.

1-6M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Historical PerspectiveHistorical Perspective

…… Muhammad ibn Musa al-Khwarizmi – 9Muhammad ibn Musa al-Khwarizmi – 9thth century century

mathematician mathematician www.lib.virginia.edu/science/parshall/khwariz.htmlwww.lib.virginia.edu/science/parshall/khwariz.html

……

1-7M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Analysis meansAnalysis means: : evaluateevaluate the costs, time and space costs, and manage the the costs, time and space costs, and manage the

resources and the methods.. resources and the methods.. A generic RAM model of computation in which instructions A generic RAM model of computation in which instructions

are executed consecutively, but not concurrently or in are executed consecutively, but not concurrently or in parallel. parallel.

Running time analysisRunning time analysis: The # of the : The # of the primitive operationsprimitive operations executed for every line of the code cexecuted for every line of the code cii (defined as machine (defined as machine independent as possible).independent as possible).

Asymptotic analysisAsymptotic analysis Ignore machine-dependent constantsIgnore machine-dependent constants Look at computational growth of Look at computational growth of T(n) T(n) as as nn→∞, →∞, n: n: input sizeinput size

that determines the number of iterationsthat determines the number of iterations: :

Relative speed (in the same machine)Relative speed (in the same machine)Absolute speed (between computers)Absolute speed (between computers)

1-8M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Euclid’s AlgorithmEuclid’s Algorithm

Problem definition: gcd(Problem definition: gcd(m,nm,n) of two nonnegative, not both ) of two nonnegative, not both zero integers zero integers m m and and nn, , m m > > n n

Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ? Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ? Euclid’s algorithm is based on repeated application of Euclid’s algorithm is based on repeated application of

equalityequalitygcdgcd((m,nm,n) = ) = gcdgcd((n, m n, m mod mod nn))

until the second number reaches to 0.until the second number reaches to 0.Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12

rr00 = m, r = m, r11 = n, = n,

rri-1i-1 = r = rii q qii + r + ri+1i+1, , 0 < r0 < ri+1i+1 < r < rii , 1 , 1i<t,i<t,

……

rrt-1t-1 = r = rtt q qtt + 0 + 0

1-9M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Step 1 If (Step 1 If (nn == 0 or m==n), return == 0 or m==n), return mm and stop; and stop; otherwise go to Step 2otherwise go to Step 2Step 2 Step 2 Divide Divide mm by by n n and assign the value of the remainder toand assign the value of the remainder to r rStep 3 Assign the value of Step 3 Assign the value of n n to to mm and the value of and the value of rr to to n. n. Go toGo to

Step 1. Step 1.

whilewhile nn ≠ 0 ≠ 0 do do {{ r ← m r ← m mod mod nn m← n m← n n ← rn ← r }}returnreturn mm

The upper bound of i iterations: iThe upper bound of i iterations: iloglog(n)+1 where (n)+1 where is is (1+sqrt(5))/2. !!! Not sure on this.. I need to check.. (1+sqrt(5))/2. !!! Not sure on this.. I need to check..

ii = O(log(max(m, n))), since = O(log(max(m, n))), since rri+1i+1 r ri-1i-1/2./2.

The Lower bound, The Lower bound, (log(max(m, n)))(log(max(m, n))), , ((log(max(m, n))).log(max(m, n))).

1-10M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Proof of Correctness of the Euclid’s algorithmProof of Correctness of the Euclid’s algorithm

Step 1, If n divides m, then gcd(m,n)=n Step 1, If n divides m, then gcd(m,n)=n Step 2, gcd(mn,)=gcd(n, m mod(n)).Step 2, gcd(mn,)=gcd(n, m mod(n)).

• If gcd(m, n) divides n, this implies that gcd must be If gcd(m, n) divides n, this implies that gcd must be n: n: gcd(m, n)gcd(m, n)n. n divides m and n, implies that n must be n. n divides m and n, implies that n must be smaller than the gcd of the pair {m,n}, nsmaller than the gcd of the pair {m,n}, ngcd(m, n)gcd(m, n)

• If m=n*b+r, for r, b integer numbers, then gcd(m, n) = If m=n*b+r, for r, b integer numbers, then gcd(m, n) = gcd(n, r). Every common divisor of m and n, also divides r. gcd(n, r). Every common divisor of m and n, also divides r.

• Proof: m=cp, n=cq, c(p-qb)=r, therefore g(m,n), divides r, Proof: m=cp, n=cq, c(p-qb)=r, therefore g(m,n), divides r, so that this yields g(m,n)so that this yields g(m,n)gcd(n,r). gcd(n,r).

1-11M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Other methods for computing gcd(Other methods for computing gcd(m,nm,n))

Consecutive integer checking algorithm, not a good way, it Consecutive integer checking algorithm, not a good way, it checks all the ..checks all the ..

Step 1 Assign the value of min{Step 1 Assign the value of min{m,nm,n} to } to tt

Step 2 Step 2 Divide Divide mm by by t. t. If the remainder is 0, go to Step 3;If the remainder is 0, go to Step 3; otherwiseotherwise, go to Step 4, go to Step 4

Step 3 Step 3 Divide Divide nn by by t. t. If the remainder is 0, return If the remainder is 0, return tt and stop; and stop; otherwise, go to Step 4 otherwise, go to Step 4

Step 4 Decrease Step 4 Decrease t t by 1 and go to Step 2by 1 and go to Step 2

Exhaustive??.. Very slow even if zero inputs would be checked.. Exhaustive??.. Very slow even if zero inputs would be checked..

O(min(m, n))O(min(m, n))..

((min(m, n))min(m, n)), when gcd(m,n) =1. , when gcd(m,n) =1.

(1) for each operation, (1) for each operation,

Overall complexity is Overall complexity is ((min(m, n))min(m, n))

1-12M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Other methods for gcd(Other methods for gcd(m,nm,n) [cont.]) [cont.]

Middle-school procedureMiddle-school procedureStep 1 Find the prime factorization of Step 1 Find the prime factorization of mm

Step 2 Find the prime factorization of Step 2 Find the prime factorization of nn

Step 3 Find all the common prime factorsStep 3 Find all the common prime factors

Step 4 Compute the product of all the common prime factorsStep 4 Compute the product of all the common prime factors and return it as gcd and return it as gcd(m,n(m,n))

Is this an algorithm?Is this an algorithm?

1-13M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Sieve of EratosthenesSieve of Eratosthenes

Input: Input: Integer Integer n n ≥≥ 2 2Output: List of primes less than or equal to Output: List of primes less than or equal to nn, sift out the numbers that are not. , sift out the numbers that are not.

for for p p ← 2← 2 to to nn do do AA[[pp] ← ] ← pp

for for p p ← 2← 2 to to nn do do if if AA[[pp] ] 0 // 0 //p p hasn’t been previously eliminated from the listhasn’t been previously eliminated from the list j j ← ← pp** pp

while while j j ≤≤ n n dodo

AA[[jj] ] ← 0← 0 // mark element as eliminated// mark element as eliminated j j ← ← jj + p+ pEx: Ex: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 252 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

22 3 5 7 9 11 13 15 17 19 21 23 25 3 5 7 9 11 13 15 17 19 21 23 25 2 2 33 5 7 11 13 17 19 23 25 5 7 11 13 17 19 23 25 2 3 2 3 55 7 11 13 17 19 23 7 11 13 17 19 23

1-14M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Asymptotic order of growthAsymptotic order of growth

A way of comparing functions that ignores constant factors and A way of comparing functions that ignores constant factors and small input sizessmall input sizes

O(O(gg((nn)): class of functions )): class of functions ff((nn) that grow ) that grow no fasterno faster than than gg((nn))

ΘΘ((gg((nn)): class of functions )): class of functions ff((nn) that grow ) that grow at same rateat same rate as as gg((nn))

ΩΩ((gg((nn)): class of functions )): class of functions ff((nn) that grow ) that grow at least as fastat least as fast as as gg((nn))

1-15M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Establishing order of growth using the definitionEstablishing order of growth using the definition

Definition:Definition: f f((nn) is in O() is in O(gg((nn)), )), O(O(gg((nn)), if order of growth of )), if order of growth of ff((nn) ) ≤≤ order of order of growth of growth of gg((nn) () (within constant multiplewithin constant multiple),),i.e., there exist a i.e., there exist a positive constant positive constant c,c, c c N N, and , and non-negative integer non-negative integer nn00 such thatsuch that

ff((nn) ) ≤≤ c gc g((nn) for every ) for every nn ≥≥ nn0 0

ff(n) is o((n) is o(gg(n)), if (n)), if ff((nn) ) ≤≤ (1/ (1/cc)) g g((nn) for every ) for every nn ≥≥ nn00 which means f grows which means f grows strictly!! more slowly than any arbitrarily small constant of g. ???strictly!! more slowly than any arbitrarily small constant of g. ???

ff(n) is (n) is ((gg(n)), (n)), c c N N, , ff((nn) ≥ ) ≥ c gc g((nn) for ) for nn ≥≥ nn0 0

ff(n) is (n) is (n) if (n) if ff(n) is both O(n) and (n) is both O(n) and (n)(n)

Examples:Examples: 1010nn is O(c is O(cnn22), c ), c ≥≥ 10, 10,

• since 10since 10nn ≤≤ 10 10nn22 for for n n ≥≥ 1 or 101 or 10nn ≤≤ nn2 2 for for n n ≥≥ 1010 55nn+20 is O(+20 is O(cncn), for all n>0, c>=25, ), for all n>0, c>=25,

• since 5since 5nn+20 +20 ≤≤ 5 5nn+20n +20n ≤≤ 25 25nn, or c>=10 for , or c>=10 for n n ≥≥ 44

1-16M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

O, O, ΩΩ, , ΘΘ

1-17M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Example of computational problem: sortingExample of computational problem: sorting

Statement of problem:Statement of problem:• Input:Input: A sequence of A sequence of nn numbers <a numbers <a11, , aa22, …, a, …, ann>>• ProblemProblem: Reorder <a: Reorder <a ´́

11, , aa´́22, …, a, …, a´́

nn> in ascending or > in ascending or descending order.descending order.

• Output desired:Output desired: a a´́ii ≤≤ aa´́

jj , for , for ii < < j j or or ii > > j j

Instance: The sequence <5, 3, 2, 8, 3>Instance: The sequence <5, 3, 2, 8, 3>

Algorithms:Algorithms:• Selection sortSelection sort• Insertion sortInsertion sort• Merge sortMerge sort• (many others)(many others)

1-18M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Selection SortSelection Sort

Input: array Input: array a[1],…,a[n]a[1],…,a[n]

Output for example: array Output for example: array aa sorted in non-decreasing sorted in non-decreasing orderorder

*** Scanning elements of unsorted part, n-1, n-2, .. Swapping. *** Scanning elements of unsorted part, n-1, n-2, .. Swapping. (n-1)*n/2=theta(n^2). Independent from the input. (n-1)*n/2=theta(n^2). Independent from the input.

Algorithm: (Algorithm: (Insertion in placeInsertion in place))

for i=1 to n swap a[i] with smallest of a[i],…a[n]

1-19M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Insertion-Sort and an idea of runtime analysis

Input size is n = length[A], and ttjj times line-4 executed times line-4 executed

1: for j ← 2← 2 to n do

2: key ← ← A[j]

//Insert A[j] into the sorted sequence A[1 . . . j − 1].

3: i ←← j - 1

4: while i > 0 and A[i] > key do

5: A[i+1] ← ← A[i]

6: i ← ← i-1

-4: end while

7: A[i+1] ← ← key

-1: end for

The total runtime for is T(n)

The Best Case c1n + c2(n-1)+c3(n-1)+ c4(n-1)+c7(n-1)

= an + b..

Times Cost

n c1

n-1 c2

-

n-1n-1 c3

ΣΣj=2:n j=2:n ttjj c4

ΣΣj=2:n j=2:n ((ttjj--1)1) c5

ΣΣj=2:n j=2:n ((ttjj--1)1) c6

n-1n-1 0 0

n-1n-1 c7

nn 00

= c1n + c2(n-1)+c3(n-1)+

c4ΣΣj=2:n j=2:n ttj j + c5 ΣΣj=2:n j=2:n ((ttjj--1)1)

+ c6 ΣΣj=2:n j=2:n ((ttjj--1)+ 1)+ c7(n-1)(n-1)

1-20M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Insertion-Sort Analysis continued

For the worst case run time.. If the array A is sorted in reverse order, For the worst case run time.. If the array A is sorted in reverse order, then.. while loop, then.. while loop,

ΣΣj=2:n j=2:n ttj j = ΣΣj=2:nj=2:n j = ΣΣj=2:nj=2:n (n(n-1)/2) -1

ΣΣj=2:n j=2:n ((ttjj--1) = 1) = ΣΣj=2:nj=2:n (j-1) = ΣΣj=2:nj=2:n n(n-1)/2

TT(n) = ((n) = (c4 + + c5 + + c6)n)n22/2 + (/2 + (c1 + + c2 + + c3 - (- (c4 - - c5 - - c6)/2 + )/2 + c7 )n + )n +

((c2 + + c3 + + c4 + + c7))

= An= An2 2 + Bn + C+ Bn + C

Average case runtime: Assume that about half of the subarray is out Average case runtime: Assume that about half of the subarray is out of order, then of order, then ttj j = j/2, which will lead a similar kind quadratic function

1-21M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

The methods The methods

Brute forceBrute force Divide and conquerDivide and conquer Decrease and conquerDecrease and conquer Transform and conquerTransform and conquer Greedy approachGreedy approach Dynamic programmingDynamic programming Iterative improvementIterative improvement Backtracking Backtracking Branch and boundBranch and bound Randomized algorithmsRandomized algorithms