7/29/2019 Intro Analysis
1/114
1
Introduction to ALgorithmicAnalysis
In this chapter, we will answer thefollowing two questions
What does it mean to be an efficientalgorithm?
How can one tell that it is more efficient thanother algorithms?
based on some easy-to-understandsearching and sorting algorithms that wemay have seen earlier.
Thursday, September 15, 11
7/29/2019 Intro Analysis
2/114
2
Searching Problem
Assume A is an array with n elements A[1],A[2], A[n]. For a given element x, we must
determine whether there is an indexj; 1 j n, such that x=A[j]
Two algorithms, among others, address this
problem Linear Search
Binary Search
Thursday, September 15, 11
7/29/2019 Intro Analysis
3/114
3
Linear Search Algorithm
Algorithm: LINEARSEARCH
Input: arrayA[1..n] ofn elements and an
elementx.Output:jifx=A[j], 1 jn, and 0 otherwise.
1. j 1
2.while (j < n) and(xA[j])
3. j j + 1
4. end while
5. if x= A[j] then return j else return 0
Thursday, September 15, 11
7/29/2019 Intro Analysis
4/114
4
Analyzing Linear Search
One way to measure efficiency is to count howmany statements get executed before the algorithmterminates
One should keep an eye, though, on statements thatare executed repeatedly.
What will be the number of element comparisons if
X first appears in the first element of A
X first appears in the middle element of A
X first appears in the last element of A
X doesnt appear in A.
Thursday, September 15, 11
7/29/2019 Intro Analysis
5/114
5
Binary Search
We can do better than linear search if we knewthat the elements of A are sorted, say in non-decreasing order.
The idea is that you can compare x to themiddle element of A, say A[middle]. If x < A[middle] then you know that x cannot be an
element from A[middle+1], A[middle+2], A[n]. Why?
If x > A[middle] then you know that x cannot be an
element from A[1], A[2], A[middle-1]. Why?
Thursday, September 15, 11
7/29/2019 Intro Analysis
6/114
6
Binary Search Algorithm
Algorithm: BINARYSEARCH
Input:An arrayA[1..n] ofn elements sorted innondecreasing order and an elementx.
Output:jifx=A[j], 1 jn, and 0 otherwise.
1. low 1; high n; j 02.while (lowhigh) and(j = 0)
3. mid (low+ high)/2
4. if x= A[mid] then j mid
5. else if x < A[mid] then high mid - 16. else low mid+ 1
7. end while
8. return j
Thursday, September 15, 11
7/29/2019 Intro Analysis
7/114
7
Worst Case Analysis of Binary
What to do: Find the maximum number of elementcomparisons
How to do:
The number of element comparisons is equal to the
number of iterations of the while loop in steps 2-7. HOW? How many elements of the input do we have in the
First iteration
Second iteration
Thrid iteration
ith iteration
The last iteration occurs when the size of array = 1.
Thursday, September 15, 11
7/29/2019 Intro Analysis
8/114
8
Theorem
The number of comparisons performed byAlgorithm BINARYSEARCH on a sortedarray of size n is at most
Thursday, September 15, 11
7/29/2019 Intro Analysis
9/114
9
Merging Two Sorted Lists
Problem Description: Given two lists (arrays) that aresorted in non-decreasing order, we need to merge theminto one list sorted in non-decreasing order.
Example:
3 7 9 12 1 2 4 13 14
Input
Output
Thursday, September 15, 11
7/29/2019 Intro Analysis
10/114
9
Merging Two Sorted Lists
Problem Description: Given two lists (arrays) that aresorted in non-decreasing order, we need to merge theminto one list sorted in non-decreasing order.
Example:
3 7 9 12 1 2 4 13 14
Input
Output
1 2 3 4 7 9 12 13 14Thursday, September 15, 11
7/29/2019 Intro Analysis
11/114
We will be interested in merging twosubarrays.
Input: A[1..n], p, q, r.
Merge A[p..q] with A[q+1..r].
10
Thursday, September 15, 11
7/29/2019 Intro Analysis
12/114
10
How to merge two subarrays?
3 7 8
A[4..6]A[1..3]B[1..6] (temp)
2 4 5
Thursday, September 15, 11
7/29/2019 Intro Analysis
13/114
10
How to merge two subarrays?
3 7 8
A[4..6]A[1..3]B[1..6] (temp)
3 7 8
2 4 5
2 4 52
Thursday, September 15, 11
7/29/2019 Intro Analysis
14/114
10
How to merge two subarrays?
3 7 8
A[4..6]A[1..3]B[1..6] (temp)
3 7 8
3 7 8
2 4 5
2 4 5
2 4 52
2 3
Thursday, September 15, 11
7/29/2019 Intro Analysis
15/114
10
How to merge two subarrays?
3 7 8
A[4..6]A[1..3]B[1..6] (temp)
3 7 8
3 7 8
3 7 8
2 4 5
2 4 5
2 4 5
2 4 5
2
2 3
2 3 4
Thursday, September 15, 11
7/29/2019 Intro Analysis
16/114
10
How to merge two subarrays?
3 7 8
A[4..6]A[1..3]B[1..6] (temp)
3 7 8
3 7 8
3 7 8
3 7 8
2 4 5
2 4 5
2 4 5
2 4 5
2 4 5
2
2 3
2 3 4
2 3 4 5
Thursday, September 15, 11
7/29/2019 Intro Analysis
17/114
10
How to merge two subarrays?
3 7 8
A[4..6]A[1..3]B[1..6] (temp)
3 7 8
3 7 8
3 7 8
3 7 8
3 7 8
2 4 5
2 4 5
2 4 5
2 4 5
2 4 5
2 4 5
2
2 3
2 3 4
2 3 4 5
2 3 4 5 7 8
Thursday, September 15, 11
7/29/2019 Intro Analysis
18/114
11
Algorithm MERGE
Algorithm: MERGE
Input: An arrayA[1..m] of elements and threeindicesp, q and r, with 1 p q
7/29/2019 Intro Analysis
19/114
12
Algorithm MERGE (Cont.)
1. s p; tq + 1; kp
2. while s q and t r
3. ifA[s] A[t] then
4. temp[k] A[s]
5. s s + 1
6. else
7. temp[k] A[t]
8. tt+ 1
9. end if
10. kk+ 1
11. end while
12. if (s = q + 1) then temp[k..r] A[t..r]
13. else temp[k..r] A[s..q]
14. end if
15.A[p..r] temp[p..r]
merge A[p..q] with A[q+1..r]
Thursday, September 15, 11
7/29/2019 Intro Analysis
20/114
13
Analyzing MERGE
Assuming arrays A[p,q] and A[q+1,r],where p =1 and r =n,
n element assignments are needed to
copy A to temp. n element assignments are needed to
copy temp to A.
The total number of element assignmentsis 2n.
Hence, the time complexity is O(n).
Thursday, September 15, 11
7/29/2019 Intro Analysis
21/114
14
Selection Sort
Algorithm: SELECTIONSORTInput:An arrayA[1..n] ofn elements.
Output:A[1..n] sorted in nondecreasing order.
1. for i 1 to n - 1
2. k i3. for j i + 1 to n
{Find the index of the ithsmallest element}
4. if A[j] < A[k] then k j
5.end for
6. if k i then interchange A[i] andA[k]
7. end for
Thursday, September 15, 11
7/29/2019 Intro Analysis
22/114
15
Selection Sort Example
42 85 9i k
Thursday, September 15, 11
7/29/2019 Intro Analysis
23/114
15
Selection Sort Example
42 85 9i k
1
Thursday, September 15, 11
7/29/2019 Intro Analysis
24/114
15
Selection Sort Example
42 85 9i k
1 2
Thursday, September 15, 11
7/29/2019 Intro Analysis
25/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
Thursday, September 15, 11
7/29/2019 Intro Analysis
26/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
2
Thursday, September 15, 11
7/29/2019 Intro Analysis
27/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
2 5
Thursday, September 15, 11
7/29/2019 Intro Analysis
28/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
54 82 92 5
Thursday, September 15, 11
7/29/2019 Intro Analysis
29/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
54 82 92 5
3
Thursday, September 15, 11
7/29/2019 Intro Analysis
30/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
54 82 92 5
3 5
Thursday, September 15, 11
7/29/2019 Intro Analysis
31/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
54 82 92 5
94 82 53 5
Thursday, September 15, 11
7/29/2019 Intro Analysis
32/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
54 82 92 5
94 82 53 5
4
Thursday, September 15, 11
7/29/2019 Intro Analysis
33/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
54 82 92 5
94 82 53 5
4 4
Thursday, September 15, 11
7/29/2019 Intro Analysis
34/114
15
Selection Sort Example
42 85 9
45 82 9
i k
1 2
54 82 92 5
94 82 53 5
94 82 54 4
Thursday, September 15, 11
A l i S l ti S t
7/29/2019 Intro Analysis
35/114
16
Analyzing Selection Sort
Iteration # of comparisons1 n-1
2 n-2
3 n-3.. ..
n-1 1
Total is Hence, time complexity is
Thursday, September 15, 11
7/29/2019 Intro Analysis
36/114
17
Insertion Sort
Algorithm: INSERTIONSORTInput:An arrayA[1..n] ofn elements.
Output:A[1..n] sorted in nondecreasing order.1. for i 2 to n
2. xA[i]3. j i - 14. while (j >0) and(A[j] > x)
5. A[j + 1] A[j]6. j j - 17. end while
8. A[j + 1] x9. end for
Thursday, September 15, 11
7/29/2019 Intro Analysis
37/114
18
Insertion Sort Example
42 85 9x=2
Thursday, September 15, 11
I i S E l
7/29/2019 Intro Analysis
38/114
18
Insertion Sort Example
42 85 9x=2
45 89
Thursday, September 15, 11
I ti S t E l
7/29/2019 Intro Analysis
39/114
18
Insertion Sort Example
42 85 9x=2
45 82 9
Thursday, September 15, 11
I ti S t E l
7/29/2019 Intro Analysis
40/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
Thursday, September 15, 11
I ti S t E l
7/29/2019 Intro Analysis
41/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
48
Thursday, September 15, 11
I ti S t E l
7/29/2019 Intro Analysis
42/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9
Thursday, September 15, 11
I ti S t E l
7/29/2019 Intro Analysis
43/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
44/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
4
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
45/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
49
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
46/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
47/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
48/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8x=4
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
49/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8x=4
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
50/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8x=4
9
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
51/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8x=4
98
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
52/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8x=4
985
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
53/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8x=4
982 5
Thursday, September 15, 11
Insertion Sort Example
7/29/2019 Intro Analysis
54/114
18
Insertion Sort Example
42 85 9x=2
45 82 9x=9
45 82 9x=8
45 92 8x=4
94 82 5
Thursday, September 15, 11
Analyzing Insertion Sort
7/29/2019 Intro Analysis
55/114
19
Analyzing Insertion Sort
The minimum number of elementcomparisons is n-1which occurs when input is sorted
The maximum number of element
comparisons is
which occurs when input is reverselysorted. Time complexity is
Thursday, September 15, 11
Bottom-Up Merge Sort
7/29/2019 Intro Analysis
56/114
20
Bottom Up Merge Sort
Informally, the algorithm does thefollowing
1. Divide the array into pairs of elements (with
possibly a single elements in case thenumber of elements is odd).
2. Merge each pair in non-decreasing order(with possibly a single pair left)
3. Repeat step 2 until there is only one pairleft.
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
57/114
21
Bottom Up Merge Sort Example
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
58/114
21
Bottom Up Merge Sort Example
5 2
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
59/114
21
Bottom Up Merge Sort Example
5 2 9 8
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
60/114
21
o o Up e ge So a p e
5 2 9 8 4 12
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
61/114
21
p g p
5 2 9 8 4 12 7 1
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
62/114
21
p g p
5 2 9 8 4 12 7 1 3 6
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
63/114
21
p g p
5 2 9 8 4 12 7 1 3 6 10
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
64/114
21
p g p
5 2 9 8 4 12 7 1 3 6 10
52 98 4 12 71 3 6 10
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
65/114
21
p g p
5 2 9 8 4 12 7 1 3 6 10
52 98 4 12 71 3 6 10
52 98
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
66/114
21
g
5 2 9 8 4 12 7 1 3 6 10
52 98 4 12 71 3 6 10
52 98 4 1271
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
67/114
21
5 2 9 8 4 12 7 1 3 6 10
52 98 4 12 71 3 6 10
52 98 4 1271 3 6 10
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
68/114
21
5 2 9 8 4 12 7 1 3 6 10
52 98 4 12 71 3 6 10
52 98 4 1271 3 6 10
52 984 1271
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
69/114
21
5 2 9 8 4 12 7 1 3 6 10
52 98 4 12 71 3 6 10
52 98 4 1271 3 6 10
52 984 1271 3 6 10
42 318 1075 9 12 6
Thursday, September 15, 11
Bottom-Up Merge Sort Example
7/29/2019 Intro Analysis
70/114
21
5 2 9 8 4 12 7 1 3 6 10
52 98 4 12 71 3 6 10
52 98 4 1271 3 6 10
52 984 1271 3 6 10
52 984 1271 3 6 10
42 318 1075 9 12 6
Thursday, September 15, 11
Algorithm BOTTOMUPSORT
7/29/2019 Intro Analysis
71/114
22
Algorithm: BOTTOMUPSORT
Input:An arrayA[1..n] ofn elements.
Output:A[1..n] sorted in nondecreasing order.1. t 1
2.while t < n
3. s t; t 2s; i 04. while i + t n
5. MERGE(A, i + 1, i + s, i + t)
6. i i + t
7.end while
8. if i + s < n then
9. MERGE(A, i + 1, i+ s, n)
10. end while
Thursday, September 15, 11
Analyzing AlgorithmWith no loss of generality, assume the size of the array is a power of 2.
7/29/2019 Intro Analysis
72/114
23
With no loss of generality, assume the size of the array is a power of 2.
In the first iteration, we have pairs that are merged using
element assignments.
In the first iteration, we have pairs that are merged using
assignments.
.
In the last iteration, we have 1 pair that are merged using element
assignments.
n
2
n
2
*2*2 = 2n
n
4
* 4 * 2 = 2n
n
4
2 * 2 *n
2
= 2n
Thursday, September 15, 11
Analyzing AlgorithmBOTTOMUPSORT
7/29/2019 Intro Analysis
73/114
24
Number of iterations is log n, since the number ofpairs is reduced by half in each iteration.
Hence, run time is 2n x log n = O(n log n).
Thursday, September 15, 11
Time Complexity
7/29/2019 Intro Analysis
74/114
25
One way of measuring the performance of analgorithm is how fast it executes. The question ishow to measure this time? Is having a digital stop watch suitable?
In general, we are not so much interested in thetime and space complexity for small inputs.
For example, while the difference in time
complexity between linear and binary search ismeaningless for a sequence with n = 10, it isgigantic for n = 230.
Thursday, September 15, 11
Complexity
F l l t t l ith A d B
7/29/2019 Intro Analysis
75/114
26
For example, let us assume two algorithms A and B
that solve the same class of problems.
The time complexity of A is 5,000n, the one for B is1.1n for an input with n elements.
For n = 10, A requires 50,000 steps, but B only 3,so B seems to be superior to A.
For n = 1000, however, A requires 5,000,000 steps,while B requires 2.51041 steps.
Thursday, September 15, 11
Complexity
C i i l i i f l i h A d B
7/29/2019 Intro Analysis
76/114
28
Comparison: time complexities of algorithms A and B
Algorithm A Algorithm BInput Size
n10
100
1,0001,000,000
5,000n50,000
500,000
5,000,0005109
1.1
n
3
2.510
41
13,781
4.81041392
Thursday, September 15, 11
Order of Growth
This means that algorithm B cannot be used for
7/29/2019 Intro Analysis
77/114
28
This means that algorithm B cannot be used for
large inputs, while algorithm A is still feasible.
So what is important is the growth of thecomplexity functions.
The growth of time and space complexity withincreasing input size n is a suitable measure forthe comparison of algorithms.
we focus on asymptotic analysis
Thursday, September 15, 11
Example
7/29/2019 Intro Analysis
78/114
29Growth rate for some function
Thursday, September 15, 11
Example
7/29/2019 Intro Analysis
79/114
30
Growth rate for same previous functions
showing larger input sizes
Thursday, September 15, 11
Running Times for Different Sizesof Inputs of Different Functions
7/29/2019 Intro Analysis
80/114
31
p
Thursday, September 15, 11
Running Times for Different Sizesof Inputs of Different Functions
7/29/2019 Intro Analysis
81/114
31
Complexity10 20 30 40 50 60
n 110-5 sec 210-5 sec 310-5 sec 410-5 sec 510-5 sec 610-5 sec
n2 0.0001 sec 0.0004 sec 0.0009 sec 0.016 sec 0.025 sec 0.036 sec
n3 0.001 sec 0.008 sec 0.027 sec 0.064 sec 0.125 sec 0.216 sec
n5 0.1 sec 3.2 sec 24.3 sec 1.7 min 5.2 min 13.0 min2n 0.001sec 1.0 sec 17.9 min 12.7 days 35.7 years 366 cent
3n 0.59sec 58 min 6.5 years 3855 cent 2108cent 1.31013cent
log2
n 310-6 sec 410-6 sec 510-6 sec 510-6 sec 610-6 sec 610-6 sec
n log2n 310-5 sec 910-5 sec 0.0001 sec 0.0002 sec 0.0003 sec 0.0004 sec
Thursday, September 15, 11
Asymptotic Analysis: Big-oh (O())
7/29/2019 Intro Analysis
82/114
32
Definition: ForT(n) a non-negatively valuedfunction, T(n) is in the set O(f(n)) if there existtwo positive constants cand n0 such that T(n)
cf(n) for all n > n0.
Meaning: For all data sets big enough (i.e.,n>n0), the algorithm always executes in lessthan or equal to cf(n) steps.
Thursday, September 15, 11
The Growth of Functions
Th id b hi d th bi O t ti i t t bli h
7/29/2019 Intro Analysis
83/114
33
The idea behind the big-O notation is to establish an
upper boundary for the growth of a function f(n) forlargen.
This boundary is specified by a function g(n) that isusually much simplerthan f(n).
We accept the constant c in the requirement f(n) cg(n) whenever n > k, because c does not growwith n.
We are only interested in large n, so it is OK iff(n) > cg(n) for n k.
Thursday, September 15, 11
The Growth of Functions
Enample:
7/29/2019 Intro Analysis
84/114
34
Enample:
Show that f(n) = n2 + 2n + 1 is O(n2).
For n > 1 we have:
n2
+ 2n + 1 n2
+ 2n2
+ n2
n2 + 2n + 1 4n2
Therefore, for c = 4 and k = 1:
f(n) cn2 whenever n > k. f(n) is O(n2).
Thursday, September 15, 11
The Growth of Functions
7/29/2019 Intro Analysis
85/114
35
Question: If f(n) is O(n
2
), is it also O(n
3
)?
Yes. n3 grows faster than n2, so n3 grows alsofaster than f(n).
Therefore, we always try to find the smallestsimple function g(n) for which f(n) is O(g(n)).
Thursday, September 15, 11
The Growth of Functions
Popular functions g(n) are
7/29/2019 Intro Analysis
86/114
36
n log n, 1, 2n
, n2
, n!, n, n3
, log nListed from fastest to slowest growth:
1log n
nn log nn2
n3
2n
n!
Thursday, September 15, 11
Asymptotic Analysis: Big-Omega (())
7/29/2019 Intro Analysis
87/114
37
Definition: ForT(n) a non-negatively valuedfunction, T(n) is in the set (g(n)) if there existtwo positive constants cand n
0such that T(n)
>= cg(n) for all n > n0.
Meaning: For all data sets big enough (i.e., n> n
0), the algorithm always executes in more
than or equal to cg(n) steps.
() notation indicates a lower bound.
Thursday, September 15, 11
Asymptotic Analysis: Big Theta (())
7/29/2019 Intro Analysis
88/114
38
When O() and () meet, we indicate thisby using () (big-Theta) notation.
Definition: An algorithm is said to be(f(n)) if it is in O(f(n)) and it is in (f(n)).
Thursday, September 15, 11
Using the limit criterion
7/29/2019 Intro Analysis
89/114
41
Let
- If , then
- If , then
- If , then41
L 0
L = c > 0
L = limn
f(n)
g(n)
f(n)=
(g(n))
f(n) = (g(n))
f(n) =O(g(n))L
Thursday, September 15, 11
Summary
limf(n)
f (n) =O(g(n)) if
7/29/2019 Intro Analysis
90/114
42
g(n) =O(f(n))f(n) = (g(n)) if
limn
g(n)
f(n) O(g(n)) if
f(n) = (g(n)) if and
g(n) =O(f(n))f(n) = (g(n)) if f(n) =O(g(n)) and
f(n) =O(g(n))f(n) = (g(n))
Thursday, September 15, 11
Examples 1
7/29/2019 Intro Analysis
91/114
44
Hence,
It follows that
Thursday, September 15, 11
Examples 2
7/29/2019 Intro Analysis
92/114
45
Hence,
It follows that
Thursday, September 15, 11
Examples 3
7/29/2019 Intro Analysis
93/114
46
Since 100 > 0,
This implies that
and
Thursday, September 15, 11
Example 4
Prove that
7/29/2019 Intro Analysis
94/114
47
Prove that
We only need to show that the two functions are not related by O or
Hence, which means
Or
Since the limit is not a constant > 0,
Thursday, September 15, 11
Example 5
7/29/2019 Intro Analysis
95/114
39
Show that log(n!) is in (n log n).
Thursday, September 15, 11
Estimating the Running Time of anAlgorithm
We need to focus on counting those
7/29/2019 Intro Analysis
96/114
50
We need to focus on counting thoseoperations which represent, in general,the behavior of the algorithm
This is achieved byCounting the
frequency ofbasic operations.
A Basic operation is an operation withhighest frequency to within a
constant factor among all otherelementary operations
Thursday, September 15, 11
Computing running time continued..
7/29/2019 Intro Analysis
97/114
54
Thursday, September 15, 11
Number of iterations (additions)
7/29/2019 Intro Analysis
98/114
16
Thursday, September 15, 11
Number of iterations (additions)
While Iteration # of additions
7/29/2019 Intro Analysis
99/114
16
While Iteration # of additions
1 n
2 n/2
3 n/4
.. ..
k n/n
Thursday, September 15, 11
Number of iterations (additions)
While Iteration # of additions
7/29/2019 Intro Analysis
100/114
16
While Iteration # of additions
1 n
2 n/2
3 n/4
.. ..
k n/n
Thursday, September 15, 11
7/29/2019 Intro Analysis
101/114
54
Thursday, September 15, 11
Computing running time continued..
Algorithm 1.9 count2
7/29/2019 Intro Analysis
102/114
51
Input: A positive integer n.Output: count = number of times Step 5 is executed.
1. count 02. for i 1 to n3. mn/i4. for j 1 to m5. count count + 16. end for
7. end for8. return count
Thursday, September 15, 11
Number of iterations (additions)
7/29/2019 Intro Analysis
103/114
16
Thursday, September 15, 11
Number of iterations (additions)
While Iteration # of additions
7/29/2019 Intro Analysis
104/114
16
1 n
2 n/2
3 n/3
.. ..
n n/n
Thursday, September 15, 11
Number of iterations (additions)
While Iteration # of additions
7/29/2019 Intro Analysis
105/114
16
1 n
2 n/2
3 n/3
.. ..
n n/n
ni=0
n
i
ni=0
n
i= n
ni=0
1
i= O(n log n).
Thursday, September 15, 11
Algorithm 1.9 count2
Input: A positive integer n.
7/29/2019 Intro Analysis
106/114
ni=0
n
i
ni=0
n
i= n
ni=0
1
i= O(n log n).
53
Output: count = number of times Step 5 is executed.1. count 02. for i 1 to n3. mn/i
4. for j 1 to m5. count count + 16. end for7. end for8. return count
Thursday, September 15, 11
Useful Summation Formulas
7/29/2019 Intro Analysis
107/114
54
Thursday, September 15, 11
Useful Summation Formulas
7/29/2019 Intro Analysis
108/114
54
Pnj=1 j =
n(n+1)2 = (n
2)
Thursday, September 15, 11
Useful Summation Formulas
7/29/2019 Intro Analysis
109/114
54
Pnj=1 j =
n(n+1)2 = (n
2)
Pn
j=0
aj = a
n+11
a1
; a 6= 1
Thursday, September 15, 11
Useful Summation Formulas
7/29/2019 Intro Analysis
110/114
54
Pnj=1 j =
n(n+1)2 = (n
2)
Pnj=0
1
2j= 2 1
n= (1)
Pn
j=0
aj = a
n+11
a1
; a 6= 1
Thursday, September 15, 11
Useful Summation Formulas
7/29/2019 Intro Analysis
111/114
54
Pnj=1
1j lnn = (logn)
Pnj=1 j =
n(n+1)2 = (n
2)
Pnj=0
1
2j= 2 1
n= (1)
Pn
j=0
aj = a
n+11
a1
; a 6= 1
Thursday, September 15, 11
Space Complexity
7/29/2019 Intro Analysis
112/114
48
Space complexity refers to the number ofmemory cells needed to carry out thecomputational steps required in an algorithmexcluding memory cells needed to hold theinput.
Compare additional space needed to carry outSELECTIONSORT to that of
BOTTOMUPSORT if we have an array with 2million elements!
Thursday, September 15, 11
Examples
What is the space complexity for
7/29/2019 Intro Analysis
113/114
49
Linear search
Binary search
Selection sort
Insertion sort Merge (that merges two sorted lists)
Bottom up merge sort
Thursday, September 15, 11
Complexity Classes and small-oh(o())
Using ()notation, one can divide the functions
7/29/2019 Intro Analysis
114/114
40
into different equivalence classes, where f(n)and g(n) belong to the same equivalence classiff(n) = (g(n))
To show that two functions belong to different
equivalence classes, the small-oh notation hasbeen introduced Definition: Let f(n) and g(n) be two functions
from the set of natural numbers to the set ofnon-negative real numbers. f(n) is said to be in
o(g(n)) if for every constant c > 0, there is apositive integern0such that f(n) < cg(n) for all nn0.
Thursday, September 15, 11