Date post: | 26-Dec-2015 |
Category: |
Documents |
Upload: | darrell-reynolds |
View: | 217 times |
Download: | 0 times |
Analysis of AlgorithmsAn algorithm is a step-by-step procedure for
solving a problem in a finite amount of time.
Input OutputMerge Sort
Quick Sort
• Which one is better?• What are the criteria?
How to evaluate algorithms?
Problem: Find Maximum from Array
2
Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A
currentMax A[0]for i 1 to n 1 do
if A[i] currentMax thencurrentMax A[i]
return currentMax
• Find max from array A: A[] = [5 8 2 8 9 1 3 7 4 6 …]
How good is algorithm arrayMax()?
3
Problem: Prefix AveragesNumbers X = 8 4 3 5 9 3 7 6 5 4 2 1 …Prefix averages A = 8 6 5 5 …
Which algorithm is better?
Analysis of Algorithms
• Data structure– A systematic way of organizing and accessing data
• Algorithm– A step-by-step procedure for performing some task in
a finite amount of time
• Running time of algorithms and data structure operations is a natural measure of “goodness”
4
Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A
currentMax A[0]for i 1 to n 1 do
if A[i] currentMax thencurrentMax A[i]
return currentMax
• Find max: A[0] … A[n-1]
• How good is algorithm arrayMax()?
A[] = [5 8 2 8 9 1 3 …]
Summations and Geometric Sums
5
n
i
nni
n
i
nni
d
i
dd
ii
n
i
b
ai
a
aaaaa
nananaananf
nnni
bfafafafif
0
1
0
12
0
2210
1
122...84212 : Example
1
1...1:Sum Geometric
...)(:Polynomial
2/)1(...21 : Example
)(...)2()1()()(:Summation
Running Time •Most algorithms transform input objects into output objects.• The running time of an algorithm typically grows with the input size.• Average case time is often difficult to determine.•We focus on the worst case running time.– Easier to analyze– Crucial to applications such as games, finance
and robotics
0
20
40
60
80
100
120
Runnin
g T
ime
1000 2000 3000 4000
Input Size
best caseaverage caseworst case
6
Experimental Studies• Write a program implementing the algorithm• Run the program with inputs of varying size and
composition• Use a method like System.currentTimeMillis( ) to get an accurate
measure of the actual running time• Plot the results
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 50 100
Input Size
Tim
e (
ms)
7
Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A
currentMax A[0]for i 1 to n 1 do
if A[i] currentMax thencurrentMax A[i]
return currentMax
Limitations of Experiments• It is necessary to implement the algorithm, which
may be difficult• Results may not be indicative of the running time on
other inputs not included in the experiment. • In order to compare two algorithms, the same
hardware and software environments must be used
8
Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A
currentMax A[0]for i 1 to n 1 do
if A[i] currentMax thencurrentMax A[i]
return currentMax
Theoretical Analysis• Uses a high-level description of the algorithm
instead of an implementation• Takes into account all possible inputs• Allows us to evaluate the relative efficiency of
any two algorithms in a way that is independent from the hardware/software environment
• Characterizes running time as a function of the input size, n.
9
Pseudocode
• High-level description of an algorithm• More structured than English
prose• Less detailed than a program• Preferred notation for
describing algorithms• Hides program design issues
10
Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A
currentMax A[0]for i 1 to n 1 do
if A[i] currentMax thencurrentMax A[i]
return currentMax
Example: find max element of an array
A[] = [5 8 2 8 9 1 3 …]
• Consider all possible inputs• Independent of
hardware/software• Characterizes running time as
a function of the input size, n.
Pseudocode Details
• Control flow– if … then … [else …]
– while … do …
– repeat … until …
– for … do …
– Indentation replaces braces
• Method declarationAlgorithm method (arg [, arg…])
Input …
Output …
• Method callvar.method (arg [, arg…])
• Return valuereturn expression
• Expressions Assignment
(like in Java) Equality testing
(like in Java)n2 Superscripts and other
mathematical formatting allowed
11
Primitive Operations
• Basic computations performed by an algorithm
• Identifiable in pseudocode• Largely independent from
the programming language• Exact definition not
important (we will see why later)
• Running time– Assumed to take a constant amount
of time in the RAM model
Examples:• Evaluating an expression• Assigning a value to a
variable• Indexing into an array• Calling a method• Returning from a method
12
currentMax 0currentMax A[0]return result
Counting Primitive Operations
By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size
13
Estimating Running Time• Algorithm arrayMax executes 8n 2 primitive
operations in the worst case. Define:a = Time taken by the fastest primitive operationb = Time taken by the slowest primitive operation
• Let T(n) be worst-case time of arrayMax. Thena (8n 2) T(n) b(8n 2)
• Hence, the running time T(n) is bounded by two linear functions
14
Algorithm arrayMax(A, n)
# operations
currentMax A[0] 2for i 1 to n 1 do 2n
if A[i] currentMax then 2(n 1)currentMax A[i] 2(n 1)
{ increment counter i } 2(n 1)return currentMax 1
Total 8n 2
Growth Rate of Running Time• Changing the hardware/software environment – Affects T(n) by a constant factor, but– Does not alter the growth rate of T(n)
• The linear growth rate of the running time T(n) is an intrinsic property of algorithm arrayMax
15
a(8n 2) T(n) b(8n 2)
Nested Loops and Quadratic Function
• Appears in nested loops, where the inner loop performs a linear number of operations and the outer loop is performed a linear number of times; i.e., nn = n2
• Also arise in the context of nested loops where the first iteration of a loop uses one operation, the second uses two operations, the third uses three operations, and so on.i.e., 1+2+3+…+(n-2)+(n-1)+n = n(n+1)/2
16
Algorithm arrayMax(A, n)Input array A of n × n integersOutput maximum element of A
currentMax A[0, 0]for i 0 to n 1 do
for j 0 to n – 1 do if A[i, j] currentMax then
currentMax A[i, j]return currentMax
3735
496
18
2
7
39
568
A
i
j
for i 0 to n 1 do for j 0 to i do …
Lower-Order Terms and Constant Factors
• The growth rate is not affected by– lower-order terms or– constant factors
• ExamplesQuadratic function:
– 105n2 108n
– 105n2
Linear function:
– 102n 105
– 102n
Different linear function:
– 3n
– 2n+10
– n
17
Big-Oh Notation• Given functions f(n) and
g(n), we say that f(n) is O(g(n)) if there are positive constantsc and n0 such that
f(n) cg(n) for n n0
• Example: 2n 10 is O(n)
2n 10 cn
(c 2) n 10
n 10(c 2)
Pick c 3 and n0 10
18
2n 10 n for n 10
Big-Oh Example
• Example: the function n2 is not O(n)– n2 cn
– n c– The above inequality
cannot be satisfied since c must be a constant
19
f(n) is O(g(n)):f(n) cg(n) for n
n0
20More Big-Oh Examples 7n-2
7n-2 is O(n)need c > 0 and n0 1 such that 7n-2 c•n for n n0
this is true for c = 7 and n0 = 1
3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 c•n3 for n
n0
this is true for c = 4 and n0 = 213 log n + 53 log n + 5 is O(log n)need c > 0 and n0 1 such that 3 log n + 5 c•log n for
n n0
this is true for c = 8 and n0 = 2not affected bylower-order terms or constant
factors
f(n) is O(g(n)):f(n) cg(n) for n
n0
Big-Oh Rules• Not affected by lower-order terms and
constant factor• If f(n) is a polynomial of degree d, then f(n) is
O(nd), i.e.,1. Drop lower-order terms2. Drop constant factors
• Use the smallest possible class of functions– Say “2n is O(n)” instead of “2n is O(n2)”
• Use the simplest expression of the class– Say “3n 5 is O(n)” instead of “3n 5 is O(3n)”
21
Big-Oh and Growth Rate• The big-Oh notation gives an upper bound on
the growth rate of a function• “f(n) is O(g(n))” means “the growth rate of f(n) is
no more than the growth rate of g(n)”• We can use the big-Oh notation to rank
functions according to their growth rate
22
f(n) is O(g(n)) g(n) is O(f(n))
g(n) grows more
Yes No
f(n) grows more
No Yes
Same growth Yes Yes
Asymptotic Algorithm Analysis• The asymptotic analysis of an algorithm
determines the running time in big-Oh notation• To perform the asymptotic analysis
1. We find the worst-case number of primitive operations executed as a function of the input size
2. We express this function with big-Oh notation
Example of Algorithm arrayMax():1. We determine that algorithm arrayMax executes at most 8n 2
primitive operations2. We say that algorithm arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard them when counting primitive operations
23
Example: Computing Prefix Averages
• The i-th prefix average of an array X is average of the first (i 1) elements of X:
A[i] X[0] X[1] … X[i])/(i+1)
• Computing the array A of prefix averages of another array X has applications to financial analysis
24
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7
X
A
X = 21 23 25 31 20
…
A = 21 22 23 25 24
…
25
Algorithm1: Prefix Averages (Quadratic)
The following algorithm computes prefix averages in quadratic time by applying the definition
X = 8 4 3 5 9 3 7 6 5 4 2 1 … A = 8 6 5 5 …
• Ignor constant factors and lower-order terms• The running time of prefixAverages1 is O(1 2 …n)
O(1 2 …n) = O(n(n 1) 2) = O(n2)
• Thus, algorithm prefixAverages1 runs in O(n2) time
26
Algorithm2: Prefix Averages (Linear)The following algorithm computes prefix averages in linear time by keeping a running sum
X = 8 4 3 5 9 3 7 6 5 4 2 1 … A = 8 6 5 5
…Algorithm prefixAverages2(X, n)
Input array X of n integersOutput array A of prefix averages of X #operationsA new array of n integers ns 0 1for i 0 to n 1 do n
s s X[i] nA[i] s (i 1) n
return A 1Algorithm prefixAverages2 runs in O(n) time
Seven Used Functions• Polynomial functions: f(n) = a0+a1n+a2n2+a3n3+…+adnd
• Exponential function: an
27
28
29Relatives of Big-OhBig-Omega f(n) is (g(n)) if there is a constant c > 0
and an integer constant n0 1 such that
f(n) cg(n) for n n0
3nlogn+2n is (nlogn) Since: 3nlogn+2n 3nlogn, for n 2
Big-Theta f(n) is (g(n)) if there are constants c’ > 0 and
c’’ > 0 and an integer constant n0 1 such that
c’g(n) f(n) c’’g(n) for n n0
3nlogn+4n+5logn is (nlogn) Since: 3nlogn 3nlogn+4n+5logn (3+4+5)nlogn for n
2
f(n) is O(g(n)):f(n) cg(n) for n n0
f(n)
cg(n)
Intuition for Asymptotic Notation
30
Big-Oh f(n) is O(g(n)) if f(n) is asymptotically less than or equal
to g(n)
Big-Omega f(n) is (g(n)) if f(n) is asymptotically greater than or
equal to g(n)
Big-Theta f(n) is (g(n)) if f(n) is asymptotically equal to g(n)
f(n) is O(g(n)) f(n) is (g(n))
31Example Uses of the Relatives of Big-Oh
f(n) is (g(n)) if it is (n2) and O(n2). We have already seen the former, for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
Let c = 5 and n0 = 1
5n2 is (n2)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 1 and n0 = 1
5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 5 and n0 = 1
5n2 is (n2)
32Example Uses of the Relatives of Big-Oh
3nlogn 3nlogn + 4n + 5 (3+4+5) nlogn for n2 3nlogn+4n+5 is (nlogn)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 1 and n0 = 1
5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 3 and n0 = 2
3nlogn+2n is (nlogn)
5n2+3nlogn+2n+5 is O(n2)5n2+3nlogn+2n+5 (5+3+2+5) n2 = cn2 for c =15, n0 = 2
A Recursive Algorithm for Computing Powers
• Power function p(x,n) = xn
– A linear recursion definition:
– Employ a squaring technique
• 27=2(26)=2(2(25))=…=128• 27=21+(6/2)2 =2(26/2)2=2(23)2=2((2(2)2)2)=128• What is running-time complexity of each
method?
33
otherwise )1,(
0 if 1),(
nxpx
nnxp
even is 0 if )2/,(odd is 0 if )2/)1(,(
0 if 1
),(2
2
nnxpnnxpx
n
nxp
A Recursive Algorithm for Computing Powers
Algorithm Power(x,n)Input: A number x and integer n 0Output: The value xn
if n = 0 then return 1if n is odd then y Power(x, (n-1)/2) return x • y • yelse y Power(x,n/2) return y • y
34
Math you need to Review
Logarithms and Exponents• properties of logarithms:
logb(xy) = logbx + logbylogb (x/y) = logbx - logbylogbxa = alogbxlogba = logxa/logxb
• properties of exponentials:a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a loga
b
bc = a c*loga
b
35
SummationsLogarithms and ExponentsProof techniquesBasic probability
Simple Justification Techniques
• By example• The “Contra” attack– Contrapositive– Contradiction
• Induction and loop invariants
36
By example
Claim:“Every element x in a set S has property P”
• To falsify the claim, we need to only produce a counterexample
• Every number of the form 2i-1 is a prime, when i is an integer greater than 1.– A counterexample: 24-1 = 15
37
Contrapositive• “If p is true, then q is true” is equivalent to
“If q is not true, then p is not true”• Let a and b be integers.
If ab is even, then a is even or b is even.Contrapositive: …?Justify by contrapositive: Suppose a = 2i+1 and b = 2j+1, for some integer i and j. Then ab = 4ij+2i+2j+1 = 2(2ij+i+j)+1; hence, ab is odd.
38
Contradition• To establish a statement q is true
First supposing q is false and then showing that this assumption leads to a contradiction
• Let a and b be integers. If ab is odd, then a is odd and b is odd.
Proof by contraditionThe opposite: suppose a is even or b is even
Assume that a is even. Then a = 2i for some integer i. Hence, ab = (2i)b = 2(ib), that is, ab is even. Contradiction: ab cannot simultaneously be odd and even. Therefore, a is odd and b is odd.
39
Induction• To say some statement q(n) is true for all n1 by
showing “if q(i) is true for i<n, then q(n) is true.”• Consider the Fibonacci function F(n), where we
define F(1)=1, F(2)=2, and F(n)=F(n-1)+F(n-2) for n>2. We claim that F(n)<2n.
Base cases: (n 2). F(1) = 1<2=21 and F(2) = 2<4=22.Induction step: (n>2). Suppose our claim is true for n’<n. Consider F(n). Since n>2, F(n) =F(n-1)+F(n-2). Moreover, since n-1<n and n-2<n, apply the inductive assumption to imply that F(n)<2n-1+2n-2, since 2n-1+2n-2<2n-1+2n-1=2 . 2n-
1=2n.