Post on 22-Jan-2016
description
transcript
Analysis Tools
What is an Algorithm?
An algorithm is a finite set of instructions that specify a sequence of operations to be carried out in order to solve a specific problem or class of problems.
Algorithm Properties
An algorithm must possess the following properties:– Finiteness: Algorithm must complete after a finite number of
instructions have been executed. – Absence of ambiguity: Each step must be clearly defined,
having only one interpretation.– Definition of sequence: Each step must have a unique defined
preceding & succeeding step. The first step (start step) & last step (halt step) must be clearly noted.
– Input/output: There must be a specified number of input values, and one or more result values.
– Feasibility: It must be possible to perform each instruction.
Analyzing an Algorithm
In order to learn more about an algorithm, we can analyze it. That is, draw conclusions about how the implementation of that algorithm will perform in general. This can be done in various ways.
– determine the running time of a program as a function of its inputs; – determine the total or maximum memory space needed for program
data; – determine the total size of the program code; – determine whether the program correctly computes the desired result; – determine the complexity of the program--e.g., how easy is it to read,
understand, and modify; and, – determine the robustness of the program--e.g., how well does it deal
with unexpected or erroneous inputs?
Analyzing an Algorithm
Different ways of analyzing the algorithm with render different results. An algorithm that runs fast is not necessarily robust or correct. An algorithm that has very little lines of code does not necessarily use less resources.
So, how can we measure the efficiency of an algorithm?
Design Considerations
Given a particular problem, there are typically a number of different algorithms that will solve that problem. A designer must make a rational choice among those algorithms:
– To design an algorithm that is easy to understand, implement, and debug (software engineering).
– To design an algorithm that makes efficient use of the available computational resources (data structures and algorithm analysis)
Example
Gauss summationsum N(N+1)/2
Gauss Summation: http://www.cut-the-knot.org/Curriculum/Algebra/GaussSummation.shtml
Easy to understand, but slow.
More efficient, but need to be as smart as 10-year-old Gauss.
Replaced by
Algorithm Analysis in CSCI 3333
But, how do we measure the efficiency of an algorithm? Note that the number of operations to be performed and the space required will depend on the number of input values that must be processed.
Benchmarking algorithms
It is tempting to measure the efficiency of an algorithm by producing an implementation and then performing benchmarking analysis by running the program on input data of varying sizes and measuring the “wall clock” time for execution. However, there are many factors that affect the running time of a program. Among these are the algorithm itself, the input data, and the computer system used to run the program. The performance of a computer is determined by
– the hardware: processor used (type and speed), memory available (cache and RAM), and disk available;
– the programming language in which the algorithm is specified; – the language compiler/interpreter used; – the computer operating system software.
Asymptotic Analysis
Therefore, the goal is to have a way of describing the inherent complexity of a program, independent of machine/compiler considerations. This means not describing the complexity by the absolute time or storage needed.
Instead, focus should be concentrated on a "proportionality" approach, expressing the complexity in terms of its relationship to some known function and the way the program scales as the input gets larger. This type of analysis is known as asymptotic analysis.
Asymptotic Analysis
There is no generally accepted set of rules for algorithm analysis. In some cases, an exact count of operations is desired; in other cases, a general approximation is sufficient.
Therefore, we strive to setup a set of rules to determine how operations are to be counted.
Analysis Rules
To do asymptotic analysis, start by counting the primitive operations in an algorithm and adding them up.
Assume that primitive operations will take a constant amount of time, such as:
– Assigning a value to a variable– Calling a function– Performing an arithmetic operation– Comparing two numbers– Indexing into an array– Returning from a function– Following a pointer reference
Example of Counting Primitive Operations
Inspect the pseudocode to count the primitive operations as a function of the input size (n)
Algorithm arrayMax(A,n):
currentMax A[0]
for i 1 to n – 1 do
if currentMax < A[i] then currentMax A[i]
return currentMax
CountArray indexing + Assignment 2
Initializing i 1Verifying i<n n
Array indexing + Comparing 2(n-1)Array indexing + Assignment 2(n-1)worstIncrementing the counter 2(n-1)Returning 1
Best case: 2+1+n+4(n–1)+1 = 5n Worst case: 2+1+n+6(n–1)+1 = 7n-2
Best, Worst, or Average Case Analysis
An algorithm may run faster on some input data than on others.Best case – the data is distributed so that the
algorithm runs fastest
Worst case – the data distribution causes the slowest running time
Average case – very difficult to calculate
For our purposes, will concentrate on analyzing algorithms by identifying the running time for the worst case data.
Estimating the Running Time
The actual running time depends on the speed of the primitive operations—some of them are faster than others
– Let t = speed of the slowest primitive operation
= worst case scenario
– Let f(n) = the worst-case running time of arrayMaxf(n) = t (7n – 2)
Growth Rate of a Linear Loop
Growth rate of arrayMax is linear. Changing the hardware alters the value of t, so that arrayMax will run faster on a faster computer. However, growth rate is still linear.
0
100
200
300
400
500
600
0 2 4 6 8n
f(n)
Slow PC10(7n-2)
Fast PC5(7n-2)
Fastest PC1(7n-2)
Growth Rate of a Linear Loop
What about the following loop?for (i=0; i<n; i+=2 ) do something
Here, the number of iterations is half of n. However, higher the factor, higher the number of loops. So, although f(n) = n/2, if you were to plot the loop, you would still get a straight line. Therefore, this is still a linear growth rate.
Growth Rate of a Logarithmic Loop
What about the following segments?
for (i=1; i<n; i*=2 ) for (i=n; i>=1; i/=2 ) do something do something
When n=1000, the loop will iterate only 10 times. It can be seen that the number of iterations is a function of the multiplier or divisor. This kind of function is said to have logarithmic growth, where f(n) = log n
Growth Rate of a Linear Logarithmic Loop
What about the following segments?for (i=1; i<n; i++ )
for (j=1; j<n; j*=2 ) do something
Here, the outer loop has a linear growth, and the inner loop has a logarithmic growth. Therefore:
f(n) = n log n
Growth Rates ofCommon Classes of Functions
Input size n
Running timef(n)
Quadratically
Exponentially
Logarithmically
Linearly
Constant
Is This Really Necessary?
Is it really important to find out the exact number of primitive operations performed by an algorithm? Will the calculations change if we miss out one primitive operation?
In general, each step of pseudo-code or statement corresponds to a small number of primitive operations that does not depend on the input size. Thus, it is possible to perform a simplified analysis that estimates the number of primitive operations, by looking at the “big picture”.
What exactly is Big-O?
Big-O expresses an upper bound on the growth rate of a function, for sufficiently large values of n.
Upper bound is not the worst case. What is being bounded is not the running time (which can be determined by a given value of n), but rather the growth rate for the running time (which can only be determined over the range of values for n).
Big-Oh Notation
Definition:Let f(n) and g(n) be functions mapping nonnegative integers to real numbers.
Then, f(n) is O(g(n)) ( f(n) is big-oh of g(n) ) if there is a real constant c>0 and an integer constant n01, such that f(n) cg(n) for every integer nn0
By the definition above, demonstrating that a function f is big-O of a function g requires that we find specific constants C and N for which the inequality holds (and show that the inequality does, in fact, hold).
Big-O Theorems
For all the following theorems, assume that f(n) is a function of n and that K is an arbitrary constant.– Theorem1: K is O(1)– Theorem 2: A polynomial is O(the term containing the highest power
of n)f(n) = 7n4 + 3n2 + 5n + 1000 is O(7n4)
– Theorem 3: K*f(n) is O(f(n)) [that is, constant coefficients can be dropped]
g(n) = 7n4 is O(n4)
– Theorem 4: If f(n) is O(g(n)) and g(n) is O(h(n)) the f(n) is O(h(n)). [transitivity]
Big-O Theorems
Theorem 5: Each of the following functions is strictly big-O of its successors:
K [constant]
logb(n) [always log base 2 if no base is shown]n
nlogb(n)n2
n to higher powers2n
3n
larger constants to the n-th powern! [n factorial]
nn
For Example: f(n) = 3nlog(n) is O(nlog(n)) and O(n2) and O(2n)
smaller
larger
Big-O Theorems
Theorem 6: In general, f(n) is big-O of the dominant term of f(n), where “dominant” may usually be determined from Theorem 5.
f(n) = 7n2+3nlog(n)+5n+1000 is O(n2)
g(n) = 7n4+3n+106 is O(3n)
h(n) = 7n(n+log(n)) is O(n2)
Theorem 7: For any base b, logb(n) is O(log(n)).
Examples
In general, in Big-Oh analysis, we focus on the “big picture,” that is, the operations that affect the running time the most – the loops
Simplify the count:1. Drop all lower-order terms
7n – 2 7n
2. Eliminate constants
7n n
3. Remaining term is the Big-Oh
7n – 2 is O(n)
More Examples
Example: f(n) = 5n3 – 2n2 + 1
1. Drop all lower order terms
5n3 – 2n2 + 1 5n3
2. Eliminate the constants
5n3 n3
3. The remaining term is the Big-Oh
f(n) is O(n3)
Determining Complexities in General
1. We can drop the constants
2. sum rule: for a sequential loops add their Big-Oh valuesstatement 1; statement 2; ... statement k;
total time = time(statement 1) + time(statement 2) + ... + time(statement k)
• if-then-else statements: the worst-case time is the slowest of the two possibilities
if (cond) { sequence of statements 1
} else {
sequence of statements 2 }
Total time = max(time(sequence 1), time(sequence 2))
Determining Complexities in General
4. for loops: The loop executes N times, so the sequence of statements also executes N times. Since we assume the statements are O(1),
total time = N * O(1) = O(N)
5. Nested for loops: multiply their Big-Oh valuesfor (i = 0; i < N; i++) { for (j = 0; j < M; j++) {
sequence of statements } }
total time = O(N) * O(M) = O(N*M)
6. In a polynomial, the term with the highest degree establishes the Big-Oh
Growth Rate Examples
Efficiency Example
ConstantO(1)
Accessing an element of an array
LogarithmicO(log n)
Binary search
LinearO(n)
Pushing a collection of elements onto a stack
QuadraticO(n2)
Bubble sort
ExponentialO(2n)
Towers of Hanoi (Goodrich, 198)
Why Does Growth Rate Matter?
n
Constant
O(1)
Logarithmic
O(logn)
Linear
O(n)
Quadratic
O(n2)
10 0.4 nsec 1.33 nsec 4.0 nsec 40.0 nsec
1,000 0.4 nsec 3.99 nsec 0.4 sec 0.4 msec
100,000 0.4 nsec 6.64 nsec 0.04 msec 4.0 sec
10,000,000 0.4 nsec 9.30 nsec 0.004 sec 1.11 hr
Assuming that it takes 0.4 nsec to process one element
Finding the Big-Oh
for( int i=0; i<n; ++i ) {
for(int j=0; j<n; ++j) {
myArray[i][j] = 0;
}
}
sum = 0;
for( int i=0; i<n; ++i ) {
for(int j=0; j<i; ++j) {
sum++;
}
}
Example that shows why the Big-Oh is so important.
Write a program segment so that, given an array X, compute array A such that, each number A[i] is the average of the numbers in X from X[0] to X[i]
1 3 5 7 9
1 2 3 4 5
X
A
X[0] + X[1] + X[2] = 9
9 / 3 = 3
Solution #1 – Quadratic time
Algorithm prefixAverages1…
for i 0 to n-1 do a 0 for j 0 to i do a a + X[j] A[i] a/(i+1)…
Two nested loops
Inner loop – loopsthrough X, adding the numbers from element 0 through element i
Outer loop – loopsthrough A, calculatingthe averages and putting the resultinto A[i]
Algorithm prefixAverages2…
s 0for i 0 to n-1 do s s + X[i] A[i] s/(i+1)…
Only one loop
Solution #2 - Linear Time
Sum – keeps track of thesum of the numbers in Xso that we don’t have to loop through X every time
Loop – loopsthrough A, addingto the sum, calculatingthe averages, and putting the resultinto A[i]
1. Both algorithms correctly solved the problem
– Lesson – There may be more than one way to write your program.
2. One of the algorithms was significantly faster
– Lesson – The algorithm that we choose can have a big influence on the program’s speed.
Evaluate the solution that you pick, and ask whether it is the most efficient way to do it.
Lessons Learned