DESIGN AND ANALYSIS OF ALGORITHMS
Text Book:
Introduction to the Design and Analysis of Algorithms.
- Anany Levitin
UNIT I
• Introduction• Notion of Algorithm• Fundamentals of Algorithmic Solving• Important Problem Types• Fundamentals of the Analysis Framework• Asymptotic Notations • Basic Efficiency Classes
Introduction
• Algorithmics is more than a branch of computer science.
• It is the core of computer science, and, in all fairness, can be said to be relevant to most of science, business, and technology.
Notion of Algorithm
• An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.
Notion of Algorithm
GCD of 2 Numbers
• Euclid’s algorithm for computing gcd(m, n)• Step 1: If n = 0, return the value of m as the
answer and stop; otherwise, proceed to Step 2.
• Step 2: Divide m by n and assign the value of the remainder to r.
• Step 3: Assign the value of n to m and the value of r to n. Go to Step 1.
GCD of 2 Numbers
ALGORITHM Euclid(m, n)
//Computes gcd (m, n) by Euclid’s algorithm
//Input: Two nonnegative, not-both-zero integers m and n
//Output: Greatest common divisor of m and n
while n != 0 do
r ←m mod n
m←n
n←r
return m
Notion of Algorithm
• Consecutive integer checking algorithm for computing gcd(m, n)
• Step 1: Assign the value of min{m, n} to t.
• Step 2: Divide m by t. If the remainder of this division is 0, go to Step 3;
otherwise, go to Step 4.
Notion of Algorithm
• Step 3: Divide n by t. If the remainder of this division is 0, return the value of t as the answer and stop; otherwise, proceed to Step 4.
• Step 4: Decrease the value of t by 1. Go to Step 2.
Notion of Algorithm
Middle-school procedure for computing gcd(m, n)
Step 1: Find the prime factors of m.
Step 2: Find the prime factors of n.
Notion of Algorithm
Middle-school procedure for computing gcd(m, n)
Step 3: Identify all the common factors in the two prime expansions found in Step
1 and Step 2.
Step 4: Compute the product of all the common factors and return it as the
greatest common divisor of the numbers given.
Notion of Algorithm
For the numbers 60 and 24, we get
60 = 2 . 2 . 3 . 5
24 = 2 . 2 . 2 . 3
gcd(60, 24) = 2 . 2 . 3 = 12.
Notion of Algorithm
Sieve of Eratosthenes method of generating prime numbers:
Notion of Algorithm
ALGORITHM Sieve(n)
//Implements the sieve of Eratosthenes
//Input: A positive integer n > 1
//Output: Array L of all prime numbers less than or equal to n
for p←2 to n do
A[p]←p
Notion of Algorithm
for p←2 to √ n do
if A[p] !=0
j ← p p∗while j ≤ n do
A[j ]←0
j ←j + p
end while
End for
Notion of Algorithm
//copy the remaining elements of A to array L of the primes
i ←0
for p←2 to n do
if A[p] = 0
L[i]←A[p]
i ←i + 1
return L
Notion of Algorithm
• Find gcd(31415, 14142) by applying Euclid’s algorithm.
Notion of Algorithm
• gcd(31415, 14142) • gcd(14142, 3131) • gcd(3131, 1618)• gcd(1618, 1513) • gcd(1513, 105) • gcd(1513, 105) • gcd(105, 43) • gcd(43, 19) = gcd(19, 5) = gcd(5, 4) • gcd(4, 1) = gcd(1, 0) = 1.
Fundamentals of Algorithmic Problem Solving
• Algorithms can be the procedural solutions to problems.
• These solutions are not answers but specific instructions for getting answers.
Fundamentals of Algorithmic Problem Solving
• Understanding the Problem
• Ascertaining the Capabilities of the Computational Device
• Choosing between Exact and Approximate Problem Solving
• Algorithm Design Techniques
Fundamentals of Algorithmic Problem Solving
• Designing an Algorithm and Data Structures
• Methods of Specifying an Algorithm
• Proving an Algorithm’s Correctness
• Analyzing an Algorithm
• Coding an Algorithm
Important Problem Types
• Sorting• Searching• String processing• Graph problems• Combinatorial problems• Geometric problems• Numerical problems
Examples
• Write an algorithm for the sorting problem that sorts an array by counting, for each of its elements, the number of smaller elements and then uses this information to put the element in its appropriate position in the sorted array.
• Apply this algorithm to sorting the list
60, 35, 81, 98, 14, 47.
Examples
ALGORITHM ComparisonCountingSort(A[0..n − 1])
//Sorts an array by comparison counting
//Input: Array A[0..n − 1] of orderable values
//Output: Array S[0..n − 1] of A’s elements sorted in nondecreasing order
Examples
for i ←0 to n − 1 do
Count[i]←0
for i ←0 to n − 2 do
for j ←i + 1 to n − 1 do
if A[i]<A[j ]
Count[j ]←Count[j ]+ 1
else Count[i]←Count[i]+ 1
Examples
for i ←0 to n − 1 do
S[Count[i]] = a[i]
Return S
Fundamentals of the Analysisof Algorithm Efficiency
• The Analysis Framework:
• There are two kinds of efficiency:
• Time efficiency
• Space efficiency.
Fundamentals of the Analysisof Algorithm Efficiency
• Time efficiency, also called time complexity, indicates how fast an algorithm in question runs.
• Space efficiency, also called space complexity, refers to the amount of memory units required by the algorithm in addition to the space needed for its input and output.
Measuring an Input’s Size
• Let’s start with the obvious observation that almost all algorithms run longer on larger inputs.
• For example, it takes longer to sort larger arrays, multiply larger matrices, and so on.
• Therefore, it is logical to investigate an algorithm’s efficiency as a function of some parameter n indicating the algorithm’s input size.
Units for Measuring Running Time
• We can simply use some standard unit of time measurement—a second, or millisecond, and so on—to measure the running time of a program implementing the algorithm.
• There are obvious drawbacks to such an approach.
Units for Measuring Running Time
• Dependence on the speed of a particular computer, dependence on the quality of a program implementing the algorithm and of the compiler used in generating the machine code, and the difficulty of clocking the actual running time of the program.
• Since we are after a measure of an algorithm’s efficiency, we would like to have a metric that does not depend on these extraneous factors.
Units for Measuring Running Time
• One possible approach is to count the number of times each of the algorithm’s operations is executed.
• The thing to do is to identify the most important operation of the algorithm, called the basic operation, the operation contributing the most to the total running time, and compute the number of times the basic operation is executed.
Units for Measuring Running Time
• As a rule, it is not difficult to identify the basic operation of an algorithm: it is usually the most time-consuming operation in the algorithm’s innermost loop.
• For example, most sorting algorithms work by comparing elements (keys) of a list being sorted with each other; for such algorithms, the basic operation is a key comparison.
Units for Measuring Running Time
• As another example, algorithms for mathematical problems typically involve some or all of the four arithmetical operations: addition, subtraction, multiplication, and division.
• Of the four, the most time-consuming operation is division, followed by multiplication and then addition and subtraction, with the last two usually considered together.
Units for Measuring Running Time
• Thus the established framework for the analysis of an algorithm’s time efficiency suggests measuring it by counting the number of times the algorithm’s basic operation is executed on inputs of size n.
Units for Measuring Running Time
• Let cop be the execution time of an algorithm’s basic operation on a particular computer, and let C(n) be the number of times this operation needs to be executed for this algorithm.
• Then we can estimate the running time T (n) of a program implementing this algorithm on that computer by the formula:
Orders of Growth
Worst-Case, Best-Case, and Average-Case Efficiencies
• The worst-case efficiency of an algorithm is its efficiency for the worst-case input of size n, which is an input (or inputs) of size n for which the algorithm runs the longest among all possible inputs of that size.
• The worst-case analysis provides very important information about an algorithm’s efficiency by bounding its running time from above.
Worst-Case, Best-Case, and Average-Case Efficiencies
• The best-case efficiency of an algorithm is its efficiency for the best-case input of size n, which is an input (or inputs) of size n for which the algorithm runs the fastest among all possible inputs of that size.
Worst-Case, Best-Case, and Average-Case Efficiencies
• However, that neither the worst-case analysis nor its best-case counterpart yields the necessary information about an algorithm’s behavior on a “typical” or “random” input.
• This is the information that the average-case efficiency seeks to provide.
• To analyze the algorithm’s average case efficiency, we must make some assumptions about possible inputs of size n.
Worst-Case, Best-Case, and Average-Case Efficiencies
Amortized Efficiency
• It applies not to a single run of an algorithm but rather to a sequence of operations performed on the same data structure.
• It turns out that in some situations a single operation can be expensive, but the total time for an entire sequence of n such operations is always significantly better than the worst-case efficiency of that single operation multiplied by n.
Amortized Efficiency
• So we can “amortize” the high cost of such a worst-case occurrence over the entire sequence in a manner similar to the way a business would amortize the cost of an expensive item over the years of the item’s productive life.
Asymptotic Notations and Basic Efficiency Classes
• The efficiency analysis framework concentrates on the order of growth of an algorithm’s basic operation count as the principal indicator of the algorithm’s efficiency.
• To compare and rank such orders of growth, computer scientists use three notations: • O(big oh), Ω(big omega), and θ(big theta).
Asymptotic Notations and Basic Efficiency Classes
• The idea of these definitions is to establish a relative order among functions.
• Given two functions, we compare their relative rates of growth.
• Example 1000n and n2
O - Notation
• A function t (n) is said to be in O(g(n)), denoted t (n) O(g(n)), if t (n) is bounded above by some ∈constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n0 such that
t (n) ≤ cg(n) for all n ≥ n0.
O - Notation
• Example:
• 100n + 5 O(n∈ 2)
O - Notation
Ω - Notation
• A function t (n) is said to be in (g(n)), denoted t (n) (g(n)), if t (n) is bounded below by ∈some positive constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n0 such that
t (n) ≥ cg(n) for all n ≥ n0.
Ω - Notation
• Example
• n3 ∈ Ω(n2)
Ω - Notation
Θ - Notation
• A function t (n) is said to be in (g(n)), denoted t (n) (g(n)), if t (n) is bounded both above ∈and below by some positive constant multiples of g(n) for all large n, i.e., if there exist some positive constants c1 and c2 and some nonnegative integer n0 such that
c2g(n) ≤ t (n) ≤ c1g(n) for all n ≥ n0.
Θ - Notation
• Example:
• 1/2 n (n-1) ∈ Θ (n2)
Θ - Notation
THEOREM
• If t1(n) O(g1(n)) and t2(n) O(g2(n)), then∈ ∈
t1(n) + t2(n) O(max{g1(n), g2(n)}).∈
PROOF
if a1 <= b1 and a2 <= b2, then
a1 + a2 < = 2 max{b1, b2}
Θ - Notation
RULES
If T1(n) = O(f(n)) and T2(n)=O(g(n)), then
(a) T1(n) + T2(n) = max (O(f(n)), O(g(n)))
(b) T1(n) * T2(n) = O(f(n) * g(n))
RULES
• If T(x) is a polynomial of degree n, then T(x) = θ(x n).
• Log k n = O(n) for any constant k. This tells us that logarithms grow very slowly.
RULES
• If two programs are expected to take
similar times, probably the best way to
decide which is faster is to code them both
up and run them.
Using Limits for Comparing Orders of Growth
• Though the formal definitions of O, , and are indispensable for proving their abstract properties, they are rarely used for comparing the orders of growth of two specific functions.
• A much more convenient method for doing so is based on computing the limit of the ratio of two functions in question.
Using Limits for Comparing Orders of Growth
Using Limits for Comparing Orders of Growth
• Compare the orders of growth of the following:
• ½ n(n − 1) and n2
• Log 2 n and √n
• n! And 2n
ASSIGNMENT
1. What is an algorithm? Explain the notion of an algorithm with a neat diagram.
2. Give and explain the important characteristics of an algorithm.
3. Write and explain the Euclid’s algorithm for computing GCD of any two integers. Hence find gcd(31415, 14142).
ASSIGNMENT
4. Explain with an example the consecutive integer checking algorithm for computing GCD of any two integers.
5. Implement Sieve’s algorithm to generate prime numbers between 1 and n. Trace the algorithm for n = 25.
6. Explain with a neat diagram, the sequence of steps to be followed in designing and analyzing an algorithm.
ASSIGNMENT
7. Write an algorithm for finding the distance between the two closest elements in an array of numbers.
8. Explain the important problem types. Give two examples for each.
9. Implement an algorithm that sorts an array by counting, for each of its elements, the number of smaller elements and then uses this information to put the element in its appropriate position in the sorted array. Apply this algorithm to sorting the list 60, 35, 81, 98, 14, 47.
ASSIGNMENT
10. Explain the terms: time and space efficiency.
11. Explain the units for measuring the running time.
12. Compare the orders of growth of various algorithms.
13. Define the following: worst case, best case and average case analysis of an algorithm.
ASSIGNMENT
14. Give the worst case, best case and average case analysis of sequential search algorithm.
15. What is amortized efficiency? Explain with an example.
ASSIGNMENT
16. For each of the following algorithms, indicate (i) a natural size metric for its inputs, (ii) its basic operation, and (iii) whether the basic operation count can be different for inputs of the same size:
a. computing the sum of n numbers
b. computing n!
c. finding the largest element in a list of n numbers
d. Euclid’s algorithm
e. sieve of Eratosthenes
f. pen-and-pencil algorithm for multiplying two n-digit decimal integers
ASSIGNMENT
17. Explain the different asymptotic notations with an example for each notation.
18. If If t1(n) O(g1(n)) and t2(n) O(g2(n)), then ∈ ∈show that t1(n) + t2(n) O(max{g1(n), g2(n)}).∈
19. Compare the orders of growth for the following:
a). ½ n(n − 1) and n2
b) Log 2 n and √n
c) n! And 2n
ASSIGNMENT
20. Indicate whether the first function of each of the following pairs has a smaller, same or larger order of growth than second function.
a. n (n+1) and 2000n2
b. 100n2 and 0.01n3
c. Log 2 n and ln n
d. 2n-1 and 2n
e. Log 2 n and log 2 n2
f. (n-1)! And n!
ASSIGNMENT
21. For each of the following functions, indicate the class (g(n)) the function belongs to. Prove your assertions.
22. Prove that if t1(n) Ω(g1(n)) and t2(n) Ω(g2(n)), ∈ ∈then t1(n) + t2(n) ∈ Ω(max{g1(n), g2(n)}).
ASSIGNMENT
23. Prove that every polynomial of degree k,
p(n) = aknk + ak−1nk−1+ . . . + a0
with ak > 0, belongs to θ (nk).
24.List the following functions according to their order of growth from the lowest to the highest:
ASSIGNMENT
25. Find the Big – oh notation for the following:a. Log n + √n
b. n + n log n
c. 6n + 2n4 + 4n5
26. With the help of a flowchart, explain the various stages of algorithm design and analysis process.
27. Explain the different problem types and give one algorithm for each type.
ASSIGNMENT
28. Use the informal definitions of O,, and to determine
whether the following assertions are true or false.