8/6/2019 Design and Analysis of Algorithms 1
1/29
.
.. ..
.Design and Analysis of Algorithms
Lecture 1: Introduction
Ahmed Khademzadehhttp://khademzadeh.mshdiau.ac.ir
Azad University of Mashhad
The slides are borrowed from Ke (Kevin) Yi. Thanks to him for his benevolence.
Lecture 1: Introduction Design and Analysis of Algorithms
http://-/?-8/6/2019 Design and Analysis of Algorithms 1
2/29
. Outline
What is this course about?
What are algorithms?
What does it mean to analyze an algorithm?
Comparing time complexity
Thoughts on algorithm design
Lecture 1: Introduction Design and Analysis of Algorithms
8/6/2019 Design and Analysis of Algorithms 1
3/29
. What is this course about?.Example (Chain Matrix Multiplication).
.. .
.
.
A = C =[
1 1 0 1].
B = D =
10
11
.
Want: ABCD =?
Method 1: (AB)(CD)
Method 2: A((BC)D)
Method 1 is much more efficient than Method 2.(Expand the expression on board)
Lecture 1: Introduction Design and Analysis of Algorithms
8/6/2019 Design and Analysis of Algorithms 1
4/29
. What is this course about?
There is usually more than one algorithm for solving aproblem.
Some algorithms are more efficient than others.
We want the most efficient algorithm.
Lecture 1: Introduction Design and Analysis of Algorithms
8/6/2019 Design and Analysis of Algorithms 1
5/29
. What is this course about?
If we a number of alternative algorithms for solving a problem,how do we know which is the most efficient?
To do so, we need to analyze each of them to determine itsefficiency.
Of course, we must also make sure the algorithm is correct.
Lecture 1: Introduction Design and Analysis of Algorithms
8/6/2019 Design and Analysis of Algorithms 1
6/29
. What is this course about?In this course, we will discuss fundamental techniques for:...1 Proving the correctness of algorithms,...2 Analyzing the running times of algorithms,...3 Designing efficient algorithms,...4 Showing efficient algorithms do not exist for some problemsNotes:
Analysis and design go hand-in-hand:
By analyzing the running times of algorithms, we
will know how to design fast algorithms.
In Data Structure course, you have learned techniques for 1,2, 3 w.r.t sorting and searching. In this course, we will discussthem in the contexts of other problems.
Lecture 1: Introduction Design and Analysis of Algorithms
8/6/2019 Design and Analysis of Algorithms 1
7/29
. What is this course about?Differences between Data Sturcture and Algorithm Design Course:.
Data Sturcture.
.. .
.
.
The name is Data Sturcture
Simpler problemsYou learn concrete problems:sorting, searching (binarytree), etc.
You know how to solvethese concrete problems.
.Algorithm Design Analysis.
.. .
.
.
The name is AlgorithmDesign Analysis
Harder problems
You learn general techniquesvia concrete problems:divide-and-conquer, dynamicprogramming, etc.
You can solve many newproblems using thesetechniques.
Lecture 1: Introduction Design and Analysis of Algorithms
8/6/2019 Design and Analysis of Algorithms 1
8/29
. Outline
What is this course about?
What are algorithms?
What does it mean to analyze an algorithm?Comparing time complexity
Thoughts on algorithm design
Lecture 1: Introduction Design and Analysis of Algorithms
C
8/6/2019 Design and Analysis of Algorithms 1
9/29
. Computational Problem.Definition... .
..
A computational problem is a specification of the desiredinput-output relationship
.Example (Computational Problem).
.. .
.
.
Sorting
Input: Sequence of n numbers a1, , an
Output: Permutation (reordering)
a
1, a
2, , a
n
such that a1 a
2 a
n
Lecture 1: Introduction Design and Analysis of Algorithms
I
8/6/2019 Design and Analysis of Algorithms 1
10/29
. Instance
.Definition... .
..A problem instance any valid input to the problem.
.Example (Instance of the Sorting Problem).
.. ..
.8, 3, 6, 7, 1, 2, 9
Lecture 1: Introduction Design and Analysis of Algorithms
Al i h
8/6/2019 Design and Analysis of Algorithms 1
11/29
. Algorithm
.Definition.
.. .
.
.
An algorithm is a well defined computational procedure thattransforms inputs into outputs, achieving the desired input-output
relationship.Definition... .
..
A correct algorithm halts with the correct output for every inputinstance. We can then say that the algorithm solves the problem
Lecture 1: Introduction Design and Analysis of Algorithms
E l I ti S t
8/6/2019 Design and Analysis of Algorithms 1
12/29
. Example: Insertion SortPseudocode:
Input: A[1 . . . n] is an array of numbers
for j=2 to n dokey = A[j];i = j-1;while i 1 and A[i]> key do
A[i+1] = A[i];
i- -;endA[i+1] = key;
end
key
Sorted UnsortedWhere in the sorted part to put key?
Lecture 1: Introduction Design and Analysis of Algorithms
H D It W k?
8/6/2019 Design and Analysis of Algorithms 1
13/29
. How Does It Work?An incremental approach: To sort a given array of length n,
at the ith step it sorts the array of the first i items by makinguse of the sorted array of the first i 1 items in the (i 1)thstep
.Example
.
.. .
.
.
Sort A = 6, 3, 2, 4, 5 with insertion sort
Step 1: 6, 3, 2, 4, 5
Step 2: 3, 6, 2, 4, 5
Step 3: 2, 3, 6, 4, 5
Step 4: 2, 3, 4, 6, 5
Step 5: 2, 3, 4, 5, 6
Lecture 1: Introduction Design and Analysis of Algorithms
O tli
8/6/2019 Design and Analysis of Algorithms 1
14/29
. Outline
What is this course about?
What are algorithms?
What does it mean to analyze an algorithm?Comparing time complexity
Thoughts on algorithm design
Lecture 1: Introduction Design and Analysis of Algorithms
Analyzing Algorithms
8/6/2019 Design and Analysis of Algorithms 1
15/29
. Analyzing AlgorithmsPredict resource utilization
...1 Running time (time complexity) focus of this course...2 Memory (space complexity)Running time: the number of primitive operations (e.g., addition,multiplication, comparisons) used to solve the problem
Depends on problem instance: often we find an upper bound:T(input size) and use asymptotic notations (big-Oh)
To make your life easier!
Growth rate much more important than constants in big-Oh.
Input size n: rigorous definition given latersorting: number of items to be sorted
graphs: number of vertices and edges
Lecture 1: Introduction Design and Analysis of Algorithms
Three Cases of Analysis: I
8/6/2019 Design and Analysis of Algorithms 1
16/29
. Three Cases of Analysis: IBest Case: An instance for a given size n that results in the fastestpossible running time.
.Example (Insertion sort).
.. .
.
.
A[1] A[2] A[3] A[n]
The number of comparisons needed is equal to
1 + 1 + 1 + + 1 n1
= n 1 = O(n)
key
Sorted Unsortedkey is compared to only the element right before it.
Lecture 1: Introduction Design and Analysis of Algorithms
Three Cases of Analysis: II
8/6/2019 Design and Analysis of Algorithms 1
17/29
. Three Cases of Analysis: IIWorst Case: An instance for a given size n that results in theslowest possible running time..Example (Insertion sort).
.. .
.
.
A[1] A[2] A[3] A[n]
The number of comparisons needed is equal to
1 + 2 + + (n 1) =n(n 1)
2= O(n2)
key
Sorted Unsortedkey is compared to everything element before it.
Lecture 1: Introduction Design and Analysis of Algorithms
Three Cases of Analysis: III
8/6/2019 Design and Analysis of Algorithms 1
18/29
. Three Cases of Analysis: IIIAverage Case: T running time averaged over all possible instancesfor the given size, assuming some probability distribution on theinstances..Example (Insertion sort)... .
.
.(n2
), assuming that each of the n! instances is equally likely(uniform distribution).
key
Sorted UnsortedOn average, key is compared to half of the elements before it.
Lecture 1: Introduction Design and Analysis of Algorithms
Three Cases of Analysis
8/6/2019 Design and Analysis of Algorithms 1
19/29
. Three Cases of Analysis
Best case: Clearly the worst
Worst case: Commonly used, will also be used in this course
Gives a running time guarantee no matter what the input isFair comparison among different algorithms
Average case: Used sometimes
Need to assume some distribution: real-world inputs areseldom uniformly random!Analysis is complicatedWill seldom be used in this course
Lecture 1: Introduction Design and Analysis of Algorithms
Outline
8/6/2019 Design and Analysis of Algorithms 1
20/29
. Outline
What is this course about?
What are algorithms?
What does it mean to analyze an algorithm?Comparing time complexity
Thoughts on algorithm design
Lecture 1: Introduction Design and Analysis of Algorithms
Example
8/6/2019 Design and Analysis of Algorithms 1
21/29
. Example
n
T(n)
Algorithm 1
Algorithm 2
Algorithm 2 is clearly superiorT(n) for Algorithm 1 is O(n3)
T(n) for Algorithm 2 is O(n2)
Since n3 grows much more rapidly, we expect Algorithm 1 totake much more time than Algorithm 2 when n increases
Lecture 1: Introduction Design and Analysis of Algorithms
Growth Rate
8/6/2019 Design and Analysis of Algorithms 1
22/29
. Growth RateGiven algorithms A and B with running times TA(n) and TB(n):
If limnTA(n)TB(n)
= 0, we say TA has a lower asymptotic growth
rate than TB. (Thus, A runs faster than B for large n.)
If limnTA(n)
TB(n) = , we say TA has a higher asymptotic growthrate than TB. (Thus, A runs slower than B for large n.)
If 0 < limnTA(n)
TB(n)
8/6/2019 Design and Analysis of Algorithms 1
23/29
. Big OhAsymptotic upper bound.Definition (Big-Oh)... .
.
.f(n) = O(g(n)): There exists constant c> 0 and n0 such thatf(n)c g(n) for n n0
When estimating the growth rate of T(n) using big-Oh:
ignore the low order terms
ignore the constant coefficient of the most significant term.Example... .
.
.Algorithm 1 runs in O(n3) time and Algorithm 2 runs in O(n2)
time.Example... .
..
It is also OK to say Algorithm 2 runs in O(n3) time, but we usuallywant to be as tight as possible
Lecture 1: Introduction Design and Analysis of Algorithms
8/6/2019 Design and Analysis of Algorithms 1
24/29
.Example.
.. .
.
.
n
2
/2 3n = O(n
2
)1 + 4n = O(n)
log10 n =log2 n
log2 10= O(log2 n) = O(log n)
sin n = O(1), 10 = O(1), 1010 = O(1)
ni=1 i2 n n2 = O(n3)n
i=1 i n n = O(n2)
210n is not O(2n)
log(n!) = log(n) + log(n 1) + + log 1 = O(n log n)
ni=1 1i = O(log n) (on board)
Lecture 1: Introduction Design and Analysis of Algorithms
Big Omega and Big Theta
8/6/2019 Design and Analysis of Algorithms 1
25/29
. g g gAsymptotic lower bound.Definition (big-Omega)... .
.
.f(n) = (g(n)): There exists constant c> 0 and n0 such thatf(n) c g(n) for n n0.
Asymptotic tight bound.Definition (big-Theta)... .
. .f(n) = (g(n)): f(n) = O(g(n)) and f(n) = (g(n)).Example.
.
. .
.
.
n2
/2 3n = (n2
)log(n!) = log(n) + log(n 1) + + log 1 log(n) + log(n 1) + + log(n/2) n/2 log(n/2)= n/2 (log n 1) = (n log n).
Lecture 1: Introduction Design and Analysis of Algorithms
Big Omega and Big Theta
8/6/2019 Design and Analysis of Algorithms 1
26/29
. g g g
.Example.
.. .
.
.
If Algorithm 1 runs in O(n3) time (we can show that any instanceof size n, the algorithm takes at most O(n3) time), and
Algorithm 1 runs in (n3
) time (there is an instance of size n onwhich the algorithm spends (n3) time),
then we say Algorithm 1 runs in (n3) time.
Usually (and in this course), it is sufficient to show only upperbounds (big-Oh).
Lecture 1: Introduction Design and Analysis of Algorithms
Outline
8/6/2019 Design and Analysis of Algorithms 1
27/29
.
What is this course about?
What are algorithms?
What does it mean to analyze an algorithm?Comparing time complexity
Thoughts on algorithm design
Lecture 1: Introduction Design and Analysis of Algorithms
Some Thoughts on Algorithm Design
8/6/2019 Design and Analysis of Algorithms 1
28/29
.
Algorithm Design, as taught in this class, is mainly aboutdesigning algorithms that have small big-Oh running times
As n gets larger and larger, O(n log n) algorithms will runfaster than O(n2) ones and O(n) algorithms will beatO(n log n) ones
Good algorithm design & analysis allows you to identify thehard parts of your problem and deal with them effectively
Too often, programmers try to solve problems using bruteforce techniques and end up with slow complicated code!
A few hours of abstract thought devoted to algorithm designoften results in faster, simpler, and more general solutions.
Lecture 1: Introduction Design and Analysis of Algorithms
Algorithm Tuning
8/6/2019 Design and Analysis of Algorithms 1
29/29
.
After algorithm design one can continue on to Algorithm tuning
concentrate on improving algorithms by cutting down on theconstants in the big O() bounds
needs a good understanding of both algorithm designprinciples and efficient use of data structures
In this course we will not go further into algorithm tuning
For a good introduction, see chapter 9 in ProgrammingPearls, 2nd ed by Jon Bentley
Lecture 1: Introduction Design and Analysis of Algorithms