+ All Categories
Home > Documents > COMP 3403 Algorithm Analysis , Part 1 Chapters 1...

COMP 3403 Algorithm Analysis , Part 1 Chapters 1...

Date post: 17-Jul-2018
Category:
Upload: dangtu
View: 223 times
Download: 0 times
Share this document with a friend
47
COMP 3403 — Algorithm Analysis Part 1 — Chapters 1 – 3 Jim Diamond CAR 409 Jodrey School of Computer Science Acadia University
Transcript
Page 1: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

COMP 3403 — Algorithm Analysis

Part 1 — Chapters 1 – 3

Jim Diamond

CAR 409

Jodrey School of Computer Science

Acadia University

Page 2: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 0

Preliminaries

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 3: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 0 1

Course Objectives

We shall cover

• review of asymptotic notation

• the goal of algorithm analysis

• recurrence equations

• order statistics, sorting and selection

• general algorithm design techniques

• graph theoretic algorithms

• string matching

• introduction to NP completeness

• other topics may be covered, time permitting

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 4: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 0 2

General Information

• Read course outline at

http://cs.acadiau.ca/~jdiamond/comp3403/comp3403.pdf

• Getting help

– ask questions in class!

– my office hours (CAR 409): MWF 8:30–930, other times TBA

and/or by appointment

– these are subject to change; I’ll let you know if they do

– if you need to see me at other times, e-mail or phone me and

we can set up a meeting time

– the teaching assistant contact hours shall be announced RSN

To pretend to know when you do not know is a disease.

To know that you do not know is the best.

— Lao Tzu

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 5: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 0 3

Reference Materials

• Introduction to the Design & Analysis of Algorithms (3rd edition) by

Anany Levitin (Addison Wesley)

• Introduction to Computer Algorithms (3rd edition) by Cormen,

Leiserson, Rivest and Stein

• The Design and Analysis of Computer Algorithms by Alfred V. Aho,

John E. Hopcroft and Jeffrey D. Ullman

• Fundamentals of Algorithmics by Gilles Brassard and Paul Bratley

• Computers and Intractability: A Guide to the Theory of

NP-Completeness by Michael R. Garey and David S. Johnson

• Computer Algorithms: Introduction to Design & Analysis (3rd edition)

by Sara Baase and Allen Van Gelder

• There are lots of reference books in the library

– reviewing other explanations of the same material may aid your

understanding

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 6: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 0 4

Course Communications

• I will not use ACORN for assignments (or anything else, in all likelihood)

– check (and read) your e-mail frequently

– look in

http://cs.acadiau.ca/~jdiamond/comp3403/assignments/

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 7: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1

Introduction

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 8: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 5

Algorithm Analysis Preliminaries: 1

• What is the difference between a computer science degree and a

programming course at a technical school?

• Sample problem: running a dating service:

– match each of N guys to (one each!) of N girls

– you have a compatibility function C(mi, fj), 1 ≤ i, j ≤ N– business plan: find a 1 – 1, onto mapping P : mi → fj which

maximizes total happiness H:

H =N∑i=1

C(i, P (i))

– <board example>

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 9: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 6

Algorithm Analysis Preliminaries: 2

• Q: how can we compute P ( )?

• A1: try all possible pairings (a BFI (q.v.) technique)

• Q’: how many pairings for N pairs?

• A’: N ! !!!

• Q”: how big is that?

• A”: n! ≈ nne−n√

2πn (Stirling’s formula)

– more specifically

nne−n√

2πn e1/(12n+1) < n! < nne−n√

2πn e1/12n

• <gnuplot examples>

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 10: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 7

Problem Categorization

• Roughly speaking, there are three types of problems:

– problems that we can solve “efficiently”

– problems that are not solvable at all

– problems that we can solve, but not “efficiently”

– for some, no efficient solution can possibly exist (provably!)

– for others, an efficient solution might exist, but no one knows

such a solution or, they’re keeping it a secret

• Dating example:

– Where does it fit? Is n! considered to be efficient?

• Does an inefficient algorithm for a problem mean the problem is “hard”?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 11: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 8

“Problem” vs. “Instance”

• Problem: a general description of something to be solved

– e.g., “Given n numbers, find the smallest.”

• Instance: a specific case of a problem

– e.g., “Find the smallest number in 2, 3, 5, 8, 13”

• Note: problems which are unsolvable or not efficiently solvable in

general may have easily solvable instances

– e.g., the halting problem

– e.g., find a Hamiltonian cycle (HC) in a connected graph– is there a HC here? – how about here?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 12: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 9

Algorithm Analysis

• We can analyze algorithms for

– space complexity:

– in a conventional computer, how much memory is used by a

given algorithm

– time complexity:

– how much CPU time is used by a given algorithm

• Issue 1:

for a given algorithm, solving a “bigger” instance usually takes longer

than solving a “smaller” instance

– how should we measure the size of an instance?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 13: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 10

Algorithm Analysis: 2

• Issue 2:

implementing an algorithm & timing it on various instances isn’t

necessarily a good way to analyze algorithms

– (why not? infer function from plot? bad implementation?

cache effects? timing inaccuracy?)

– we could count the operations done to solve an instance

– all operations?!? Usually not necessary.

– often, the “total” work is proportional to the amount of work

done by some selected operations

– examples:

– sorting: (people usually) just count the number of comparisons

of the data items

– matrix operations: just count the number of (scalar)

multiplications & maybe the additions

(Q (GEQ?): what is the relationship between the additions and

the multiplications in a matrix multiplication operation?)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 14: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 11

Algorithm Analysis: 3

• Issue 3:

Do all operations take the same (or similar) amount of time?

– Consider operations on integers:

– if all operands are “small”, we can assume all operations take

the same amount of time

– if operands can become arbitrarily large, then we can’t assume

that all operations take the same amount of time

• We consider two models of arithmetic operations:

Uniform model: all operations take the same amount of time

– this model is good for most purposes

Logarithmic model: operation times vary with the size of the operands

– this model is appropriate when very large numbers are used

– got any examples (of algorithms which use very large numbers)?

– why is the name “logarithmic”?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 15: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 1 12

Algorithm Analysis: 4

• Issue 4:

Should we consider the best case, worst case or average case analysis?

– best case analysis: (usually) not of much interest

– worst case analysis: usually easier to do than average case analysis

– average case may be of most interest

– e.g., sorting with quicksort

– e.g., linear programming using the simplex method

• Issue 5:

Are we interested in algorithm complexity or problem complexity?

– Algorithm complexity: what is the complexity of (a specific

implementation of?!) a given algorithm?

– worst and/or average case

– Problem complexity: what is the complexity of the best algorithm

that solves the problem? Best average case or best worst case.

– solving this requires you to find lower bounds

– usually difficult to do! How can you know you have the best?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 16: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2

Fundamentals of the Analysis of Algorithm Efficiency

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 17: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 13

Goal of Algorithm Analysis

• Suppose we can measure the size of a problem instance; call it N

• Goal: given an algorithm A, find functions T (n) & S(n) such that the

amount of time and space needed by A to solve an instance of size N

are T (N) and S(N), respectively

• Note: if the time or space used varies for different instances of size N ,

we may want to consider the best, average and/or worst case versions

of T (n) and S(n)

• Note: in algorithm analysis T (n) and S(n) are ≥ 0 ∀n (n ≥ 0)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 18: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 14

Algorithm Analysis Details

• Normally we are interested in the “big picture”

• E.g., suppose we have analyzed an algorithm and discovered that on an

input of size n,

T (n) = k2n2 + k1n+ k0

• In rare cases, we might be concerned with the k1n and k0 terms, but

normally we are not

• The specific values of ki might vary from one computer architecture to

another — this lessens their relevance

• In most cases we are not even concerned with k2:

T (n) will be a quadratic function on any computer

• <Sample plots>

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 19: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 15

Algorithm Analysis: Asymptotic Behaviour

• To allow us to concentrate on the highest-order term(s), we introduce

some definitions

(1) “Big O”

g(n) ∈ O(f(n)) when ∃ k > 0, n0 ∈ IN such that

|g(n)| < k · |f(n)| ∀n > n0

• Note: in this course we only consider space and time functions which are

– defined for n ≥ 0 (so we can drop “ | | ”)

• We say “g(n) is bounded above by f(n)”

• NOTE: traditionally, people wrote g(n) = O(f(n)), not g(n) ∈ O(f(n))

– arguably, using “∈” is superior notation

– in this context, you can think of O(f(n)) as the set of all

functions bounded above (loosely speaking) by f(n)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 20: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 16

Algorithm Analysis: Asymptotic Behaviour: 2

(2) “Little O”

g(n) ∈ o(f(n)) when limn→∞

g(n)

f(n)= 0

(3) “Theta”

g(n) ∈ Θ(f(n)) when ∃ k1, k2 > 0, n0 ∈ IN such that

k1 · f(n) < g(n) < k2 · f(n) ∀n > n0–

(4) “Big omega”

g(n) ∈ Ω(f(n)) if f(n) ∈ O(g(n))

(5) “Little omega”

g(n) ∈ ω(f(n)) if f(n) ∈ o(g(n))

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 21: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 17

Examples of Asymptotic Notation

• 33n2 + 5n+ 10100 ∈ O(n2)

• n2 ∈ O(n3)

• n2 ∈ O(n2)

• n2 ∈ o(n3)

• n3 ∈ Ω(n2)

• n4 6∈ o(n4) Are all

• n3 ∈ Ω(n3) of these

• n3 ∈ ω(n2) correct?

• n2 ∈ Θ(n2)

• n2 6∈ Θ(n3)

• n4 6∈ Θ(n3)

• 6.02 · 1023 ∈ O(1)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 22: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 18

Upper and Lower Bounds

• Defn: An upper bound for the complexity of a problem is the

complexity of some algorithm which solves the problem

• E.g., an upper bound for multiplication of two n× n matrices is O(n3)

we say “matrix multiplication is O(n3)”

or “matrix multiplication is in O(n3)”

• Q: can we say “matrix multiplication is O(n4)” ?

A: yes, but this is sloppier than necessary

• Defn: A lower bound for the complexity of a problem is a minimum

amount of time (or space) necessary to solve the problem

• E.g., a lower bound for the complexity of matrix multiplication is Ω(n2).

(Why?)

Sloppy lower bound: Ω(n).

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 23: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 19

Objectives of Algorithm Analysis

• (1) find as small an UB as possible

(2) find as large a LB as possible

• E.g., Problem: given m integers n1, n2, . . . , nm, find the maximum.

Just consider the number of comparisons.

– LB: trickier. (See B&vG page 40 for another discussion.)

Define a graph G = (V,E) in which there is a vertex vnifor i = 1..m

Initially E = Ø

Whenever a given algorithm compares 2 numbers, we add a

corresponding edge to E

As long as the graph has parts that aren’t connected, we can’t

know the maximum. (Why?)

To make the graph connected we need at least m− 1 edges. (Why?)

... we need at least m− 1 comparisons to find the max

– Since we have an alg whose complexity matches our LB, we have an

optimal algorithm

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 24: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 20

Analyzing Non-Recursive Algorithms

• Typically fairly straightforward; here is an informal, quick overview. . .

see the textbook for details

(1) Decide on the which operation is the most significant

– e.g., comparisons in a sorting algorithm usually

(2) Find the (nested) loop where the innermost operations are done

the most frequently

(3) Count the number of operations done inside this innermost loop

• Caution: you may have an algorithm with a triply-nested loop which

does less work than another doubly-nested loop

• Caution: if you call a function from inside a loop, you have to take into

account whether that function can be performed in constant time, linear

(i.e., O(n)) time, O(n2) time, and so on

– GEQ: in this case, do you add the function complexity to the loop

complexity, or do you multiply the function complexity by the loop

complexity?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 25: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 21

Analyzing Recursive Algorithms

• Both more difficult and more interesting than non-recursive algs!

• Consider the following function:

fact(n)

if n < 2return 1

elsereturn n * fact(n-1);

• Note: we have

– a recursive call which solves a smaller case

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 26: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 22

Analyzing fact()

• Pictorial representation:

fact(n) n!↓ ↑ fact(n)

fact(n-1) n-1! ↓ ↑ if n < 2... ... return 1↓ ↑ else

fact(2) = 2 return n * fact(n-1);↓ ↑

fact(1) = 1

• In this case, the size of the problem is the magnitude of the argument

• Count the number of multiplications for time complexity:

T (0) = T (1) = 0

T (n) = 1 + T (n− 1) for n ≥ 2

• This is a recurrence equation; solving recurrence equations is

(somewhat) similar to solving differential equations

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 27: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 23

Solving a Simple Recurrence Equation

• Equation to solve:

T (0) = T (1) = 0 (α)

T (n) = 1 + T (n− 1) (β)

• Solve by inspection (i.e., start by guessing a solution):

T (1) = 0

T (2) = 1 + T (1) = 1 + 0 = 1

T (3) = 1 + T (2) = 1 + 1 = 2

T (4) = 1 + T (3) = 1 + 2 = 3...

my guess: T (n) = n− 1 for n ≥ 1 (H)

• I must prove that my guess is correct

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 28: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 24

Proving Guess (H)

• Proof by induction is appropriate here

recall these formulae:

T (0) = T (1) = 0 (α)

T (n) = 1 + T (n− 1) (β)

T (n) = n− 1 for n ≥ 1 (H)

• (1) Induction is on n for n ≥ 1

(2) Base case: substitute n = 1 into (H):

T (1) = 1− 1 = 0, which is true according to (α)

(3) Induction hypothesis: assume (H) is correct for n− 1 (n > 1):

i.e., T (n− 1) = n− 2

(4) Inductive case: substitute the IH into (β):

T (n) = 1 + T (n− 1) (β)

= 1 + n− 2

= n− 1

• Therefore† (H) is true for n ≥ 1

† by the principle of mathematical induction

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 29: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 25

First Differences

• Sometimes we can analyze recurrence equations by looking at “first

differences”:

T (1) = 0

∆ = 1

T (2) = 1

∆ = 1

T (3) = 2

∆ = 1

T (4) = 3

• Note that the difference is constant

– or, what sort of function has a constant for its derivative?

– or, what is the integral of a constant?∫c dx = ?

• Conclude that T (x) = c · x+ k

– solve for c and k: c = 1, k = −1 two unknowns: choose two x’s to give two eqns

– so T (n) = n− 1

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 30: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 26

A More Complicated Recurrence Equation

Fib(n) /* Defined only for n >= 0 */ /* Sequence is 0, 1, 1, 2, 3, 5, 8, ... */

if n < 2 return nelse return Fib(n - 1) + Fib(n - 2)

• Analysis: count number of recursive calls (why?)

i.e., T (n) is the number of recursive calls when computing Fib(n)

T (0) = T (1) = 0

T (n) = 2 + T (n− 1) + T (n− 2) for n > 1

• Solve (guess) by inspection?

T (2) = 2 + T (1) + T (0) = 2 + 0 + 0 = 2

T (3) = 2 + T (2) + T (1) = 2 + 2 + 0 = 4

T (4) = 2 + T (3) + T (2) = 2 + 4 + 2 = 8

T (5) = 2 + T (4) + T (3) = 2 + 8 + 4 = 14

T (6) = 2 + T (5) + T (4) = 2 + 14 + 8 = 24

T (7) = 2 + T (6) + T (5) = 2 + 24 + 14 = 40

• Not an obvious pattern. . .

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 31: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 27

First, Second and Third Differences

• Differences of 0, 0, 2, 4, 8, 14, 24, 40, 66, 108, 176, . . .

1st difference 2nd difference 3rd difference

0

2 2

2 0 −2

4 2 2

6 2 0

10 4 2

16 6 2

26 10 4

42 16 6

68 26 10

• Ignoring the first couple of entries, these sets of “discrete derivatives”

give the same values

• What would that say about the efficiency of this algorithm?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 32: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 28

Solution of Fibonacci Recurrence Equation

• The solution is not trivial. . . (see textbook for details)

T (n) = −2−4√

5(−2

1−√5

)n5(1−

√5)

+4√

5(−2

1+√5

)n5(1 +

√5)

• Note: if F (n) is the n-th Fibonacci number,

then if F (1) = F (2) = 1,

F (n) =

√5

5

[(1

2+

√5

2

)n

−(

1

2−√

5

2

)n ]

or if F (0) = F (1) = 1

F (n) =

(√5

10+

1

2

)[(1

2+

√5

2

)n

−(

1

2−√

5

2

)n ]

• (This is just different “boundary conditions”, for those of you who know

about differential equations)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 33: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 29

Fibonacci Numbers in Various Places

• Fibonacci numbers turn up in all sorts of places in the “real world”

• Here’s one:

• The right panel instead applies the Perrin sequence.

See http://mathworld.wolfram.com/FibonacciNumber.html

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 34: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 30

Yet Another Recurrence Equation Example

• Suppose we have a recurrence equation for T (n) and we calculate a few

values as follows:T (n) 1st diffs 2nd diffs

T (1) = 103

T (2) = 135

2

T (3) = 187

2

T (4) = 259

2

T (5) = 34

• The fact that the 2nd diffs are constant suggests that T (n) is a

quadratic function

– T (1) = a+ b+ c

T (2) = 4a+ 2b+ c

T (3) = 9a+ 3b+ c

– Solve this system of linear equations: a = 1, b = 0, c = 9

... T (n) = n2 + 9

• Need to prove that this guess is a solution to the original R.E.

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 35: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 31

R.E.s for “Divide and Conquer” Algorithms

• Suppose you solve a problem of size n by

(a) breaking it into 2 problems of size n/2,

(b) solving these 2 problems recursively, and

(c) combining the 2 solutions in O(n) time

• Specifically, T (n) = 2T (n/2) + bn for n > 1

T (1) = b

• Look at the first few 2k values:

T (21) = T (2) = 2T (1) + 2b = 2b+ 2b = 4b

T (22) = T (4) = 2T (2) + 4b = 8b+ 4b = 12b

T (23) = T (8) = 2T (4) + 8b = 24b+ 8b = 32b

Guess: T (n) = bn(lg n+ 1)

(or we could say T (2k) = b2k(k + 1) )

• . . . continued next slide

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 36: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 32

R.E. for “Divide and Conquer” Algorithms: 2

• Prove that T (n) = bn(lg n+ 1) is a solution to previous slide’s R.E.

• Base case (n = 1):

T (1) = b · 1 · (lg 1 + 1)

= b · (0 + 1)

= b–

• Induction step: assume the guess for T (n) is true for all k < n

T (n) = 2T

(n

2

)+ bn

= 2

[bn

2

(lgn

2+ 1

)]+ bn (use I.H.)

= bn(lg n− 1 + 1) + bn

= bn(lg n+ 1) QED

• What if n 6= 2k? See Appendix B of Levitin (“smoothness rule”).

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 37: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 33

Generalized Version for Divide and Conquer

• Suppose we solve a problem of size n > 1 by splitting it into a problems,

each of size n/c, and using bn additional operations to form the

complete answer (a, b and c positive constants)

i.e., T (n) =

b n = 1

aT (nc ) + bn n = ck, n > 1

• The solution to this recurrence equation is

T (n) ∈

Θ(n) a < c

Θ(n lg n) a = c

Θ(nlogc a) a > c

• (Note: this is sometimes called the “master equation”)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 38: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 34

Proof of “Master” Equation

• First, expand the recurrence equation to see what’s happening:

T (n) = bn+ aT

(n

c

)= bn+ a

(bn

c+ aT

(n

c2

))= · · ·

= bn+ abn

c+ a2

bn

c2+ a3

bn

c3+ · · ·

= bn

(1 +

a

c+a2

c2+a3

c3+ · · ·

)• Guess:

T (n) = bn

logc n∑i=0

(a

c

)i

(∗)

• Note: eventually, we prefer a closed-form solution rather than a

summation; we could either “solve” the sum (three cases!) and then

prove the solutions are correct, or we could prove the sum is correct and

then “solve” the sum

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 39: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 35

Proof of “Master” Equation: 2

• Guess (repeated from the previous slide):

T (n) = bn

logc n∑i=0

(a

c

)i

(∗)

• First: see if (∗) satisfies the base case:

T (1) = b · 10∑

i=0

(a

c

)i

= b–

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 40: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 36

Proof of “Master” Equation: 3

• Guess (repeated): T (n) = bn

logc n∑i=0

(a

c

)i

(∗)

• Induction step: assume (∗) is true for all k < n (in particular, n/c)

T (n) = aT

(n

c

)+ bn

= a

bnc

(logc n)−1∑i=0

(a

c

)i+ bn by I.H.

= bn

1 +a

c

(logc n)−1∑i=0

(a

c

)i

= bn

1 +

logc n∑i=1

(a

c

)i

= bn

logc n∑i=0

(a

c

)i

= (∗) QED/2

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 41: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 37

Proof of “Master” Equation: 4

• Still need to evaluate the sum; three cases

– recall that if a 6= 1,k∑

i=0

ai =1− ak+1

1− a=ak+1 − 1

a− 1

• Case 1: a < c T (n) = bn

(ac

)1+logc n − 1ac − 1

= bn1−

(ac

)1+logc n

1− ac

< bn1

1− ac

since ac < 1

– Exercise (GEQ?): show T (n) ∈ Θ(n)

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 42: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 2 38

Proof of “Master” Equation: 5

• Case 2: a = c T (n) = bn

logc n∑i=0

1

= bn(1 + logc n)

∈ Θ(n lg n) (Why?)

• Case 3: a > c T (n) = bn

(ac

)1+logc n − 1ac − 1

=bnc

a− c

((a

c

)1+logc n

− 1

)

=bnc

a− c

(a

c

)nlogc

ac − bnc

a− calog b = blog a

=bnc

a− c

(a

c

)n(logc a)−1 − bnc

a− c

=ab

a− cnlogc a − bnc

a− c∈ Θ

(nlogc a

)QED

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 43: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 3

Brute Force

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 44: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 3 39

Brute Force Sorting

• Consider selection sort of n numbers (bubble sort very similar):

– find the smallest number in the list (n− 1 comparisons)

– swap it with the first number in the list

• Conceptually simple (but uses more time than necessary)

T (n) =n−1∑i=1

n∑j=i+1

1 =n−1∑i=1

(n− (i+ 1) + 1

)=

n−1∑i=1

(n− i) =n−1∑i=1

n−n−1∑i=1

i

∈ Θ(n2)

• Note that in selection sort the number of swaps is bounded above by n,

whereas in bubble sort the number of swaps is bounded above by the

number of comparisons

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 45: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 3 40

Brute Force String Matching

• Problem: find a string s in a text T , where m = |s| and n = |T |

• Brute force approach:

– compare s to T , starting at T [0]

– if s == T [0 ..m− 1] return success

– it not found before tail of T runs out, return failure

• Q: what is the time complexity of this algorithm?

– in worst case, must do m · (n−m+ 1) = Θ(mn) comparisons

• Worst case occurs if strings don’t match, but only at the last char of s

– in general, a non-match will occur earlier in s, or a matching

substring of T might be found

• Surprising (?) result: on random texts, this algorithm runs in

Θ(m+ n) = Θ(n) time! GEQ: why is Θ(m + n) = Θ(n)?

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 46: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 3 41

Closest-Pair Problem: Brute Force

• Problem: given n points(xi, yi)

in the plane, find the two closest

points

• BFI solution #1:

(a) compute the (Euclidean) distance between each pair of points:

di,j =√

(xi − xj)2 + (yi − yj)2

(b) find min1≤i6=j≤n

di,j

– problem: just because there is a sqrt() function in standard

libraries doesn’t mean it is fast

• BFI solution #2:

(a) make the amazing observation that

√x <√y ⇐⇒ x < y

(b) optimize BFI #1 by comparing the collection of

di,j = (xi − xj)2 + (yi − yj)2 values

Jim Diamond, Jodrey School of Computer Science, Acadia University

Page 47: COMP 3403 Algorithm Analysis , Part 1 Chapters 1 3socs.acadiau.ca/~jdiamond/comp3403/notes/slides-pt1-bw.pdf · Chapter 0 3 Reference Materials Introduction to the Design & Analysis

Chapter 3 42

Finding the Convex Hull: Brute Force

• A set S of points in the plane (or n-dimensional space) is said to be

convex if, for any two points p1 and p2 in S, all the points in the line

segment from p1 to p2 are also in S

• The convex hull of a set of points S in the plane (or n-dimensional

space) is the smallest convex set containing S

• Problem: given a set of points S, find the convex hull of S

• Example: S = pins sticking out of a board, find the convex hull by

stretching a rubber band around all the points and letting it go

• BFI solution #1:

– for each pair of points, compute the line joining them and see if all

points fall on one side of the line or the other

– watch out for other points on that line!

• BFI solution #2:

– find the point with largest y coordinate (break ties by choosing min

x coordinate)

– walk around the set of points in a clockwise direction

Jim Diamond, Jodrey School of Computer Science, Acadia University


Recommended