CSC2105: Algorithms Mashiour Rahman [email protected] American International University Bangladesh.

CSC2105: AlgorithmsCSC2105: Algorithms

Mashiour RahmanMashiour [email protected]@aiub.edu

American International University BangladeshAmerican International University Bangladesh

CSC2105CSC2105 Mashiour RahmanMashiour Rahman 22

LiteratureLiterature

Introduction to Algorithms, Second Edition, Introduction to Algorithms, Second Edition, Thomas H. Cormen, Charle E. Leiserson, Thomas H. Cormen, Charle E. Leiserson, Ronald L. Rivest, Clifford Stein (CLRS). Ronald L. Rivest, Clifford Stein (CLRS).

Fundamental of Computer Algorithms, Ellis Fundamental of Computer Algorithms, Ellis Horowitz, Sartaj Sahni, Sanguthevar Horowitz, Sartaj Sahni, Sanguthevar Rajasekaran (HSR).Rajasekaran (HSR).

Helpful link for Problem Solving : Helpful link for Problem Solving : http://acm.uva.es/problemset/http://acm.uva.es/problemset/


The Goals of this CourseThe Goals of this Course

The main things we will learn in this course:The main things we will learn in this course: To To thinkthink algorithmically algorithmically and get the spirit of how and get the spirit of how

algorithms are designed.algorithms are designed. To get to know a To get to know a toolboxtoolbox of of classical classical algorithms.algorithms. To learn a number of algorithm design To learn a number of algorithm design techniquestechniques

(such as divide-and-conquer).(such as divide-and-conquer). To reason (in a precise and formal way) about the To reason (in a precise and formal way) about the

efficiencyefficiency and the and the correctnesscorrectness of algorithms. of algorithms.


GeneralGeneral

Algorithms are first solved on paper and later Algorithms are first solved on paper and later keyed in on the computer.keyed in on the computer.

The most important thing is to be The most important thing is to be simple simple andand preciseprecise..

During lectures:During lectures: Interaction is welcome; ask questions.Interaction is welcome; ask questions. Additional explanations and examples if desired.Additional explanations and examples if desired. Speed up/slow down the progress.Speed up/slow down the progress.


PrerequisitesPrerequisites

Introduction to programmingIntroduction to programming Data types, operationsData types, operations Conditional statementsConditional statements LoopsLoops Procedures and functionsProcedures and functions C/C++/JavaC/C++/Java

Computer lab (edit, compile, execute)Computer lab (edit, compile, execute)


HistoryHistory

Name: Persian mathematician Mohammed al-Name: Persian mathematician Mohammed al-Khowarizmi, in Latin became Algorismus.Khowarizmi, in Latin became Algorismus.

First algorithm: Euclidean Algorithm, greatest First algorithm: Euclidean Algorithm, greatest common divisor, 400-300 B.C.common divisor, 400-300 B.C.

1919thth century – Charles Babbage, Ada Lovelace. century – Charles Babbage, Ada Lovelace. 2020thth century – Alan Turing, Alonzo Church, century – Alan Turing, Alonzo Church,

John von Neumann.John von Neumann.


Data Structures and AlgorithmsData Structures and Algorithms

Data structureData structure Organization of data to solve the problem at hand.Organization of data to solve the problem at hand.

AlgorithmAlgorithm Outline, the essence of a computational procedure, Outline, the essence of a computational procedure,

step-by-step instructions.step-by-step instructions. ProgramProgram

Implementation of an algorithm in some Implementation of an algorithm in some programming language. programming language.


Overall PictureOverall Picture

Using a computer to Using a computer to help solve problems.help solve problems. Precisely specify the Precisely specify the

problem.problem. Designing programsDesigning programs

architecture architecture algorithmsalgorithms

Writing programsWriting programs Verifying (testing) Verifying (testing)

programsprograms

Data Structure and Algorithm Design Goals

Implementation Goals

Correctness Efficiency

Robustness

Adaptability

Reusability


Overall Picture/2Overall Picture/2

This course is This course is notnot about: about: Programming languagesProgramming languages Computer architectureComputer architecture Software architectureSoftware architecture Software design and implementation principlesSoftware design and implementation principles

Issues concerning small and large scale programming.Issues concerning small and large scale programming. We will only touch upon the theory of We will only touch upon the theory of

complexity and computability.complexity and computability.


Algorithmic problemAlgorithmic problem

Infinite number of input Infinite number of input instancesinstances satisfying the satisfying the specification. For example:specification. For example:

A sorted, non-decreasing sequence of natural numbers. A sorted, non-decreasing sequence of natural numbers. The sequence is of non-zero, finite length:The sequence is of non-zero, finite length:

1, 20, 908, 909, 100000, 1000000000.1, 20, 908, 909, 100000, 1000000000. 3. 3.

Specification of input

?Specification of output as a function of input


Algorithmic SolutionAlgorithmic Solution

Algorithm describes actions on the input instanceAlgorithm describes actions on the input instance.. There may be many correct algorithms for the There may be many correct algorithms for the

same algorithmic problemsame algorithmic problem..

Input instance, adhering to the specification

Algorithm Output related to the input as required


Definition of an AlgorithmDefinition of an Algorithm

An An algorithmalgorithm is a sequence of is a sequence of unambiguousunambiguous instructions for solving a problem, i.e., for instructions for solving a problem, i.e., for obtaining a obtaining a required outputrequired output for any for any legitimate legitimate inputinput in a finite amount of time. in a finite amount of time.

Properties:Properties: PrecisionPrecision DeterminismDeterminism FinitenessFiniteness

EfficiencyEfficiency CorrectnessCorrectness GeneralityGenerality


How to Develop an AlgorithmHow to Develop an Algorithm

Precisely definePrecisely define the problem. Precisely specify the the problem. Precisely specify the input and output. Consider all cases. input and output. Consider all cases.

Come up with a Come up with a simple plansimple plan to solve the problem at to solve the problem at hand.hand. The plan is language independent.The plan is language independent. The precise problem specification influences the plan.The precise problem specification influences the plan.

Turn the plan into an implementationTurn the plan into an implementation The problem representation (data structure) influences the The problem representation (data structure) influences the

implementation.implementation.


Preconditions, PostconditionsPreconditions, Postconditions

It is important to specify the preconditions and It is important to specify the preconditions and the postconditions of algorithms:the postconditions of algorithms: INPUTINPUT: precise specifications of what the : precise specifications of what the

algorithm gets as an input.algorithm gets as an input. OUTPUTOUTPUT: precise specifications of what the : precise specifications of what the

algorithm produces as an output, and how this algorithm produces as an output, and how this relates to the input. The handling of special cases relates to the input. The handling of special cases of the input should be described.of the input should be described.


Analysis of AlgorithmsAnalysis of Algorithms

Efficiency:Efficiency: Running timeRunning time Space usedSpace used

Efficiency as a function of the input size:Efficiency as a function of the input size: Number of data elements (numbers, points).Number of data elements (numbers, points). The number of bits of an input number .The number of bits of an input number .


The RAM modelThe RAM model

It is important to choose the level of detail.It is important to choose the level of detail. The RAM model:The RAM model:

Instructions (each taking constant time), we Instructions (each taking constant time), we usually choose one type of instruction as a usually choose one type of instruction as a characteristiccharacteristic operation that is counted: operation that is counted:

Arithmetic (add, subtract, multiply, etc.)Arithmetic (add, subtract, multiply, etc.) Data movement (assign)Data movement (assign) Control flow (branch, subroutine call, return)Control flow (branch, subroutine call, return) ComparisonComparison

Data types – integers, characters, and floats Data types – integers, characters, and floats


Analysis of Insertion SortAnalysis of Insertion Sort

for j 2 to n do keyA[j] // Insert A[j] into A[1..j-1] ij-1 while i>0 and A[i]>key do A[i+1]A[i] i-- A[i+1]:=key

costc1

c2

0 c3

c4

c5

c6

c7

timesnn-1n-1n-1

n-1

2

n

jjt

2( 1)

n

jjt

2( 1)

n

jjt

Time to compute the Time to compute the running timerunning time as a function of as a function of the the input size input size (exact analysis).(exact analysis).


Analysis of Insertion Sort/2Analysis of Insertion Sort/2

The running time of an algorithm is the sum of The running time of an algorithm is the sum of the running times of each state-ment.the running times of each state-ment.

A statement with cost c that is executed n A statement with cost c that is executed n times contributes c*n to the running time.times contributes c*n to the running time.

The total running time T(n) of insertion sort isThe total running time T(n) of insertion sort is T(n) = c1*n + c2(n-1) + c3(n-1) + c4 +T(n) = c1*n + c2(n-1) + c3(n-1) + c4 + c5 + c6 + c7(n-1)c5 + c6 + c7(n-1)

2

n

jjt

2( 1)

n

jjt

2

( 1)n

jjt


Analysis of Insertion Sort/3Analysis of Insertion Sort/3

Often the performance depends on the details Often the performance depends on the details of the input (not only the length n).of the input (not only the length n).

This is modeled by tThis is modeled by tj.j.

In the case of insertion sort the time tIn the case of insertion sort the time t jj depends depends

on the original sorting of the input array.on the original sorting of the input array.


Performance AnalysisPerformance Analysis

Often it is sufficient to count the number of iterations Often it is sufficient to count the number of iterations of the core (innermost) part.of the core (innermost) part. No distinction between comparisons, assignments, etc (that No distinction between comparisons, assignments, etc (that

means roughly the same cost for all of them).means roughly the same cost for all of them). Gives precise enough results.Gives precise enough results.

In some cases the cost of selected operations In some cases the cost of selected operations dominates all other costs.dominates all other costs. Disk I/O versus RAM operations.Disk I/O versus RAM operations. Database systems.Database systems.


Best/Worst/Average CaseBest/Worst/Average Case

Analyzing insertion sort’sAnalyzing insertion sort’s Best caseBest case: elements already sorted, : elements already sorted, ttjj=1, =1, running running

time = n-1, i.e., time = n-1, i.e., linearlinear time. time. Worst caseWorst case: elements are sorted in inverse order, : elements are sorted in inverse order,

ttjj=j-1=j-1, running time = , running time =

(n(n22-n)/2, -n)/2, i.e.,i.e., quadratic quadratic time. time. Average caseAverage case: : ttjj=j/2, =j/2, running time = running time =

(n(n22+n-2)/4, +n-2)/4, i.e.,i.e., quadratic quadratic time. time.

2( 1)

n

jjt


Best/Worst/Average Case/3Best/Worst/Average Case/3

For inputs of all sizes:For inputs of all sizes:

1n

2n

3n

4n

5n

6n

Input instance size

Run

ning

tim

e

1 2 3 4 5 6 7 8 9 10 11 12 …..

best-case

average-case

worst-case


Best/Worst/Average Case/4Best/Worst/Average Case/4

Worst case Worst case is usually used:is usually used: It is an upper-bound.It is an upper-bound. In certain application domains (e.g., air traffic In certain application domains (e.g., air traffic

control, surgery) knowing the control, surgery) knowing the worst-caseworst-case time time complexity is of crucial importance.complexity is of crucial importance.

For some algorithms For some algorithms worst caseworst case occurs fairly often- occurs fairly often- The The average caseaverage case is often as bad as the is often as bad as the worst caseworst case.. Finding the Finding the average case average case can be very difficult.can be very difficult.


Analysis of Linear SearchAnalysis of Linear Search

INPUTINPUT: A[1..n] – a sorted array of integers, : A[1..n] – a sorted array of integers, q – q – an integer.an integer. OUTPUTOUTPUT: an index : an index j j such that A[such that A[jj] = ] = q. NILq. NIL if if j j (1(1jjnn): A[): A[jj] ] qq j := 1j := 1while while j j nn and and A[j] A[j] q q dodo j++j++if if j j nn thenthen returnreturn j jelse return else return NILNIL

INPUTINPUT: A[1..n] – a sorted array of integers, : A[1..n] – a sorted array of integers, q – q – an integer.an integer. OUTPUTOUTPUT: an index : an index j j such that A[such that A[jj] = ] = q. NILq. NIL if if j j (1(1jjnn): A[): A[jj] ] qq j := 1j := 1while while j j nn and and A[j] A[j] q q dodo j++j++if if j j nn thenthen returnreturn j jelse return else return NILNIL

Worst case running time: n Average case running time: n/2 Best case running time: 0


Binary SearchBinary Search

INPUTINPUT: A[1..n] – a sorted array of (increasing) integers, : A[1..n] – a sorted array of (increasing) integers, q – q – an integer.an integer. OUTPUTOUTPUT: an index : an index j j such that A[such that A[jj] = ] = q. NILq. NIL, if , if j j (1(1jjnn): A[): A[jj] ] qq l := 1; r := l := 1; r := nndodo

m := m := (l+r)/2(l+r)/2if if A[m] = q A[m] = q thenthen return return mmelse if else if A[m] > q A[m] > q then then r := m-1r := m-1else else l := m+1 l := m+1

while while l <= rl <= rreturn return NILNIL

INPUTINPUT: A[1..n] – a sorted array of (increasing) integers, : A[1..n] – a sorted array of (increasing) integers, q – q – an integer.an integer. OUTPUTOUTPUT: an index : an index j j such that A[such that A[jj] = ] = q. NILq. NIL, if , if j j (1(1jjnn): A[): A[jj] ] qq l := 1; r := l := 1; r := nndodo

m := m := (l+r)/2(l+r)/2if if A[m] = q A[m] = q thenthen return return mmelse if else if A[m] > q A[m] > q then then r := m-1r := m-1else else l := m+1 l := m+1

while while l <= rl <= rreturn return NILNIL

Idea: Have a left and right bound. Elements to the right of r are bigger than the search element. Equivalent for l.

In each step reduce the range of the search space by half.


Analysis of Binary SearchAnalysis of Binary Search

How many times the loop is executed?How many times the loop is executed? With each execution the difference between With each execution the difference between ll and and rr is cut in half. is cut in half.

Initially the difference is Initially the difference is n.n. The loop stops when the difference becomes 0 (less than The loop stops when the difference becomes 0 (less than

1)1) . . How many times do you have to cut How many times do you have to cut n n in half to get in half to get

0?0? log n log n – better than the brute-force approach of – better than the brute-force approach of

linear search (linear search (nn).).


Linear Search vs Binary SearchLinear Search vs Binary Search

Costs of linear search: nCosts of linear search: n Costs of binary search: log(n)Costs of binary search: log(n) Should we care?Should we care? Phone book with 200’000 entries:Phone book with 200’000 entries:

n = 200’000n = 200’000 log n = log 200’000 = 17.6log n = log 200’000 = 17.6


Asymptotic AnalysisAsymptotic Analysis

Goal: Goal: tto simplify the analysis of the running time by o simplify the analysis of the running time by getting rid ofgetting rid of details, which are affected by specific details, which are affected by specific implementation and hardware implementation and hardware ““rounding” of numbers: rounding” of numbers: 1,000,0011,000,001 1,000,0001,000,000 ““rounding” of functionsrounding” of functions: 3: 3nn22 nn22

Capturing the essence: how the running time of an Capturing the essence: how the running time of an algorithm increases with the size of the input algorithm increases with the size of the input in the in the limitlimit.. Asymptotically more efficient algorithms are best for all Asymptotically more efficient algorithms are best for all

but small inputs but small inputs


Asymptotic NotationAsymptotic Notation

The “big-Oh” The “big-Oh” OO--NotationNotation asymptotic upper boundasymptotic upper bound f(n) = O(g(n)),f(n) = O(g(n)), if there exists if there exists

constants constants c>0c>0 and and nn00>0, s.t. >0, s.t. f(n) f(n) c g(n) c g(n) for for n n nn00

f(n)f(n) and and g(n)g(n) are functions are functions over non-negative integersover non-negative integers

Used for Used for worst-caseworst-case analysisanalysis

)(nf( )c g n

0n Input Size

Run

ning

Tim

e


The “big-Omega” The “big-Omega” NotationNotation asymptotic lower boundasymptotic lower bound f(n) = f(n) = (g(n))(g(n)) if there exists if there exists

constants constants c>0c>0 and and nn00>0, s.t. >0, s.t. c g(n) c g(n) f(n) f(n) for for n n nn00

Used to describe Used to describe best-case best-case running times or lower bounds running times or lower bounds of algorithmic problems.of algorithmic problems. E.g., lower-bound of searching in E.g., lower-bound of searching in

an unsorted array is an unsorted array is (n). (n). Input Size

Run

ning

Tim

e

)(nf

( )c g n

0n

Asymptotic Notation/2Asymptotic Notation/2



Simple Rule: Drop lower order terms and Simple Rule: Drop lower order terms and constant factors.constant factors. 50 50 n n log log n n is O(is O(n n log log n)n) 77n n - 3 is O(- 3 is O(nn)) 88nn22 log log n n + 5+ 5nn22 + + n n is O(is O(nn22 log log nn))

Note: Although (50 Note: Although (50 n n log log nn) is) is O(O(nn55), it is ), it is expected that an approximation is of the expected that an approximation is of the smallest possible order.smallest possible order.


The “big-Theta” The “big-Theta” NotationNotation asymptoticly tight boundasymptoticly tight bound f(n) =f(n) = (g(n))(g(n)) if there exists if there exists

constants constants cc11>0, c>0, c22>0,>0, and and nn00>0, >0, s.t. s.t. for for n n nn00 cc11 g(n) g(n) f(n) f(n) c c22 g(n) g(n)

f(n)f(n) == (g(n)) (g(n)) if and only if if and only if f(n)f(n) == (g(n)) (g(n)) and and f(n)f(n) == (g(n))(g(n))

O(f(n)) O(f(n)) is often abused instead is often abused instead of of (f(n)) (f(n))

Input Size

Run

ning

Tim

e

)(nf

0n


)(ngc 2

)(ngc 1


A Quick Math Refresher A Quick Math Refresher

Arithmetic progressionArithmetic progression

Geometric progressionGeometric progression given an integer given an integer nn00 and a real number and a real number 0<a0<a11

geometric progressions exhibit exponential growthgeometric progressions exhibit exponential growth

12

0

11 ...

1

nni n

i

aa a a a

a

0

(1 )1 2 3 ...

2

n

i

n ni n


MiscellaneousMiscellaneous

Manipulating logarithms:Manipulating logarithms:

aalog b = log b / log alog b = log b / log a

log alog abb = b log a = b log a

Manipulating summations:Manipulating summations:

j

jj

jj

jj baba )(

j

jj

j acca


SummationsSummations

The running time of insertion sort is The running time of insertion sort is determined by a nested loop.determined by a nested loop.

Nested loops correspond to summations:Nested loops correspond to summations:

for j := 2 to n key := A[j] i := j-1 while i>0 and A[i]>key A[i+1] := A[i] i := i-1 A[i+1] := key

for j := 2 to n key := A[j] i := j-1 while i>0 and A[i]>key A[i+1] := A[i] i := i-1 A[i+1] := key

n

jj

2)1(


Proof by InductionProof by Induction

We want to show that property We want to show that property P P is true for all is true for all integers integers n n nn00..

BasisBasis: prove that : prove that PP is true for is true for nn00.. Inductive stepInductive step: prove that if : prove that if PP is true for all is true for all kk such such

that that nn0 0 k k n n – 1 then – 1 then PP is also true for is also true for n.n. ExampleExample

BasisBasis

0

( 1)( ) for 1

2

n

i

n nS n i n

1

0

1(1 1)(1)

2i

S i


Proof by Induction/2Proof by Induction/2

0

1

0 0

2

( 1)( ) for 1 k 1

2

( ) ( 1)

( 1 1) ( 2 )( 1)

2 2( 1)

2

k

i

n n

i i

k kS k i n

S n i i n S n n

n n n nn n

n n

Inductive StepInductive Step


Correctness of AlgorithmsCorrectness of Algorithms

An algorithm is An algorithm is correctcorrect if for any legal input it if for any legal input it terminates and produces the desired output.terminates and produces the desired output.

Automatic proof of correctness is not possible.Automatic proof of correctness is not possible. There are practical techniques and rigorous There are practical techniques and rigorous

formalisms that help to reason about the formalisms that help to reason about the correctness of (parts of) algorithms.correctness of (parts of) algorithms.


Partial and Total CorrectnessPartial and Total Correctness

Partial correctnessPartial correctness

Any legal input

Algorithm Output

IF this point is reached, THEN this is the desired output

Total correctness

Any legal input

Algorithm Output

INDEED this point is reached, AND this is the desired output


AssertionsAssertions

To prove partial correctness we associate a number of To prove partial correctness we associate a number of assertionsassertions (statements about the state of the (statements about the state of the execution) with specific checkpoints in the algorithm.execution) with specific checkpoints in the algorithm. E.g., E.g., AA[1], …, [1], …, AA[j][j] form an increasing sequenceform an increasing sequence

PreconditionsPreconditions – assertions that must be valid – assertions that must be valid beforebefore the execution of an algorithm or a subroutine the execution of an algorithm or a subroutine ((INPUTINPUT).).

PostconditionsPostconditions – assertions that must be valid – assertions that must be valid afterafter the execution of an algorithm or a subroutine the execution of an algorithm or a subroutine ((OUTPUTOUTPUT).).


Pre/post-conditionsPre/post-conditions

Example:Example: Write a pseudocode algorithm to find the two Write a pseudocode algorithm to find the two

smallest numbers in a sequence of numbers (given smallest numbers in a sequence of numbers (given as an array).as an array).

INPUT: an array of integers A[1..n], n > 0INPUT: an array of integers A[1..n], n > 0 OUTPUT: (m1, m2) such thatOUTPUT: (m1, m2) such that

m1<m2 and for each im1<m2 and for each i[1..n]: m1 [1..n]: m1 A[i] and, if A[i] and, if A[i] A[i] m1, then m2 m1, then m2 A[i]. A[i].

m2 = m1 = A[1] if m2 = m1 = A[1] if j,ij,i[1..n]: A[i]=A[j][1..n]: A[i]=A[j]


Loop InvariantsLoop Invariants

Invariants:Invariants: assertions that are valid any time they are assertions that are valid any time they are reached (many times during the execution of an reached (many times during the execution of an algorithm, e.g., in loops)algorithm, e.g., in loops)

We must show three things about loop invariants:We must show three things about loop invariants: Initialization:Initialization: it is true prior to the first iteration. it is true prior to the first iteration. Maintenance:Maintenance: ifif it is true before an iteration, it is true before an iteration, thenthen it is true it is true

after the iteration.after the iteration. Termination:Termination: when a loop terminates the invariant gives a when a loop terminates the invariant gives a

useful property to show the correctness of the algorithmuseful property to show the correctness of the algorithm


Example: Binary Search/1Example: Binary Search/1

InitializationInitialization: : l = 1, r = n l = 1, r = n the invariant holds because there are the invariant holds because there are no elements to the left of no elements to the left of l l or to the right of or to the right of r.r.

l=1 yields l=1 yields j,i j,i [1..0]: A[i]<q [1..0]: A[i]<q this holds because [1..0] is emptythis holds because [1..0] is empty

r=n yields r=n yields j,i j,i [n+1..n]: A[i]>q [n+1..n]: A[i]>q this holds because [n+1..n] is emptythis holds because [n+1..n] is empty

l := 1r := ndo m := (l+r)/2 if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= rreturn NIL


We want to show that q is not in A if NIL is returned.

Invariant: i[1..l-1]: A[i]<q (Ia) i[r+1..n]: A[i]>q (Ib)



MaintenanceMaintenance: l, r, m = : l, r, m = (l+r)/2(l+r)/2 A[m]!=q & A[m]>q, r=m-1, A sorted implies A[m]!=q & A[m]>q, r=m-1, A sorted implies

kk[r+1..n]: A[k]>q (Ib)[r+1..n]: A[k]>q (Ib) A[m]!=q & A[m]<q, l=m+1, A sorted implies A[m]!=q & A[m]<q, l=m+1, A sorted implies

kk[1..l-1]: A[k]<q (Ib)[1..l-1]: A[k]<q (Ib)

Invariant: i[1..l-1]: A[i]<q (Ia) i[r+1..n]: A[i]>q (Ib)

l := 1r := ndo m := (l+r)/2 if A[m] = q then return m else if A[m] > q then r := m-

1 else l := m+1 while l <= rreturn NIL

l := 1r := ndo m := (l+r)/2 if A[m] = q then return m else if A[m] > q then r := m-

1 else l := m+1 while l <= rreturn NIL



TerminationTermination: l, r, l<=r: l, r, l<=r Two cases:Two cases:

l:=m+1 we get l:=m+1 we get (l+r)/2(l+r)/2+1 > l+1 > l r:=m-1 we get r:=m-1 we get (l+r)/2(l+r)/2-1 < r-1 < r

The range gets smaller during each iteration and the loop will The range gets smaller during each iteration and the loop will terminate when l<=r no longer holds.terminate when l<=r no longer holds.

Invariant: i[1..l-1]: A[i]<q(Ia) i[r+1..n]: A[i]>q (Ib)




Example: Insertion Sort/1Example: Insertion Sort/1

InvariantInvariant:: outside while loopoutside while loop

A[1...j-1] is sortedA[1...j-1] is sorted A[1...j-1] A[1...j-1] A Aorigorig

inside while loop:inside while loop: A[1...i], key, A[i+1…j-1]A[1...i], key, A[i+1…j-1] A[1...i] is sortedA[1...i] is sorted A[i+1…j-1] is sortedA[i+1…j-1] is sorted A[k] > key, i+1<=k<=j-1A[k] > key, i+1<=k<=j-1

for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key

for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key



outside while loopoutside while loop A[1...j-1] is sortedA[1...j-1] is sorted A[1...j-1] A[1...j-1] A Aorigorig


Initialization: j=2: the invariant holds, A[1…1] is trivially sorted. i=j-1: A[1...j-1], key, A[j…j-1] where key=A[j]

A[j…j-1] is empty (and thus trivially sorted) A[1…j-1] is sorted (invariant of outer loop)

for j := 2 to length(A)do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key




outside while loopoutside while loop A[1…j-1] is sortedA[1…j-1] is sorted A[1...j-1] A[1...j-1] A Aorigorig


Maintenance: (A[1…j-1] sorted) + (insert A[j]) A[1…j]

sorted). A[1...i-1], key, A[i,i+1…j-1] satisfies conditions

because of condition A[i]>key and A[1...j-1] being sorted.





outside while loopoutside while loop A[1…j-1] is sortedA[1…j-1] is sorted A[1...j-1] A[1...j-1] A Aorigorig


Termination: main loop, j=n+1: A[1…n] sorted. A[i]key: (A[1...i], key, A[i+1…j-1]) = A[1…j-1] is

sorted i=0: (key, A[1…j-1]) = A[1…j-1] is sorted.




RecursionRecursion

An object is recursiveAn object is recursive if if it contains itself as part of it, orit contains itself as part of it, or iit is defined in terms of itself.t is defined in terms of itself.

Factorial: n!Factorial: n! How do you compute 10!?How do you compute 10!? n! = 1 * 2 * 3 *...* nn! = 1 * 2 * 3 *...* n n! = n * (n-1)!n! = n * (n-1)!


Factorial FunctionFactorial Function

Pseudocode of factorial:Pseudocode of factorial:fac1fac1INPUTINPUT: n: n – – a natural number.a natural number. OUTPUTOUTPUT: n! (factorial of n) : n! (factorial of n) fac1(n)fac1(n) if if n < n < 22 thenthen returnreturn 1 1 else return else return n * fac1(n-1)n * fac1(n-1)

fac1fac1INPUTINPUT: n: n – – a natural number.a natural number. OUTPUTOUTPUT: n! (factorial of n) : n! (factorial of n) fac1(n)fac1(n) if if n < n < 22 thenthen returnreturn 1 1 else return else return n * fac1(n-1)n * fac1(n-1)

A recursive procedure includes aA recursive procedure includes a Termination condition (determines when and how Termination condition (determines when and how

to stop the recursion).to stop the recursion). One (or more) recursive calls.One (or more) recursive calls.


Tracing the ExecutionTracing the Execution

fac(3)

fac(1) 1

fac(2) 2 * fac(1)

1

2

6

3 * fac(2)


BookkeepingBookkeeping

fac(5) 5*fac(4)

fac(4) 4*fac(3)

fac(3) 3*fac(2)

fac(2) 2*fac(1)

fac(1) 1

1

2

6

24

120

The computer maintains an activation stack for The computer maintains an activation stack for active procedure calls (-> compiler active procedure calls (-> compiler construction). Example for fac(5).construction). Example for fac(5).


Variants of FactorialVariants of Factorial

fac2fac2INPUTINPUT: n: n – – a natural number.a natural number. OUTPUTOUTPUT: n! (factorial of n) : n! (factorial of n) facfac22(n)(n) if if n n 22 thenthen returnreturn 1 1 return return n * fac2(n-1)n * fac2(n-1)

fac2fac2INPUTINPUT: n: n – – a natural number.a natural number. OUTPUTOUTPUT: n! (factorial of n) : n! (factorial of n) facfac22(n)(n) if if n n 22 thenthen returnreturn 1 1 return return n * fac2(n-1)n * fac2(n-1)

fac3fac3INPUTINPUT: n: n – – a natural number.a natural number. OUTPUTOUTPUT: n! (factorial of n) : n! (factorial of n) facfac33(n)(n) return return n * fac3(n-1)n * fac3(n-1) if if n n 22 thenthen returnreturn 1 1

fac3fac3INPUTINPUT: n: n – – a natural number.a natural number. OUTPUTOUTPUT: n! (factorial of n) : n! (factorial of n) facfac33(n)(n) return return n * fac3(n-1)n * fac3(n-1) if if n n 22 thenthen returnreturn 1 1


Analysis of the solutionsAnalysis of the solutions

fac2 is correct fac2 is correct The return statement in the if clause terminates the The return statement in the if clause terminates the

function and, thus, the entire recursion.function and, thus, the entire recursion. fac3 is incorrectfac3 is incorrect

Infinite recursion. The termination condition is Infinite recursion. The termination condition is never reached.never reached.


Fibonacci NumbersFibonacci Numbers

DefinitionDefinition ffib(ib(11) = 1) = 1 ffib(ib(22) = 1) = 1 ffib(n) = ib(n) = ffib(n-ib(n-11) + ) + ffib(n-ib(n-22)), n>2, n>2

Numbers in the series:Numbers in the series: 1, 1, 2, 3, 5, 8, 13, 21, 34, ...1, 1, 2, 3, 5, 8, 13, 21, 34, ...


Fibonacci Fibonacci ImplementationImplementation

fibfibINPUTINPUT: n: n – – a natural numbera natural number larger than 0 larger than 0.. OUTPUTOUTPUT: : fib(n), the nth Fibonacci number.fib(n), the nth Fibonacci number. fibfib(n)(n) if if n n 22 thenthen returnreturn 1 1 else return else return fib(n-1) + fib(n-2) fib(n-1) + fib(n-2)

fibfibINPUTINPUT: n: n – – a natural numbera natural number larger than 0 larger than 0.. OUTPUTOUTPUT: : fib(n), the nth Fibonacci number.fib(n), the nth Fibonacci number. fibfib(n)(n) if if n n 22 thenthen returnreturn 1 1 else return else return fib(n-1) + fib(n-2) fib(n-1) + fib(n-2)

Multiple recursive calls are possible.Multiple recursive calls are possible.


Fibonacci Fibonacci Implementation/2Implementation/2

intint f fibib((intint i) { i) { ifif ( (i i <<= = 2) {2) { returnreturn 1;} 1;} elseelse { { returnreturn f fibib(i-1)(i-1) + fib(i-2) + fib(i-2); }; }}}

intint main() { main() { printf(“Fibonacci of 5 is %d\n", fac(5));printf(“Fibonacci of 5 is %d\n", fac(5)); }}

intint f fibib((intint i) { i) { ifif ( (i i <<= = 2) {2) { returnreturn 1;} 1;} elseelse { { returnreturn f fibib(i-1)(i-1) + fib(i-2) + fib(i-2); }; }}}

intint main() { main() { printf(“Fibonacci of 5 is %d\n", fac(5));printf(“Fibonacci of 5 is %d\n", fac(5)); }}


Tracing fib(4)Tracing fib(4)

fib(4) fib(3) + fib(2)

fib(2) 1


11

3

fib(2) 1

fib(1) 1

21


BookkeepingBookkeeping


fib(2) 1fib(1) 1

2

3

Activation stack for fib(4).Activation stack for fib(4).

fib(3) fib(2) + fib(1) 11

1

fib(2) 1


Mutual RecursionMutual Recursion

Recursion doesRecursion does not not always occur because a always occur because a procedureprocedure calls itself. calls itself.

Mutual recursion Mutual recursion occurs ifoccurs if two two proceduresprocedures call call each other.each other.

B A


Mutual Recursion ExampleMutual Recursion Example

Problem: Determine whether a Problem: Determine whether a natural natural numbernumber is even.is even.

Definition of even:Definition of even: 0 is even0 is even N is odd if N-1 isN is odd if N-1 is eveneven N is even if N-1 is oddN is even if N-1 is odd


ImplementationImplementation of even of even

Can it be used to determine whether a number Can it be used to determine whether a number is odd?is odd?

evenevenINPUTINPUT: n: n – – a natural numbera natural number..OUTPUTOUTPUT: : true if n is even; false otherwisetrue if n is even; false otherwise oddodd(n)(n) if if n = 0n = 0 then return then return TRUETRUE return !return !even(n-1)even(n-1)

even(n)even(n) if if n = 0 n = 0 thenthen returnreturn TRUE TRUE else return !else return !odd(n-1)odd(n-1)

evenevenINPUTINPUT: n: n – – a natural numbera natural number..OUTPUTOUTPUT: : true if n is even; false otherwisetrue if n is even; false otherwise oddodd(n)(n) if if n = 0n = 0 then return then return TRUETRUE return !return !even(n-1)even(n-1)

even(n)even(n) if if n = 0 n = 0 thenthen returnreturn TRUE TRUE else return !else return !odd(n-1)odd(n-1)


Is Recursion Necessary?Is Recursion Necessary?

Theory: You can always resort to iteration and Theory: You can always resort to iteration and explicitly maintain a recursion stack.explicitly maintain a recursion stack.

Practice: Recursion is elegant and in some cases the Practice: Recursion is elegant and in some cases the best solution by far.best solution by far.

In the previous examples recursion was never In the previous examples recursion was never appropriate since there exist simple iterative appropriate since there exist simple iterative solutions.solutions.

Recursion is more expensive than corresponding Recursion is more expensive than corresponding iterative solutions since bookkeeping is necessary.iterative solutions since bookkeeping is necessary.


SortingSorting

Sorting is a classical and important algorithmic Sorting is a classical and important algorithmic problem.problem.

We look at sorting arrays (in contrast to files, which We look at sorting arrays (in contrast to files, which restrict random access).restrict random access).

A key constraint is the efficient management of the A key constraint is the efficient management of the spacespace In-place sorting algorithmsIn-place sorting algorithms

The efficiency comparison is based on the number of The efficiency comparison is based on the number of comparisons (C) and the number of movements (M).comparisons (C) and the number of movements (M).


SortingSorting

Simple sorting methods use roughly n * n Simple sorting methods use roughly n * n comparisonscomparisons Insertion sortInsertion sort Selection sortSelection sort Bubble sortBubble sort

Fast sorting methods use roughly n * log n Fast sorting methods use roughly n * log n comparisons.comparisons. Merge sortMerge sort Heap sortHeap sort QuicksortQuicksort


Sort

Example 2: SortingExample 2: SortingINPUTsequence of n numbers

a1, a2, a3,….,anb1,b2,b3,….,bn

OUTPUTa permutation of the input sequence of numbers

2 5 4 10 7 2 4 5 7 10

Correctness (requirements for the output)For any given input the algorithm halts with the output:

• b1 < b2 < b3 < …. < bn

• b1, b2, b3, …., bn is a permutation of a1, a2, a3,….,an

Correctness (requirements for the output)For any given input the algorithm halts with the output:

• b1 < b2 < b3 < …. < bn

• b1, b2, b3, …., bn is a permutation of a1, a2, a3,….,an


InsertionInsertion Sort Sort

A1 nj

3 6 84 9 7 2 5 1

i

Strategy• Start with one sorted card.• Insert an unsorted card at the

correct position in the sorted part.

• Continue until all unsorted cards are inserted/sorted.

Strategy• Start with one sorted card.• Insert an unsorted card at the

correct position in the sorted part.

• Continue until all unsorted cards are inserted/sorted.

A 44 55 12 42 94 18 06 67 44 55 12 42 94 18 06 67 12 44 55 42 94 18 06 67 12 42 44 55 94 18 06 67 12 42 44 55 94 18 06 67 12 18 42 44 55 94 06 67 06 12 18 42 44 55 94 67 06 12 18 42 44 55 67 94


Insertion Sort/2Insertion Sort/2

n

j

j2

The number of comparisons during the jth The number of comparisons during the jth iteration is iteration is at least 1: Cmin = = n-1 at least 1: Cmin = = n-1 at most j-1: Cmax = = (n*n-n-2)/2at most j-1: Cmax = = (n*n-n-2)/2

INPUT: A[1..n] – an array of integersOUTPUT: a permutation of A such that A[1]A[2]…A[n]for j := 2 to n do key = A[j] i := j-1 while i > 0 and A[i] > key do A[i+1] := A[i]; i-- A[j+1] := key

INPUT: A[1..n] – an array of integersOUTPUT: a permutation of A such that A[1]A[2]…A[n]for j := 2 to n do key = A[j] i := j-1 while i > 0 and A[i] > key do A[i+1] := A[i]; i-- A[j+1] := key

n

j 21

n

jj

21


Insertion Sort/3Insertion Sort/3

The number of comparisons during the jth The number of comparisons during the jth iteration is:iteration is: j/2 in average: Cavg = = (n*n+n–2)/4j/2 in average: Cavg = = (n*n+n–2)/4

The number of movements is Ci+1: The number of movements is Ci+1: Mmin = = 2*(n-1), Mmin = = 2*(n-1),

Mavg = = (n*n+5n-6)/4 Mavg = = (n*n+5n-6)/4

Mmax = = (n*n+n-2)/2Mmax = = (n*n+n-2)/2

n

j 22

2/2

n

jj

n

jj

2

n

jj

212/


SelectionSelection Sort Sort

A1 nj

1 3 42 5 7 8 9 6

i

Strategy• Start empty handed.• Enlarge the sorted part by

switching the first element of the unsorted part with the smallest element of the unsorted part.

• Continue until the unsorted part consists of one element only.

Strategy• Start empty handed.• Enlarge the sorted part by

switching the first element of the unsorted part with the smallest element of the unsorted part.

• Continue until the unsorted part consists of one element only.

A 44 55 12 42 94 18 06 67 06 55 12 42 94 18 44 67 06 12 55 42 94 18 44 67 06 12 18 42 94 55 44 67 06 12 18 42 94 55 44 67 06 12 18 42 44 55 94 67 06 12 18 42 44 55 94 67 06 12 18 42 44 55 67 94


Selection Sort/2Selection Sort/2

The number of comparisons is indepen-dent The number of comparisons is indepen-dent of the original ordering (this is less natural of the original ordering (this is less natural behavior than insertion sort):behavior than insertion sort): C = = (n*n-n)/2C = = (n*n-n)/2

INPUT: A[1..n] – an array of integersOUTPUT: a permutation of A such that A[1]A[2]…A[n]for j := 1 to n-1 do key := A[j]; ptr := j for i := j+1 to n do if A[i] < key then ptr := i; key := A[i]; A[ptr] := A[j]; A[j] := key

INPUT: A[1..n] – an array of integersOUTPUT: a permutation of A such that A[1]A[2]…A[n]for j := 1 to n-1 do key := A[j]; ptr := j for i := j+1 to n do if A[i] < key then ptr := i; key := A[i]; A[ptr] := A[j]; A[j] := key

1

1

n

jj


Selection Sort/3Selection Sort/3

The number of movements is: The number of movements is: Mmin = = 3*(n-1) Mmin = = 3*(n-1)

Mmax = = (n*n–n)/4 + 3*(n-1)Mmax = = (n*n–n)/4 + 3*(n-1)

1

13

n

j

1

13

n

jj


BubbleBubble Sort Sort

A1 nj

1 3 42 5 7 9 8 6

i

Strategy• Start from the back and

compare pairs of adjacent elements.

• Switch the elements if the larger comes before the smaller.

• In each step the smallest element of the unsorted part is moved to the beginning of the unsorted part and the sorted part grows by one.

Strategy• Start from the back and

compare pairs of adjacent elements.

• Switch the elements if the larger comes before the smaller.

• In each step the smallest element of the unsorted part is moved to the beginning of the unsorted part and the sorted part grows by one.

A 44 55 12 42 94 18 06 67 06 44 55 12 42 94 18 67 06 12 44 55 18 42 94 67 06 12 18 44 55 42 67 94 06 12 18 42 44 55 67 94 06 12 18 42 44 55 67 94 06 12 18 42 44 55 67 94 06 12 18 42 44 55 67 94


Bubble Sort/2Bubble Sort/2

The number of comparisons is indepen-dent of The number of comparisons is indepen-dent of the original ordering:the original ordering: C = = (n*n-n)/2C = = (n*n-n)/2

INPUT: A[1..n] – an array of integersOUTPUT: a permutation of A such that A[1]A[2]…A[n]for j := 2 to n do for i := n to j do if A[i-1] < A[i] then key := A[i-1]; A[i-1] := A[i]; A[i]:=key

INPUT: A[1..n] – an array of integersOUTPUT: a permutation of A such that A[1]A[2]…A[n]for j := 2 to n do for i := n to j do if A[i-1] < A[i] then key := A[i-1]; A[i-1] := A[i]; A[i]:=key

n

jj

21


Bubble Sort/3Bubble Sort/3

The number of movements is: The number of movements is: Mmin = 0 Mmin = 0

Mmax = = 3*n*(n-1)/2Mmax = = 3*n*(n-1)/2

Mavg = = 3*n*(n-1)/4 Mavg = = 3*n*(n-1)/4

n

jj

2)1(3

2/)1(32

n

jj






The divide- and- conquer design The divide- and- conquer design paradigmparadigm

1. Divide 1. Divide the problem (instance) into the problem (instance) into subproblems.subproblems.

2. Conquer 2. Conquer the subproblems by solving them the subproblems by solving them recursively.recursively.

3. Combine 3. Combine subproblem solutions.subproblem solutions.



Example: Example: merge sortmerge sort1. Divide: 1. Divide: Trivial.Trivial.2. Conquer: 2. Conquer: Recursively sort 2 subarrays.Recursively sort 2 subarrays.3. Combine: 3. Combine: Linear- time merge.Linear- time merge.Recurrence for merge sortRecurrence for merge sort T( n) T( n) = = 2 2 T( n/ 2) T( n/ 2) # subproblems# subproblems subproblem sizesubproblem size

++ O( n) O( n) work dividing and combiningwork dividing and combining



Binary searchBinary searchFind an element in a sorted array:Find an element in a sorted array:1. Divide: 1. Divide: Check middle element.Check middle element.2. Conquer: 2. Conquer: Recursively search 1 subarray.Recursively search 1 subarray.3. Combine: 3. Combine: Trivial.Trivial.Recurrence for binary searchRecurrence for binary searchT(n)T(n) = = 11 T(n/2)T(n/2) ++ ΘΘ(1)(1) # subproblems# subproblems subproblem subproblem work work size dividing and size dividing and combiningcombining





ConclusionConclusion

• • Divide and conquer is just one of several Divide and conquer is just one of several powerful techniques for algorithm design.powerful techniques for algorithm design.

• • Divide- and- conquer algorithms can be Divide- and- conquer algorithms can be analyzed using recurrences and the master analyzed using recurrences and the master method (so practice this math).method (so practice this math).

• • Can lead to more efficient algorithmsCan lead to more efficient algorithms




















Date post:	26-Dec-2015
Category:	Documents
Upload:	griffin-asher-pitts
View:	218 times
Download:	0 times

CSC2105: Algorithms Mashiour Rahman [email protected] American International University Bangladesh.

Documents