Date post: | 31-Dec-2015 |
Category: |
Documents |
Upload: | cornelia-lewis |
View: | 234 times |
Download: | 2 times |
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-11
CSC401 – Analysis of Algorithms CSC401 – Analysis of Algorithms
Chapter 5 – 2. Chapter 5 – 2.
Divide-and-ConquerDivide-and-ConquerObjectives:Objectives: Introduce the Divide-and-conquer paradigm and Introduce the Divide-and-conquer paradigm and
variants: Decrease-and-Conquer and Transform-and-variants: Decrease-and-Conquer and Transform-and-ConquerConquer
Review the Merge-sort algorithmReview the Merge-sort algorithm Solve recurrence equationsSolve recurrence equations
– Iterative substitutionIterative substitution– Recursion treesRecursion trees– Guess-and-testGuess-and-test– The master methodThe master method
Discuss integer and matrix multiplicationsDiscuss integer and matrix multiplications
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-22
Divide-and-ConquerDivide-and-ConquerDivide-and conquerDivide-and conquer is a general algorithm design is a general algorithm design paradigm:paradigm:– DivideDivide: divide the input data : divide the input data SS in two or more disjoint in two or more disjoint
subsets subsets SS11, S, S22, …, …– RecurRecur: solve the subproblems recursively: solve the subproblems recursively– ConquerConquer: combine the solutions for : combine the solutions for SS11,, SS22, …, into a solution , …, into a solution
for for SS
The base case for the recursion are subproblems of The base case for the recursion are subproblems of constant sizeconstant sizeAnalysis can be done using Analysis can be done using recurrence equationsrecurrence equations
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-33
Merge-Sort ReviewMerge-Sort ReviewMerge-sort on an input Merge-sort on an input sequence sequence SS with with nn elements consists of elements consists of three steps:three steps:– DivideDivide: partition : partition SS into into
two sequences two sequences SS11 and and SS22 of about of about nn22 elements elements eacheach
– RecurRecur: recursively sort : recursively sort SS11 and and SS22
– ConquerConquer: merge : merge SS11 and and SS2 2 into a unique sorted into a unique sorted sequencesequence
Algorithm mergeSort(S, C)Input sequence S with n
elements, comparator C Output sequence S sorted
according to Cif S.size() > 1
(S1, S2) partition(S, n/2)
mergeSort(S1, C)
mergeSort(S2, C)
S merge(S1, S2)7 2 9 4 2 4 7 9
7 2 2 7 9 4 4 9
7 7 2 2 9 9 4 4
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-44
Recurrence Equation AnalysisRecurrence Equation AnalysisThe conquer step of merge-sort consists of merging The conquer step of merge-sort consists of merging two sorted sequences, each with two sorted sequences, each with nn22 elements and elements and implemented by means of a doubly linked list, takes implemented by means of a doubly linked list, takes at most at most bnbn steps, for some constant steps, for some constant bb..Likewise, the basis case (Likewise, the basis case (nn < 2) < 2) will take at will take at bb most most steps.steps.Therefore, if we let Therefore, if we let TT((nn) ) denote the running time of denote the running time of merge-sort:merge-sort:
We can therefore analyze the running time of merge-We can therefore analyze the running time of merge-sort by finding a sort by finding a closed form solutionclosed form solution to the above to the above equation.equation.– That is, a solution that has That is, a solution that has TT((nn) ) only on the left-hand side.only on the left-hand side.
2if)2/(2
2if )(
nbnnT
nbnT
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-55
Iterative SubstitutionIterative SubstitutionIn the iterative substitution, or “plug-and-chug,” In the iterative substitution, or “plug-and-chug,” technique, we iteratively apply the recurrence technique, we iteratively apply the recurrence equation to itself and see if we can find a patternequation to itself and see if we can find a pattern
Note that base, T(n)=b, case occurs when 2Note that base, T(n)=b, case occurs when 2ii=n. =n. That is, i = log n. That is, i = log n. So,So,Thus, T(n) is O(n log n).Thus, T(n) is O(n log n).
ibnnT
bnnT
bnnT
bnnT
bnnbnT
bnnTnT
ii
)2/(2
...
4)2/(2
3)2/(2
2)2/(2
))2/())2/(2(2
)2/(2)(
44
33
22
2
nbnbnnT log)(
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-66
The Recursion TreeThe Recursion TreeDraw the recursion tree for the recurrence relation Draw the recursion tree for the recurrence relation and look for a pattern:and look for a pattern:
depthdepth T’sT’s sizesize
00 11 nn
11 22 nn22
ii 22ii nn22ii
…… …… ……
2if)2/(2
2if )(
nbnnT
nbnT
timetime
bnbn
bnbn
bnbn
……
Total time = bn + bn log n(last level plus all previous levels)
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-77
Guess-and-Test MethodGuess-and-Test MethodIn the guess-and-test method, we guess a closed In the guess-and-test method, we guess a closed form solution and then try to prove it is true by form solution and then try to prove it is true by induction:induction:
Guess: T(n) < cnlogn.Guess: T(n) < cnlogn.
Wrong: we cannot make this last line be less than Wrong: we cannot make this last line be less than cn log ncn log n
nbncnncn
nbnncn
nbnnnc
nbnnTnT
loglog
log)2log(log
log))2/log()2/((2
log)2/(2)(
2iflog)2/(2
2if )(
nnbnnT
nbnT
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-88
Guess-and-Test Method: Guess-and-Test Method: Part 2Part 2Recall the recurrence equation:Recall the recurrence equation:
Guess #2: T(n) < cn logGuess #2: T(n) < cn log22 n. n.
– if c > b.if c > b.
So, T(n) is O(n logSo, T(n) is O(n log22 n). n).In general, to use this method, you need to have a In general, to use this method, you need to have a good guess and you need to be good at induction good guess and you need to be good at induction proofs.proofs.
ncn
nbncnncnncn
nbnncn
nbnnnc
nbnnTnT
2
2
2
2
log
loglog2log
log)2log(log
log))2/(log)2/((2
log)2/(2)(
2iflog)2/(2
2if )(
nnbnnT
nbnT
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-99
Master MethodMaster MethodMany divide-and-conquer recurrence Many divide-and-conquer recurrence equations have the form:equations have the form:
The Master Theorem:The Master Theorem:
dnnfbnaT
dncnT
if)()/(
if )(
.1 somefor )()/( provided
)),((is)(then),(is)(if 3.
)log(is)(then),log(is)(if 2.
)(is)(then),(is)(if 1.
log
1loglog
loglog
nfbnaf
nfnTnnf
nnnTnnnf
nnTnOnf
a
kaka
aa
b
bb
bb
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1010
Master Method, ExamplesMaster Method, ExamplesThe form:The form:
The Master The Master Theorem:Theorem:
dnnfbnaT
dncnT
if)()/(
if )(
.1 somefor )()/( provided
)),((is)(then),(is)(if 3.
)log(is)(then),log(is)(if 2.
)(is)(then),(is)(if 1.
log
1loglog
loglog
nfbnaf
nfnTnnf
nnnTnnnf
nnTnOnf
a
kaka
aa
b
bb
bb
nnTnT )2/(4)(Example 1:Example 1:– Solution: a=4, b=2, logba=2, f(n)=n, so case 1
says T(n) is O(n2).
Example 2:– Solution: a=b=2, logba=1, f(n)=nlogn, so case 2
says T(n) is O(n log2 n).
nnnTnT log)2/(2)(
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1111
Master Method, ExamplesMaster Method, Examples
The form:The form:
The Master The Master Theorem:Theorem:
dnnfbnaT
dncnT
if)()/(
if )(
.1 somefor )()/( provided
)),((is)(then),(is)(if 3.
)log(is)(then),log(is)(if 2.
)(is)(then),(is)(if 1.
log
1loglog
loglog
nfbnaf
nfnTnnf
nnnTnnnf
nnTnOnf
a
kaka
aa
b
bb
bb
Example 3:Example 3:– Solution: a=1, b=3, logba=0, f(n)=nlogn, so case 3 says
T(n) is O(nlogn).
Example 4:– Solution: a=2, b=8, logba=3, f(n)=n2, so case 1 says
T(n) is O(n3).
2)2/(8)( nnTnT
nnnTnT log)3/()(
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1212
Master Method, ExamplesMaster Method, ExamplesThe form:The form:
The Master The Master Theorem:Theorem:
1)2/()( nTnT
dnnfbnaT
dncnT
if)()/(
if )(
.1 somefor )()/( provided
)),((is)(then),(is)(if 3.
)log(is)(then),log(is)(if 2.
)(is)(then),(is)(if 1.
log
1loglog
loglog
nfbnaf
nfnTnnf
nnnTnnnf
nnTnOnf
a
kaka
aa
b
bb
bb
Example 5:Example 5:– Solution: a=9, b=3, logba=2, f(n)=n3, so case 3 says
T(n) is O(n3).Example 6: (binary search)– Solution: a=1, b=2, logba=0, f(n)=1, so case 2 says
T(n) is O(log n).Example 7: (heap construction)– Solution: a=2, b=2, logba=1, f(n)=logn, so case 1
says T(n) is O(n).
3)3/(9)( nnTnT
nnTnT log)2/(2)(
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1313
Iterative “Proof” of the Master TheoremIterative “Proof” of the Master TheoremUsing iterative substitution, let us see if we can Using iterative substitution, let us see if we can find a pattern:find a pattern:
We then distinguish the three cases asWe then distinguish the three cases as– The first term is dominantThe first term is dominant– Each part of the summation is equally dominantEach part of the summation is equally dominant– The summation is a geometric seriesThe summation is a geometric series
1)(log
0
log
1)(log
0
log
2233
22
2
)/()1(
)/()1(
. . .
)()/()/()/(
)()/()/(
))/())/((
)()/()(
n
i
iia
n
i
iin
b
b
b
b
bnfaTn
bnfaTa
nfbnafbnfabnTa
nfbnafbnTa
bnbnfbnaTa
nfbnaTnT
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1414
Integer MultiplicationInteger MultiplicationAlgorithm: Multiply two n-bit integers I and Algorithm: Multiply two n-bit integers I and J.J.– Divide step: Split I and J into high-order and low-order Divide step: Split I and J into high-order and low-order
bitsbits
– We can then define I*J by multiplying the parts and We can then define I*J by multiplying the parts and adding:adding:
– So, T(n) = 4T(n/2) + n, which implies T(n) is O(nSo, T(n) = 4T(n/2) + n, which implies T(n) is O(n22).).– But that is no better than the algorithm we learned in But that is no better than the algorithm we learned in
grade school.grade school.
ln
h
ln
h
JJJ
III
2/
2/
2
2
lln
hln
lhn
hh
ln
hln
h
JIJIJIJI
JJIIJI
2/2/
2/2/
222
)2(*)2(*
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1515
An Improved Integer Multiplication AlgorithmAn Improved Integer Multiplication Algorithm
Algorithm: Multiply two n-bit integers I and J.Algorithm: Multiply two n-bit integers I and J.– Divide step: Split I and J into high-order and low-order bitsDivide step: Split I and J into high-order and low-order bits
– Observe that there is a different way to multiply parts:Observe that there is a different way to multiply parts:
– So, T(n) = 3T(n/2) + n, which implies T(n) is O(nSo, T(n) = 3T(n/2) + n, which implies T(n) is O(nloglog22
33), by the ), by the Master Theorem.Master Theorem.
– Thus, T(n) is O(nThus, T(n) is O(n1.5851.585).).
ln
h
ln
h
JJJ
III
2/
2/
2
2
lln
hllhn
hh
lln
llhhhlhhlllhn
hh
lln
llhhhllhn
hh
JIJIJIJI
JIJIJIJIJIJIJIJI
JIJIJIJJIIJIJI
2/
2/
2/
2)(2
2])[(2
2]))([(2*
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1616
Matrix Multiplication AlgorithmMatrix Multiplication AlgorithmProblem:Problem: Given n×n matrices X and Y, find their Given n×n matrices X and Y, find their product Z=X×Y, product Z=X×Y,
General algorithmGeneral algorithm: O(n: O(n33))
Strassen’s AlgorithmStrassen’s Algorithm::– Rewrite the matrix product as: Rewrite the matrix product as: – Where Where I=AE+BG, J=AF+BHI=AE+BG, J=AF+BH
K=CE+DG, L=CF+DHK=CE+DG, L=CF+DH– Define 7 submatrix products:Define 7 submatrix products:
S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E)S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E)
S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F)S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F)– Construct: Construct:
I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7-S3-I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7-S3-S5S5
– Running time: T(n) = 7T(n/2)+bnRunning time: T(n) = 7T(n/2)+bn22 for some constant b, for some constant b, which gives: O(nwhich gives: O(nlog7log7), better than O(n), better than O(n33))
1
0
],[],[],[n
k
jkYkiXjiZ
HG
FE
DC
BA
LK
JI
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1717
Decrease and ConquerDecrease and Conquer Reduce problem instance to smaller Reduce problem instance to smaller
instance of the same problem and extend instance of the same problem and extend solutionsolution
Solve smaller instanceSolve smaller instance Extend solution of smaller instance to Extend solution of smaller instance to
obtain solution to original problemobtain solution to original problem Also referred to as Also referred to as inductiveinductive or or
incrementalincremental approach approach
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1818
Examples of Decrease and ConquerExamples of Decrease and ConquerDecrease by one:Decrease by one:– Insertion sortInsertion sort– Graph search algorithms:Graph search algorithms:
DFSDFSBFSBFSTopological sortingTopological sorting
– Algorithms for generating permutations, subsetsAlgorithms for generating permutations, subsets
Decrease by a constant factorDecrease by a constant factor– Binary search Binary search – Fake-coin problemsFake-coin problems– multiplication multiplication à la russeà la russe– Josephus problemJosephus problem
Variable-size decreaseVariable-size decrease– Euclid’s algorithmEuclid’s algorithm– Selection by partitionSelection by partition
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-1919
Decrease by OneDecrease by OneGenerating all subsetsGenerating all subsets– Given n elements in the setGiven n elements in the set– Take one element from the setTake one element from the set– Generate all subsets of remaining elementsGenerate all subsets of remaining elements– Insert the element taken to the subsets generatedInsert the element taken to the subsets generated– Base case: empty set only has a single subset – itselfBase case: empty set only has a single subset – itself– Time efficiency?Time efficiency?
Generating permutationsGenerating permutations– Given n symbolsGiven n symbols– Take one symbol from the setTake one symbol from the set– Generate all permutations of remaining symbolsGenerate all permutations of remaining symbols– Insert the symbol taken to the permutations generated Insert the symbol taken to the permutations generated – Base case: a single symbol has only one permutation -- Base case: a single symbol has only one permutation --
itself itself – Time efficiency?Time efficiency?
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2020
Variable-size-decrease: Variable-size-decrease: Binary search treesBinary search trees
Arrange keys in a binary tree with Arrange keys in a binary tree with the the binary search tree propertybinary search tree property::
k
<k >k • What about repeated keys?What about repeated keys?
Example 1:Example 1: 5, 10, 3, 1, 7, 12, 9
Example 2:Example 2: 4, 5, 7, 2, 1, 3, 6
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2121
Searching and insertion in binary Searching and insertion in binary search treessearch trees
Searching – straightforwardSearching – straightforward
Insertion – search for key, insert at leaf where Insertion – search for key, insert at leaf where search terminatedsearch terminated
All operations: worst case # key comparisons = All operations: worst case # key comparisons = hh + 1+ 1
log log nn ≤ ≤ h h ≤ ≤ nn – 1 with average (random files) – 1 with average (random files) 1.41log 1.41log nn
Thus all operations have:Thus all operations have:– worst case: worst case: ΘΘ((nn) ) – average case: average case: ΘΘ(lg(lgnn) )
Bonus:Bonus: inorder traversal produces sorted list inorder traversal produces sorted list ((treesorttreesort))
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2222
Transform and ConquerTransform and ConquerSolve problem by transforming into:Solve problem by transforming into:
a more convenient instance of the same problem a more convenient instance of the same problem ((instance simplificationinstance simplification))– presortingpresorting– Gaussian eliminationGaussian elimination
a different representation of the same instance a different representation of the same instance ((representation changerepresentation change))– balanced search treesbalanced search trees– heaps and heapsortheaps and heapsort– polynomial evaluation by Horner’s rulepolynomial evaluation by Horner’s rule– Fast Fourier TransformFast Fourier Transform
a different problem altogether (a different problem altogether (problem reductionproblem reduction))– reductions to graph problemsreductions to graph problems– linear programming linear programming
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2323
Instance simplification - PresortingInstance simplification - Presorting
Solve instance of problem by transforming into Solve instance of problem by transforming into another simpler/easier instance of the same another simpler/easier instance of the same problemproblem
Presorting:Presorting:Many problems involving lists are easier when list is Many problems involving lists are easier when list is
sorted.sorted.
searching searching
computing the median (selection problem)computing the median (selection problem)
computing the modecomputing the mode
finding repeated elementsfinding repeated elements
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2424
Selection ProblemSelection ProblemFind the kFind the kth th smallest element in A[1],…,A[smallest element in A[1],…,A[nn]. ].
Special cases:Special cases:– minimumminimum: k : k = 1= 1– maximummaximum: k : k = = nn– medianmedian: k : k = = nn/2/2
Presorting-based algorithm Presorting-based algorithm – sort listsort list– return A[return A[kk]]
Partition-based algorithm (Variable decrease & conquer):Partition-based algorithm (Variable decrease & conquer):– pivot/split at A[pivot/split at A[ss] using partitioning algorithm from quicksort] using partitioning algorithm from quicksort– if if ss==k k return A[ return A[ss] ] – else if else if s<k s<k repeat with sublist A[repeat with sublist A[ss+1],…A[+1],…A[nn]. ]. – else if s>k else if s>k repeat with sublist A[1],…A[repeat with sublist A[1],…A[ss-1]. -1].
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2525
Notes on Selection ProblemNotes on Selection Problem
Presorting-based algorithm: Presorting-based algorithm: ΩΩ((nnlglgnn) + ) + ΘΘ(1) = (1) = ΩΩ((nnlglgnn) ) Partition-based algorithm (Variable decrease & Partition-based algorithm (Variable decrease & conquer):conquer):– worst case: T(worst case: T(nn) =T() =T(nn-1) + (-1) + (nn+1) -> +1) -> ΘΘ((nn22) ) – best case: best case: ΘΘ((nn) ) – average case: T(average case: T(nn) =T() =T(nn/2) + (/2) + (nn+1) -> +1) -> ΘΘ((nn) ) – Bonus:Bonus: also identifies the also identifies the kk smallest elements (not just smallest elements (not just
the the kkthth))
Special cases max, min: better, simpler linear Special cases max, min: better, simpler linear algorithm (brute force)algorithm (brute force)Conclusion:Conclusion: Presorting does Presorting does notnot help in this case. help in this case.
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2626
Finding repeated elementsFinding repeated elementsPresorting-based algorithm: Presorting-based algorithm: – use mergesort (optimal): use mergesort (optimal): ΘΘ((nnlglgnn) ) – scan array to find repeated scan array to find repeated adjacentadjacent elements: elements:
ΘΘ((nn) )
Brute force algorithm: Brute force algorithm: ΘΘ((nn22) )
Conclusion:Conclusion: Presorting yields Presorting yields significantsignificant improvementimprovement
Similar improvement for modeSimilar improvement for mode
What about searching?What about searching?
ΘΘ((nnlglgnn))
CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-2727
Taxonomy of Searching AlgorithmsTaxonomy of Searching Algorithms
Elementary searching algorithmsElementary searching algorithms– sequential searchsequential search– binary searchbinary search– binary tree searchbinary tree search
Balanced tree searchingBalanced tree searching– AVL treesAVL trees– red-black treesred-black trees– multiway balanced trees (2-3 trees, 2-3-4 trees, B trees)multiway balanced trees (2-3 trees, 2-3-4 trees, B trees)
HashingHashing– separate chainingseparate chaining– open addressingopen addressing