Algoritmen & Datastructuren 2014 – 2015
Greedy Algorithms
Philip Dutré & Benedict Brown Dept. of Computer Science, K.U.Leuven
What is a greedy algorithm?
Copyright Ph.Dutré,Spring 2015 2
Greedy algorithms make the choice that seems best now Hopefully lots of locally good choices gives a good outcome Example 1: Driving directions: always head toward your goal
What is a greedy algorithm?
Copyright Ph.Dutré,Spring 2015 3
Greedy algorithms make the choice that seems best now Hopefully lots of locally good choices gives a good outcome Example 1: Driving directions: always head toward your goal (but suboptimal)
What is a greedy algorithm?
Copyright Ph.Dutré,Spring 2015 4
Example 2: Numbering fragments sequential fragments should be near Greedy: next fragment is closest unnumbered fragment Not optimal, but reasonable approximation to very hard problem
What is a greedy algorithm?
Copyright Ph.Dutré,Spring 2015 5
Example 3: Coin Exchange Suppose you want to count out a certain amount of money,
using the fewest possible bills and coins A greedy algorithm would do this would be:
At each step, take the largest possible bill or coin that does not overshoot
Example: To make $6.39, you can choose: a $5 bill a $1 bill, to make $6 a 25¢ coin, to make $6.25 A 10¢ coin, to make $6.35 four 1¢ coins, to make $6.39
For US money, the greedy algorithm always gives the
optimum solution
What is a greedy algorithm?
Copyright Ph.Dutré,Spring 2015 6
Example 3: Coin Exchange In some (fictional) monetary system, “quatloos” come in:
1 quatloo, 7 quatloo, and 10 quatloo coins
Using a greedy algorithm to count out 15 quatloos, you would get: A 10 quatloo piece Five 1 quatloo pieces (for a total of 15 quatloos) this requires six coins
Better solution: two 7 quatloo pieces and one 1 quatloo piece this only requires three coins
The greedy algorithm for this problem results in a solution, but not in an optimal solution
Real problem: Activity Scheduling
Copyright Ph.Dutré,Spring 2015 8
We want to schedule as many non-overlapping activities as possible. Tennis matches: as many on centre court as possible Hubble: observations can occur only when orbit is right Classrooms: class and practice sessions limited by available
rooms
Real problem: Activity Scheduling
Copyright Ph.Dutré,Spring 2015 9
Too many schedules to try them all Greedy approach: Schedule an activity …
.. then schedule as many non-overlapping activities as possible If we always pick well, the result should be pretty good or
perfect!
Activity Scheduling, Heuristic 1
Copyright Ph.Dutré,Spring 2015 10
Schedule the next non-conflicting activity to start This can lead to suboptimal choices: We schedule one long activity instead of four short ones
We schedule this activity first, but this leaves no room for the other 4 (and we want to optimize for the number of activities)
Activity Scheduling, Heuristic 2
Copyright Ph.Dutré,Spring 2015 11
Schedule the shortest non-conflicting activity This can lead to suboptimal choices : We schedule one short activity and blocking two longer ones.
We schedule this activity first, but this leaves no room for the other 2 (and we want to optimize for the number of activities)
Activity Scheduling, Heuristic 3
Copyright Ph.Dutré,Spring 2015 12
Schedule the activity with the fewest conflicts This can lead to suboptimal choices: We schedule the central activity and two others, instead
of the top four.
We schedule this activity first (fewest conflict), and we then only have place for 2 others (and we want to optimize for the number of activities)
Activity Scheduling, Heuristic 4
Copyright Ph.Dutré,Spring 2015 13
Schedule the non-conflicting activity that finishes first
1st 2nd 3rd 4th
Activity Scheduling, Heuristic 4
Copyright Ph.Dutré,Spring 2015 14
Let A = {a1, … , an} be the set of activities selected by heuristic 4, sorted by finish time. The activities do not overlap, so they are also sorted by start time.
Let A’ = {a’1, … , a’n} be an optimal set of activities. Suppose a1 != a’1:
By heuristic 4, a1 is the first activity to finish, so it must finish before a’1 . Therefore, a1 cannot conflict with a’2 (since A’ is a solution the problem),
and the schedule {a1, a’2, … , a’n}, is also a valid optimal schedule (same number of activities as A’).
By induction, we can keep replacing a’i 's with ai, while maintaining an optimal schedule. Therefore A is an optimal solution.
Real problem: File Compression
Copyright Ph.Dutré,Spring 2015 15
Computers usually encode characters using a fixed number of bits: ASCII: 7 bits per character (127 characters) ISO-8859-1: 8 bits per character (256 characters) Unicode: 16 bits per character (64k characters) [ ... Unicode 7.0 June 2014]
But: Most documents contain fewer than 64k distinct characters
UTF-8 use fewer bits to represent common characters? Roman alphabet (ASCII): 1 byte per character Accented letters, Greek, Cyrillic, Arabic, Hebrew, etc.: 2 bytes Chinese, etc.: 3 bytes Obscure Chinese characters, historic characters, etc.: 4 bytes.
Cfr. Morse Code Assign short codes to common letters e = . , t = _ , ? = .._ _.. On average, Morse code yields 1.66:1 compression for English text
File Compression
Copyright Ph.Dutré,Spring 2015 16
No algorithm can compress every bitstring
Proof 1 [by contradiction] Suppose you have a universal data compression algorithm U
that can compress every bitstring. Given bitstring B0, compress it to get smaller bitstring B1. Compress B1 to get a smaller bitstring B2. Continue until reaching bitstring of size 1.
Implication: all bitstrings can be compressed to 1 bit!
Proof 2 [by counting] Suppose your algorithm can compress all 1,000-bit strings. 21000 possible bitstrings with 1,000 bits. Only 1 + 2 + 4 + … + 2998 + 2999 = 21000 -1 can be encoded with
≤ 999 bits.
Run-Length Encoding
Copyright Ph.Dutré,Spring 2015 18
Representation. Use 4-bit counts to represent alternating runs of 0s and 1s: 15 0s, then 7 1s, then 7 0s, then 11 1s.
Huffman Coding
Copyright Ph.Dutré,Spring 2015 20
How can we compress more than Morse code? Pick the codes for each character in a specific text We can usually get 2:1 compression for English text
How do we know the length of each code? Morse code uses dots and dashes for letters, and gaps to end codes Computers have only 1s and 0s (no gaps) Prefix codes: No code is a prefix of any other code
e.g. If e = 101, no other letter's code can start with 101
Simple tree representation for prefix codes:
Optimal prefix codes
Copyright Ph.Dutré,Spring 2015 21
Best code for each letter? Intuition: length of code should be proportional to letter's
frequency
Construct prefix tree by greedily linking two least frequent characters Combined characters form “new character” with frequency =
sum of frequencies of linked characters.
Implementation requires a priority queue: ~n.log2 n
Proof of optimality
Copyright Ph.Dutré,Spring 2015 23
Lemma 1: Any optimal code tree is complete.
(nodes with only 1 child are impossible)
Proof of optimality
Copyright Ph.Dutré,Spring 2015 24
Lemma 2: There exists an optimal subtree in which the two least frequent letters are
siblings at the maximum depth. Proof:
Every leaf node has a sibling or the tree would not be full (lemma 1). The two least frequent letters x and y must be at the maximum depth.
If not, there must be a letter w at maximum depth that occurs more frequently, and that would not be optimal.
If x and y are not siblings, swap y with x's sibling. The number of bits for all letters remains unchanged.
Proof of optimality
Copyright Ph.Dutré,Spring 2015 25
Base Case:
Induction: Assume an optimal tree where the two least frequent letters x and y are merged into a single letter z
Hypothesis: The tree created by adding x and y as children of z is optimal
Proof of optimality
Copyright Ph.Dutré,Spring 2015 26
Contradiction: Expanded tree is not optimal. So start from optimal expanded tree and merge x and y