Greedy Algorithms and Heuristics

Optimization problems, Greedy Algorithms, Optimal Substructure and Greedy choice

Learning & Development Teamhttp://academy.telerik.com

Telerik Software Academy

Table of Contents1. Optimization Problems2. Greedy Algorithms and Failure

Cases3. Optimal Greedy Algorithms

Optimal Substructure and Greedy Choice

Demo: Proving Optimality of a Greedy Solution

4. The Set Cover Problem5. Notable Greedy algorithms

2

Optimization ProblemsNot "just" Looking for a Solution

Optimization Problems Finding the best solution of a problem From all solution candidates i.e. most optimal solution candidate Not just any solution

Two categories – based on variables Continuous – variable values are

continuous Discrete – a.k.a. Combinatorial 4

Optimization Problems Importance of optimization problems Optimal solutions reduce cost Optimal solutions are more likely to

be realistically possible E.g. finding any route between to a city In theory enables getting there In practice it might be too long to

travel Finding the shortest route is much

better

Greedy AlgorithmsPicking Locally Best Solution

Greedy Algorithms Greedy algorithms are a category of algorithms Can solve some optimization

problems Usually more efficient than all other

algorithms For the same problems

Greedy algorithms pick The best solution From their current position & point

of view i.e. they make local solutions

Greedy Algorithms: Example

Playing against someone, alternating turns

Per turn, you can take up to three coins

Your goal is to have as much coins as possible

Greedy Algorithms: Example

Things to notice in the way you played to winAlways take the max number of coins

i.e. make the current optimal solution

You don't consider what the other player does

You don't consider your actions' consequences

The greedy algorithm works optimally here

Greedy Algorithms A greedy algorithm solves a problem in steps At each step

Algorithm picks the best action available

Best action is determined regardless of future actions or states

i.e. greedy algorithms assume Always choosing a local optimum Leads to the global optimum

10

Greedy Algorithms Main components of a Greedy algorithm A candidate set A selection function A feasibility function An objective function and a solution

function Basically, a greedy algorithms have

A set of possible actions A way to pick the best action at a

time A way to determine the solution is

reached

Greedy Algorithms Greedy algorithms exploit problem structure E.g. where the solution is the sum

of the optimal solutions to subproblems

E.g. where greedy choices don't lead to bad overall positions

Many currency systems' denomination units are suited to greedy processing Ironic, isn't it?

Greedy Algorithms: Example

Consider the US currency denominations

Problem: gather a sum of money, using the least possible number of bills/coins Suppose you have infinite supplies

of each denomination1¢

5¢

10¢

25¢

50¢

1$Sum to make: 1.29$

1$

25¢

1¢

1¢

1¢ 1¢

Greedy Algorithms: Example

Greedy algorithm to make a sum With minimum number of coins Given the US currency system

This will work for most 1-2-5 series based systemsWe want to achieve the sum S

We start with the sum Q = 0We have a set of coins C = { 1, 5, 10 … }At each step• pick the largest value V from C

such that Q + V less than or equal to S

• Increase Q by V //i.e. add a coin of value VRepeat until Q == S//the number of repetitions is the number of needed coins

Greedy for Sum of CoinsLive Demo

Greedy Failure Cases Greedy algorithms are often not optimal Even can reach the unique worst

possible solutions for some problems

Example: Largest sum path (starting at top)

Example: Coin denominations 4, 10, 25; Greedy will fail to make the sum 41which is 25 + 4 * 4

Optimal Greedy AlgorithmsOptimal Substructure, Greedy Choice

Property, Proving Optimality of a Greedy Approach

Optimal Greedy Algorithms

Suitable problems for greedy algorithms often have these properties: Greedy choice property Optimal substructure

Any problem having the above properties Guaranteed to have an optimal

greedy solution Matroids – way to prove greedy optimality If a problem has the properties of a

matroid, it is guaranteed to have an optimal greedy solution

Optimal Greedy Algorithms

Greedy choice property An optimal solution to the problem

begins with a greedy choice Subproblems that arise can be

solved by consequent choices Also enforced by optimal

substructure

Optimal Greedy Algorithms

Optimal substructure After each greedy choice The problem remains an

optimization problem Of the same form as the original

problem i.e. the optimal solution to the

problem contains optimal solutions to the subproblems

Solving a Problem Optimally with

GreedyGreedy for the Activity Selection Problem and Proving its Optimality

Proving Greedy Optimality

The Activity Selection Problem (a.k.a. Conference Scheduling Problem) Given a set of activities S = {a1, a2, … an} Each with a start & finish time: ai = (si, fi)

Activities are "compatible" if they don't overlap i.e. their intervals do not intersect

What is the maximum-size subset of compatible activities? i.e. which is the largest list of

compatible activities that can be scheduled

Proving Greedy Optimality

The Activity Selection Problem Can have several optimal solutions

In the following case {a1, a4, a8, a11} is optimal

Another optimal is {a2, a4, a9, a11}Index 1 2 3 4 5 6 7 8 9 10 11

Start (si) 1 3 0 5 3 5 6 8 8 2 12Finish (fi) 4 5 6 7 8 9 10 11 12 13 14

Proving Greedy Optimality

A greedy algorithm for the task:

Greedy characteristic of above algorithm Taking the earliest finish activity

gives more time for other activities i.e. choose the "maximum

remaining time"

• Select activity with the earliest finish from S

• Remove activities in S conflicting with selected

• i.e. non-compatible activities are removed

• Repeat the until no activities remain in S

Proving Greedy Optimality

To prove the discussed greedy is optimal Need to prove the problem has a

greedy choice property Need to prove the problem has

optimal substructure Observations

1. If there exists an optimal solution, it is a subset of activities from S

2. In any solution, the first activity to start is the first to finish

Proving Greedy Optimality

Let A be an optimal solution (subset of S) Sort activities in A by finish time.

Let k be the index of the earliest activity in A

If k = 1 => A begins with a greedy choice

If k != 1 => Let B = (A – {k}) + {1}. Prove B is

optimal: f1 <= fk (from sorting and k > 1) => f1 doesn't overlap any activity in B, so B is a solution

B has the same size (number of activities) as A Hence, B is also optimal

Proving Greedy Optimality

So far we proved that: A solution starting with a greedy

choice exists The greedy choice solution is also

optimal Hence we proved the problem exhibits the greedy choice property There exists an optimal solution,

starting with a greedy choice Now, we need to prove the problem has optimal substructure

Proving Greedy Optimality

We have selected activity 1 (greedy 1st choice) Thus, we reduced to the same

problem form Without activities in S which

intersect activity 1 If A is optimal to S, then:

A' = A – {1} is optimal to S' = all activities of S which start after f1

Can A' be non-optimal to S' ? If exists B' with more activities than A' (from S')

Adding activity 1 to B' gives B with more activities (from S) than A -> contradiction

Proving Greedy Optimality

We just proved the problem has optimal substructure Each greedy choice leads us to a

problem of the same form The new problem's solution is a

subset of the initial problem's solution

i.e. all local solutions joined form the global optimal solution

We have proven both properties, so our greedy algorithm is optimal

Activity Selection ProblemLive Demo

The Set Cover ProblemUsing Greedy for Approximation

Set Cover Problem Greedy algorithms can sometimes find optimal solutions This is not their only application

There exist problems, for which An optimal solution is to complex to

find i.e. calculating an optimal solution

is infeasible Sometimes greedy algorithms

provide good approximations of the optimal result

Set Cover Problem The Set Cover Problem (SCP) is such a problem

SCP formulation: Given a set {1,2,…,m} called "the

Universe" (U) And a set S{{…},…} of n sets whose

union = U Find the smallest subset of S, the

union of which = U (if it exists) So, we have a target set, and a set

of sets with repeating elements (i.e. redundant elements) How do we find the smallest number

of sets, which in union make the target set

Set Cover Problem The SCP turns out very complex

The optimal solution is NP-complete i.e. infeasible for calculations

(unless P = NP) However, relatively good solutions can be achieved through a greedy approach: At each step, pick the set

containing the largest number of uncovered elements

Notable Greedy Algorithms

Several Common Greedy Algorithms

Notable Greedy Algorithms

Dijkstra's algorithm for finding the shortest path between two vertices in a weighted graph (with no

negative cycles) At each step, of all reached edges, pick: the one that, along with the path to

it, constitutes a minimal sum The algorithms is proven optimal

Immediately after it reaches a vertex, the path generated to it is guaranteed to be optimal i.e. no need to traverse all vertices

Notable Greedy Algorithms

Prim and Kruskal's algorithms for a minimum spanning tree (MST) are greedy algorithms Prim:

pick the smallest edge, not in the MST so far

Kruskal: pick the smallest edge, connecting

two vertices, not in the same tree Both algorithms have the same

complexity

Notable Greedy Algorithms

The prefix tree generation algorithm in Huffman coding is greedy Greedy: pick the

two smallest-value leaves/nodes and combine them

Left move: 0,Right move: 1

46 15

27

54100

0

0

0

0

01

1

11

1 A: 00B: 100C: 01D: 1010E: 11F: 1011

Frequency % 22 12 24 6 27 9Symbol A B C D E F

Huffman CodingLive Demo

форум програмиране, форум уеб дизайнкурсове и уроци по програмиране, уеб дизайн – безплатно

програмиране за деца – безплатни курсове и уроцибезплатен SEO курс - оптимизация за търсачки

уроци по уеб дизайн, HTML, CSS, JavaScript, Photoshop

уроци по програмиране и уеб дизайн за ученициASP.NET MVC курс – HTML, SQL, C#, .NET, ASP.NET MVC

безплатен курс "Разработка на софтуер в cloud среда"

BG Coder - онлайн състезателна система - online judge

курсове и уроци по програмиране, книги – безплатно от Наков

безплатен курс "Качествен програмен код"

алго академия – състезателно програмиране, състезанияASP.NET курс - уеб програмиране, бази данни, C#, .NET, ASP.NET

курсове и уроци по програмиране – Телерик академия

курс мобилни приложения с iPhone, Android, WP7, PhoneGapfree C# book, безплатна книга C#, книга Java, книга C# Дончо Минков - сайт за програмиране

Николай Костов - блог за програмиранеC# курс, програмиране, безплатно

?? ? ?

??? ?

?

? ?

??

?

?

? ?

Questions?

?

Greedy Algorithms

http://algoacademy.telerik.com

References & Further Reading

The following materials were very helpful in the making of this lecture and we recommend them for further reading: Lecture @ Boston University (Shang-

Hua Teng) http://www.cs.bu.edu/~

steng/teaching/Fall2003/lectures/lecture7.ppt

Lecture @ University of Pennsylvania http://www.cis.upenn.edu/~

matuszek/cit594-2005/Lectures/36-greedy.ppt

Book on Algorithms @ Berkeley University http://www.cs.berkeley.edu/~

vazirani/algorithms/chap5.pdf