Randomized QS. Ch5, CLRS Randomized QS. Ch5, CLRS Dr. M. Sakalli, Marmara UniversityDr. M. Sakalli, Marmara UniversityPicture 2006, RPIPicture 2006, RPI

2-2M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Randomized AlgorithmRandomized Algorithm

o Randomize the algorithm so that it works well with Randomize the algorithm so that it works well with high probability on all inputshigh probability on all inputs

o In this case, randomize the order that candidates In this case, randomize the order that candidates arrivearrive

o Universal hash functions: randomize selection of Universal hash functions: randomize selection of hash function to usehash function to use

2-3M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Probabilistic Analysis and Probabilistic Analysis and Randomized AlgorithmsRandomized Algorithmso Hiring problemHiring problem

n candidates interviewed for a positionn candidates interviewed for a position One interview each day, if qualified hire the new, fire the current employed.One interview each day, if qualified hire the new, fire the current employed. Best = Meet the most qualified one at the 1Best = Meet the most qualified one at the 1 stst interview. interview. for k = 1 to nfor k = 1 to n

Pay Pay ccii interview cost;. interview cost;.

if candidate(k) is better if candidate(k) is better Sack current, bestSack current, bestk; // sack current one and hire(k); k; // sack current one and hire(k);

Pay Pay cchh hiring (and sacking) cost; hiring (and sacking) cost;

o Worst-case cost is increased quality: Worst-case cost is increased quality: ncnch h + + ncnchh, , cchh>> c cii, then O(, then O(ncnchh) )

o Best-case, the least cost: Best-case, the least cost: ((n-1) ((n-1) cci i + + cchh), prove that hiring problem is ), prove that hiring problem is (n)(n)

o Suppose applicants arriving in Suppose applicants arriving in a random qualification ordera random qualification order and and suppose that suppose that the randomness is equally likelythe randomness is equally likely to be any one of the to be any one of the nn! ! of permutations 1 through of permutations 1 through n.n.

o A A uniform random permutationuniform random permutation..

2-4M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Probabilistic Analysis and Randomized AlgorithmsProbabilistic Analysis and Randomized Algorithms

o An algorithm is randomizedAn algorithm is randomized, not just if , not just if its behaviorsits behaviors is is controlled by but if the output is generated controlled by but if the output is generated by a random-by a random-number generatornumber generator..

o C rand(a,b), with boundaries are inclusive.. Equally likely C rand(a,b), with boundaries are inclusive.. Equally likely (probable) outputs of X = {X(probable) outputs of X = {Xii} for i=1:n.} for i=1:n.

o Expected value: E[X] = Expected value: E[X] = ΣΣiin n xxii * (probability density of each * (probability density of each

x) = E[X] = x) = E[X] = ΣΣiin n xxii Pr{x=x Pr{x=xii}}

o In the hiring case, In the hiring case, o Defining an indicator, Indicator random variable, XDefining an indicator, Indicator random variable, X ii = I = I

{ indicator of candidate if hired (or of coin), 1(H) or 0(T)}{ indicator of candidate if hired (or of coin), 1(H) or 0(T)}

2-5M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o H and T, three fair coins, head 3$s, every tail H and T, three fair coins, head 3$s, every tail 2$s, and expected value of earnings. 2$s, and expected value of earnings.

o HHHHHH = 9$s, 1/8, = 9$s, 1/8, o HHTHHT= 4$s, 3/8,= 4$s, 3/8,o HTTHTT = -1$s, 3/8,= -1$s, 3/8,o TTTTTT = -6$s, 1/8= -6$s, 1/8

o E[earnings]= 9/8+12/8-3/8-6/8=12/8=1.5E[earnings]= 9/8+12/8-3/8-6/8=12/8=1.5

2-6M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Probabilistic Analysis and Randomized AlgorithmsProbabilistic Analysis and Randomized Algorithms

o Lemma 1. Given a sample Lemma 1. Given a sample space Sspace S, and , and an event an event ee in the sample space, let in the sample space, let XXee=I(e)=I(e) be indicator of be indicator of occurrence of e, then, E[Xoccurrence of e, then, E[Xee]=Pr{e}.]=Pr{e}.

Proof: Fr the definition of expected value, Proof: Fr the definition of expected value, E[XE[XAA] = ] = ΣΣee

n n xxee Pr{x=e} = 1* Pr{X Pr{x=e} = 1* Pr{Xee} + 0 * Pr{notX} + 0 * Pr{notXee} }

= Pr{X= Pr{Xee}, where note=S-e. }, where note=S-e.

o In the binary case, equally distributed, uniform In the binary case, equally distributed, uniform distribution, E[e] = Pr{Xdistribution, E[e] = Pr{Xee} = 1/2. } = 1/2.

2-7M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o X = X = ΣΣj=1j=1nn{x{xii}}

o Expected value of a candidate been hired is the Expected value of a candidate been hired is the probability of candidate hired. probability of candidate hired. E[xE[xjj]= Pr{if candidate(i) is hired}, ]= Pr{if candidate(i) is hired}, Candidate i is hired if better than previous i-1 candidates. Candidate i is hired if better than previous i-1 candidates. The probability of being better is 1/i. The probability of being better is 1/i. Then, E[xThen, E[xii] = Pr{x] = Pr{xii}= 1/i}= 1/i

o Expected value of hiring (the average number of Expected value of hiring (the average number of hired ones out of n arriving candidates in random hired ones out of n arriving candidates in random rank) E[X]. Uniform distribution, equally likely, 1/i. rank) E[X]. Uniform distribution, equally likely, 1/i.

2-8M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o Expected value of hiring (the average number of hired ones out of n arriving Expected value of hiring (the average number of hired ones out of n arriving candidates in random rank) E[X]. Uniform distribution, equally likely, 1/i. candidates in random rank) E[X]. Uniform distribution, equally likely, 1/i.

o E[X] = E[E[X] = E[ΣΣj=1j=1n n IIii Pr{x Pr{xii}], I}], Iii={1, 0} indicator random value here.={1, 0} indicator random value here.

o E[X] = E[X] = ΣΣiinnE[xE[xii], ], from linearity of expected valuefrom linearity of expected value. . . .

o ΣΣi=1:(n)i=1:(n)(1/i) = 1+1/2+1/3…, harmonic number (divergent).. (1/i) = 1+1/2+1/3…, harmonic number (divergent).. Int 1 to (n+1), Int 1 to (n+1), (1/x)=ln(n+1)<=ln(n)+O(1)(1/x)=ln(n+1)<=ln(n)+O(1)

o ΣΣi=2:(n)i=2:(n)(1/i) = 1/2+1/3….. (1/i) = 1/2+1/3….. Int 1 to (n+1),(1/x) =ln(n)<=ln(n)Int 1 to (n+1),(1/x) =ln(n)<=ln(n)o ΣΣi=1:(n)i=1:(n)(1/i) = 1+1/2+1/3….. (1/i) = 1+1/2+1/3….. ln(n)+1 ln(n)+1

o E(X) = ln(n)+O(1), E(X) = ln(n)+O(1),

o Expected value of all hirings.. Expected value of all hirings.. Upper boundary is ln(n).Upper boundary is ln(n).

o Lemma5.2: When candidates presented in random, the cost of hiring is Lemma5.2: When candidates presented in random, the cost of hiring is O(cO(chhlgn). Proof from Lemma5.1.lgn). Proof from Lemma5.1.

o How to randomize.. Some random outputs of permutations will not be How to randomize.. Some random outputs of permutations will not be random. Would it matter?.. random. Would it matter?..

2-9M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o In the case of dice, Pr{heads}= 1/2, in which case, for n tries, E[X] = In the case of dice, Pr{heads}= 1/2, in which case, for n tries, E[X] = E[E[ΣΣj=1j=1

nn{1/2}] =n/2. {1/2}] =n/2.

o Biased CoinBiased Coin Suppose you want to output 0 and 1 with the probabilities of 1/2Suppose you want to output 0 and 1 with the probabilities of 1/2 You have a coin that outputs 1 with probability p and 0 with probability 1-p for You have a coin that outputs 1 with probability p and 0 with probability 1-p for

some unknown 0 < p < 1some unknown 0 < p < 1 Can you use this coin to output 0 and 1 fairly?Can you use this coin to output 0 and 1 fairly? What is the expected running time to produce the fair output as a function of p?What is the expected running time to produce the fair output as a function of p?

o Let SLet Sii be the probability that we successfully hire the best qualified be the probability that we successfully hire the best qualified candidate AND this candidate was the ith one interviewedcandidate AND this candidate was the ith one interviewed

o Let M(j) = the candidate in 1 through j with highest scoreLet M(j) = the candidate in 1 through j with highest score

o What needs to happen for SWhat needs to happen for Sii to be true? to be true? Best candidate is in position i: BBest candidate is in position i: B ii

No candidate in positions k+1 through i-1 are hired: ONo candidate in positions k+1 through i-1 are hired: O ii

These two quantities are independent, so we can multiply their probabilities to get These two quantities are independent, so we can multiply their probabilities to get SSii

2-10M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o Characterizing the running time of a randomized algorithm.Characterizing the running time of a randomized algorithm.o E[T(n)] = E[T(n)] = ΣΣj=1j=1

nn ttiiPr{Pr{ttii}}

void qksrt( vector<int> & data)void qksrt( vector<int> & data){{

RandGen rand;RandGen rand;qksrt(data, 0, data.lngth() - 1, rand );qksrt(data, 0, data.lngth() - 1, rand );

}}

void qksrt( vector<int> & dt, int srt, int end, RandGen & rand )void qksrt( vector<int> & dt, int srt, int end, RandGen & rand ){ {

if( start < end )if( start < end ){{

int bnd = partition(dt, srt, end, rand.RandInt(srt, end ) );int bnd = partition(dt, srt, end, rand.RandInt(srt, end ) ); qksrt(dt, start, bnd, rand);qksrt(dt, start, bnd, rand); qksrt(dt, bnd + 1, end, rand );qksrt(dt, bnd + 1, end, rand );}}

}}

2-11M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o we are equally likely to get each possible split, since we are equally likely to get each possible split, since we choose the pivot at random.we choose the pivot at random.

o Express the expected running time as:Express the expected running time as:o T(0) = 0;T(0) = 0;

o T(n) = T(n) = ΣΣk=0k=0nn-1-1

1/n[T(k)+T(n-k+1)+n]; 1/n[T(k)+T(n-k+1)+n];

o T(n) = n + T(n) = n + ΣΣk=0k=0nn-1-1

1/n[T(k)+T(n-k+1)]; 1/n[T(k)+T(n-k+1)];o Bad split, choosing from 1Bad split, choosing from 1stst or 4 or 4thth quarter, that is quarter, that is

ii kk ii+ + ¼[ ¼[jj--ii+1] or +1] or j j - ¼[- ¼[jj--ii+1] +1] kk jj o Good split Good split i i + ¼[+ ¼[ii--jj+1] +1] kk j j - ¼[- ¼[jj--ii+1] +1]

2-12M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o Mixing good and bad coincidence with equal likelihood. Mixing good and bad coincidence with equal likelihood. o T(n) = n + T(n) = n + ΣΣk=0k=0

nn-1-1 1/n[T(k)+T(n-k+1)]; 1/n[T(k)+T(n-k+1)];

o T(n) = n + (2/n) T(n) = n + (2/n) ΣΣj=n/2j=n/2nn-1-1

[T(k)+T(n-k+1)]; [T(k)+T(n-k+1)];= n + (2/n){= n + (2/n){ΣΣk=n/2k=n/2

33nn/4/4 [T(k)+T(n-k+1)] + [T(k)+T(n-k+1)] + ΣΣk=3k=3nn/4/4

nn-1-1 [T(k)+T(n-k+1)]}; [T(k)+T(n-k+1)]};

n + (2/n){ n + (2/n){ ΣΣk=n/2k=n/233nn/4/4

[T(3n/4)+T(n/4)] + [T(3n/4)+T(n/4)] + ΣΣk=3k=3nn/4/4nn-1-1

[T(n-1)+T(0)]}; [T(n-1)+T(0)]}; n + (2/n)(n/4){[T(3n/4)+T(n/4)] + [T(n-1)]};n + (2/n)(n/4){[T(3n/4)+T(n/4)] + [T(n-1)]}; n +(1/2){[T(3n/4)+T(n/4)] + [T(n-1)]};n +(1/2){[T(3n/4)+T(n/4)] + [T(n-1)]};

o Prove that for all n T(n) Prove that for all n T(n) cnlog(n), T(n) is the statement cnlog(n), T(n) is the statement obtained above. Inductive proof, for n=0, and n=n, obtained above. Inductive proof, for n=0, and n=n,

o Probability of many bad splits is very small. With high Probability of many bad splits is very small. With high probability the list is divided into fractional pieces which is probability the list is divided into fractional pieces which is enough balance to get asymptotic n log n running time. enough balance to get asymptotic n log n running time.

n + (1/2)[c(3n/4) log(3n/4) + c(n/4) log(n/4)] +(1/2)c(n-1)log(n-1) n + (1/2)[c(3n/4) log(3n/4) + c(n/4) log(n/4)] +(1/2)c(n-1)log(n-1)

2-13M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

o MIT notessMIT notesso LL((nn)= 2)= 2UU((nn/2) + Θ(/2) + Θ(nn) ) luckyluckyo UU((nn)= )= LL((n n –1) + Θ(–1) + Θ(nn) ) unluckyunluckyo LL((nn)= 2)= 2UU((nn/2 –1) + 2Θ(/2 –1) + 2Θ(n/2n/2)+ Θ()+ Θ(nn))o LL((nn)= 2)= 2UU((nn/2 –1) + Θ(/2 –1) + Θ(nn))o LL((nn)= Θ()= Θ(nlognnlogn))o And there is more there.. And there is more there..

2-14M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Computing SComputing S

o BBii = 1/n = 1/n

o OOii = k/(i-1) = k/(i-1)

o SSii = k/(n(i-1)) = k/(n(i-1))

o S = S = i>ki>k S Sii = k/n = k/n i>ki>k 1/(i-1) is probability of success 1/(i-1) is probability of success

o k/n (Hk/n (Hnn – H – Hkk): roughly k/n (ln n – ln k)): roughly k/n (ln n – ln k)

o Maximized when k = n/eMaximized when k = n/eo Leads to probability of success of 1/eLeads to probability of success of 1/e