Characterization of Truthful MAB Mechanisms for Multi-Slot...

IntroductionSponsored Search Auctions and MAB Mechanisms

Conclusion

Characterization of Truthful MAB Mechanisms forMulti-Slot Sponsored Search Auctions

Sujit Prakash Gujar

Advisor : Y NarahariE-Commerce Lab

Department of Computer Science and AutomationIndian Institute of Science, Bangalore-12

Joint Work with Akash Das SarmaPresented At Xerox Research Center Europe, Grenoble

August 10, 2010

Sujit Prakash Gujar (CSA, IISc) Truthful MAB Mechanisms August 10, 2010 1 / 42


Conclusion

Mechanism DesignSpace of Mechanisms

Agenda

Introduction to Mechanism Design

Sponsored Search Auctions

Multi-Armed Bandit (MAB) Mechanisms

State of the Art

Research Gaps

Our Contributions and Results

Directions for Future Work



Conclusion


Mechanism Design

Game Theory: Analysis of strategic interaction among players

Mechanism Design: Reverse engineering of game theory

Mechanism Design is the art of designing rules of a game to achievea specific outcome in presence of multiple self-interested agents,each with private information about their preferences.



Conclusion


Mechanism Design






Conclusion


Mechanism Design






Conclusion




Conclusion




Conclusion




Conclusion




Conclusion




Conclusion


Incentive Compatibility and Individual Rationality

Individual Rationality

No agent is worse off by participating in the mechanism



Conclusion


Auctions

First Price Auction (FPA) for selling a single item.

The bidder with the highest bid wins.She pays as much as she has bid for.

Second Price Auction (SPA) for selling a single item.

The bidder with the highest bid wins.She pays as much as the second highest bid.

Vickrey 1 showed : The truth revelation is dominant strategy insecond price auction.

1W. Vickrey. Counter speculation, auctions, and competitive sealed tenders.Journal of Finance, 16(1):8-37, March 1961.



Conclusion


Auctions


The bidder with the highest bid wins.

She pays as much as she has bid for.







Conclusion


Auctions









Conclusion


Auctions









Conclusion


Auctions




The bidder with the highest bid wins.

She pays as much as the second highest bid.





Conclusion


Auctions









Conclusion


Auctions









Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion


Space of Mechanisms



Conclusion

Sponsored Search AuctionsState-of-the-ArtResearch GapsOur Approach




Conclusion



A Successful Application of Mechanism Design in E-Commerce:Sponsored Search Auctions

Most popular sites: search engines viz. Google, Yahoo!, Bing

Search engines display ads relevant to the search query

Typically limited number of slots for display of such slots

There is an auction running behind the scene for each keyword fordisplay of such ads



Conclusion


Challenges in Allocating the slots to the Agents

Billions of dollars

Consider an example

2 agents competing for a single slotAgent 1 bids $1 and Agent 2 bids $0.7

The probability of receiving click on ad for agent 1 is 0.5and agent 2 receives a click if her ad is displayedDisplaying ad of the agent 2 is beneficial for the search engine

The probability of ad receiving a click is referred to asClick-Through-Rate (CTR)

While allocating one should consider the CTRs



Conclusion



Billions of dollars

Consider an example

2 agents competing for a single slotAgent 1 bids $1 and Agent 2 bids $0.7The probability of receiving click on ad for agent 1 is 0.5and agent 2 receives a click if her ad is displayedDisplaying ad of the agent 2 is beneficial for the search engine





Conclusion



Billions of dollars

Consider an example

2 agents competing for a single slotAgent 1 bids $1 and Agent 2 bids $0.7The probability of receiving click on ad for agent 1 is 0.5and agent 2 receives a click if her ad is displayedDisplaying ad of the agent 2 is beneficial for the search engine





Conclusion


Need for Mechanism Design

Agents are strategic

It is critical to know the value that an agent obtains for each clickshe receives, but this is private information

× Popular Mechanisms GFP and GSP are not truthful

Mechanism Design is a natural tool

Design of truthful sponsored search auctions: Aggrawal et. al.2

Design of optimal sponsored search auctions: Garg and Narahari 3

2G. Aggarwal, A. Goel, and R. Motwani. Truthful auctions for pricing searchkeywords. ACM EC’06.

3D. Garg and Y. Narahari. An Optimal Mechanism for Sponsored Search Auctionsand Comparison with other Mechanisms. In, IEEE Transactions on AutomationScience and Engineering, 2009.



Conclusion


Need for Combining Mechanism Design with MachineLearning

The previous work relied on the knowledge of the CTRs

× Generally neither the search engine nor the advertisers haveknowledge of the CTRs for the ads

Typically the same set of advertisers compete for a particularkeyword

The CTRs can be learnt over the course of repeated auctions

When there is a single slot, the problem is the same as Multi-ArmedBandit (MAB) Problem



Conclusion


Multi-Armed Bandit (MAB) Mechanism

Regret: Measure of a performance of a learning algorithm

Regret = Worst case Loss in Social Welfare

One would like to learn CTRs with minimal regret

This calls for combining techniques from Mechanism Design Theoryand Machine Learning

Agents may manipulate the learning algorithm for the underlyingMAB problem

Goal is to design an auction mechanism that is truthful as well aslearns CTRs

Such mechanisms are called as MAB Mechanisms



Conclusion


State of The Art

If agents are not strategic: R(T ) = O(T 1/2)

Nikhil Devanur and Sham M. Kakade, The price of truthfulness forpay-per-click auctions. In Proceedings of the 10th ACM Conferenceon Electronic Commerce, pages 99-106, 2009.

Moshe Babaioff, Yogeshwer Sharma, and Aleksandrs Slivkins,Characterizing Truthful Multi-Armed Bandit Mechanisms, InProceedings of the 10th ACM Conference on Electronic Commerce,pages 79-88, 2009.

Onno Zoeter, On a form of advertiser cheating in sponsored searchand a dynamic-VCG solution. In Proceedings of Workshop onTargeting and Ranking for Online Advertising, TROA, 2008.



Conclusion


State of The Art







Conclusion


State of The Art







Conclusion


State of The Art







Conclusion


Research Gaps

MAB Mechanisms for sponsored search: Previous work assumes asingle slot

The techniques do not immediately generalize to multi-slot case

The problem that we address

Goal

Characterize truthful Multi-Armed Bandit (MAB) mechanisms for theallocation of advertisers to multiple slots in sponsored search auctionsunder various assumptions on Click-Through Rates (CTRs).



Conclusion


Research Gaps




Goal




Conclusion


Research Gaps




Goal




Conclusion


Research Gaps




Goal




Conclusion


Our Contributions

Generalize notion of pointwise monotonicity: Strong Monotonicityand Weak Monotonicity

Introduce notion of Type-I Separatedness and Type-IISeparatedness

Characterization of truthful MAB mechanisms for unknown andunconstrained CTRs

Necessary conditions for a MAB mechanism to be truthful withvarious assumptions on CTRs

Sufficient conditions for a MAB mechanism to be truthful withvarious assumptions on CTRs



Conclusion


Notation

m Number of slots

i Index of an agent, i = 1, 2, . . . , k

j Index of a slot, j = 1, 2, . . . , m

T Total number of rounds

t A particular round. t ∈ 1, 2, . . . , TAij(t) = 1 if an agent i is allocated slot j in round t

= 0 otherwiseA(t) (Aij(t))i∈K ,j∈M

A = (A(1), A(2), . . . , A(T )), Allocation rule

ρij(t) = 1 if agent i gets a click in slot j in round t= 0 otherwise

ρ(t) (ρij(t))i∈K ,j∈M

ρ = (ρ(1), ρ(2), . . . , ρ(T ))

vi Agent i ’s valuation of a click to her ad

bi Bid by agent i

b Bid vector, indicating bids of all the agents= (bi , b−i ) = (b1, b2, . . . , bk)



Conclusion


Ci (b, ρ) Total number of clicks obtained by an agent iin T rounds

Pi (b, ρ) Payment made by agent i

P(b, ρ) = (P1(. ), P2(. ), . . . , Pk(. )), Payment rule

Ui (vi , b, ρ) Utility of an agent i in T rounds= viCi (b, ρ)− Pi (b, ρ)

b+i A real number > bi

αi Click probability associated with agent i

βj Click probability associated with slot j

µij The probability that an ad of an agent i receivesclick when the agent is allotted slot j .

N(b, ρ, i , t) Set of slot agent pairs in round tthat influence agent i in some future rounds

Table: Notation



Conclusion


Myerson4 designed an optimal auction for the seller.

He characterized the truthful mechanisms

Technique developed by him applies to most single parameterdomains

Myerson Characterization: Let (A,P) be a normalized mechanismfor the MAB mechanism design problem. It is truthful withunrestricted payment computation if and only if for any givenrealization ρ the corresponding click-allocation C is non-decreasingand the payment rule is given by,

Pi (bi , b−i ; ρ) = bi · Ci (x , b−i ; ρ)−∫ bi

0

Ci (x , b−i ; ρ)dx

4R. B. Myerson. Optimal auction design. Math. Operations Res., 6(1):58-73, Feb.1981.



Conclusion


Characterization of Truthful MAB mechanism for singleslot case

Babaioff5 et. al. showed,

Theorem

Consider the MAB mechanism design problem. Let A be anon-degenerate, deterministic allocation rule. Then mechanism (A,P) isnormalized and truthful for some payment rule P if and only if A ispointwise monotone and weakly separated.

5M. Babaioff, Y. Sharma, and A. Slivkins. Characterizing truthful multi-armedbandit mechanisms. ACM EC’09



Conclusion


Allocation Rule Properties

Normalized: Payment is non negative and no agent pays more thanwhat she has bid for. That is,

∀ i ,∀ b, ρ, Pi (·) ≥ 0 and Pi (·) ≤ bi · Ci (·)

Non-degenerate: For any agent at each bid, small perturbation inbid does not affect her allocation. That is, ∀ i ,∀ bi ∀b−i , ρ, ∃ I 3 bi such that,

Ai (x , b−i ; ρ) = Ai (bi , b−i ; ρ) ∀x ∈ I



Conclusion


Monotonicity

Definition (Strong Pointwise Monotonicity)

An allocation rule is said to be strongly pointwise monotone if it satisfies:For any fixed (b−i , ρ), if an agent i with bid bi is allocated a slot j inround t, then ∀ b+

i > bi , she is allocated the same slot j in round t.

Higher bid: agent receives the same slot in round t,Lower bid:, she may receive the same slot or may lose the impression.

Definition (Weak Pointwise Monotonicity)

We call an allocation rule weak pointwise monotone if, for any given(b−i , ρ), and bid b+

i > bi , Aij((bi , b−i ), ρ, t) = 1 ⇒Aij′((b

+i , b−i ), ρ, t) = 1 for some slot j ′ ≤ j , ∀t.



Conclusion


Monotonicity

Definition (Strong Pointwise Monotonicity)

An allocation rule is said to be strongly pointwise monotone if it satisfies:For any fixed (b−i , ρ), if an agent i with bid bi is allocated a slot j inround t, then ∀ b+

i > bi , she is allocated the same slot j in round t.

Higher bid: agent receives the same slot in round t,Lower bid:, she may receive the same slot or may lose the impression.

Definition (Weak Pointwise Monotonicity)

We call an allocation rule weak pointwise monotone if, for any given(b−i , ρ), and bid b+

i > bi , Aij((bi , b−i ), ρ, t) = 1 ⇒Aij′((b

+i , b−i ), ρ, t) = 1 for some slot j ′ ≤ j , ∀t.



Conclusion


Example

Consider, k = 4 agents, m = 2 slots, T = 1000 rounds

CTRs, µij decreasing for each agent i

Let an allocation A be:

For the first 100 rounds, advertisements of four agents displayed inround robin fashionFor the remaining 900 rounds, the advertisements are displayed thatmaximize the expected sum of valuations of the clicks

A described here is weakly pointwise monotone



Conclusion


More Definitions

Definition (Influential Set)

Given a bid vector, b, a realization ρ and round t, an influential setI (b, ρ, t) is the set of all agent-slot allocation pairs (i , j), such that (i)Aij(b, ρ, t) = 1 and (ii) a change in ρij(t) will result in a change in theallocation in a future round. t is referred to as an influential round.Agent i is referred to as an influential agent and j as influential slot w.r.tround t.



Conclusion


Separatedness

Definition (Type-I Separated)

We call an allocation rule Type-I separated if for a given (b−i , ρ), ifN((bi , b−i ), ρ, i , t) is an i-influential set, then∀ (i ′, j ′) ∈ N((bi , b−i ), ρ, i , t), Ai ′j′ = 1 when the agent i increases herbid to b+

i .

When an agent i increases her bid, while the other parameters are keptfixed, the allocation in the originally influential slots does not change,though the influentiality of that agent-slot pair may be lost.



Conclusion


Definition (Type-II Separated)

We call an allocation rule Type-II separated if for a given (b−i , ρ) and twobids of agent i , bi and b+

i > bi , N((bi , b−i ), ρ, i , t) ⊆ N((b+i , b−i ), ρ, i , t).

This means that when an agent i increases her bid, while the otherparameters are kept fixed, the allocation in the originally influential slotsdoes not change and they remain influential.



Conclusion


Example

Consider, k = 4 agents, m = 2 slots, T = 2 rounds

Let an allocation A be:

First round, the ad of the agent i is displayed in the slot i , i = 1, 2.If any ad receives click, same ad is retained in same slotIf the ad in slot i is not clicked

i If bi < bi+2, in round 2, the ad of the agent i + 2 is displayed in slot iii Else, the original ad is retained

A described here is Type-I separated

A described here is not Type-II separated (If b1 > b3, agent 1 is notinfluential for herself)



Conclusion


Our Results

Number of Learning Parameter Solution Allocation rule Worst CaseSlots (m) (CTR) Concept Characterization Regret

m = 1 Unrestricted DSIC Pointwise monotone & Θ(T 2/3)Exploration separated

m ≥ 1 Unrestricted DSIC Strongly pointwise Θ(T )monotone andType-I separated

m ≥ 1 Higher Slot Click DSIC Weakly pointwise monotone & regret analysisPrecedence Type-I separated open

(Necessary Condition)m ≥ 1 CTR Pre-estimates Truthful in Weakly Pointwise monotone & regret analysis

available expectation Type-I separated (Necessary) openType-II Separated (Sufficient)

m ≥ 1 Separable CTR Truthful in Weakly Pointwise monotone & Ω(T 2/3)expectation Type-I separated (Necessary) (Experimental

Type-II Separated (Sufficient) Evidence)

Table: Results



Conclusion


Theorem

Theorem

Let (A,P) be a deterministic, non-degenerate mechanism for the MAB,multi-slot sponsored search auction, with unconstrained and unknown µij .Then, mechanism (A,P) is DSIC iff A is strongly pointwise monotoneand Type-I separated. Further, the payment scheme is given by,

Pi ((bi , b−i ), ρ) = biCi ((bi , b−i ), ρ)−∫ bi

0

Ci ((x , b−i ), ρ)dx .



Conclusion


Sketch of the Proof

Monotonic Allocation - # clicks should increase with her bid

For the unknown and unconstrained CTRs this leads to a need ofstrong pointwise monotonicity

Payment should be in the form as given by Theorem 1This can be derived from Myerson Characterization

Payment should be computable from the observed clicks: Type -Iseparatedness is necessary

Type-I separatedness along with the strong pointwise monotonicity issufficient for truthful implementation



Conclusion


Implications of Strong Monotonicity

In a particular round, for a fixed game instance, if an agent isallocated a particular slot, she has to be allocated same slot at allhigher bids

This is a very strong necessity

It leads to instances on which regret is O(T )



Conclusion








Conclusion








Conclusion


What Next?

Three escape routes



Conclusion


Higher Order Click Precedence

Assume, if there is a click on ad in slot j , then all previous slots receiveclick.

Proposition

Consider the setting in which realization ρ follows Higher Slot ClickPrecedence. Let (A,P) be a deterministic non-degenerate DSICmechanism for this setting. Then the allocation rule A must be weakpointwise monotone and Type-I separated. Further, the payment schemeis given by,

Pi (bi , b−i ; ρ) = biCi (bi , b−i ; ρ)−∫ bi

0




Conclusion


Higher Order Click Precedence

Assume, if there is a click on ad in slot j , then all previous slots receiveclick.

Proposition

Consider the setting in which realization ρ follows Higher Slot ClickPrecedence. Let (A,P) be a deterministic non-degenerate DSICmechanism for this setting. Then the allocation rule A must be weakpointwise monotone and Type-I separated. Further, the payment schemeis given by,

Pi (bi , b−i ; ρ) = biCi (bi , b−i ; ρ)−∫ bi

0




Conclusion


Truthful In Expectation

Definition (Truthful in Expectation)

A mechanism is said to be truthful in expectation over µ, the CTRpre-estimate, if each of the agents believes that the number of clicks sheobtains is indeed

∑t

∑j(µijAij(·)), which is the number of clicks she will

obtain if the CTR pre-estimate is perfectly accurate.



Conclusion


CTR Pre-estimates are Available

Theorem

Let (A,P) be a mechanism for this stochastic multi-round auction settingwhere A is a non-degenerate, deterministic and fair allocation rule. Then,(A,P) is truthful in expectation over µ if A is weakly pointwise monotoneand Type-II separated and the payment scheme is given by,

Pi (b, ρ) =T∑

t=1

m∑j=1

µijbiAij(b, ρ, t)−∫ bi

0

Aij(x , b−i , ρ, t)dx

Also, if a mechanism (A,P) is truthful, then it is weakly pointwisemonotone, Type-I separated, the payment is given as above and iscomputable.



Conclusion


Separable CTRs

Theorem

Let (A,P) be a mechanism for the stochastic multi-round auction settingwhere A is a non-degenerate, deterministic and fair allocation rule. Then,(A,P) is truthful in expectation over µ′ if A is weakly pointwisemonotone and Type-II separated and the payment scheme is given by,

Pi (b, ρ) =T∑

t=1

m∑j=1

µ′ijbiAij(b, ρ, t)−∫ bi

0

Aij(x , b−i , ρ, t)dx

Also, if a mechanism (A,P) is truthful, then it is weakly pointwisemonotone, Type-I separated, the payment is given as above, and iscomputable.



Conclusion

Our ResultsSummaryReferences

Our Results

Number of Learning Parameter Solution Allocation rule Worst CaseSlots (m) (CTR) Concept Characterization Regret

m = 1 [2] Unrestricted DSIC Pointwise monotone & Θ(T 2/3)Exploration separated

m ≥ 1 Unrestricted DSIC Strongly pointwise Θ(T )monotone andType-I separated

m ≥ 1 Higher Slot Click DSIC Weakly pointwise monotone & regret analysisPrecedence Type-I separated open

(Necessary Condition)m ≥ 1 CTR Pre-estimates Truthful in Weakly Pointwise monotone & regret analysis

available expectation Type-I separated (Necessary) openType-II Separated (Sufficient)

m ≥ 1 Separable CTR Truthful in Weakly Pointwise monotone & Ω(T 2/3)expectation Type-I separated (Necessary) (Experimental

Type-II Separated (Sufficient) Evidence)

Table: Results



Conclusion


Summary

We have seen,

Characterization of truthful mechanisms with Unknown andUnconstrained CTRs

Need of a strong monotonicity puts severe restrictions on truthfulallocation rules

How the relation across the CTRs can be exploited



Conclusion


Summary

We have seen,






Conclusion


Summary

We have seen,






Conclusion


Direction for future work

Fill in the gaps in necessary and sufficient conditions

Regret analysis

Weaker solution concepts,

1 Approximate ex-post Nash Incentive Compatibility (Parkes et al)2 Bayesian Incentive Compatibility

Randomized mechanisms6

6M. Babaioff, R Kleinberg, and A Slivkins, Truthful Mechanisms with ImplicitPayment Computation, in ACM Conference on Electronic Commerce (EC’10)



Conclusion




Regret analysis







Conclusion




Regret analysis







Conclusion




Regret analysis

Weaker solution concepts,1 Approximate ex-post Nash Incentive Compatibility (Parkes et al)

2 Bayesian Incentive Compatibility





Conclusion




Regret analysis

Weaker solution concepts,1 Approximate ex-post Nash Incentive Compatibility (Parkes et al)2 Bayesian Incentive Compatibility





Conclusion




Regret analysis

Weaker solution concepts,1 Approximate ex-post Nash Incentive Compatibility (Parkes et al)2 Bayesian Incentive Compatibility





Conclusion


Literature Review

For Sponsored Search Auctions: [3, 4, 5]Web Information: http://advertising.yahoo.com/,http://adwords.google.com

For Mechanism Design: [5, 6, 7]

Multi-Armed Bandit Problems: [8, 9]

MAB Mechanisms for sponsored search: [10, 2, 11, 12, 13]



Conclusion


R. B. Myerson.Optimal auction design.Math. Operations Res., 6(1):58–73, Feb. 1981.

M. Babaioff, Y. Sharma, and A. Slivkins.Characterizing truthful multi-armed bandit mechanisms: extendedabstract.In Proceedings of the 10th ACM Conference on ElectronicCommerce, pages 79–88, Stanford, California, 2009.

G. Aggarwal, A. Goel, and R. Motwani.Truthful auctions for pricing search keywords.In EC ’06: Proceedings of the 7th ACM conference on Electroniccommerce, pages 1–7, New York, NY, USA, 2006.

B. Edelman, M. Ostrovsky, and M. Schwarz.Internet advertising and the generalized second price auction: Sellingbillions of dollars worth of keywords.



Conclusion


In 2nd Workshop on Sponsored Search Auctions in conjunction withthe ACM Conference on Electronic Commerce (EC’06), Ann Arbor,MI, June 2006.

Y. Narahari, D. Garg, N. Rama Suri, and H. Prakash.Game Theoretic Problems in Network Economics and MechanismDesign Solutions.Advanced Information and Knowledge Processing Series, Springer,London, 2009.

Dinesh Garg, Y Narahari, and Sujit Gujar.Foundations of mechanism design: A tutorial - part 1: Key conceptsand classical results.Sadhana - Indian Academy Proceedings in Engineering Sciences,33(Part 2):83–130, April 2008.

Dinesh Garg, Y Narahari, and Sujit Gujar.Foundations of mechanism design: A tutorial - part 2: Advancedconcepts and results.



Conclusion


Sadhana - Indian Academy Proceedings in Engineering Sciences,33(Part 2):131–174, April 2008.

H. Robbins.Some aspects of the sequential design of experiments.Bulletin of American Maths Society, 58, Number 5:527–535, 1952.

P. Auer, N. Cesa-Bianchi, and P. Fischer.Finite-time analysis of the multiarmed bandit problem.Machine Learning, 47(2-3):235–256, 2002.

N. R. Devanur and S. M. Kakade.The price of truthfulness for pay-per-click auctions.In Proceedings of the 10th ACM Conference on ElectronicCommerce, pages 99–106, 2009.

R. Gonen and E. Pavlov.An adaptive sponsored search mechanism delta -gain truthful invaluation, time, and budget.



Conclusion


In Third International Workshop on Internet and NetworkEconomics, WINE07,, volume 4858 of Lecture Notes in ComputerScience, pages 341–346. Springer, 2007.

R. Gonen and E. Pavlov.An incentive-compatible multi-armed bandit mechanism.In Proceedings of Principles of Distributed Computing, PODC ’07,pages 362–363, New York, NY, USA, 2007. ACM.

Akash Das Sarma, Sujit Gujar, and Y. Narahari.Multi-armed bandit mechanisms for multi-slot sponsored searchauctions.CoRR, abs/1001.1414, 2010.Working Paper, Dept of CSA, Indian Institute of Science, Bangalore.



Conclusion


Questions?



Conclusion


Thank You!!!


Date post:	17-Oct-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Characterization of Truthful MAB Mechanisms for Multi-Slot...

Documents