Date post: | 26-Mar-2015 |
Category: |
Documents |
Upload: | noah-maloney |
View: | 218 times |
Download: | 4 times |
Crowdsourcing and All-Pay Auctions
Milan VojnovićMicrosoft Research
Joint work with Dominic DiPalantino
UC Berkeley, July 13, 2009
Examples of Crowdsourcing• Crowdsourcing = soliciting solutions via open calls to
large-scale communities– Coined in a Wired article (’06)
• Taskcn– 530,000 solutions posted for 3,100 tasks
• Innocentive– Over $3 million awarded
• Odesk– Over $43 million brokered
• Amazon’s Mechanical Turk– Over 23,000 tasks
2
Examples of Crowdsourcing (cont’d)
• Yahoo! Answers– Lunched Dec ’05– 60M users / 65M answers (as of Dec ’06)
• Live QnA– Lunched Aug ’06 / closed May ’09– 3M questions / 750M answers
• Wikipedia
3
Incentives for Contribution• Incentives
– Monetary
$$$
– Non-momentary
Social gratification and publicityReputation pointsCertificates and “levels”
• Incentives for both participation and quality
4
Incentives for Contribution (cont’d)• Ex. Taskcn
5
Reward range (RMB)
Cont
est d
urati
onN
umbe
r of s
ubm
issi
ons
Num
ber o
f reg
istr
ants
Num
ber o
f vie
ws
100 RMB $15 (July 09)
Incentives for Contribution (cont’d)• Ex. Yahoo! Answers
6
Points Levels
Source: http://en.wikipedia.org/wiki/Yahoo!_Answers
Questions of Interest
• Understanding of the incentive schemes– How do contributions relate to offered rewards?
• Design of contests– How do we best design contests?– How do we set rewards?– How do we best suggest contests to players and
rewards to contest providers?
7
Strategic User Behavior
• From empirical analysis of Taskcn by Yang et al (ACM EC ’08) – (i) users respond to incentives, (ii) users learn better strategies– Suggests a game-theoretic analysis
8
User Strategies on Taskcn.com User Strategies on Taskcn.com
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion9
Single Contest Competition
10
c1
c2
c3
c4
R
ci = cost per unit effort or quality produced
contest offeringreward Rplayers
Single Contest Competition (cont’d)
11
Outcome
-c1b1
R - c2b2
-c3b3
-c4b4
c1
c2
c3
c4
b1
b2
b3
b4
R
All-Pay Auction
12
Outcome
-b1
v2 - b2
-b3
-b4
v1
v2
v3
v4
b1
b2
b3
b4
Everyone pays their bid
Competing Contests
13
R1
R2
RJ
...
Rj...
contestsusers
1
2
u
N
),,( ,1, Juuu vvv
juv ,
......
Incomplete Information Assumption
Each user u knows
= total number of usersN
= his own skilluv
= skills are randomly drawn from FF
14
We assume F is an atomless distribution with finite support [0,m]
Assumptions on User Skill1) Player-specific skill
random i.i.d. across u (ex. contests require similar skills or skill determined by player’s opportunity cost)
),,( uu vvv
2) Contest-specific skill
random i.i.d. across u and j (ex. contests require diverse skills)
),,( ,1, Juu vvv
juv ,
uv
15
Bayes-Nash Equilibrium
• Mixed strategy
• EquilibriumSelect contest of highest expected profit
where expectation with respect to “beliefs” about other user skills
)(, vju = prob. of selecting a contest of class j
jub , = bid
16Contest class = set of contests that offer same reward
User Expected Profit
• Expected profit for a contest of class j
v
Ncjjjj dxxFpRvg
0
1)(1)(
= prob. of selecting a contest of class j
jp
= distribution of user skill conditional on having selected contest class j
()jF
17
vn
jn
jjujj dxxFvFvRnvg0
, )()(),(
)),((E)( Mvgvg jj
),1(Bin~ jpNM
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion18
Equilibrium Contest Selection
m
0
1
2
3
4
5
1v2
v3
v4
2
3
4
skilllevels
contestclasses
19
Threshold Reward
• Only K highest-reward contest classes selected with strictly positive probability
)(
11:max
~],1[
],1[
1
1
RHJ
RiK ii
Ni
1
11
)(
AkkJ
JA
N
A
k RRH
Ak
kA JJ
20kJ = number of contests of class k
Partitioning over Skill Levels
• User of skill v is of skill level l if
KlRH
RJvF
l
lll
N ~,,1 for ,
)(11)(
],1[],1[
11
),[ 1 ll vvv
where
KKlv l ,,~
for ,0
21
Contest Selection
• User of skill l, i.e. with skill selects a contest of class j with probability
Klj
ljR
R
vl
kk
j
j N
N
,,10
,,1)(
1
11
11
),[ 1 ll vvv
22
Participation Rates
• A contest of class j selected with probability
KKj
Kj
R
RH
Jp Nj
K
Kj
,,1~
0
~,,1
)(111
1
1
]~
,1[
]~
,1[
23
• Prior-free – independent of the distribution F
Large-System Limit
• For positive constants
where K is a finite number of contest classes
J
NNlim
kk
N J
J lim
kkN Np lim
Kkkk ,,1 , , ,
KRRR 21
24
Skill Levels for Large System
• User of skill v is of skill level l if
KlR
RvF
l
l
kk
ll
lk
~,,1 for ,log1)( 1
/
],1[
],1[
),[ 1 ll vvv
where
KKlvl ,,1~ for ,0
25
Participation Rates for Large System
• Expected number of participants for a contest of class j
,K,Kj
Kj
R
RK
kk
j
Kj
Kk
1~
0
~,,1log ~
1
/]~
,1[ ]~
,1[
],1[],1[
1
/:max~ iik eRRiKi
kki
26
• Prior-free – independent of the distribution F
Contest Selection in Large System• User of skill l, i.e. with skill selects a
contest of class j with probability
Klj
ljJv lj
,,10
,,11
)( ],1[
),[ 1 ll vvv
m
0
1
2
34
5
123
4
1/3
1/3
1/3
27
• For large systems, what matters is which contests are selected for given skill
Proof Hint for Player-Specific Skills
28
• Key property – equilibrium expected payoffs as showed
vm0 v1v2v3
g1(v)
g2(v)
g3(v)
g4(v)
4321 RRRR
Outline
• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion29
Contest-specific Skills
• Results established only for large-system limit
• Same equilibrium relationship between participation and rewards as for player-specific skills
30
Proof Hints
• Limit expected payoff – For each ],0[ mv
veRvg jjjN
)(lim
• Balancing – Whenever 0j
keReR kjkj all for ,
• Asserted relations for follow from above
),,( 1 K 31
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion32
System Optimum Rewards
33
K
kkk
K
kkkk RCRU
11
)())((
RR
K
kkk
1
maximise
over
subject to
SYSTEM
• Set the rewards so as to optimize system welfare
Example 1: zero costs(non monetary rewards)
34
Assume are increasing strictly concave functions. Under player-specific skills, system optimum rewards:
()kU
KjN
UcR
N
jj ,,1 ,
)(1
)1(1'
for any c > 0 where is unique solution of
K
kkkU
1
1' )(
• Rewards unique up to a multiplicative constant – only relative setting of rewards matters
Example 1 (cont’d)
35
• For large systems
Assume are increasing strictly concave functions. Under player-specific skills, system optimum rewards:
()kU
KjceR jUj ,,1 ,)(1'
for any c > 0 where is unique solution of
K
kkkU
1
1' )(
Example 2: optimum effort
36
• Consider SYSTEM with
)))(1(1())(( )(Rjjjj
jeRmRR
)))((())(( RVRU jjjjj
)()1()( )(jj
Rj RDeRC j
exerted effort
{cost of
giving Rj (budget constraint)
{
prob. contest attended
{
Utility:
Cost:
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion37
Taskcn• Analysis of rewards and participation across
tasks as observed on Taskcn– Tasks of diverse categories: graphics, characters,
miscellaneous, super challenge– We considered tasks posted in 2008
38
Taskcn (cont’d)
39
reward
number of views
number of registrants
number of submissions
Submissions vs. Reward
• Diminishing increase of submissions with reward
40
Graphics Characters Miscellaneous
linear regression
Submissions vs. Rewardfor Subcategory Logos
• Conditioning on the more experienced users, the better the prediction by the model
41
any rate once a month every fourth day every second day
• Conditional on the rate at which users submit solutions
model
Same for the Subcategory 2-D
42
any rate once a month every fourth day every second day
model
Conclusion• Crowdsourcing as a system of competing contests
• Equilibrium analysis of competing contests– Explicit relationship between rewards and participations
• Prior-free– Diminishing increase of participation with reward
• Suggested by the model and data
• Framework for design of crowdsourcing / contests
• Base results for strategic modelling– Ex. strategic contest providers
43
More Information
• Paper: ACM EC ’09
• Version with proofs: MSR-TR-2009-09– http://research.microsoft.com/apps/pubs/default.
aspx?id=79370
44