1
Only Valuable Experts Can be
ValuedMoshe Babaioff, Microsoft Research, Silicon Valley
Liad Blumrosen, Hebrew U, Dept. of Economics
Nicolas Lambert, Stanford GSB
Omer Reingold, Microsoft Research, Silicon Valley and Weizmann.
2
Probabilities of Events• Often, estimating probabilities of
future events is important.• Examples:
– Weather: probability of rain tomorrow
– Online advertising: what is the click probability of the next visitor on our web-site?
– What % of Toyota cars is defective?
– Many applications in financial markets.
3
Contracts and Screening
Uncertain about the probability of a future event.
Claims he knows this probability.
Averse to uncertainty, is willing to pay $$$ to reduce it.
May be uninformedand pretend to be informed to get $$$…
A decision maker (Alice)
An expert (Bob)
4
Contracts and Screening
• Goal: screen experts.• That is, design contracts such
that:
– Informed experts will:1. accept the contract2. reveal the true probability.
– Uninformed experts will reject the contract.
• Contracts can be based on outcomes only: True probabilities are never revealed.
5
This work• We characterize settings where Alice can
separate good experts from bad experts.
• We discuss what is a “valuable” expert, and its relation to screening of experts.
6
Outline• Model
• With prior:– An easy impossibility result– A positive result
• No priors
• Extensions
7
Model (1/4)• Ω - Finite set of outcomes.• p - A (true) distribution over Ω.
– Unknown to Alice• Φ - The set of possible distributions.
– Φ may be restricted, examples to come…
• Bayesian assumptions:Prior f on Φ.
• f is known to Alice, Bob.
No Prior on Φ.
In the beginning of this talk Later….
8
Model (2/4)• Contract: π(q,ω)
payment to Bob when reporting q when outcome is ω.– Bob is risk neutral.
• Bob’s expected payment depends on what he knows:
– Informed: π(q,p) = Eω~p [ π(q,ω) ]– Uninformed: Ep~f Eω~p [ π(q,ω) ] =
π(q,E[p])
Reported probability
Realized outcome
Notation: E[payment] upon reporting q when the true probability is p.
9
Model (3/4)• Bob δ-accepts the contract if he has
a report q with payment > δ– Otherwise, we say that Bob δ-rejects.
• For avoiding handling ties, we aim that for δa> δr :– an informed Bob will δa -accept – an uninformed Bob will δr –reject
δa
δr
0
10
Model (4/4)
• We actually study a more general model: – experts are ε-informed.– Mixed strategies are allowed.
• This talk: perfectly informed, pure strategies.
11
Example• A binary event: Ω = { , }
Pr( ) = αp Φ = { (α , 1- α) | α [0,1] }
• Alice does not know p.– Knows, however, that α ~ U[0,1] .Sees an a-priori probability of ½.
• Bob claims he knows the realization of p.
• Contract: Bob reports α. Is paid according to or .
12
Main message
• The ability to screen experts closely relates to the structure of Φ– Roughly, on whether Φ is convex or not.
• If all possible experts are “valuable” to Alice, then screening is possible.
13
An Easy Impossibility
• Reason: when the true probability is E[p] a true expert accepts the contract.
Proof:o Informed experts always accept the contract π.That is, for all p, we have π(p,p) > δa.
o Then, an uninformed agent can get > δa by reporting E[p]:Ep~f Eω~p [π(E[p],ω)] =π(E[p],E[p]) > δa
Proposition: screening is impossible.
14
Valuable Experts• So screening is impossible when E[p] Φ.• But experts knowing that the true
distribution is E[p] are not really valuable to Alice.
• In the binary-outcome example:−Alice’s prior is U[0,1], so she believes
that Pr( ) = ½.−An expert knowing that p=½ is not that
helpful…What if all experts are “valuable”?
15
Possibility result• Theorem: when E[p] is not in Φ (and Φ is
closed), screening is possible.– When all experts are valuable, we can
screen...
• Immediate questions:−Is non-convex Φ natural?−Can we expect Φ not to contain E[p]?
E[p]
Φ
16
Non-convex Φ: examples
• Example 1: a coin which is either fair (p=1/2) or biased (p=3/4)– For any (non-trivial) prior, E[p] not in Φ.
• Example 2: many standard distributions are not closed under mixing.– E.g., uniform, normal, etc.
1/2 3/4
17
Non-convex Φ: examples
• Example 3: The binary outcome example. But now, Alice observes two samples.– For example, we wish to know the failure rate
in cars, and we thoroughly check 2 random Toyotas.
18
Non-convex Φ: examples
Ω = { ( , ), ( , ), ( , ) , ( , ) }Φ = { ( α2, α(1- α), (1- α)α, (1-α)2 ) } α [0,1]
• Example 3: The binary outcome example.
But now, Alice observes two samples.
For every prior, E[p] is not in Φ, and thus screening is possible.
Φ is not convex!− Moreover, for every p,p’,
the convex combination of the above vectors is not in Φ.
19
Possibility result• Theorem: when E[p] is not in Φ (and Φ is
closed), screening is possible.Φ
• One definition before proceeding to the proof…
20
Scoring rules• Scoring rules:
– Contracts that elicit distributions from experts.• S(q,ω) = payment for an expert reporting q
when the realized outcome is ω.– A scoring rule is strictly proper if the expert is
always strictly better off by reporting the true distribution.
– Strictly proper scoring rules are known to exist [Brier ‘50, , Good ’52, Savage ‘71,…]
• We want, in addition, to screen good experts from bad.
21
Possibility result• Theorem: when E[p] is not in Φ (and Φ is
closed), screening is possible.Φ• Proof:
• Let s be some strictly proper scoring rule.− On the full probability space
• The following contracts screens experts:
rq
ra qpEsqqspEsqsq
)'],[()','(inf
)],[(),(2),('
We need to show:1. An informed expert δa-accepts.2. An uninformed expert δr-rejects.
22
Possibility result• Theorem: when E[p] is not in Φ (and Φ is
closed), screening is possible.Φ• Proof:
• Let s be some strictly proper scoring rule.• On the full probability space
• The following contracts screens experts:
rq
ra qpEsqqspEsqsq
)'],[()','(inf
)],[(),(2),('
An informed expert reporting the truth p gains: rrrapp 2),(
≥1
23
Possibility result• Theorem: when E[p] is not in Φ (and Φ is
closed), screening is possible.Φ• Proof:
• Let s be some strictly proper scoring rule.• The following contracts screens experts:
Since s is strictly proper, for every q:
An uninformed expert will gain: rrrapEq 02])[,(
0])[],[(])[,( pEpEspEqs
rq
ra qpEsqqspEsqsq
)'],[()','(inf
)],[(),(2),('
24
Outline• Model
• With prior:– An easy impossibility result– A positive result
• No priors
• Extensions
25
Related work• [Olszewski & Sandroni 2007] studied a
similar model:– A binary event with unknown probability p.– No priors:
• An uninformed expert accepts a contract if it is good in the worst case.
• Theorem [O&S]:
– Use min-max theorems.– The probability space is convex.
All informed experts accept a contract
An uninformed expert also accepts it
26
Valuable Experts: No Prior
• We claim: invaluable agents are also behind this impossibility.– But what is a valuable agent without priors?
• What are Alice’s utility function and actions?– A(p) : Alice’s action when she knows p.– U( A(p),p ): utility maximizing actions.
Theorem:Φ is convex (and closed)
There exists p such thatU(A(p),p)=U(A(“reject”),p)no prior on Φ
• Interpretation: if Φ is convex then some informed expert is not valuable.
27
No prior: positive result• We have an analogues positive result for
the no-prior case:non-convex Φ screening is
possible.
• (we use the with-prior positive result in the proof)
28
Outline• Model
• With prior:– An easy impossibility result– A positive result
• No priors
• Extensions
29
Extension: forecasting• A related line of research is forecasting:
– An unbounded sequence of events.– An expert provides a forecast before
each event occurs.– Goal: test the expert.
30
Forecasting: related work
• Negative results are known:– Informed experts pass the test uninformed
experts can do it too. [e.g., Foster & Vohra ‘97, Fudenberg & Levine ‘99]
– When forecasting is possible, decisions can be delayed arbitrarily. [Olszewski & Sandroni ‘09]
• Some works around this impossibility:– [Olszewski & Sandroni ‘09] show a counter-example
by constructing non-convex set of distributions. – [Al-Najjar & Sandroni & Smorodinsky & Weinstein ‘10]
Describe a class of distribution such that decisions can be made in time. The relevant class of distributions also admits non-convexities.
31
Extension: forecasting• We extend our approach to forecasting
settings.– In the works.
• We characterize conditions on the set of distributions that allow expert testing.
• Analysis is more involved, but the ideas are similar.– Results relate to the convexity of Φ.– For example: two samples at each period enable
testing.
32
Summary• A decision maker want to hire an expert.
– For learning the probability of some future event.– The expert may be a charlatan.
• Can the decision maker separate good experts from bad ones?
• We characterize the settings where such screening is possible.– With or without priors on Φ.
• We design screening contracts.
33
Thanks!