Experiment 1
Estimate the percentage of red dots in multiple rounds Solo condition Group condition: individuals observe estimates of other individuals
Procedure Round 1: individuals estimate independently of other individuals Round 2 and 3: individuals see same display again
in group condition, individuals also see estimates of other individuals from previous round
Each individual/group was tested on 36 displays with each 3 rounds. The displays consisted of 4 repetitions of 10,20,…,90% of red dots
giving 36 different displays
3
Round 2: what is the percentage of red circles?
5
Individual 1: 75Individual 2: 85Individual 3: 72Individual 4: 78
Estimates from previous round
Number of individuals per group
6
1 2 3 4 5 6 7 8 9 10 110
1
2
3
4
5
6
7
8
Group
Num
ber
of
pe
ople
in g
rou
p
MAD per individual over rounds
7
0 10 20 30 40 50 60 702
4
6
8
10
12
14
16
18
Individuals (sorted by mean error)
Mea
n e
rror
ove
r q
uest
ion
s
round 1round 2round 3
MAD per question over rounds
8
0 50 100 150 200 250 300 3500
5
10
15
20
25
30
35
Questions (sorted by mean error)
Mea
n e
rror
ove
r q
uest
ion
s
round 1round 2round 3
why are somequestions so mucheasier than others?
Individual estimates across rounds for different displays
9
0 50 1000
50
100
150
10%
0 50 1000
20
40
60
80
100
20%
0 50 1000
20
40
60
30%
0 50 1000
20
40
60
80
10040%
0 50 1000
50
100
50%
0 50 1000
20
40
60
60%
0 50 1000
20
40
60
70%
0 50 1000
20
40
60
80
80%
0 50 1000
50
100
150
90%
(note: across groups too)
10
0 20 40 60 80 100
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Que
stio
ns
Group 1
0 20 40 60 80 100
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Que
stio
ns
Group 1
all raw data from one group with six individuals: in each box we see the responses for three rounds (bottom row in each box is round 1). Dashed line represents ground truth
11
0 20 40 60 80 100
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
Que
stio
ns
Group 2
0 20 40 60 80 100
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
Que
stio
ns
Group 2
Higher accuracy on round 1 when individuals do not change their estimates in subsequent rounds
12
0 20 40 60 80 1000
10
20
30
40
50
60
70
80
90
Mean estimate change from round 1 to round 2
Mea
n e
rror
at
rou
nd 1
Correlation = 0.708
0 10 20 30 40 50 600
10
20
30
40
50
60
70
80
90
Std estimates across round
Mea
n e
rror
at
rou
nd 1
Correlation = 0.751
If people change their estimate, they generally increase their accuracy
13
0 20 40 60 80 100-20
0
20
40
60
80
100
Change in estimate from round 1 to round 2
MA
D(
rou
nd 1
) -
MA
D(
roun
d 2
)Correlation = 0.875
Experiment 2
Estimate percentages for trivia questions in multiple rounds
Solo and group conditions 21 groups
Similar setup as in experiment 1 36 different trivia questions three estimation rounds per question
Major difference with experiment 1 No gain in perceptual information between rounds (no opportunity
to extract more information from same display)
15
Number of Individuals in 21 groups (Exp 2)
16
0 5 10 15 200
1
2
3
4
5
6
7
8
9
10
Group
Num
ber
of
pe
ople
in g
rou
p
Trivia Questions1. "What percent of people in the U.S. over the age of 25 have obtained a college degree?"
2. "What percent of U.S. women age 25 and older have completed high school?"
3. "What percent of the worldwide use of energy is by the United States?"
4. "What percent of the worldwide population lives in the United States?"
5. "What percent of U.S. energy consumption is with fossil fuels?"
6. "In 2000, the richest 1% of adults owned what percentage of the world's total assets?"
7. "What percentage of Americans are age 0-14?"
8. "What percentage of Americans are Protestant?"
9. "What percentage of Americans are black?"
10. "What percentage of Americans over the age of 15 can read and write?"
11. "What percentage of the American population is below the poverty line?"
12. "In 2005, what was the U.S. military expenditure as percent of the national income?"
13. "What percentage of people in the world are age 0-14?"
14. "What percentage of people in the world are Christian?"
15. "Across the world, what percentage of people's first language is Mandarin (a dialect of Chinese)?"
16. "Across the world, what percentage of people over the age of 15 can read and write?"
17. "Across the world, what percentage of the world labor force is in agriculture (not industry or services)?"
18. "What percentage of the world's surface is water (not land)?"
19. "What percentage of people in the world are aged 65 or over?"
20. "What percentage of people in the world are Muslim?"
21. "What percentage of college freshmen have never used a typewriter?"
22. "What percentage of the U.S. population will eat macaroni and cheese sometime in the next two weeks?"
23. "What percentage of Americans drink a soda pop nearly every day with breakfast?"
24. "In the United States, what percentage of households routinely speak a language other than English?"
25. "In the United States, what percentage of firms are owned by women?"
26. "What percentage of American men serve time in prison at some point in their life?"
27. "What percentage of Americans are regular cigarette smokers?"
28. "What percentage of Americans have no health insurance?"
29. "What percentage of children diagnosed with cancer make a full recovery?"
30. "What percentage of a woman's body consists of water?"
31. "What percentage of people grind their teeth while sleeping?"
32. "What percentage of the earth's water is fresh (not salt) water?"
33. "What percentage of people struck by lightning immediately die?"
34. "What percentage of Americans are left handed?"
35. "What percentage of the total number of bones in the human body are in the feet?"
36. "What percentage of Utah's population is Mormon?"
17
Distribution of answers to trivia questions (1)(regardless of what group an individual is part of)
18
0 20 40 60 80 1000
5
10
15
20
What percent of people in the U.S. over theage of 25 have obtained a college degree?
0 20 40 60 80 1000
5
10
15
20
25
What percent of U.S. women age 25 and olderhave completed high school?
0 20 40 60 80 1000
5
10
15
What percent of the worldwide use of energyis by the United States?
0 20 40 60 80 1000
5
10
15
20
25
30
What percent of the worldwide populationlives in the United States?
0 20 40 60 80 1000
5
10
15
20
25
What percent of U.S. energy consumption iswith fossil fuels?
0 20 40 60 80 1000
5
10
15
In 2000, the richest 1% of adults owned whatpercentage of the world's total assets?
0 20 40 60 80 1000
5
10
15
20
25
30
What percentage of Americans are age 0-14?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of Americans are Protestant?
0 20 40 60 80 1000
5
10
15
20
25
30
What percentage of Americans are black?
0 20 40 60 80 1000
5
10
15
20
What percentage of Americans over the age of15 can read and write?
0 20 40 60 80 1000
5
10
15
20
25
30
What percentage of the American populationis below the poverty line?
0 20 40 60 80 1000
5
10
15
In 2005, what was the U.S. military expenditureas percent of the national income?
green = round 1, blue=round 2, red = round 3
Distribution of answers to trivia questions (2)
19
0 20 40 60 80 1000
10
20
30
What percentage of people in the world areage 0-14?
0 20 40 60 80 1000
5
10
15
20
What percentage of people in the world areChristian?
0 20 40 60 80 1000
10
20
30
40
Across the world, what percentage of people'sfirst language is Mandarin (a dialect of Chinese)?
0 20 40 60 80 1000
5
10
15
Across the world, what percentage of peopleover the age of 15 can read and write?
0 20 40 60 80 1000
5
10
15
20
Across the world, what percentage of theworld labor force is in agriculture (not industry or services)?
0 20 40 60 80 1000
10
20
30
40
50
What percentage of the world's surface iswater (not land)?
0 20 40 60 80 1000
5
10
15
20
What percentage of people in the world areaged 65 or over?
0 20 40 60 80 1000
5
10
15
20
What percentage of people in the world areMuslim?
0 20 40 60 80 1000
5
10
15
20
What percentage of college freshmen havenever used a typewriter?
0 20 40 60 80 1000
5
10
15
20
What percentage of the U.S. population willeat macaroni and cheese sometime in the next two weeks?
0 20 40 60 80 1000
5
10
15
20
What percentage of Americans drink a sodapop nearly every day with breakfast?
0 20 40 60 80 1000
5
10
15
20
In the United States, what percentage ofhouseholds routinely speak a language other than English?
Distribution of answers to trivia questions (3)
20
0 20 40 60 80 1000
5
10
15
20
25
In the United States, what percentage offirms are owned by women?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of American men serve timein prison at some point in their life?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of Americans are regularcigarette smokers?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of Americans have nohealth insurance?
0 20 40 60 80 1000
5
10
What percentage of children diagnosed withcancer make a full recovery?
0 20 40 60 80 1000
10
20
30
40
What percentage of a woman's body consistsof water?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of people grind their teethwhile sleeping?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of the earth's water isfresh (not salt) water?
0 20 40 60 80 1000
2
4
6
8
10
12
What percentage of people struck bylightning immediately die?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of Americans are lefthanded?
0 20 40 60 80 1000
5
10
15
20
25
What percentage of the total number ofbones in the human body are in the feet?
0 20 40 60 80 1000
2
4
6
8
10
12
What percentage of Utah's population isMormon?
Experiment 2 mean errors by individual
21
0 20 40 60 80 100 12010
15
20
25
30
35
40
Individuals (sorted by mean error)
Mea
n e
rror
ove
r q
uest
ion
s
round 1round 2round 3
Experiment 2 mean errors by question
22
0 5 10 15 20 25 30 355
10
15
20
25
30
35
40
45
Questions (sorted by mean error)
Mea
n e
rror
ove
r q
uest
ion
s
round 1round 2round 3
4 questions have very high mean error
Removed the following questions
Four questions had high error rates across individuals 12 16 29 33
(12) In 2005, what was the U.S. military expenditure as percent of the national income?
(16) Across the world, what percentage of people over the age of 15 can read and write?
(29) What percentage of children diagnosed with cancer make a full recovery?
(33) What percentage of people struck by lightning immediately die?
Removed these questions from dataset
23
Replotting mean error per individual
24
0 20 40 60 80 100 1200
5
10
15
20
25
30
35
Individuals (sorted by mean error)
Mea
n e
rror
ove
r q
uest
ion
s
round 1round 2round 3
error very low
25
0 20 40 60 80 100
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Que
stio
ns
Group 2
0 20 40 60 80 100
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Que
stio
ns
Group 2
all raw data from one group with two individuals: in each box we see the responses for three rounds (bottom row in each box is round 1). Dashed line represents ground truth
26
0 20 40 60 80 100
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Que
stio
ns
Group 6
0 20 40 60 80 100
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Que
stio
ns
Group 6
27
0 20 40 60 80 100
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Que
stio
ns
Group 4
0 20 40 60 80 100
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Que
stio
ns
Group 4
Higher accuracy on round 1 when individuals do not change their estimates in subsequent rounds
28
0 20 40 60 80 1000
10
20
30
40
50
60
70
80
90
100
Mean estimate change from round 1 to round 2
Mea
n e
rror
at
rou
nd 1
Correlation = 0.369
0 10 20 30 40 50 600
10
20
30
40
50
60
70
80
90
100
Std estimates across roundM
ean
err
or a
t ro
und
1
Correlation = 0.389
(weak effect though)
If people change their estimate, they generally increase their accuracy on next round
29
0 20 40 60 80 100-80
-60
-40
-20
0
20
40
60
80
100
Change in estimate from round 1 to round 2
MA
D(
rou
nd 1
) -
MA
D(
roun
d 2
)Correlation = 0.547
Wisdom of Crowds Analysis
How accurate is the group estimate compared to the individual estimates? Does the group estimate improve over rounds or do group processes start to correlate the errors, lowering the accuracy of group estimates?
31
MAD for experiment 1 (solo condition)
32
Exp 1 Solo 1 2 3Individual level
11.71 9.82 9.55Combined groups level
Mean 2.19 1.51 1.71Median 1.28 2.56 2.67Mode 5.00 4.72 5.56
Rounds
MAD for experiment 1 (group condition)
33
Exp 1 GroupIndividual level
8.68 6.96 6.50Group level
Mean 5.23 4.69 4.78Median 5.19 5.16 5.11Mode 6.08 5.66 5.58
Combined groups levelMean 1.78 2.75 3.11
Median 2.78 3.50 3.72Mode 3.89 3.61 3.33
MAD for experiment 2 (solo condition)
34
Exp 2 SoloIndividual level
16.97 17.34 17.32Combined groups level
Mean 14.48 14.38 14.70Median 13.01 12.05 12.45Mode 13.54 12.62 13.42
(note: very few individuals in this condition)
MAD for experiment 2 (group condition)
35
Exp 2 GroupIndividual level
17.83 16.03 15.66Group level
Mean 15.43 14.66 14.44Median 14.54 14.20 14.04Mode 15.31 14.67 14.60
Combined groups levelMean 14.28 13.23 12.98
Median 12.34 12.04 11.90Mode 10.12 11.13 11.20
Interpretation of Experiment 1
In the solo condition, accuracy increases because of increased information -- individuals can inspect the same display again in rounds 2 and 3
In the group condition, accuracy increases because of some combination of the increase in perceptual information available and some group dynamic
Possible reasons for difference in accuracy for round 1 in solo and group conditions: Between subject differences -- individuals in group conditions could be
inherently better “Somebody is watching” effect. People try harder when they are in
groups Improved calibration. People are learning, cross trials, by seeing
everybody else's guesses. People never get feedback on what the RIGHT answer is, but they do see other people's responses, and they can use these to calibrate their own responses.
36
Interpretation of Experiment 2
Accuracy did not increase in solo condition not surprising because they have to answer the same trivia
question within short time period caveat: there were a few solo individuals in this experiment.
Individual accuracy does increase in the group condition this strongly suggests that a group process is responsible
37
Interpretation
Over rounds the group estimate does improve, but the effect is fairly small
Therefore, communication between individuals does not necessarily harm the “wisdom of crowd effect”
38
Notes / Ideas
Current experiment is a variant of the Delphi method Look up published papers on the effectiveness of the Delphi
method. Some overview papers have claimed that communication between individuals can improve group accuracy
Can we do an experiment identical to Experiment 1 but the perceptual information is only shown in round 1? This would make the experiment comparable to the trivia questions
Can we do an experiment identical to Experiment 1 but where people do not know at round 1 whether they are in a solo or group condition? This should by design remove any differences found at round 1 and facilitate any differences found at round 2 and 3.
40
Next steps: Bayesian Aggregation Methods (not implemented yet)
Can we improve on these simple group estimation procedures such as the mean or median and get an even better group estimate? Can we build a model that removes any possible biases due to group processes because the model “knows” about such processes?
We need a model that analyzes individuals and groups across questions and rounds taking into account that each individual’s estimate is a blend of their own prior and the estimates of other individuals. This leads to recursive Bayesian models where at one level, we have a model for how people integrate their own information with other people’s estimates and at another level, how we take the output of individuals and make inferences on how people make inferences about other people in order to best infer the group estimate
41