The PollyVote
Combining forecasts for
U.S. Presidential Elections
Andreas Graefe, Karlsruhe Institute of Technology
J. Scott Armstrong, Wharton School, University of Pennsylvania
Randall Jones, Jr., University of Central Oklahoma
Alfred Cuzán, University of West Florida
The full paper to this talk can be downloaded at: tinyurl.com/combiningelections.
Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences PerspectiveNovember 21, 2010
Background on the PollyVote project
The PollyVote project was begun in 2003 to demonstrate the value of forecasting principles by applying them to election forecasting.
The initial focus was on combining forecasts.
Performance of the PollyVote
The PollyVote combined forecasts to obtain highly accurate forecasts of U.S. Presidential Election outcomes:– Prospectively for 2004 and 2008 (MAE: 0.4 percentage points)
– Retrospectively for 1992 to 2000
Across these five elections, the PollyVote was on average more accurate than each of its components: - Polls- Prediction markets- Experts- Statistical models
Polly achieved this without knowing anything about politics.
Power of combining
Question: What is the ratio of students per teacher in primary schools in Romania?
Judge Estimate Error
1 18 .5
2 19 1.5
Typical error of individual estimate 1
Combined estimate 18.5 1
Error reduction through combining 0%
Judge Estimate Error
1 18 .5
2 16 1.5
Typical error of individual estimate 1
Combined estimate 17 0.5
Error reduction through combining 50%
Procedure and conditions for combining forecasts
Procedure: Mechanically combine forecasts equal weights
(unless you have strong evidence for differential weights)
Conditions:1. Several forecasts available2. Uncertainty about which forecasts is most accurate
(although combing is often beneficial even when the best method is known beforehand)
Conditions for when combining is most beneficial:1. Different forecasting methods are available2. Forecasts rely upon different data
Benefits of combining
1. Improves accuracy
2. Avoids large errors
3. Provides an additional assessment of uncertainty
4. Can be used for nearly all forecasting problems.
5. Simple to describe and apply.
Costs of combining
1. Requires expertise with various methods
2. Higher expenses with more methods
7
Prior research
Meta-analysis of 30 studies on combining: 12% error reduction vs. error of typical component.
Recommendation: Combine forecasts from different methods that use different information
[Armstrong, 2001]
However, few studies have focused on the ex ante conditions of when combining is most beneficial.
8
9
Polly’sComponents
Polly‘s components
PollsIEM
predictionmarket
ExpertsQuantitative
models
10
Polly’sComponents
Polls
Problem:
• Polls often unreliable, especiallyearly in campaign
• Large differences in results of individual polls conducted aroundthe same time
Polls
Within component Combining
11
Polly’sComponents
IEMprediction
market
Within component Combining
• Polly’s prediction market: Iowa Electronic Markets (IEM)
• 7-day rolling average of daily marketprices
• Adjust for overreactions of marketsuch as information cascades
IEM prediction market
12
Polly’scomponents Experts
Within component Combining
• Survey of experts
• Assumptions: Experts possess
• Information from polls
• Knowledge about the effect of debates, campaigns, etc.
Experts
13
Polly’scomponents
Quantitativemodels
Within combining Combining
Models focus on 2 to 7 variables, most often
Incumbent‘s popularity
State of economy
Individual accuracy of modelsvaries across elections
Quantitative models
14
Mean error reduction(93 days prior to Election Day,1992 to 2008)
Polly’scomponents
Gains from combining within components
Polls IEM Experts Models
Within components Combining Combining Combining Combining
14% 9% 21%18%
Polly’scomponents
Combining across components
Polls IEM Experts Models
Within components Combining Combining Combining Combining
Across componentsCombining(unweighted
average)
PollyVote-Prediction
Mean error reduction(93 days prior to Election Day,1992 to 2008)
Polly’scomponents
Gains from combining across components
Polls(combined)
IEM(combined)
Experts(combined)
Models(combined)
PollyVote-Prediction
50% 1% 32%43%
Mean error reduction(93 days prior to Election Day,1992 to 2008)
Polly’scomponents
Gains from combining within & across components
TypicalPoll
OriginalIEM
TypicalExperts
TypicalModels
PollyVote-Prediction
58% 10% 58%52%
If combining forecasts is so useful,
why is it seldom used?
18
1. Managers do not believe combining helps
In four experiments with MBAs at INSEAD, most subjects did not realize that the error of the average forecast would be less than the error of the typical forecast.
Most subjects thought that averaging forecasts would yield average performance.
[Larrick & Soll, 2006]
19
2. Some forecasters mistakenly believethey are combining properly
People often use unaided judgment to assign differential weights to individual forecasts.
Informal combining is likely to be harmful as people can select a forecast that suits their biases.
20
3. Managers, forecasters, and researchers are persuaded by complexity
Simple models often predict complex problems better than more complex ones.
[Hogarth, in press]
These findings are difficult to believe. There is a strong belief that complex models are necessary to solve complex problems.
21
4. Forecasters build reputation with extreme forecasts
Forecasters do not want to get lost in the crowd.
More extreme forecasts usually gain more attention and the media is more likely to report them.
[Batchelor, 2007]
5. People mistakenly believe they can identify the most accurate forecast
In a series of experiments, when given two estimates as advice, most people chose one instead of averaging them – and thereby reduced accuracy.
[Soll & Larrick, 2009]
Why doesn’t the PollyVote capture mass media attention?
The PollyVote varies little and, basically, is never wrong. Thus, no entertainment value.
Instead of accuracy, voters want excitement – and hope for their candidate.
24
Accuracy problem is solved for major elections
PollyVote deviation averaged 0.4% for the 2004 and 2008 U.S. presidential elections and substantial improvements are scheduled for 2012.
Polly is available to researchers and practitioners for elections in the U.S., as well as in other countries.
25
Applications of combining
All organizations can benefit from combining.
References
Armstrong, J. S. (2001). Combining forecasts. In: J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners, Norwell: Kluwer, pp.417-439.
Batchelor, R. (2007). Bias in macroeconomic forecasts, International Journal of Forecasting, 23, 189-203.
Hogarth, R. (in press). When simple is hard to accept. In P. M. Todd, G. Gigerenzer, & The ABC Research Group (Eds.), Ecological rationality: Intelligence in the world. Oxford: Oxford University Press.
Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science,52, 111-127.
Soll, J. B. & Larrick, R. P. (2009). Strategies for revising judgment: How (and how well) people use others’ opinions, Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 780-805.