Multiverse analysis of natural experimentsSystematic execution, presentation and interpretation of robustness analyses
Dave Balzer and Nico Sonntag
November 17, 2020
Multiverse analysis of natural experiments November 17, 2020 1 / 21
Forking paths and researcher degrees of freedom
Reported
Analyzed
a) Traditional analysis b) Robustness analysis c) Multiverse analysis
Multiverse analysis of natural experiments November 17, 2020 2 / 21
Multiverse analysis
It is possible to calculate all plausible model combinations automatically. We build onapproaches that have been around for some time. For example:
• “I Just Ran Four Million Regressions” (Sala-i-Martin 1997)• “Multimodel analysis” und Stata module mrobust (Young & Holsteen 2017)• “Multiverse analysis” (Steegen et al. 2016)• “Specification curves (Simonsohn et al. 2015)• “Coefficient stability plots” (Rao 2020)
Our own approach emphasizes
• Relevance for sociology and natural experiments• Includes further degrees of freedom• The aim is to assess which decisions are particularly critical for results
Multiverse analysis of natural experiments November 17, 2020 3 / 21
Harmless?
Multiverse analysis of natural experiments November 17, 2020 4 / 21
Unexpected Event during Surveys Design
Terrorist attack during ESS field work as a natural experiment (Legewie 2013):
• Randomisation of whole periods (before/after attack)
• Estimate: ATE (?)
Multiverse analysis of natural experiments November 17, 2020 5 / 21
Unexpected Event during Surveys Design
“In many ways, this identification strategy resembles a regression discontinuity design.”
δ
-500
0
500
1000
1500
2000
outc
om
e
-100 -50 0 50 100
forcing variable
• Randomisation only around the threshold
• Correctly specified functional form
• Estimate: LATE (local average treatment effect)
Multiverse analysis of natural experiments November 17, 2020 6 / 21
Unexpected Event during Surveys Design
If longitudinal data are available, a difference-in-differences model could also beestimated:
Differenz 1
Differenz 2 }Treatment-Effekt (ATT)
Zeitpunkt 1 Zeitpunkt2
Ou
tco
me
Kontrollgruppe
Treatmentgruppe
• (As-if) randomisation with regard to time trends (common trends assumption)
• Estimate: ATT
Multiverse analysis of natural experiments November 17, 2020 7 / 21
Our research question
Multiverse analysis of natural experiments November 17, 2020 8 / 21
Our research question
Ele
ctio
n
ControlGroup Treatment
Group(N = 2660)(N = 1817)
010
020
030
0
Num
ber
of R
espo
nden
ts/D
ay
29.03.2005 19.04.2005 10.05.2005
Interview Date
Benedict XVI
Ele
ctio
n
ControlGroup Treatment
Group(N = 5750)
(N = 3580)
010
020
030
040
050
0
Num
ber
of R
espo
nden
ts/D
ay
20.02.2013 13.03.2013 03.04.2013
Interview Date
Francis
Multiverse analysis of natural experiments November 17, 2020 9 / 21
Our research question
To what extent did the 2005 and 2013 papal elections influence reported religious activity?
Features of our research project
• GSOEP allows the analysis of longitudinal data
• We are able to compare two similar events with the same data set
• Flexible study design: illustrative application of multiverse analysis
Multiverse analysis of natural experiments November 17, 2020 10 / 21
Religious and national identity
Mechanisms:
• Interplay between religious and national identity
• Do publicly visible religious leaders increase the salience of religion in their countryof origin?
Hypotheses:
H1: Only the 2005 papal election, but not the 2013 papal election, should haveincreased reported religious activity.
H2: The 2005 papal election primarily influences respondents with low religiousactivity.
H3: The 2005 papal election primarily affects respondents without establishedreligious identity.
Multiverse analysis of natural experiments November 17, 2020 11 / 21
Data summary
Election Benedict XVI (2005):
• GSOEP v33: years 2001, 2005, 2007
• Total N: 64 342
• N ± 3 weeks: 20 296
Election Francis (2013):
• SGOEP v33: years 2011, 2013, 2015
• Total N: 86 872
• N ± 3 weeks: 44 078
• Outcome 2013 scaled differently!
0 10 20 30 40 50 60percent
every week
every month
less frequently
never
Rel
igio
us p
artici
pation
(se
rvic
e vi
sit)
Years 2001, 2005, 2007. N = 63,940.
Multiverse analysis of natural experiments November 17, 2020 12 / 21
Multiverse analysis: Election Benedict XVI (2005)
An exemplary multiverse analysis includes several variants of a regression discontinuitydesign:
• Time trend: linear, quadratic or cubic
• Constant or changing slope after treatment (interaction time trend × treatment)
• Day 0: treatment or control group
• Sub-sample: Catholics vs. non-Catholics
• Bandwidth: [7; 42] days before and after the election
3× 2× 2× 2× 36 = 864 models
We focus on 144 models with bandwidths at weekly intervals (7, 14, 21 ... 42 days).
Multiverse analysis of natural experiments November 17, 2020 13 / 21
Multiverse analysis: Election Benedict XVI (2005)
7
28
141421
7777
21
71421
42
7
35
2114
28
77
2828
1421
42
14
4235
7
3535
14
4235
7
42
14
2828
14
42
77
21
35
7
35
14
28
77
42
1414211421
35
14
28212821142121
2835
14
42
147
28
42
28
4235
14
42
28353535
28
4242
21
42
2121
42
2121
4242352821
42
7
3528
424242353535
283535
2828283528
14
423528
42
21
35
14212121
7
21141414
77
28
7
3528
42
21
714
7b-widthcatholic = 0catholic = 1
day 0 controlday 0 treatment
cubicquadratic
linearinteract
no interact
Specifications
-1.5
-.75
β 0
.75
1.5
Specification curve: RDD
Multiverse analysis of natural experiments November 17, 2020 14 / 21
Multiverse analysis: Election Benedict XVI (2005)
Another exemplary multiverse analysis covers variants of the panel models:
• Fixed- vs. random effects
• Year 2007 in analysis sample yes/no
• Day 0: treatment or control group or exclusion
• Sub-sample: Catholics vs. non-Catholics
• Bandwidth: [7; 42] days before and after the election
2× 2× 3× 2× 36 = 864 models
Again, we focus on 144 models with bandwidths at weekly intervals (7, 14, 21 ... 42days).
Multiverse analysis of natural experiments November 17, 2020 15 / 21
Multiverse analysis: Election Benedict XVI (2005)
777714
7771414
7
211414
2128
142114212828
2114
35
14
2828
77
35
21
4235
7
28
7
21
7
28
7
212114
42
14
35
7
28
7
423528282828
35
714
423535
4242
2828
42
7
35
21
7
35423528352835353535
42423535
4242
21
7
42352835
7
3542
7
4242
2828282828
7
2128
7
28
424235424242
2121
35
21211421
42
211414
4235
141414
42
212121212121141414141414
b-widthcatholic = 1catholic = 0
day 0 missing
day 0 controlday 0 treatment
w 2007w/o 2007
FERE-.3
-.15
β 0
.15
.3
Specification curve: Panel/DiD
Multiverse analysis of natural experiments November 17, 2020 16 / 21
Multiverse analysis: Crucial decisions
RD specifications:
• Overall, the results do not confirm the hypothesis
• Little systematic relationship between decisions and effect size• Non-Catholic respondents• Day of election = treatment
Panel specifications:
• Some evidence of a positive treatment effect
• Three patterns• Non-Catholic respondents• Bandwidth of 14 or 21 days• Inclusion of the year 2007
Multiverse analysis of natural experiments November 17, 2020 17 / 21
Multiverse analysis: Election Benedict XVI (2005)
0
20
40
60
80
freq
ency
-.8 -.4 0 .4 .8β
catholic = 1
0
20
40
60
80
freq
ency
-.8 -.4 0 .4 .8β
catholic = 0
0
10
20
30
freq
ency
0 .2 .4 .6 .8 1p value
0
10
20
30
freq
ency
0 .2 .4 .6 .8 1p value
0/1 1/2 2/3
cutpoint
Multiverse analysis of natural experiments November 17, 2020 18 / 21
Conclusion and discussion
Multiverse analyses help to increase transparency
• Very few natural experiments are “harmless” in the sense that they unambiguouslycall for a particular research design
• Thus, there are many researcher degress of freedom in the analysis of naturalexperiments
Open questions
• Shifting the problem to another level?
• Overburdening the readers?
• Best possible (graphic) presentation of results?
• Difficulty of model comparisons
• Too pessimistic about theory-driven model selection?
Multiverse analysis of natural experiments November 17, 2020 19 / 21
Thank you for your attention! Pax et bonum!
Multiverse analysis of natural experiments November 17, 2020 20 / 21
References
Angrist, J. D. and J.-S. Pischke (2008). Mostly harmless econometrics: An empiricist’s companion. Princeton universitypress.
Legewie, J. (2013). Terrorist events and attitudes toward immigrants: A natural experiment. American journal ofsociology 118(5), 1199–1245.
Rao, A. (2020). Coefficient Stability Plots. https://github.com/AakaashRao/starbilit.
Sala-i-Martin, X. X. (1997). I just ran four million regressions. American Economic Review 87.
Simonsohn, U., J. P. Simmons, and L. D. Nelson (2020). Specification curve: Descriptive and inferential statistics on allreasonable specifications. Available at SSRN: https://ssrn.com/abstract=2694998.
Steegen, S., F. Tuerlinckx, A. Gelman, and W. Vanpaemel (2016). Increasing transparency through a multiverse analysis.Perspectives on Psychological Science 11(5), 702–712.
Young, C. and K. Holsteen (2017). Model uncertainty and robustness: A computational framework for multimodelanalysis. Sociological Methods & Research 46(1), 3–40.
Multiverse analysis of natural experiments November 17, 2020 21 / 21