Randomisation Bias and Post-Randomisation Selection Bias in RCTs:
Barbara Sianesi Institute for Fiscal Studies
September 14, 2006
The role of non-experimental methods in the ERA demonstration
Randomised Controlled Trials in the Social Sciences: Challenges and Prospects
The University of York
Talk Outline
RCTs are the gold standard in evaluation BUT not immune from limitations
1. parameter retrieved2. outcomes that can be looked at
Judicious combination with non-experimental methods can enhance (under suitable assumptions!) what can be learnt from a RCT
Excellent example to illustrate this: ERA
What is ERA
a new package of support financial incentives (job retention,
training) to assist ND25+ and NDLP customers
obtain, retain and advance in work evaluated via RA (~14,000 in 6 districts)
Issue #1
Some eligibles in the ERA Districts did not reach the decision stage or refused to take part in research
scheme
Experimental contrast unbiased estimate of ERA impact for those who have reached the RA stage & have agreed to participate.
ERA impacton ERA
participants
Staff• discretion. choice of marketing strategy
Customer• finding job unlikely• finding job likely but no desire to stay in touch with JCP• antipathy to government, systems of support, mandatory programmes• resistant to change, …
Staff• Some may not think customer would benefit / be interested.• Some non-ERA advisers may think customers close to job entry may provide quick win.
?
?
Non-participants
Why do non-participants pose a potential issue?
Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts.
Benchmark: pilot/control area evaluation In ideal scenario
staff would offer ERA to any eligible (no discretion)
all eligibles would participate (no need for consent)
Why do non-participants pose a potential issue?
Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts.
But: ERA tested only on a subset of ERA eligibles in ERA Districts – the participants.
How to view this
a) Interested in impact on eligibles but only get impact on participants.
“Randomization Bias occurs when random assignment causes the type of persons participating in a program to differ from the type that would participate in the program as it normally operates.”
(Heckman and Smith, 95, p.99)
How to view this
b) Focus on what the RCT consistently estimates (impact on participants) and interested in how well it generalizes to wider population (impact on eligibles).
Issue of External Validity.How representative of the full eligible population?
When are non-participants a problem?
E(impact | eligibles) What we want
When are non-participants a problem?
E(impact | eligibles)
=E(impact | eligible partic.)
what we want
what we get
When are non-participants a problem?
E(impact | eligibles)
=E(impact | eligible partic.) Prob(eligible partic.)
what we want
observed
E(impact | eligibles)
=E(impact | eligible partic.) Prob(eligible partic.)
what we get
When are non-participants a problem?
E(impact | eligibles)
=E(impact | eligible partic.) Prob(eligible partic.)
+E(impact | eligible non-part.)
what we want
what we get
observed
?
When are non-participants a problem?
E(impact | eligibles)
=E(impact | eligible partic.) Prob(eligible partic.)
+E(impact | eligible non-part.) Prob(eligible non-
part.)
what we want
what we get
?
observed
When are non-participants a problem?
eligibles – partic = (nonpartic – partic)Probnonpart
bias = p
It depends on: 1. relative size of eligible non-participant
group 2. whether it is very different from RA
group
I. Descriptive Analysis
I. Extent of non-participation
II. How different
a. observable characteristics
b. outcomes (non-participants vs. controls)
II. Non-Experimental Analysis
Issue #2
Does ERA enhance hourly wages (productivity) wage growth (advancement) employment duration (retention)
Cannot use experimental contrast due to post-randomisation selection bias into employment.
?
To conclude …
“Since experiments can answer only a subset of the questions of interest to the evaluator, it remains important to build up the stock of basic social science knowledge required to successfully utilize non-experimental methods, both in themselves and as tools for more extensive analyses of experimental data.”
Heckman and Smith (1995, p.95)