Randomised controlled trials

Peter John

Causation in policy evaluation

OutcomeIntervention

Other agency actions

External environment

Do not do this (before and after study)

A Randomised Controlled Trial

• A RCT is an experiment where two or more groups of subjects who are compared with a control group that does not get the treatment

• Randomisation between treated and control subjects ensures that there are no other outcome differences between the groups than from the intervention

• Focus on comparing average outcomes in each group

Figure 7.1 Simple Random Allocation Design

                                                                                                                                                                                                                                                                                                                                      

Simple Random Allocation Design

Policy experiments

• A distinct form of RCT which involves the policy-maker delivering an intervention. Goes back to the 1920s - e.g. school milk experiment (sabotaged by teachers)

• Policy-maker randomises, researcher evaluates• Different to participation experiments, such as

GOTV + various forms of experiment we in NNTT• Can be part of an official evaluation, but new

model whereby done informally

The new nirvana?• Promoted by UK Behavioural Insights Team – had

little money, but look at what it did with it: RCTs on tax letters, texting for court fines, charitable giving, DVLA reminders, organ donations with DVLA, job centres, work with firms

• Massive expansion in development field• US interest - education federal funding, social

policy, welfare to work• Use of experiments by economists (e.g.Duflo)• Now experiments in local government

Example: Essex County Council

Blue Badge Renewals: Channel ShiftPeter John and Toby Blume

Essex County Council

Research design

• We set out to test whether local authorities can ‘nudge’ residents to renew their Blue Badge permits online rather than by post – channel shift

• Three types of message were tested for Blue Badge permit holders over a period of two months (December 2014-Jan 2015)

• Two months to go

Treatments and sample

• 2798 renewals were sent out: 699 allocated to control, 698 to the simplified condition, 698 to the incentive group and 699 to the messenger group

• Treatments:– Simplification– Messenger - the messenger treatment included a

picture and testimonial from another Blue Badge holder of how they have renewed online and encouraged others to do the same.

– Incentive - appealing to common good

Incentive letter

Results• Simplification significantly increased online renewal

rates compared against the control group by 7.3 percentage points.

• Offering an intrinsic incentive – that does not offer a personal benefit but rather a collective incentive, so the individual is prompted to act for the benefit of others – also significantly increased renewal rates compared with the control group by 7.3 percentage points.

• The use of a peer messenger – another Blue Badge holder – to encourage online renewal did not appear to have an eff

Control Simplified Incentive Messenger0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

3037.3 37.3

31.8

40.432.2 36.1

37.4

29.6 30.5 26.630.9

Online Paper Not renewed

Results

The seven things you need to plan to do a RCT

1. Establish a question that can be answered by a trial

2. Work out the units3. Determine sample size 4. Specify the treatment arm or arms5. Set out randomisation procedure6. Specify outcomes and their measurement7. Address ethical issues and data protection

Step 1: Establish a question that can be answered by a trial

• Not all questions can• Need to be a good question – you do not

know the answer• Is the theory plausible (a mechanism you

expect)? Theory of change (ToC)• Can you intervene in a way that changes

behaviour or outcomes (or attitudes) in a set time period?

Step 2: Work out the units

• Can be individuals, households, streets, communities, larger areas, firms, other organisations, classes

• Can be nested in units, so individuals in streets, or students in classes

More on units

• Governed by a set of practical question about availability

• Data records will drive this• Costs of measurement (sequeway to sample

size)• Recruitment – there will be drop out >

selection bias

Step 3: work out sample size

• A topic of its own involving use of statistics• Basic message is that RCTs need large groups• Rough minimum of 400 per group• Avoid too small trials• Sample size calculators can help (be careful

with the online ones)

More on size

• To calculate the right sample size, you need to have an idea of what the expected effect is going to be, usually expressed as point differences between two groups, (e.g. .5 and .6 for a ten percentage point difference).

• You need to specify in advance what probability level you are prepared to accept. Conventionally we have a two-sided test at .05

• You need to set the statistical power that is desirable, which is usually 80 per cent.

Step 4: Specify the treatment arm or arms

• Need to ensure the treatment reflects what you want to measure,

• Need for precision in delivery• Ensure the treatment is not doing too

many things at once or that you break up different elements into treatment arms

Step 5: Set out randomisation procedure

• Allocate subjects using random numbers: > http://www.random.org/

• Or create a random number seed e.g. in Excel

• Good idea to ask an outside person to randomise to ensure it is implemented effectively

• You can test for balance across T and C

Step 6: Specify outcomes and their measurement

• Need something measurable that is standard across the units and T&C, e.g. achievement, volunteering

• Can be interval (that is in equal units, such as money donated) or categorical (e.g. volunteer not not volunteer) or even ordered (e.g. who arrives first), can be measured over time, or before and after (desirable but not essential)

Step 7: Address ethical and data protection issues

• All research needs to be guided by ethical issues – consent, avoidance of harm, respect for participants (e.g. right to withdrawal, privacy), maintain safety

• Trials often thought to be vulnerable because of randomisation may ‘deny’ benefits or impose costs (in practice these happen anyway)

• Also may involve deception that needs to be justified