Building Evidence in Education:Conference for EEF Evaluators

11th July: Theory12th July: Practice

www.educationendowmentfoundation.org.uk

Panel session 1: Grand designs

© Institute for Fiscal Studies

Experimental design for Achieve TogetherEllen Greaves and Luke Sibieta

© Institute for Fiscal Studies

Achieve Together

• Bring together three programmes in a school:– Teach First– Teaching Leaders– Future Leaders

• Intensive human capital investment• Original motivation was also to encourage schools to

work together and to engage the local community and organisation in school-improvement Cluster-design (two clusters) Difficult to evaluate quantitatively

© Institute for Fiscal Studies

Achieve Together

• Two pilots:

1. Area-based design

2. School-level human capital investment

© Institute for Fiscal Studies

Achieve Together

• Two pilots:

1. Area-based design• One-cluster in Bournemouth

• 4 primary schools and 6 secondary schools

• Involvement of local community/organisations

• Process evaluation

2. School-level human capital investment

© Institute for Fiscal Studies

Achieve Together

• Two pilots:

1. Area-based design

2. School-level human capital investment• School-level intervention

• No co-ordination within clusters or involvement of external organisations

• 48 schools recruited for RCT (24 treatment, 24 control)

• Quantitative evaluation and process evaluation

Power calculations

0 0.1 0.2 0.3 0.4 0.5Model 1 0.048 0.203 0.283 0.345 0.398 0.444Model 2 0.052 0.220 0.306 0.373 0.430 0.480Model 3 0.044 0.186 0.259 0.315 0.363 0.406Note: Model 1 reports the detectable effect size when the variance of the outcome unexplained by attributes of the pupils (including prior attainment) is 60%. Model 2 reports a less optimistic scenario (70% unexplained), whilst Model 3 is more optimistic (50% unexplained)

© Institute for Fiscal Studies

Challenges

• School-level RCT began to look clustered...• Cluster based recruitment

• Co-ordination between schools

• Complicates and creates risks for evaluation:

1. What can we learn from the evaluation?

2. How will the power calculations be affected?

© Institute for Fiscal Studies

Challenges

• School-level RCT began to look clustered...• Cluster based recruitment

• Co-ordination between schools

• Complicates and creates risks for evaluation:

1. What can we learn from the evaluation?• Is positive impact due to the human capital approach?

• Or better co-ordination between schools?

• Our findings would be inconclusive

2. How will the power calculations be affected?

© Institute for Fiscal Studies

Challenges

• School-level RCT began to look clustered...• Cluster based recruitment

• Co-ordination between schools

• Complicates and creates risks for evaluation:

1. What can we learn from the evaluation?

2. How will the power calculations be affected?• Required treatment effect from power calculations with

clustering at the school level looked ambitious...

• Clustering may increase the intra-cluster correlation and increase the challenge

© Institute for Fiscal Studies

Power calculations and the risk of clustering

• The ICC may be higher when schools are close together, all else equal (pupils in an area are more similar than pupils in different areas) Minor

© Institute for Fiscal Studies

Power calculations and the risk of clustering

• The ICC may be higher when schools are close together, all else equal (pupils in an area are more similar than pupils in different areas) Minor

• The ICC may be higher when schools work together Less minor Co-ordination across schools may make the pilot more

homogeneous, so pupils’ outcomes will become more correlated

In the extreme case, we can think of the unit of treatment as the cluster

Uncertain risk for detectable effect size Distinct from cluster-based delivery where treatment may

be more heterogeneous across schools

© Institute for Fiscal Studies

Solutions

• Discussion

• Negotiate the focus with the project team

• Highlight the importance of:• Clear conclusions from this important evaluation

• Maximising the probability that the effect size is detectable

© Institute for Fiscal Studies

Conclusions

• Cluster-based interventions increase risk for detectable effect size

• Fine distinction between cluster-based interventions and cluster-based delivery

• Most crucial to define a clear treatment, and for the project team to understand the importance of this

• Negotiation with project team is essential, and can be done!

© Institute for Fiscal Studies

Panel session 1: Grand designs

Piloting regression discontinuity design to evaluate the effectiveness of a Saturday school

programme

Carole Torgerson, Andy Wiggins, Victoria Menzies and Kirsty Younger

Durham University

USING RANDOMISED DESIGNS TO EVALUATE WRITING INTERVENTIONS FOR CHILDREN DURING

THE TRANSITION BETWEEN PRIMARY AND SECONDARY SCHOOL

Conference for EEF Evaluators: Building evidence in education

11th and 12th July

Hannah Ainsworth, Research Fellow, York Trials Unit, University of York

Buckley, H., Heaps, C., Jefferson, L., Mitchell, N., Torgerson, C, and Torgerson, D.

TRIAL DESIGN

Individually Randomised (Discover)

Cluster Randomised (Calderdale)

Split Plot Design (Exeter)

BACKGROUND

EEF Transitions Round: Evaluate 3 different writing interventions with aim to

improve children’s writing skills during transition for primary school to secondary school

In all 3 trials: the primary outcome measure will be the combined score on the 2 writing tasks within the Progress in English 11 (Long Form), GL Assessment (PiE11LF).

Secondary outcomes will be scores on the reading, spelling and grammar components of the PiE11LF

DISCOVER SUMMER SCHOOL

Research question:

“What is the effectiveness of the Discover summer writing workshop intervention compared with a “business as usual” control group on the writing abilities of participating children?”

DISCOVER: INTERVENTION

Intervention group Discover Children’s Story Centre summer writing workshop 4 week programme Specifically aimed at children who are less able writers. The summer school programme will include:

Poetry and literacy sessions, including opportunities to engage with professional poets and writers

Other ‘enrichment activities,’ including drama and sports activities

Trips to West End performances and visits to the Olympic Park.

Control group Business as Usual Offered opportunity to attend a Saturday writing workshop in the

autumn term.

DISCOVER: TRIAL DESIGN

Individually randomised controlled trial Aim to recruit 250 pupils, randomised on a 1:1

ratio to the intervention and control groups

Year 6 pupils will be eligible to be included if they:

1) Are predicted to achieve between level 3c and level 4c in English by the end of Key Stage 2 (based on teacher assessment).

2) Attend a participating primary school

3) Attend a participating secondary school

DISCOVER: TRIAL DESIGN DIAGRAM

DISCOVER: CURRENT PROGRESS

Progress to date Recruitment: 124 pupils Randomisation ratio changed to 3:2 to allow greater numbers in the

intervention group in order to fill more spaces in the summer school Pupils have now been allocated to Intervention/Control Summer school taking place 29th July to 23rd August Post-test being delivered in first 2 weeks of autumn term.

Challenges encountered to date Short time frame involved – parents already had plans for summer holidays

(reduced consent rates, but also expected to reduce summer school attendance rates)

Potential contamination as other summer schools available to control group (and intervention group) pupils

Identification of pupils challenging in transition from Year 6 to Year 7 as must be able to complete post-test to be eligible

CALDERDALE IMPROVING WRITING QUALITY

Research question: What is the effectiveness of the Improving

Writing Quality programme compared with “business as usual” on the writing skills of participating children?

CALDERDALE: INTERVENTIONIntervention Group The intervention will be offered to all pupils in Year 6 Pupils who are predicted to achieve Level 3 or an insecure

Level 4 in English (based on teacher assessment) will continue to receive intervention in Year 7 (Autumn term).

Memorable experiences for pupils in Year 6. Professional development for primary and secondary school

teachers in key elements of the writing intervention Self-Regulated Strategy Development (SRSD), including discussion, modelling and planning.

Control Group Business as Usual Primary schools will be trained following year

CALDERDALE: TRIAL DESIGN

Pragmatic cluster randomised controlled trial.

Approximately 24 schools (feeding into 3

secondary schools) randomised on a 1:1 ratio to intervention and control groups

CALDERDALE: TRIAL DESIGN DIAGRAM

CALDERDALE: CURRENT PROGRESS

Progress to date Recruitment: 23 Primary Schools (possible 24); 3 secondary

schools (possible 4) Successful Information Meeting, to explain intervention and

evaluation, hosted by delivery partner and evaluators Opt Out consent for data sharing: 22 parents opted out Randomised Primary Schools in March 2013 Intervention delivery on-going Post test December 2013

Challenges encountered to date Secure provision of pupil level data at appropriate time

EXETER: GRAMMAR FOR WRITINGResearch questions:

1. “What is the effectiveness of the whole class Grammar for writing intervention compared with a “business as usual” control group on writing skills of participating children?”

2. “What is the effectiveness of the whole class Grammar for writing intervention plus additional small group intervention compared with a “business as usual” control group on writing skills of participating children?”

3. “What is the effectiveness of the whole class Grammar for writing intervention plus additional small group intervention compared with the whole class Grammar for writing intervention only on writing skills of participating children?”

EXETER: INTERVENTIONIntervention:

Cluster Level 15 sequential guided writing sessions delivered during literacy

classes for the whole glass Writing sessions delivered using a number of techniques

(teacher led, DVD, power point, group exercises)

Individual level Whole class intervention plus additional guided writing

sessions delivered in a small group setting

Control

Cluster Level Business as usual

Individual Level Whole class intervention only

EXETER: TRIAL DESIGN Pragmatic cluster randomised trial with a split

plot design Two levels of randomisation – class and

individual

Class Within each school, one class randomly allocated to

whole class intervention and one to business as usual

Individual Within intervention class, pupils expected to achieve

Level 3 or borderline Level 4 in English by the end of Key Stage 2 (based on teacher assessment), individually randomised to whole class plus small group intervention or whole class only intervention

EXETER: TRIAL DESIGN DIAGRAM

EXETER: CURRENT PROGRESS

Progress to date: 55 schools recruited & randomised 2 schools have since dropped out Intervention delivered between 27th May – 5th July Post intervention testing between 1st July – 18th July

Challenges encountered: Short time frames challenging Numerous protocol deviations

CONCLUSIONS

Large variety of randomised designs available Consider specific nature and circumstances Consider carefully the exact research question/s

you will be able to answer with chosen design

Any Questions?