Operationalizing IE:Case Study example: Textbooks and

Teacher Training in Sierra Leone

APEIE Workshop, May 10-14 2010

Operational Key Steps

2. IE Design

4. InterventionStarts

1. Policy

5. Survey

6. Analysis, Reports, Policy Advice3. Baseline

Survey

IE Design – methodological issues

Available data and required data Identify control group and treatment group

– Randomization– Transparency

Sampling and sample sizes– Power calculations– Budget and access constraints

Identify outcome indicators

Data Instruments

Utilize survey experts Who do you survey?

– Households, community, facilities, government…

Pilot instruments– Clear questions and coding– Includes all information needed for analysis and

covariates

Intervention Roll Out

Possible contamination of evaluation results!– Intervention benefits rolled out to control and

treatment schools evaluation compromised– Intervention or treatment schools only receive

additional benefit evaluation compromised

Monitor roll out– Train implementers – they need to know why!– Hire supervisors– Continue supervision throughout roll out

Background

In June 2007, the World Bank and a team from the Sierra Leone Ministry of Education and Sports (MOES) met to rigorously evaluate the impact of two pilot studies on student performances.

Two questions– What is the effect of textbooks on student performance?– What is the effect of teacher training on student

performances?

Random Selection– 5 districts in Sierra Leone

Kailahun, Kambia, Pujehun, Western Rural, Western Urban– Neediest schools in district (using EMIS data)

Student to teacher ratio, presence of latrine, school construction

360 schools total Random Assignment

– Comparison schools: receive regular allotment of books– Books schools: receive additional books– Books and training schools: receive additional books +

teacher training

How do we design the study to get rigorous results?

Timeline

Sept: WB and MOE agree on pilot

Oct: Met with local councilors to select districts

Nov: Baseline data collection and teacher training manual developed

Nov: Mid-term data collection

June: Baseline data collection

Dec: Roll out of textbooks and teacher training

June: Roll out of textbooks

2007 2008 2009

Dec: Mid-term data collection

NO teacher training up to this point

REALITY

TARGET

What happened?

Met with several challenges on the ground that caused significant delays in the project:

– Textbook distribution– Teacher Training– Data Collection

Challenge 1: Textbook Distribution

Target date: December 2007 Actual date: June 2008 Number of books:

– MOES accidentally distributed original books, so had to use Sababu books

– had fewer books than project initially called for Communication – information not filtering down

– Sababu not properly informed – Service Providers unwilling/unable to change distribution– Headteachers in comparison schools unaware of project

Challenge 1: Textbook Distribution continued

Textbook distribution: June 2008 Data collection: December 2008 Consequences:

– Not enough books distributed in some districts to make a statistical comparison

– Distribution happened very late and at a bad time (right before summer vacation)

Allowing for only a small window for students to utilize the extra textbooks

– Some Comparison schools received books

Challenge 2: Teacher Training

Target date: December 2007 Actual date: has not yet occurred Time and budget constraints The ‘+’ part of the ‘intervention +’ so took a

back seat to the textbook distribution

Challenge 3: Mid-term Data Collection

Target date: November 2008 Actual date: December 2009 Delayed funding disbursement Non-existent schools (poor quality of initial

data) Headteachers unaware of project – not

willing for enumerators to collect data

Solutions:

Back up list of schools on hand for enumerators Better communication with all levels of organization

involved Constant project sensitization Greater monitoring of distribution Inclusion of Process Surveys Allow for sufficient time for procurement and

disbursement

Next Steps

Analyze mid-term data – Was there a significant effect due to textbooks?– Has education in Sierra Leone improved in the past 2

years?– Where in the textbook distribution are there major

informational stop gaps?

Move forward on teacher training Write up findings and circulate report

Conclusion

Impact evaluations a useful tool for policy considerations

But need to understand every process involved in the impact evaluation

– Textbook distribution, not just textbooks Study can be ineffective if you don’t have full buy-in

and understanding at all levels of government, all service providers, and headteachers:

– SENSITIZATION!– COMMUNICATION!– MONITORING!

THANK YOU