Learning about learningThe GEC approach to M&E
UKFIET Conference
Joseph Holden & Jason Calvert
15th September 2015© PEAS
Structure
2 | UKFIET 2015 Conference
Challenges with the approach
- In school vs. out-of-school approaches- Enumerators and testing- PbR + Accountability vs. flexibility- Cost and capacity
1
2
The GEC approach to M&E
- Background - Focus on outcomes and PbR- Quantitative benchmark for achievement- Mixed methods for outputs- The evaluator model
The GEC, 37 projects working in 18 countries
3 | UKFIET 2015 Conference
The GEC has three major outcomes…
4 | UKFIET 2015 Conference
A L LATTENDANCE
Girls that stay and attend school throughout life cycle of the project
LEARNING
Girls supported by GEC with improved learning outcomes
LEVERAGE
Additional funds secured & sustainability mechanisms established
PbR Indicators: Attendance
Net enrolment
PbR Indicators:
Literacy
Numeracy
Non-PbR Indicators:
Match funding
Influence on policy and community perceptions
A rigorous approach, means control groups…
5 | UKFIET 2015 Conference
Time
Learning Outcomes
Control Group
Intervention Group Counterfactual
Intervention Group
GEC Intervention Commences
Control Group Trajectory
Intervention Group Trajectory
Outcomes are defined as ‘additional’ learning
With a consistent approach to target setting…
6 | UKFIET 2015 Conference
y
75th
percentile
μ
Fre
quen
cy
Score
y
μ 75th
percentile
Fre
quen
cyScore
Distribution of test scores for girls in one grade above in baselineExpected distribution of girls in the intervention group at midline
The target is the expected improvement on the learning test over and
above control group
y
μ = Mean z
Fre
qu
en
cy
Score
σ
σ
Y = 0.2 SD
Evidence supports the focus on learning...
7 | UKFIET 2015 Conference
EGRA words per minute (wpm) for Grade 5 girls. Selected GEC projects + estimate from United States, compared to estimate of speed required for comprehension
8 | UKFIET 2015 Conference
But also a mixed methods approach…
IMPACT LONG TERM STUDY
OUTCOMES
OUTCOME SPREADSHEET
&MID / ENDLINE
OUTPUT LOGFRAME
INPUTSMILESTONES
& VfM TABLES
Improved Employment, Reduced Child Marriage etc.
Learning
Improved teaching skills
Teacher training
Improved access to materials
Distribution of textbooks
Attendance
Access to appropriate
finance
Bursaries
Safe mode of transport
Van / Bus Economy
Efficiency
Effectiveness
9 | UKFIET 2015 Conference
• The independent, External Evaluator:
– Helps to design and finalise tools
– Collects data on learning indicator and information on attendance
– Conducts HHS
– Conducts qualitative research including output indicators if requested
– Produces data and draft report
• The project:
– Contracts the evaluation
– Helps design and finalise tools
– Feeds in theory of change to evaluation
– Conducts own qualitative research including informing logframe indicators, as well as own attendance spot checks
– Quality controls the outputs from the evaluator
And projects must be independently evaluated
Part two: Challenges
10 | UKFIET 2015 Conference
1
2 Challenges with the approach
- In school vs. out-of-school approaches- Enumerators and testing- PbR + Accountability vs. flexibility- Cost and capacity
The GEC approach to M&E
- Background - Focus on outcomes and PbR- Quantitative benchmark for achievement- Mixed methods for outputs- The evaluator model
11 | UKFIET 2015 Conference
• GEC projects employ a longitudinal study, tracking a cohort of girls across 3 years, representative of their beneficiary population.
Finding the same girls is difficult, if OOS…
• Projects have done so using:
– The cohort via a household survey
– The cohort via schools
• There are large statistical benefits to tracking a cohort – reduces standard errors + allows for other factors to be used in regressions
BASELINE MIDLINEENDLINE
12 | UKFIET 2015 Conference
Tests are not perfect, particularly at higher levels..
• Challenges have arisen over the course of the baseline. These include:
• Complexity of enumeration and data collection:
– More training of enumerators at baseline would have been beneficial.
– EGRA – the use and reporting of the timed dimensions of the test.
– EGMA – the need to weight sections equally.
– ASER / UWEZO – scoring tests clearly given a binary, level-based approach to assessments.
• The findings of ceiling or floor effects
– Adaptations have been needed and re-testing where “ceiling” or “floor” effects have been found.
PbR challenges: accountability vs flexibility…
13 | UKFIET 2015 Conference
14 | UKFIET 2015 Conference
To ensure complete and good quality data, and that enumerators and data processors (e.g. data entry clerks and translators) are collecting and processing data in the correct way, fieldwork monitoring and data quality assurance should always be divided into three key components:
1. Pre-fieldwork processes
2. Fieldwork
3. Post-fieldwork processes
Investing in all three processes may save time and resources in the future. MANY GEC PROJECT EVALUATORS HAVE LACKED CAPACITY AT THESE STAGES.
Cost and capacity constraints for fieldwork…
15 | UKFIET 2015 Conference
Respondent Fatigues
Attendance data
- Accuracy- Spot checks
Organisational Capacity
- Enumeration and - data issues
PbR relatedRisks
- Stat significance
Ethical Concerns
- Control groups
Fragile States
- Security- Access
Balance between evaluation and implementation
So, a host of challenges, this is not easy….
16 | UKFIET 2015 Conference
80,000The amount of girls in the full GEC cohort (approx.)
Outcomes model VfM methodology used across other DFID projects
Quantitative and qualitative data contributing towards detailed
understanding on the context and education barriers in
developing countries
But also some achievements!