Learning about learningThe GEC approach to M&E

UKFIET Conference

Joseph Holden & Jason Calvert

15th September 2015© PEAS

Structure

2 | UKFIET 2015 Conference

Challenges with the approach

- In school vs. out-of-school approaches- Enumerators and testing- PbR + Accountability vs. flexibility- Cost and capacity

1

2

The GEC approach to M&E

- Background - Focus on outcomes and PbR- Quantitative benchmark for achievement- Mixed methods for outputs- The evaluator model

The GEC, 37 projects working in 18 countries

3 | UKFIET 2015 Conference

The GEC has three major outcomes…

4 | UKFIET 2015 Conference

A L LATTENDANCE

Girls that stay and attend school throughout life cycle of the project

LEARNING

Girls supported by GEC with improved learning outcomes

LEVERAGE

Additional funds secured & sustainability mechanisms established

PbR Indicators: Attendance

Net enrolment

PbR Indicators:

Literacy

Numeracy

Non-PbR Indicators:

Match funding

Influence on policy and community perceptions

A rigorous approach, means control groups…

5 | UKFIET 2015 Conference

Time

Learning Outcomes

Control Group

Intervention Group Counterfactual

Intervention Group

GEC Intervention Commences

Control Group Trajectory

Intervention Group Trajectory

Outcomes are defined as ‘additional’ learning

With a consistent approach to target setting…

6 | UKFIET 2015 Conference

y

75th

percentile

μ

Fre

quen

cy

Score

y

μ 75th

percentile

Fre

quen

cyScore

Distribution of test scores for girls in one grade above in baselineExpected distribution of girls in the intervention group at midline

The target is the expected improvement on the learning test over and

above control group

y

μ = Mean z

Fre

qu

en

cy

Score

σ

σ

Y = 0.2 SD

Evidence supports the focus on learning...

7 | UKFIET 2015 Conference

EGRA words per minute (wpm) for Grade 5 girls. Selected GEC projects + estimate from United States, compared to estimate of speed required for comprehension

8 | UKFIET 2015 Conference

But also a mixed methods approach…

IMPACT LONG TERM STUDY

OUTCOMES

OUTCOME SPREADSHEET

&MID / ENDLINE

OUTPUT LOGFRAME

INPUTSMILESTONES

& VfM TABLES

Improved Employment, Reduced Child Marriage etc.

Learning

Improved teaching skills

Teacher training

Improved access to materials

Distribution of textbooks

Attendance

Access to appropriate

finance

Bursaries

Safe mode of transport

Van / Bus Economy

Efficiency

Effectiveness

9 | UKFIET 2015 Conference

• The independent, External Evaluator:

– Helps to design and finalise tools

– Collects data on learning indicator and information on attendance

– Conducts HHS

– Conducts qualitative research including output indicators if requested

– Produces data and draft report

• The project:

– Contracts the evaluation

– Helps design and finalise tools

– Feeds in theory of change to evaluation

– Conducts own qualitative research including informing logframe indicators, as well as own attendance spot checks

– Quality controls the outputs from the evaluator

And projects must be independently evaluated

Part two: Challenges

10 | UKFIET 2015 Conference

1

2 Challenges with the approach

- In school vs. out-of-school approaches- Enumerators and testing- PbR + Accountability vs. flexibility- Cost and capacity

The GEC approach to M&E

- Background - Focus on outcomes and PbR- Quantitative benchmark for achievement- Mixed methods for outputs- The evaluator model

11 | UKFIET 2015 Conference

• GEC projects employ a longitudinal study, tracking a cohort of girls across 3 years, representative of their beneficiary population.

Finding the same girls is difficult, if OOS…

• Projects have done so using:

– The cohort via a household survey

– The cohort via schools

• There are large statistical benefits to tracking a cohort – reduces standard errors + allows for other factors to be used in regressions

BASELINE MIDLINEENDLINE

12 | UKFIET 2015 Conference

Tests are not perfect, particularly at higher levels..

• Challenges have arisen over the course of the baseline. These include:

• Complexity of enumeration and data collection:

– More training of enumerators at baseline would have been beneficial.

– EGRA – the use and reporting of the timed dimensions of the test.

– EGMA – the need to weight sections equally.

– ASER / UWEZO – scoring tests clearly given a binary, level-based approach to assessments.

• The findings of ceiling or floor effects

– Adaptations have been needed and re-testing where “ceiling” or “floor” effects have been found.

PbR challenges: accountability vs flexibility…

13 | UKFIET 2015 Conference

14 | UKFIET 2015 Conference

To ensure complete and good quality data, and that enumerators and data processors (e.g. data entry clerks and translators) are collecting and processing data in the correct way, fieldwork monitoring and data quality assurance should always be divided into three key components:

1. Pre-fieldwork processes

2. Fieldwork

3. Post-fieldwork processes

Investing in all three processes may save time and resources in the future. MANY GEC PROJECT EVALUATORS HAVE LACKED CAPACITY AT THESE STAGES.

Cost and capacity constraints for fieldwork…

15 | UKFIET 2015 Conference

Respondent Fatigues

Attendance data

- Accuracy- Spot checks

Organisational Capacity

- Enumeration and - data issues

PbR relatedRisks

- Stat significance

Ethical Concerns

- Control groups

Fragile States

- Security- Access

Balance between evaluation and implementation

So, a host of challenges, this is not easy….

16 | UKFIET 2015 Conference

80,000The amount of girls in the full GEC cohort (approx.)

Outcomes model VfM methodology used across other DFID projects

Quantitative and qualitative data contributing towards detailed

understanding on the context and education barriers in

developing countries

But also some achievements!