Monitoring – Fit for purpose?Rachel Norman, PhD Researcher &Dr Richard Franceys, Senior Lecturer

1

The Research Inquiry

2

WHAT IS CONSIDERED AN APPROPRIATE LEVEL OF M&E TO ENSURE AN ANSWER OF YES TO THE FOLLOWING QUESTIONS?

Is it there? Is it working continuously? Is it used by all? Is it having an impact?

Where does the responsibility lie in terms of defining what is appropriate?

Does an appropriate level of M&E currently exist in the sector?

Is the implementation of M&E ensuring improved service delivery?

WASH Sector Service Delivery

WHAT COST WHY HOW USE

Research Overview

3

Research Aim To understand better how much WASH sector stakeholders are investing in monitoring and evaluation and how the data generated is being used.

Working Hypothesis Monitoring and evaluation in the WASH sector in lower-

income countries is not value for money …… because it is not used.

Objectives 1. To identify the evolution of M&E. 2. To examine the conceptual framework of Cost of M&E. 3. To examine budgeted and absolute costs of M&E at

global, national and programme level. 4. To explore the underlying purpose and use of M&E. 5. To examine the conceptual framework of VfM and

identify whether there is a core set of indicators. 6. To understand the potential for harmonisation and

alignment with the SDGs and HR frameworks.

Case Study Strategy – Global / Kenya / Uganda Mixed method design whereby qualitative data is used to validate quantitative data collection, analysis and visualisation

Data collection, analysis and visualisation

• Approx. 90 stakeholders identified• Approx. 900 academic articles sourced and reviewed• Approx.1000 (Global, Kenya, Uganda), documents & data archives

sourced and analysed:

• Approx. 90 interviews carried out.

LITERATURE REVIEWS DOCUMENT & DATA ARCHIVE REVIEWS

INTERVIEWS SURVEYS

4

What are monitoring and evaluating?

Literature shows:• 1960s: Coverage data• 1980s: Minimum Evaluation Procedures• 2000s: Guidelines and Toolkits providing

hundreds of indicators to choose to monitor

5

What are monitoring and evaluating?

Country level data analysis shows:• Increasing number of indicators being reported over

time.

6

Where are the indicators coming from?

Kenya: National: NIMES & KJAS

Sector: SWaP, JAR, Regulator

7

Uganda:National: NIMES & JAF & Baraza

Sector: SWaP, JSR, JTR

Programmes and Projects: Organisations, institutions, agencies

National and International public, private and civil society organisations

Clustered data

• Approximately 50% of indicator entries relate to Service level data

• Coverage is still the most frequently reported indicator

• Functionality reporting is limited

8

Service(Consumer)

Service Provider (Regional & Local)

Sector (National)

•Is it there?•Is it working?•Is it being used by the right people?

•Is it having an impact?

•Regional & Local

•National

Why are we monitoring and evaluating?

KENYA• ~ 50% of data records

reported a purpose

UGANDA• < 50% of data records

reported a purpose

9

How is M&E carried out?

Theory• Academic research

• 644,000 articles of which 3% related to WASH

• 79 different tools methods and approaches of which 57% related to WASH such as:

• Randomised controlled trials

• Cost-benefit analysis• Benchmarking• Needs assessments

10

How is M&E carried out? (cont.)

Practically WASH M&E in pictures?

11

• Practical application• Driven by accountability

requirements• Lack of understanding of

the options

What is the cost of M&E?

• M&E…. ‘are expensive activities yet there has been little analysis of whether they provide value for money’ (Maddock, 1993)

• Information systems….. ‘because of the lack of published research in this area, no established methodology could readily be adapted to this study…’ (Rommelmann et al, 2003)

• Multiplicity of M&E systems: ‘they also have a cost in terms of the muddle that it creates…’ (Anonymous, 2011)

12

What is the current spending on M&E?

• Possible spending of sector budget on M&E - Kenya

• MWI Sector Actual / Budget figures are sourced from 2011 Environment Water & Irrigation Sector Report 2010

• Possible spending of sector budget on M&E - Uganda

• WSS sub-sector figures are sourced from MWE (2010) SPR and MWE (2011) SPR

13

Is information actually USED to improve service delivery?

Very limited reported/documented information on use:Used for internal and external accountabilityUsed for internal management decisions

Stakeholder interviews suggest use:• Through learning and knowledge/information sharing• Policy formulation and to inform new planning • Performance monitoring.

14

Analysis of stakeholder interviews on Use

Kenya Uganda

15

Discussion

• Who, What and Why?• Common goal and yet multiplicity of indicators• Internally or externally driven or supported?• The definition crisis causing duplication

• Who, How and Why?• Capacity and capability is an issue• Research and development is an opportunity• Roles and responsibilities are a requisite

16

Discussion (cont.)

• What Cost• Accountability and yet no accountability• Too much or too little – difficult to quantify?• Embedded or extracted?

• How Used• Limited resources – a repetitive explanation• Used but no audit trail• Lead time on use

17

Interim Conclusions

• What is considered an appropriate level of M&E?• Where does the responsibility lie in terms of

defining what is appropriate?• Does an appropriate level of M&E currently exist

in the WASH sector?• Is the implementation of M&E ensuring improved

service delivery?

18

Interim Conclusions (cont.)

19