Tim Brennan and John McKibbin

REFLECTIONS ON MONITORING A LARGE-SCALE CIVIL SOCIETY WASH INITIATIVE

A/Prof Juliet Willetts, Bruce Bailey, Dr Paul Crawford

THINKCHANGEDO

Lessons for sector monitoring and potential contributions from NGOs

IRC Symposium 2013 Addis Ababa

Griffin nrm

Key messages• Three lessons for sector monitoring from donor monitoring:1. Investing in a ‘learning focus’ as part of sector or donor

monitoring is critical

2. Tensions between specificity and flexibility in choosing indicators and frameworks must be addressed carefully

3. An actor-centred ‘theory of change’ provided a helpful, readily understandable structure

• A wide range of strategies are available to NGOs to support sector monitoring (but so far are little adopted)

A large-scale civil society WASH fund… $32.5 million 11 CSOs 21 countries

45 activities$200K- >$1.6m

782,000 people with toilets

564,000 people with water

Our role in the Fund… diverse activities of the monitoring review panel (MRP)…

• Engaged for life of the program

• Designed the performance framework and reporting requirements

• Conducted analysis and synthesis – learning papers

• Undertook monitoring of Fund activities

• Provided an M&E capacity building role

• Undertook two independent evaluations of the Fund

Lesson 1 from donor monitoring: Invest in a learning focus

• ‘Developmental evaluation’ approach – what did we do?:• Collected ‘real-time data’ to inform on-

going decision making and adaptations• Included descriptive free-form case

studies as well as indicators• Used evaluation questions (not just

indicators)• Facilitated, structured learning ‘events’• On-going one-on-one feedback to NGOs • Learning ‘products’ that synthesise

lessons across contexts

Developmental evaluation adapts evaluation (or monitoring) to “emergent and dynamic realities in complex environments” (Patton, 2011, p1).

Lesson 2 from donor monitoring: Balance specificity and flexibility

• Why is specificity needed in sector monitoring?

• Why is flexibility needed in sector monitoring?

• How can these needs be balanced?

7 mandatory indicators

19 ‘result areas’

Flexible guidance on

what to monitor or

report

We achieved balance by defining some indicators very tightly, and making space in other ways for more diverse, flexible inputs

Lesson 3 from donor monitoring: Actor-centred theory of change was helpful

• Monitoring is usually about measuring ‘change’

• ‘Theories of change’ simplify reality to make it easier to measure, communicate and understand how we think change happens

• We can then also test our assumptions about how change happens

DELIVERABLES:

E.g. training, mentoring, advocacy, technical advice, campaigns etc.

ENABLING OUTCOMES:

Outcome 4: improved WASH governance

Outcome 5:strengthened local capacity

Outcome 6: improved gender equality

Outcome 7: improved WASH evidence base

CORE OUTCOMES:

Outcome 1: increased access to sanitation

Outcome 2:increased access to water

Outcome 3:improved hygiene behaviour

CSO

Delivery Team

Institutional &

community partners

Poor communities

& households

Theory of change behind the performance arrangements

NGO roles to support sector monitoring: a ‘strategy map’Strategy Causal Persuasive Supportive

I-1 I-2 I-3Focused on a particular individual or group

E-1 E-2 E-3Focused on the enabling environ-ment

Strategy map was developed by IDRC as a part of ‘outcome mapping’ (Earl et al., 2002)

Strategy Causal Persuasive Supportive

I-1 I-2 I-3Focused on a particular individual or group

Direct role in monitoring own direct implementation activities.

Providing awareness raising, education or specific training to community members or other partners.

Providing frequent, sustained, on-going mentoring and support; or multipurpose capacity building; or developing support structures, committees & networks.

E-1 E-2 E-3Focused on the enabling environ-ment

Engaging in policy dialogue, directly causing changes in incentives, rules; playing an advocacy role.

Dissemination of information to broad audience; persuasive environment for a specific behaviour or attitude; conducting workshops/conf’s.

Building partnerships, providing collective support and promoting networking and coordination; also supporting higher levels of government or local research networks.

Examples of NGO roles and their benefit to sector monitoring

• Direct monitoring role (I-1 strategy map)

“one NGO in Bangladesh showed that government (Department of Public Health Engineering) data for sanitation coverage of 38-58% in three ‘upazilas’ in 2009 was actually at 5-28% when the NGO conducted their baseline study in 2010, with these large differences gaining attention from relevant government staff”

Examples of NGO roles and their benefit to sector monitoring• Systematic strategies to build local government expertise in monitoring (I-3)

“one NGO working in Indonesia and Bangladesh directly supported existing government monitoring systems for sanitation by working in a direct support role for local government”

Examples of NGO roles and their benefit to sector monitoring• Document and share own learning/innovations and promote their uptake(E-2)

“An NGO working in Bhutan achieved adoption of elements of their project monitoring framework which used a qualitative information scale for examining the hygienic status of toilets, and also includes a scale for examining handwashing with soap.

Their ideas were taken up and integrated into government monitoring systems, including Ministry of Health’s Annual Survey in February/March each year, and inclusion of indicators to sanitary access and hygienic toilet usage in the next Five Year Development Plan from mid-2013 as Key sector Result Areas” (Halcrow, pers comm. 2013)”

So what? What needs to change for NGOs to better support sector monitoring?

Government staff should:• Consider and communicate the range of roles which

NGOs might play in support of national initiatives

Program designers (both NGOs and donors) should:

• Make efforts to adopt strategies explored in this presentation, particularly considering strategies not yet in use

M&E specialists:• Should align monitoring and evaluation systems of

WASH projects to sector monitoring, instigate the sharing of information with government

Key messages• Three lessons for sector monitoring from donor monitoring:1. Investing in a ‘learning focus’ as part of sector or donor

monitoring is critical

2. Tensions between specificity and flexibility in choosing indicators and frameworks must be addressed carefully

3. An actor-centred ‘theory of change’ provided a helpful, readily understandable structure

• A wide range of strategies are available to NGOs to support sector monitoring (but so far are little adopted)

Thank you

Contacts: Juliet.Willetts@uts.edu.au, pcrawford@aid-it.com, bbailey@griffin-nrm.com.au

Griffin nrm