Logo

Zenebe Uraguchi

HELVETAS Swiss Intercooperation Bangladesh

Regional Workshop25.09.2013

Ben Tre, Vietnam

Monitoring and Results Measurement (MRM) System

Logo Why implement MRM system based on the DCED standard ?

Internally & externally

Cornerstone of good project steering Guidance for strategic direction for

achieving impacts sustainably

Externally

Demonstrates impact to the donors & other stakeholders

Contributes to credibility of projects

Internally

Fundamental to the learning process by providing vital information that projects need

Enhances making good decisions in order to maximize impacts

DCED: Donor Committee for Enterprise Development

Logo What is included in the (DCED) standard

The DCED is a forum; 23 members (bilateral donors, including SDC, multilateral agencies and private foundations

Member agencies and others share experiences and formulate a common understanding of good practices

7. Reporting

The DCED Standard

8. Managing the

System

4. Estimating attributable changes

5. Capturing wider changes in the system

1. Articulating the

results chains

2. Defining indicators

of change

3. Measuring changes in indicators

6. Tracking programme costs

Logo What is new as a system?

• Do events unfold as planned? Yes? No? Why?• An internal system for MRM instead of external consultants

One-off events

External consultants

Internal MRM system

Improving intervention logics

Logo

Source: Peter Roggekamp, Hans Posthumus

What we don’t want to do?

series of changes resulting from activities of the project

local systems –people, businesses, government, civil society organizations and the relationships, rules, policies and cultural norms that govern how they interact and work together

Goal

Logo

Source: Peter Roggekamp, Hans Posthumus

What we want to do?

LogoWhy MRM is important!

Source: Peter Roggekamp, Hans Posthumus

Logo

Source: Peter Roggekamp, Hans Posthumus

LogoLet us talk about the ball & goal

Systematic Intervention

Market trigger

Market uptake

Enterprise performance

Impact

The ball is the series of changes resulting from activities of the project

A results chain or impact logic clearly outlines the series of changes project staff expect, starting with project activities and ending with goal level impacts

LogoAn example from RLDP, Tanzania

Farmers attend demo plot sessions

Farmers understand technology

Farmers apply technology

MSK establishesdemo plots

Farmers‘ groups sign contract with MSK

MSK distributes inputs

Farmers apply inputs

Farmers increase yield

Farmers increase net income

MSK sensitize farmers and fgroups

LogoAn example from Samriddhi, Bangladesh

Facilitating technical capacity of market

actors

Improving access to technologies/good practices & inputs

Forming & strengthening

producers’ groups

Facilitating access to adapted financial

services & products

Systemic intervention

Market trigger

Enhanced capacity of local service providers

on technical knowledge & skills

Increased engagement of private sector

enterprises

Enhanced capacity of local service providers & producers’ groups in business planning & financial literacy

Improved organizational &

business skills of local service

providers

Service provision for affordable & quality services & inputs available; improved linkages with private sector enterprises & financial service providers established; improved &/or new technologies &

practices in bull fattening available

Better functioning market system

Market uptake

Increased production & productivity

Increased additional employment & net income by producers

Improved performance

Impact

Producers start establishing functional

relationships with private sector

enterprises

Producers apply improved bull rearing

practices using quality inputs & services

tPoducers start developing & implementing business plans by

identifying & negotiating with a range of financial service providers

Logo

An example on results chain: direct & indirect outreach

Training manual developed with the support of the project / other players e.g., Department of Livestock Services

Activity

Market trigger

Market uptake

Improved performance

Impact

Appropriate private sector company identified; deal made

Local service providers selected

Private sector company provided training to local service providers on improved bull fattening technology

Local service providers improved their skills & knowledge on improved bull fattening technology

Local service providers provided information on services to producers

Producers are aware of service availability & are willing to receive & pay for the services

Producers started using the services & adopted improved bull fattening technology

Farmers fattened bulls in short cycles

Farmers received better price for their bulls

Increased profit for farmers

Additional income of xx Taka for yy farmers

Other farmers are influenced to fatten bulls using new

technology

Additional farmers received better price for their bulls

Increased profit for indirect farmers

Indirect

outreach

LogoHow about developing a results chain from your project?

Systemic intervention

Market system changes

Market uptake

Improved performance

Impact

Tips:

1)Please select one intervention 2)Please be as detailed as possible

in the impact logic (changes at each level)

LogoResults Chain and Logframes

Compatible

Results chain increase the relevance of Logframe as management tool

Results chain provide details of the process for achieving results

LogoCompatible with Logframe

•

• • •

• • •

• • •

Log Frame

Results chain

LogoDefining Indicators

# box

indicator

12

11

10

9

8

7

6

5

4

3

2

12

3

4

5 6

7 8

9

1

10

11

12

LogoHow to formulate indicators

• Each box needs at least 1 indicator• Use quantitative and qualitative indicators

- Behaviour, perception, motivation, …• Indicators should be S M A R T !

- Specific- Measureable- Attributable- Relevant- Timely

LogoIndicators….

We should always ask: “Is each piece of information essential to decision making?”

What is a good indicator?

Relevance

Precision

% growth in a business membership organization

% change (year-on-year) of total businesses paying full membership fees in business

membership organization

Lazy & loose

Tight & precise

Time bounded-ness (e.g., profits in past month vs.

in past year )

Realistic

LogoDefining indicators of change

An indicator

shows what success looks like

outputs are produced & outcomes are achieved

if

indicators specify how the project will measure

therefore

the changes anticipated in the results chains are really occurring

whether

if so to what extent?+ how?

if not then why?

LogoIndicators

indicators to be derived from the logic of the results chain

Farmers reduced post-harvest loss of vegetables

Farmers adopt improved plastic crates for harvesting

and grading vegetables

Number of farmers who adopted plastic crates

% reduction of wastage or % increase in the sales of

produce

Quality of produce

+

Quantitative indicator

Quantitative indicator

tracking these indicators very important frequency of tracking will depend on the specific targets or projections

set by the project

Quantitative indicator

LogoIndicators & quantitative projection

Projections: change in the value of an indicator due to the project by a specific date

What to do?

1) At the start of activities, make clear projections about expected impacts

Why we do this?

give staff targets to aim forprovide staff with feedback if an

intervention is on track

2) Clear projection of key quantitative indicators

give an indication of whether particular investments are worth their cost & impacts

3) Do market research, have field observations or other credible sources (e.g., secondary resources)

provide realistic / credible projection

4) Periodically update projections (actual vs. projection)

Help reflect new data collected on indicators of change

LogoAn example from PakistanAssumption Calucation Projection

50% success rate 445*50% 222 garment producers improveOct-13

GPs produce 6 units per month

150Rs/unit*6units

Garment producers increase by 900Rs/month. Apr-13

Margins increase by 150Rs/unit

445 garment producers earn +150/unit Apr-13

75% success rate 594*75% 445 garment producers produceApr-13

90% success rate 660*90% 594 garment producers improveFeb-13

Each sales agent services 30 garment producers

22*30 660 garment producers servicedFeb-13

75% uptake rate 30*75% 22 women adoptDec-12

30 women trainedOct-12

Women adopt new service agents business model

Sales agents deliver services to garment producers

Garment producers improve knowledge of production techniques

Garment producers produce high quality garments

Garment producers earn higher margins

Garment producers increase incomes

Garment producers improve social status in house holds & community

Project trains

women to adopt new

sales agents business model

Constraint: Limited mobility of female garment producers (GPs)Identified solution: Sales agents (SAs) travel and provide services

LogoData collection tools

ObservationIn-depth

interview

Key informant interview (KII)

Company / partner records

Focus Group Discussions

(FGD)

Outsourced surveys

Secondary

Sources

In-house

surveys

LogoObservation

Source: HPC 2012

LogoSecondary sources

Source: HPC 2012

LogoCompany records

Source: HPC 2012

LogoFocus Group Discussion (FGD)

Source: HPC 2012

LogoStakeholder consultation

Source: HPC 2012

LogoIn-depth interviews

Source: HPC 2012

LogoIn house surveys

Source: HPC 2012

LogoOutsourced surveys

Source: HPC 2012

LogoBalance between rigor and cost

Rigour

Costs

Observations

Big surveys

LogoMix of methods and triangulation

2

3

4

5 6

7 8

9

1

10

11

12

Same tool for several boxes For one box: use several simple tools for

triangulation

Tool 1

Tool 2

Tool 3

Document the process, not only findings; state the limitations

LogoMeasuring change• Necessary to measure change• Ideally Pre-Intervention• Sources:- Previous monitoring data- Previous market analyses- Additional data collection and

analysis e.g. through surveys, interiviews, FGD, etc.

• Usually multiple baseline are needed across interventions

Impact

Indicator

LogoMissing baseline

• What if baseline has not been collected?• What if the beneficiaires have not been known at the start of

the intervention?

- Use secondary data;- Derive a baseline from retrospective research;- Use data from other regions with similar demographics etc.- Use partners‘ records

LogoResults Measurement Plans: example

Level Impact chain Questions Indicators How? Who ? When? What do we have?

Impact Additional income

Whether farmers and laborers have increased income?

Net increase in income

Survey; secondary sources

Research firm; sector team

June xx Census from xx; secondary source

Market uptake

Farmers cultivated early varietyVegetables

Do the farmers cultivate early variety vegetable?Do they get proper information?

No. of farmers who cultivate earlyvariety vegetable?Farmers' knowledge level

Survey Research firm; sector team

July XX

Producers improved their knowledge oncultivation techniques of early varietyVegetables

How many producers have been reached throughpromotional activities?How much have they learnt from the promotional activities?

No. producers aware;change in knowledge of producers

Observation of theorganizers, reports

Sector team

June xx

LogoEstimating attributable changes

“it is better to be approximately right than precisely wrong.“John Maynard Keynes

The outcomes might have happened without

the intervention (counterfactual)

Target groups might have

been better off without the intervention

(displacement or unintended

consequences)

The outcomes might not be due to the intervention, but due to other factors (e.g.,

projects, weather changes)

Logo

b= before

Impact= a-c

a= after

c= changes without the project

Time

Impact

Intervention starts Intervention ends

LogoAttribution: Influencing factors

Impact

External factors

Causality

Many

Few

Easy to establish

More difficult

to establish

Adapted from A. Miehlbradt

LogoAttribute plausibly

Impact

External factors

Causality

Many

Few

Easy to establish

More difficult

to establish

Increasing complexity

Demonstrate rigorously

Validate transparently

Estimate contribution transparently

Adapted from A. Miehlbrandt

LogoLessons from Samriddhi

Samriddhi had its “mock” audit for the compliance of its MRM system with the DCED standard in 2013.

= Complies with the Standard = Additional work required = Current system does not address

The project was assessed to have 75% of compliance with the standard, which was good.

Areas of improvement include:

1)Developing results chain not per value chain but by intervention2)Collecting specific and detailed information from individual

producers, and not from producers’ groups

LogoSuccess factors for Samriddhi’s MRM system

1) Designing MRM system to support decision making at all levels

2) Availability of MRM manual describing and elaborating the “nitty-gritty” of the MRM system

3) Making MRM system part of everyone’s job

4) Giving equal emphasis to qualitative and quantitative data with core indicators

5) Developing an integrated MRM tool

6) Ensuring staff training as a priority

7) Cooperation from the donor and good steering of the project

Logo

Thank you

LogoComparing different tools

Tool What is it Purpose Sampling guide Tips

Observation quick assessment of activity effectivenessgathers qualitativeinformation; collects preliminary information before the use of other tools

validate findings from other tools; understand the process of change

required for regular monitoring purposes

undertaken in the normal course of work or plannedeffective if a checklist of required information is prepared before a visitMay include informal discussions; familiarity with context required

In-depth interviews

are detailed and require a lot of time toextract explanations

In-depth analysis of change Validating facts through details

number depends on the context of thestudy or impact assessmentrespondents are intentionally chosen for relevant information

watch out for coached responses from the respondentsuse follow up questions and probes to elicit more extensive responses

LogoComparing different tools

Tool What is it Purpose Sampling guide Tips

Key Informant interviews (KII)

qualitative in‐depth interviews with individuals whohave first‐hand knowledge of the issues

collect information from a wide range ofindividuals who can provide important and significant insight into the issue

Sample size ranges from 2‐5 individuals

must have first‐hand knowledge about the issues associated with theIntervention diverse set of representatives are preferable

Pocket surveys

gather information on quantitative indicators when significant rigor isnot needed

validate the detailed findings of in‐depth interviews, with a largersample size

sample size ranges from 20‐40

questions should be straight‐forward and generally directed at getting facts

Focus Group Discussions (FGD)

conducted when consensus is needed on a topic, or to get a common view of attendants

explore the process of change extensively; provide deeper understanding ofwhy changes are taking place

the group consists of 8‐15 people

guideline should be prepared on need of info

LogoComparing different tools

Tool What is it Purpose Sampling guide Tips

Formal surveys

conducted for 'special cases' whichrequire us to generate specific quantitative information

validate the findings of an informal survey or an FGD

Depends on the size of the population; usually between 200-400

Questionnaire is structured with closed ended questions, based on the checklist Simple questions to get facts