Basic Principles of Monitoring and Evaluation

Basic Principles of Monitoring and Evaluation

Wednesday, August 10, 20162:00 PM (EDT)

Dr. Beverly J Nyberg

Webinar Series: 81

2

3

1. Measure and communicate the scope, scale, distinctiveness

and results of faith‐based groups’ work…

The Lancet, Vol 386, Oct. 31, 2015

4

Goals of this webinar

To help participants:1. Recognize the importance of M&E for your organization 2. Create a desire to have a robust M&E implementation process3. Understand the components of an M&E system4. Assess where you are at in the M&E implementation process5. Set goals for your next step for M&E implementation6. Provide resources for you to take that next step for M&E

implementation

5

M&E Session Overview1. WHY: Discuss why M&E is

important

2. WHAT: Describe the components of an M&E system and related terms

3. HOW: Suggest resources for each of those components

6

Monitoring – Routine tracking of data to learn to what extent implementation is occurring and progress is being made, e.g. tracking inputs and outputs.

Monitoring data indicates:• how programs are conducted. Is the intervention on track or on budget? (inputs & activities)• program’s level of performance. Is it reaching the desired number of households? (outputs)

Evaluation – Systematic collection of information to learn why and/or to what extent a program has been successful/unsuccessful. Ascertains whether the inputs and outputs have produced the desired results – that is, evaluating their effectiveness

Evaluation data indicates:• whether programs achieved what they were intended to achieve (outcomes & impact)

What is M& E?

7

Inputs

• Money/funds• Trained personnel/staff• Other resources and supplies• Other contributions (volunteers)

8

What you need to implement the program

Activities

• Provide training workshops for personnel and site managers

• Teach classes• Develop curriculum • Supervise

9

What the program does/provides to accomplish its objectives

Outputs

• Materials distributed• Number of students taught• People reached• Intervention sessions completed

10

What the program delivers

Outcomes

Changes in • knowledge • behavior • attitudes • skills

11

The results of the program or changes that occur both immediately or some time after activities are completed

Impact

Possible examples:• Health and longevity improved (nutrition programs)• Violence against children decreases (parenting programs, etc.)• Decreases in HIV mortality and morbidity (HIV programming)• Improved economy (education programs/savings and loans groups)

12

The wider effect of the program on long-term results

Indicator

13

a written measure used to demonstrate change in a situation, or the progress in, or results of, an activity, project, or program.

Example: (OVC_SERV) Number of beneficiaries served by PEPFAR OVC programs for children and families affected by HIV.

What’s the difference?Monitoring –

Evaluating –

Tracks what we are doing (primarily inputs & outputs) to assess whether programs are performing according to plan. Are we doing things right?

Assesses if we have achieved the desired results (outcomes & impact)Are we doing the right things?

Review

14

Monitoring Data 1

• Provides regular feedback that measures change over time in any of the program components such as costs, personnel and program implementation

15

Global: HKID Funding by Year, 2005‐2014

$0

$50,000,000

$100,000,000

$150,000,000

$200,000,000

$250,000,000

$300,000,000

$350,000,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

16

Global: OVC Served by Year, 2005‐2014*

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

*2014 contains people served over 18 years old through OVC programs. All other years contain only under 18 year olds.

17

Monitoring Data 2

• Can be analyzed to track the efficiency of achieving program goals and objectives

18

HKID Funding per OVC Served by Year, 2005‐2014**

$0

$20

$40

$60

$80

$100

$120

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

** 2014 only includes the under 18 year‐olds of those countries that disaggregated by age. 19

Monitoring Data 3

• An unexpected change in monitoring data may trigger the need for a more formal evaluation of activities

20

Global: Served by OVC Programs by Age and Sex, 2014

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

<1 1 to 4 5 to 9 10 to 14 15 to 17 18+

Male

Female

21

Evaluation Data

• Can be used to assess the effectiveness, relevance and impact of achieving your program’s goals.

• Often involves measuring changes in knowledge, attitudes, behaviors, skills, community norms, utilization of services

• Provides feedback that helps programs analyze the consequences, outcomes and results of its actions

22

Program Evaluation

23

Programstart

ProgramendTIME

Change inprogramoutcomes

With program

Without programProgrameffect

0

0.5

1

1.5

2

2.5

Pre Post

Physical abuse p<.001

00.51

1.52

2.53

3.5

Pre Post

Emotional Abuse p<.001/.002

Caregiver

Teen

18.5

19

19.5

20

20.5

21

21.5

22

22.5

Positive Parenting p<.001/<.01

7

8

9

10

11

12

13

14

Pre Post

Poor monitoring p<.002/<.04)

Caregiver

Teen

Results of parenting program

Cluver, L – SinovuyoTeen

24

Why do M & E?• To ensure effective and efficient use of resources and provide accountability to donors

• To assess whether the project has achieved its objectives ‐ has the desired effects

• Program improvement• To learn from our activities, and provide information to design future projects

• To make decisions about project management and service delivery and broader policy

To ImprovePrograms

To ImprovePrograms

Reporting/Accountability

Reporting/Accountability

Assess achieve-ments

Assess achieve-ments

Poll

25

to integrate M&E into our programs?

How do we begin…

26

Fundamental Steps to Carry out M&E1. Construct a logic model/logframe and/or a theory of change

for your program2. Develop an M&E plan3. Select indicators4. Collect information on the selected indicators 5. Analyze the information gathered6. Compare results with program initial goals, objectives and LM7. Share/Use the data

27

Poll

Basic Logic Model

28

Logic Model causal pattern

• Funds• Staff• Resources

Inputs

• What you will do

Activities• Participants• Products

Outputs

• Short• Intermediate• Long

Outcomes

29

30

Be clear about the assumptions you are making about what will cause/ bring about the desired outcomes

Inputs Activities OutcomesOutputs

?

Bought & distributed backpacks

EducationalSupport: $$, staff

1,500 OVC received backpacks

Child is enrolled in and regularly attends school.

Short-term outcome

Intermediate outcome

Child graduates from secondary school.

Gets a great job!Long Term outcome Impact Economy improves31

AB

32

Theory of Change

Long Term Outcome

Necessary Precondition





33

10.4

5.14.6

2.2

0.0

2.0

4.0

6.0

8.0

10.0

12.0

No intervention Child grant Free school Both interventions

FEMALES:%PROBABILITYOFINCIDENCEOFTRANSACTIONALSEX(modeledpercentageprobabilitiesusingmultivariatelogisticregressioncoefficients)

Using research to design your program

Cluver, Orkin, Yakubovich 34

0

0.5

1

1.5

2

2.5

Pre Post

Physical abuse p<.001

00.51

1.52

2.53

3.5

Pre Post

Emotional Abuse p<.001/.002

Caregiver

Teen

18.5

19

19.5

20

20.5

21

21.5

22

22.5

Positive Parenting p<.001/<.01

7

8

9

10

11

12

13

14

Pre Post

Poor monitoring p<.002/<.04)

Caregiver

Teen

Results of parenting program

Cluver, L – SinovuyoTeen

Evidence‐based programming

35

Difference between LM & TOC

Logic Models (LM)graphically illustrate program components. Creating one helps stakeholders clearly identify outcomes, inputs, and activities • Representation • List of Components • Descriptive

Theories of Change (ToC)Link outcomes and activities to explain HOW and WHY the desired change is expected to come about• Critical Thinking • Pathway of Change • Explanatory

36

LOGIC MODEL Objective/goal: Planned Work Intended Result Situation/ problem statement:

Inputs/Resources Activities Outputs Outcomes/Impact What we invest: In order to accomplish our goal we need the following:

What we will do: In order to

achieve our goal we will do the following:

Participants: Who we reach

Products: What we create

Short‐term outcomes: At the end of

program/service we expect changes in skills, awareness,

knowledge, attitudes, motivations, behaviors

or behaviors that participants intend to

change.

Intermediate outcomes: Several

months to a few years after the end of the program we expect changes in actions (behavior, practice, decisions, policies) by participants based on what they learned.

Long term/Impact: A year or several years

(7‐10 yrs) after program completion we expect changes in conditions, policies, or enduring changed

behavior.

ASSUMPTIONS: EXTERNAL FACTORS:

37

Resources for

Step 1 ‐ Construct a logic model and/or a theory of change for your program

• Logic Model:• W.K. Kellogg Foundation Logic Model Development Guide• http://www.smartgivers.org/uploads/logicmodelguidepdf.pdf

• Theory of Change• Theory of Change website: http://www.theoryofchange.org/• http://www.dochas.ie/Shared/Files/4/Theory_of_Change_Tool_Manual.pdf

38

Fundamental Steps to Carry out M&E1. Construct a logic model and/or a theory of change for your

program2. Develop an M&E plan3. Select indicators4. Collect information on the selected indicators5. Analyze the information gathered6. Compare results with program initial goals, objectives and LM7. Share/Use the data

39

Program Components as They Relate to M&E

Processes/Activities Outputs Impact

SituationAssessments

Input & Output Monitoring Outcome Monitoring

Outcome Evaluation

InputsPlanning Outcomes

Impact Monitoring

Impact Evaluation

Agree on the scope and objectives of an M&E plan

40

41

Defining the Scope Scope refers to the extent of the activity you will undertake in an M&E effort. Scope is determined by several factors (questions):

• What should be monitored and evaluated?• When should programs be monitored and evaluated?• How much will M&E cost?• Who should be involved in monitoring?• Who should carry out the evaluation?• Where should M&E take place?• How will we collect the data?

Resources for

Step 2 ‐ Develop an M&E plan• Measure Evaluation template for developing an M&E plan: http://www.cpc.unc.edu/measure/resources/training/materials/avian‐and‐pandemic‐influenza‐monitoring‐and‐evaluation‐1/me‐work‐plan‐template‐for‐avian‐influenza

• http://www.tools4dev.org/resources/monitoring‐evaluation‐plan‐template/

• http://www.thehealthcompass.org/how‐to‐guides/how‐develop‐monitoring‐and‐evaluation‐plan

• http://usaidprojectstarter.org/content/project‐me‐plan

42



43

Develop indicators for each phase of your program

Processes/Activities Outputs Impact

Input & Output Monitoring Outcome Monitoring

Inputs Outcomes

Impact Monitoring

44

Monitoring & Evaluation level of effort

Some All Most Few

Levels of Monitoring & Evaluation Efforts

• Resources• Staff• Funds• Materials • Facilities• Supplies• Training

• Materials availability• Trained staff • Quality of services

• Knowledge increased• Behavior change• Attitude change • Changes in trends• Increase in social support

InputsOutputs

Short-term and intermediate effects

Long-term effects

Outcomes

Impact

45

LOGIC MODEL Objective/goal: Situation/ problem statement:

Inputs/Resources Activities Outputs Outcomes/Impact What we invest: In order to accomplish our goal we need the following:

What we will do: In order to

achieve our goal we will do the following:

Participants: Who we reach

Products: What we create

Short‐term outcomes: At the end of

program/service we expect changes in skills, awareness,

knowledge, attitudes, motivations, behaviors

or behaviors that participants intend to

change.

Intermediate outcomes: Several

months to a few years after the end of the program we expect changes in actions (behavior, practice, decisions, policies) by participants based on what they learned.

Long term/Impact: A year or several years

(7‐10 yrs) after program completion we expect changes in conditions, policies, or enduring changed

behavior.

Indicators

ASSUMPTIONS: EXTERNAL FACTORS:

46

What data do we collect?

Example of PC cascade of nutritional indicators

Inputs Output Short‐term outcome

Intermediate outcome

# of PC Volunteers trained in nutritional service provision

# of individuals who received food or other nutritional services

# of individuals benefiting from community or household gardens

# of individuals who improved their nutritional status as a result of receiving nutritional services

47

Indicator reference sheet – OVC_SERV (PEPFAR)• Define indicator –

• Numerator: Number of beneficiaries served by PEPFAR OVC programs for children and families affected by HIV.

• Denominator (if applicable) (NA in this case)• Disaggregation –

• Age & sex are important: <1, 1‐9, 10‐14M, 10‐14F, 15‐17M, 15‐17 F, 18‐24 M, 18‐24 F, 25+ M, 25+ F

• Status: Active, Graduation, Transferred, Exited without graduation• Service area: Education support, Parenting/Caregiver programs, Social Protection (including cash transfer), Economic Strengthening, Other service areas in line with PEPFAR 2012 guidance for OVC programming.

• Frequency of collecting the data –• Every 6 months (end of Q2 & Q4)

• Tools to collect the data –• Attendance records, service delivery records

48

OVC_SERV Indictor Reference SheetDescription: Number of beneficiaries served by PEPFAR OVC programs for children and families affected by HIV.

Numerator: Number of beneficiaries served by PEPFAR OVC programs for children and families affected by HIV.

The numerator is the sum of the Program Participation Disaggregation: active beneficiaries, graduated beneficiaries, transferred, and beneficiaries who have exited without graduation in the reporting period, from the PEPFAR OVC Program.

This indicator is a direct (output) measure of the number of individuals receiving PEPFAR OVC program services for children and families affected by HIV/AIDS.

This indicator tracks progress on how successful the program is at graduating beneficiaries and also tracks “exited without graduation” (such as loss‐to‐follow up, aging out without transition plan, moved, or died).

Transferred to existing host‐country programs, where the host‐country program provides a sustainable response to OVC needs.

Graduation will vary based on local criteria for achieving stability in the household.

Denominator N/A 49

USAID Indicator Reference Sheet

50

51

GOOD=4Child seems happy, hopeful, and content.

FAIR=3Child is mostly happy but occasionally he/she is anxious, or withdrawn. Infant may be crying, irritable or not sleeping well some of the time.

BAD=2Child is often withdrawn, irritable, anxious, unhappy or sad. Infant may cry frequently or often be inactive.

VERY BAD=1Child seems hopeless, sad, withdrawn, wishes could die or wants to be left alone. Infant may refuse to eat, sleep poorly or cry a lot.

Factor 5A. Emotional Health

Goal: The child is happy and content with a usually positive mood and hopeful outlook.

52

Resources for Step 3 – Select & define indicators

• Child Status Index materials • http://www.cpc.unc.edu/measure/resources/tools/child‐health/child‐status‐index/child‐status‐index‐tool‐kit

• Guide for how to develop indicators • http://www.thehealthcompass.org/how‐to‐guides/how‐develop‐indicators

• World Vision – child wellbeing outcome indicators• http://www.wvi.org/sites/default/files/Compendium%20of%20Indicators‐English.pdf

• Guide for indicator reference sheet – USAID• https://www.usaid.gov/sites/default/files/documents/1861/Recommended_PIRS_for_USAID_indicators_0.pdf

53



54

How to collect dataData collection methods:• Memory• Paper & pencil• Cell phone• Tablet/computer

Cell phone apps• Digital improve data quality• The most feasible of digital methods• Remote Data Collection: 10 of the best apps for gathering data in the field https://zapier.com/learn/ultimate‐guide‐to‐forms‐and‐surveys/best‐data‐collection‐apps/ (cost)

• Free ones, too – SocialCops (Google Play)55

56

57

How to collect dataWhere to store data:• Paper records (never get analyzed)• Digital

• Excel• Access• Other databases• Web based is more accessible and

can be in real time

58



59

Data Analysis

Monitor• Who?• What?• Where?

Starts with a good question. What do you want to know that the data can tell you?

Evaluate• Why?• How?

60

Analyze the data• Methods for data analysis

• Qualitative and quantitative demand different methods of analysis but both are types of data are important to include

• Software (excel, access, SPSS, etc.) • Don’t be afraid to ask for help

• How to represent the data• Tables and charts with data• Graphics can be more powerful and easier to read• Graphics can also be very complex• Adding GIS coordinates can add another dimension to your analysis

61


0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

<1 1 to 4 5 to 9 10 to 14 15 to 17 18+

Male

Female

62

OVC_SERV: Number of OVC receiving services (PEPFAR, FY15)

63

TX_CURR: Number on treatment

PLHIV:Number of people living with HIV

(PEPFAR, FY15)

64



65

Compare Results

• Go back to your logframe/logic model or Theory of Change and start asking a lot of questions:

• Did you accomplish your goals? Did you reach your targets?• Are you doing things right?• Are you doing the right things?• Were your underlying assumptions about the program effects correct?• Were there external factors that impacted your results?

66

Global: OVC Served by Year, 2005‐2014*

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

*2014 contains people served over 18 years old through OVC programs. All other years contain only under 18 year olds.

67


0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

<1 1 to 4 5 to 9 10 to 14 15 to 17 18+

Male

Female

68

Disparityinnewinfectionsamongyoungpeople

Source: UNAIDS 2014

NewHIVInfectionsinSub‐SaharanAfricabyAge&Sex,2013

69



70

Share your data!

• Share up for stakeholders• Share with program implementers to see what they have accomplished and where there is room for improvement

71

Where are you at in the process?

1. Construct a logic model and/or a theory of change for your program

2. Develop an M&E plan3. Select indicators4. Collect information on the selected indicators5. Analyze the information gathered6. Compare results with program initial goals, objectives and LM7. Share/Use the data

72

A few tips re: M & E

• Begin with focusing on monitoring

• Hire experts to do your evaluation

• Use program interventions that have already been proven to be effective (e.g. Sinovuyo Parenting Program)

• Terms are not used consistently – which can be confusing

73

Additional M&E Websites• MEASURE Evaluation:

• www.cpc.unc.edu/measure• http://www.cpc.unc.edu/measure/resources/training/online‐courses‐and‐resources

• Health Metrics Network: www.who.int/healthmetrics• John Snow Inc.: www.jsi.com• HIV Global Partners: www.globalhivmeinform.org

74

75

•What are your next steps?

Thank you!

76

Date post:	04-Apr-2023
Category:	Documents
Upload:	khangminh22
View:	0 times
Download:	0 times

Basic Principles of Monitoring and Evaluation

Documents