Evaluating the impact of OVC programs: standardizing our methods

Jenifer Chapman, PhD

Senior OVC Advisor

MEASURE Evaluation

Overview

Background

Purpose of the OVC Survey Tools

Process of development

Guiding principles

Structure and content

When to use the Tools

When not to use the Tools

Using the data

Tools in a Toolbox

There is no single data collection tool that can meet all OVC program targeting, case management and M&E requirements.

This set of survey tools responds to distinct information needs related to program planning and evaluation, and fills a tools gap.

The problem

High investment in OVC programs BUT impact is unclear & questions regarding “what works” in improving household well-being

Part of the challenge: lack of standardized measures and tools for child and household outcomes (well-being)

4

A proposed solution

Standardized questionnaires for use in a survey of children ages 0-17 years and their adult caregivers

The purpose

Standardize population-level child and caregiver well-being data beyond what is available from routine surveys

Produce actionable data to inform programs and enable mid-course corrections

Enable comparative assessments of child and caregiver well-being and household economic status across a diverse set of interventions and regions

Who are these tools for?

Local and international research institutions and other implementing organizations with evaluation agenda

Our Process

Two step, participatory process:

Build consensus around core impact indicators for PEPFAR-funded OVC programs

Develop OVC program evaluation (survey) tools

Distilling the core indicators

Step 1: Extensive literature search

Step 2: Gaps (HES, PSS) filled through targeted research

Result: >600 child/HH wellbeing questions/indicators

Step 3: Analysis against 8 criteria

Result: shorter list of questions for discussion

9

Inclusion criteria

1. Measures impact/outcomes

2. Amenable to change from program interventions

3. Relevant across a wide range of interventions

4. Contributes to a holistic vision of child wellbeing

5. Verifiable through another source

6. Easy to implement

7. Relevant across different regions / countries

8. Relevant or easily adapted across age and sex

10

Finalizing the core indicators

External working group: solicited review from 49 stakeholders

Received feedback from > 25 individuals/groups

Finalized core set of 12 child and 3 household measures

11

From indicators to tools: Guiding principles

Questionnaires measure program outcomes

Program outcome data should be collected by trained data collectors

A documented protocol is required

Protocol with tools needs to undergo ethical approval both in the country of data collection and in the US

Tools require pilot testing in new settings before use

Developing the tools

Tools drafted with strong stakeholder input

Draft tools piloted in Zambia (and Nigeria) Cognitive interviews to

test key concepts (e.g. social support)

Household pre-test of full tools, procedures

Structure and content

1. Caregiver questionnaire (including questions on household)

2. Child questionnaire (ages 0-9 years), administered to caregiver

3. Child questionnaire (ages 10-17), administered to child with parental consent & child assent

Sections Core questions Optional modulesSection 1: Household schedule

Household schedule* (10) Changes in household

composition (4)

 

Section 2: Background Information on Caregiver and Household

Demographic information* (7) Work* (3) Access to money (3) Shelter (1)

Household Economic Status (forthcoming)

Progress out of Poverty Index or similar (country specific)

Section 3: Food Security Household food security (6) Dietary Diversity (1)

Section 4: Caregiver Well-being and Attitudes

General health (2) Caregiver support (4) Parental self-efficacy (1)

Perceptions and experience of child discipline, violent discipline (forthcoming)

Gender roles and decisionmaking power* (9)

Section 5: HIV/AIDS Testing, Knowledge, Attitudes

Basic HIV/AIDS knowledge* (7) HIV testing* (3) Attitudes to condom educ (1)

HIV/AIDS attitudes* (4)

Section 6: Access to HIV Prevention, Care & Support

Household access to services (1)

 

*DHS, bold=core indicator

Caregiver questionnaire

Sections Core questions Optional modulesSection 1: Child Health and Welfare

Confirm demographics (5) General health & disability (4) Birth certificate (2) Vaccinations (11) Fever (<5 years)* (1) Diarrhea (<5 years)* (1) Experience of neglect (2) Slept under mosquito net* (1) HIV testing experience* (2)

Fever: extended* (4) Diarrhea: extended* (3) Health for children

living with HIV/AIDS (forthcoming)

Section 2: Education and Work

School attendance*, progression/repeats, drop-outs, missed school days (5+ years) (9)

Work for wages (2) Early childhood stimulation (2)

 

Section 3: Food Consumption

Food consumption (2+ years) (8) • Dietary diversity (1)

Section 4: Access to HIV Prevention, Care & Support

Child access to services (1)  

Section 5: Anthropometric Measures (of Children)

Weight*, Height*, MUAC  

Child questionnaire (ages 0-9)

*DHS, bold=core indicator

Child questionnaire (ages 10-17)Sections Core questions Optional modulesSection 1: Background Information on Child

Confirm demographics* (5) Identity of caregiver (1)

 

Section 2: Diary Daily log (6)  Section 3: Education School attendance*,

progression/repeats, drop-outs (9)

 

Section 4: Chores & Work Chores (3) Work (7)

 

Section 5: Food & Alcohol Consumption

Food consumption (8) Alcohol consumption (3)

Dietary diversity (1)

Section 6: Health, Support & Protection

Birth certificate (2) General health & disability (3) General support (4)

Health for children living with HIV/AIDS (forthcoming)

Perceptions/experience of violence (forthcoming)

Section 7: HIV Testing, Knowledge, and Attitudes

Basic HIV/AIDS knowledge* (7) HIV testing * (3)

Child development knowledge (6) HIV/AIDS attitudes and beliefs (4)

Section 8: Sexual Experience • Sexual behavior (13-17 yrs) (5)

Section 9: Access to HIV Prevention, Care & Support

Child access to services (1)  

Section 10: Anthropometric Measures: Weight and Height

Weight, Height, MUAC  

When are these the right tools?

Tools are useful if your question is:

1.Is my program having, or did my program have an impact on the children and households it reached?

2.What are the characteristics of children and their caregivers in my country, state/province or district/area, in terms of education, health, protection, and psychosocial support?

3.Where do the children most in need of program support live?

4.Approximately how many children need services or support?

5.What are the needs of my program’s registered beneficiaries, in terms of education, health, protection, and psychosocial support?

These are not the right tools for you if…

You want to know: Which children in selected communities to targetHow a particular child/household is faring Which households, children or caregivers are worst off What services to provide or refer for a particular child / householdThe number of children/households that are receiving program support, and the types of support receivedWhether staff are carrying out their responsibilities Whether interventions are being implemented as planned

And, why a special OVC survey?

DHS and MICS take a general population sample difficult to discern the program’s contribution

DHS and MICS include some, but not all of the OVC core indicators

Using the Data

Representative sample of program beneficiaries

Data collected at one point in time

Data collected at two points in time

Representative sample of the general population

Beneficiary sample: 1 point in time

Often called: Baseline, Midline or Endline

If baseline or midline: Use data for program planning or design, or mid-course corrections

Example: high food insecurity found at baseline

Result: Change in workplan, PMP agreed between partner and USG, emphasizing food security

If endline: Use data to inform follow-on activities

Beneficiary sample: 2 (or more) points in time

Commonly referred to as an “evaluation” Baseline data should be used immediately Evaluation results inform future programming, policy But, change in wellbeing from time 1 to time 2, does

not mean program is 100% responsible Much stronger result if:

Comparison group is added (counterfactual)

Panel study / cohort

General population sample

Commonly called a Situation Analysis

Use data for needs-based resource allocation at national or sub-national level (not individual or community level)

Example:

Nigeria OVC Situation Analysis

Triangulate

Analyze data alongside DHS and MICS data

Check for differences and similarities

Determine whether differences or similarities make sense, or point to a data quality issue

Implementation so far

Zambia: Impact evaluation of savings and internal lending communities on OVC wellbeing

Baseline data available early Fall

Nigeria: Baseline survey of OVC umbrella grant mechanism beneficiaries in 10 states (planning phase)

You said a toolkit?

Tools & Manual Data analysis guide Template protocol with

consent/assent forms Data collector training

materials And then what?

Revising as we learn

Supporting countries to implement

Where can I find out more?

Go to our website:http://www.cpc.unc.edu/measure/our-work/ovc

Keep in touch on Child Status Net:http://childstatus.net/

Email: Jenifer Chapman: jchapman@futuresgroup.com and Janet Shriberg: jshriberg@usaid.gov

The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A-00-08-00003-00. Views expressed are not necessarily those of PEPFAR, USAID or the United States government.

MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.

Questions?