Development Project Planning 6

Monitoring and Evaluation

Data Sources

Ground Rules

The point is

1) for you to get benefit from the process

2) not to disturb other people

What are the rules?

This Session

Recap – Indicators

Mathematics

Data Systems and Collection

Levels of Indicators

What is an Indicator ?

a variable …

that measures one aspect of a project

that is directly related to the program’s objectives.

DATA SYSTEMS

Components of a clearly defined Data System:

o multiple, well defined indicators

o a variety of appropriate data sources

o baseline and target values

o feasible data collection plan and budget• specified frequency• identified responsibility

LEVELS OF DATA

Policy or Program

Population

Service Environment

Client

Spatial / Geographic

DATA SOURCES AND TOOLS

POLICY/PROGRAM LEVEL

Gives information about the ‘big picture’

The context, the nation or region

How countries compare to each other

DATA SOURCES AND TOOLS

POLICY/PROGRAM LEVEL

Sources: Official documents (e.g. government reports,

legislative and administrative documents) National budgets or other accounts data Policy inquiries Reputational rankings (e.g. program effort scores)

Tools: Index questionnaires (give rankings e.g. morbidity) Special and contract studies

DATA SOURCES AND TOOLSSERVICE ENVIRONMENT LEVEL

Gives information about how the services are being run

DATA SOURCES AND TOOLSSERVICE ENVIRONMENT LEVEL

Sources: Administrative records (e.g., service statistics, HMIS data, financial) Service delivery point information (e.g., audit information,

inventories, facility survey data) Staff or provider information (performance / competency

assessments, training records, staff/provider data, quality of care data)

Client visit registers/compilationsTools: Health Service Information Systems Facility sample surveys Performance monitoring reports Service Delivery Point records

HEALTH MANAGEMENT INFORMATION SYSTEMS (HMIS)

An important way of monitoring routine data over time is through a Health Management Information System. HMIS is a system for routine reporting on service delivery. Often it is national. Best if data collected from a full set of service delivery points, and on topics like:

o Costso Stockoutso Birthso Mortalityo Morbidityo Numbers of clients seen, referred (inpatient; outpatient)o Numbers of clients by types of service

DATA SOURCES AND TOOLS

INDIVIDUAL LEVEL

Gives information about how individual people are being served

DATA SOURCES AND TOOLS

INDIVIDUAL LEVEL

Sources: Case surveying (e.g., epidemiology of disease) Medical records Interview data Provider-Client interactions (check the clinical,

technical or interpersonal skills)Tools: Case reports Client register analysis Patient flow analysis Direct observation

DATA SOURCES AND TOOLS

POPULATION LEVEL

Gives information about everyone in the country

o How things change over time

o how countries compare

DATA SOURCES AND TOOLS

POPULATION LEVEL

Sources: Government Census Office Vital registration systems (e.g., birth and death

certificates) Sentinel systems – watch a specific thing (e.g. H2N1) Sample households or individuals Special population samples (demographic or

occupational group, or geographic sector)Tools: Birth certificates Household/Individual/Special surveys Census forms

DATA SOURCES AND TOOLSSPATIAL / GEOGRAPHIC

Sources: Satellite imagery and aerial photography Digital line graphs and elevation models Cadastral maps (land ownership)

Tools: Global Positioning System GPS Computer software programs GIS

Take a breakTake a break

•Back in 15 minutesBack in 15 minutes

Some Maths

A couple of statistical ideas

PERCENTAGES

Consist of a fraction – the number measured divided by the maximum possible

times by 100 (per cent = for each hundred)

Number counted = “Numerator” Number possible = “Denominator”

% = Numerator

Denominatorx 100

Example: PERCENTAGES

Number of projects at HU Ratanakiri: 7 Number of EU funded projects: 4 What percentage of projects at HU RTK ate

EU funded? Numerator = 4, Denominator = 7

% = 4

7x 100 = 57%

Percentile

A percentile tells you where an individual or subgroup fits into a larger group as a percentage

If you say “Sam’s IQ is at the 66th percentile” it means that his IQ is in the first 66% of the population

Normal Distribution

Many population statistics rely on the rule that the measure results will scatter in a “normal distribution”

There is an average (“mean”) with half above and half below

Most individuals are close to the mean

Normal Distribution

Percentile 0.1 2.2 15.8 50 84.1 97.7 99.9

Quintile

Divide a group into five equal groups according to a ranking e.g income

Each group is a quintile

Enables comparisons between locations e.g someone in Cambodia earning $2 a

week and someone in the USA earning $40 a week will both be in the lowest quintile for income.

Example: Disabilities by income quintile (Indigenous Australia)

DIFFERENT DATA SOURCES - SAME INDICATOR

Different data sources can be used to measure the same indicator. In some cases, changes to the metric are required depending on data sources selected.

% of live births attended by a trained TBA in last 12 months

Option 1 – using health service data: Numerator: # of live births in the district attended by trained TBA

in last 12 months Denominator: # of live births in the district in last 12 months

Option 2 – using information from village visits: Numerator: # of women having a live birth in last 12 months

reporting being attended by a trained TBA Denominator: # of women having a live birth in last 12 months

DIFFERENT DATA SOURCES - SAME INDICATOR

When a choice is available look at the + and - of different data sources.% of children 12-23 months receiving all three polio vaccines Numerator: # of children 12-23 months receiving all three polio

vaccines Denominator: # of children 12-23 monthsData sources: Option 1: numerator from individual client records at facilities,

denominator from census Option 2: numerator from HMIS, denominator from census Option 3: numerator and denominator from sample household

survey (interview with mother) Option 4: numerator and denominator from sample household

survey (review of child’s immunization card)

Some MEASURE Indicators

The Handouts come from USAID MEASURE program, which promotes good M&W practice

Disaggregate: to separate into partshow this indicator is specific to one group

PopulationHealthEnvironment

Value-Added Indicators

Value-added indicators show groups of people or sectors that may not have been targeted in the intervention but have an impact from the intervention.

Value-Added Indicator: Example

The project may intend to increase the use of condoms by providing condoms to local merchants and providing merchants with sales training.

In the process, the livelihood of that merchant is helped by providing an alternative stream of income.

The diversification of livelihood for that merchant is a value-added result.

Integration Indicators

Integrated programs have several advantages over stand-alone population, child health, or environment programs.

Integrated programs are cost-effective compared to the cost of single-sector programs.

Integrated programs also recruited a greater number of men to family planning efforts and a greater number of women and adolescents to environment/ conservation efforts. Integrated programs also improved the perceived valueof family planning efforts by packaging them with health interventions.

Example: Number of cross sector linkages formed

Global Fund Guide

Lists many indicators

Thankyou

Produced by Tony Hobbs

Health Unlimited,

Ratanakiri, Cambodia

www.healthunlimited.org

With the support of Australian Volunteers Internationalwww.australianvolunteers.com

© 2009 HU. Use with Acknowledgement