Food and Nutrition Technical Assistance II Project (FANTA-2)Academy for Educational Development 1825 Connecticut Ave., NW Washington, DC 20009Tel: 202-884-8000 Fax: 202-884-8432 E-mail: [email protected] Website: www.fanta-2.org
Layers:Layers:Adaptations for DP’s Adaptations for DP’s
M&E needsM&E needs
Food and Nutrition Technical Assistance II Project (FANTA-2)Academy for Educational Development 1825 Connecticut Ave., NW Washington, DC 20009Tel: 202-884-8000 Fax: 202-884-8432 E-mail: [email protected] Website: www.fanta-2.org
Goal of session
• Discussed in this session: – What Layers is – Types of questions asked in Layers– Where Layers is currently being implemented
• Also :– Is Layers adaptable to MYAPs’ M&E systems?
• Highly structured to address Mission needs• Can it help PVO needs?• What issues may be addressed? Outcomes? MTEs?
Session format
• Part 1:– Presentation – Q&As
• Part 2:– Working group session: potential for use of
Layers by MYAPs
• Part 3:– Plenary wrap up
4
What is Layers?
• Layers is a monitoring system developed to help USAID Missions fulfill their responsibilities to monitor Food for Peace Act, Title II Programs.
• According to DCHA/FFP enabling regulation:
“USAID Missions are expected to monitor CS's management of the commodities and use of grant funds. (…) Oversight and monitoring should include regularly scheduled visits to distribution centers and warehouses…”
5
Typical (pre-Layers) Title II monitoring• Title II sites are periodically visited by USAID staff to verify
compliance on commodity storage, disposal and distribution.
• If discrepancies are found, the partner is notified.
Weaknesses of this approach:
• Only commodities monitored, not program activities.
• Site selection based on convenience--no systematic sampling.
• Only sites visited are assessed; no program-wide monitoring.
• Designed for USAID’s use; little helpful feedback to partners.
• No methodological guidance to field monitors (sampling, indicators).
6
What Layers brings:• Allows monitoring of program activities as
well as commodity management
• Draws from a random sample of sites: results are representative of the entire program
• Uses standardized indicators
• Shares findings with partners to improve program performance
7
What is LQAS (Lot Quality Assurance Sampling)?
• LQAS is a sampling method developed to control the quality of manufactured goods produced in ‘lots.’
• LQAS takes a small random sample and tests the sample for quality.
• The sample will tell if program activities (agriculture, health, etc.) are meeting/not a performance benchmark
• The sample size is chosen so that there is a high probability of determining what indicators in a given activity are meeting or not meeting the performance benchmark.
• A MYAP may get a “No” on some indicators and a “Yes” on others within the same activity type.
8
Limitations and words of cautions about LQAS and Layers
• Layers only shows whether a CS has met the benchmark for an indicator or not (e.g. does it meet 95% “fail free” standard?). It does not provide point estimates (i.e. “% of sites where standard is met”)
• LQAS is good at accurately identifying a CS that is above the upper benchmark or below the lower benchmark. Classification errors are unfrequent but may occur when CS results are “borderline” (fall between UT & LT).
• Planned sample size must be met—else, error rates increase which may lead to erroneous conclusion that the CS performs below the benchmark).
9
• A popular sample size is n=19. This does not have to be so: what sets the final n is: benchmark level, UT/LT, error rate
• Calculating the right sample size for a LQAS survey is done with easy-to-use tools found on the FANTA-2 website, athttp://www.fantaproject.org/layers/reference.shtml
• The tool tells us:
– How many sites to sample (e.g., 25)
– How many of those sites must receive a “yes” (e.g., 21) in order to meet the benchmark for that indicator (called the “decision rule”).
– Alpha and beta error rates based on sample size and upper and lower thresholds
Sampling for LQAS
10
11
Indicators Used by Layers
• Layers can monitor many aspects of a program to ensure the overall success of the CS in delivering services.
• Several modules of indicators/questionnaires already exist. Those are modified and adapted to each country/program.
• New indicators can also be created.
12
Examples of Common Layers Indicators by Type of Activity
Commodity Mgt MCHN FFW Education
Commodity movements
–Ledger/physical inventories agree?
–Tracking system in place and adequate?
Immunization services
–Cold chain OK?
–Any vaccines stock-outs, past due?
Infrastructures
–Structure respects EIA, engineering standards?
Education
–Attendance as expected?
–Kitchen adequate?
Storage conditions
- Commodities clean and secure?
Nutrition intervention
–Staff knowledgeable?
–Posters present?
–Scales calibrated?
Irrigation, water systems
–Maintenance ensured?
Teaching quality
–Teachers present, equipped?
–School building meets standards?
Data entry using PDAs Use of PDAs facilitates the work of enumerators
• PDAs reduce the likelihood of error– Automatically selects form/module/question to ask– Verify coherence with earlier responses– Minimizes coding error, poor transcript – Data entry errors eliminated
• PDAs are economical– No need to print paper forms– Changes to the questionnaires easily made mid course– Data entry is eliminated, & info is immediately usable
• PDAs are flexible:– GPS and other electronic equipment can be adapted– Reference tools can be included (manuals, calculators, etc)– Decision making tools can be included (drill down,
decision/diagnosis trees)BUT
• PDAs require programming, specialized training and IT support
14
15
16
17
Layers Data Analysis
• Analysis with LQAS is simple:
– Count the number of “yes” responses for each indicator.
– Apply the decision rule in each case.
# of “Yes” responses ≥ decision rule: YES – this indicator meets the benchmark
# of “Yes” responses < decision rule: NO - this indicator does not meet the benchmark
Setting up Layers: what’s involved?
• Staff infolved:– Mission FFPO– PVO (IT Manager, M&E manager, COP)– FANTA/2, TANGO
• Steps:– Initial planning– Sampling (create TUS, chose sites)– Adapt model (questionnaire by sector, etc)– Train data collectors – Collect, tabulate and analyze data– Communicate results, recommendations to Awardees
• Entire process: 7 mo (first run); shorter afterward
18
19
Timeline for setting up Layers in a new Country*
Assumes 1mo data collection via external contractor
Final Step: Sample LAYERS Report Letter
1. Findings for commodity warehouses (documentation, management and storage)
Successes:• Routine documentation procedures at MCHN sites are followed
appropriately. • Storage management is appropriate: storage sites are clean,
weatherproof and safe; and the food stored in those sites is kept correctly.
Challenges:• The ledgers did not concur with existing inventories in many sites.
This is most serious and needs urgent attention.• Minor problems related to ventilation, rodent infestation and use of
the warehouse for other purposes were detected in a few sites.
Letter ends with a Summary of Recommendations. PVOs expected to reply and state how they will address the recommendations.
Where is Layers used?
• Currently used in Haiti, Guatemala, Madagascar, Ethiopia
• Currently undergoing preparation in Chad, Mali, Uganda, Malawi, Bangladesh
• Will be progressively implemented in all FFP focus countries, as they turn over to new Development Programs
21
Extensions of Layers
• Outcome Monitoring– Same general approach as Layers– Adds the following dimensions:
• Focuses on Outcome level indicators (random sample of target population, not sites)
• Implemented in several “supervision areas”– parametric results possible (means, etc) – Identifies areas that need reinforcement
• Can feed into other activities (Mid Term Evaluations? Operations Research? Etc)
22
Focuses on Outcome Indicators
EXTERNAL PROCESSES
Human resourcesFinancial resources Physical facilities Equipment, etc
INPUTS
Training Setting up systems Supervision of activitiesNetworking
PROCESSES
N of mothers trained N of working cold chains FP methods distributed
OUTPUTS OUTCOMES
EBF, ORS, etcImmunization rateCPR
IMPACTS
Better nutritionReduced morbiditySmaller families
Monitoring indicators(yearly measurements)
Evaluation indicators(BL, MTE, Final Eval)
Rainfaill, prices, policies, etc.
Program-level info Beneficiary-level info
Why outcome and not impact? Relates to expected size of change bt/years
OM Indicators are highly streamlinedMalaria
1. % HHs with child< 5 having at least one ITN2. % children <5y in malaria-risk areas who selpt under ITN the previous night3. % women who received 2+ doses of SP for malaria during last pregnancy4. % children <5y w/fever in last 2w who received AMT w/in 24hrs from onset of fever
Maternal and Child Health (MCH)5. % women who gave birth who had a PP visit w/in 3 days6. % newborns receiving essential newborn care7. % women seen at ANC 4+ times during last pregnancy8. % births attended by a doctor, nurse or trained midwife9. % children 12-23mo who received 3rd dose of DTP by 12mo 10. % children 12-23mo who received vit A supplement in last 6 months11. % children <5yr who are > -2 SD of median W/A12. % infants <6mo exclusively breastfed in past 24hrs 13. % children 0-59mon w/diarrhea in last 2 weeks treated with ORS14. % children 0-59 mo w/ARI in last 2 weeks taken to appropriate health provider15. % children 12-23mo fed a MAD
Family Planning Reproductive Health (FP/RH)16. % women 15-49yr (or partner) using a modern method of contraception17. % need satisfied by modern method of FP18. % women 15-49yr stating desire to space birth 36mo+ or to limit births
Use of PMAs (Program Management Areas )
n=21 This design allows us to saywhether the region as a whole passes or fails the benchmark
What PMAs Add
n=21
n=21
n=21
n=21
n=21
n=21 n=21 n=21 n=21 n=21n=105
Dividing the total area into PMAs allows to:(i) State whether each PMA fulfills the benchmark orients corrective action(ii) Compare PMAs to one another identify “positive deviants”(iii) Generate parameters from the total (n=105) estimate coverage (means, etc)
Sampling design: dictated by key indicators*
Subsample Group of Indicators this Subsample will Document
Mothers of Children 0-5mo
Exclusive Breast Feeding rates
Mothers of Children 0-11mo
Pre/Postpartum Care, including Birth Attendance, Essential Newborn Care, Pre/Postpartum visits, Vit A, TT, and Iron Folate
Mothers of Children 12-23mo
Immunization and Vitamin A, Infant and Young Child Feeding
Mothers of Children 0-59mo
Malaria, Illness management (diarrhea, fever, ARI), Anthropometry
Women of Reproductive age (15-49yrs)
Family Planning, including CPR, Needs Satisfied, Birth Spacing; as well as Reproductive Health
*Recall: LQAS requires that all samples be completed
Putting the pieces together: Example of Sampling Matrix with
5 PMAs, 5 Subsamples
SUBSAMPLES
PMAs Motherchildren
0-5m
Motherchildren 0-11m
MotherChildren
12-23
Motherschildren
0-59m
Women Reprod. age
(15-49y)
Total
Chimaltenango HZ 21 21 21 21 21 105
Quetzaltenango HZ 21 21 21 21 21 105
Quiche HZ 21 21 21 21 21 105
San Marcos HZ 21 21 21 21 21 105
Totonicapan HZ 21 21 21 21 21 105
TOTAL 105 105 105 105 105 525
Part 2: Adapting Layers to Awardees’ needs
• How do we adapt those concepts to the needs of T-II Awardees?– Yearly monitoring of outputs? outcomes?– Use in MTE?– Use in BL/FE?– Use in special studies?
• Issues to keep in mind:– Integration between Mission and MYAP data– Why sample? (if PVOs must visit all sites)– Monitor what? (Outputs, Outcomes, Impacts?)
30
Method for Part 2
• Breakout in working groups. • Each WG discusses a possible use of Layers.
Examples of options (others welcome): – Yearly monitoring– Mid term evaluations– Baseline/Final evaluations– Special studies
• Report in plenary after. Have audience review and critique each approach
31
32
This presentation is made possible by the generous support of the American people through the support of the Office of Health, Infectious Disease and Nutrition, Bureau for Global Health, and the Office of Food for Peace, Bureau for Democracy, Conflict and Humanitarian Assistance, United States Agency for International Development (USAID) under terms of Cooperative Agreement No. GHN-A-00-08-00001-00, through the Food and Nutrition Technical Assistance II Project (FANTA-2), managed by the Academy for Educational Development (AED). The contents are the responsibility of AED and do not necessarily reflect the views of USAID or the United States Government.
Food and Nutrition Technical Assistance II Project (FANTA-2)Academy for Educational Development 1825 Connecticut Ave., NW Washington, DC 20009Tel: 202-884-8000 Fax: 202-884-8432 E-mail: [email protected] Website: www.fanta-2.org