26
CONSISTENCY AND RESULTS EVALUATION IN MEXICO SEMPTEMBER 2008 Gonzalo Hernández Licona
CONSISTENCY AND RESULTS EVALUATION IN MEXICO
SEMPTEMBER 2008
Gonzalo Hernández Licona
The Context: Performance Evaluation System (SED)
Consistency and Results Evaluation
Results-Based M&E System
Strategic goals: Ministries
Program indicators (Results):Logical Framework
Results
Follow up on evaluation results
Consistency and Results
Evaluation
Impact Evaluation
Process Evaluation
Performance Evaluation
Strategic Evaluation
Annual Evaluation Program
Planning and Indicators Evaluation
Programs aligned toStrategic goals
Performance Evaluation System (SED). Systematic information about the program’s performance
I
II
I
I
EE
I
IBeginning in 2009:
% Budget Spent
0
20
40
60
80
100
1er trimester2o trimester3er trimester4to trimester
2009(Scholarship Program)
SED. Indicators 1
Number of scholarships granted
0
50,000
100,000
150,000
200,000
250,000
1er trimester 2o trimester 3er trimester 4to trimester
2009
Goal
(Scholarship Program)
SED. Indicators 2
SED. Indicators 3
Coverage Indicator
0
10
20
30
40
50
60
70
80
90
10020
06
2008
2010
2012
2014
2016
2018
2020
2022
2024
(Scholarship Program)
Impact in educational years
0
6
7
8
9
10
11
12
13
14
1520
08
2009
2010
2011
2012
2013
2014
2015
2016
2017
(Scholarship Program)
Beneficiaries
Control Group
SED. Evaluation 1
Consistency and Results Evaluation. What is it? The main goal: to what extent social programs are
oriented toward results, based on their own documentation and processes?
It’s a diagnosis about the institutional and management capacity of the programs to achieve results.
The idea was to have a homogeneous instrument, with measurable answers, in order to compare performance among programs.
The evaluation took place in 2007-2008 for 119 programs.
This evaluation is a baseline showing what type of tools (programs) the new Government has to achieve its goals.
The next evaluation will take place in 3 years time.
Consistency and Results Evaluation. Characteristics The evaluation has 100 questions about the
following topics, for every program: Design, Strategic Planning, Target Population and Coverage, Operation, Perception of Beneficiaries and Final Results.
The Design topic is mainly based on the Logical Framework.
80 of the questions have a binary answer (YES/ NO)
The evaluations were done by external institutions (universities, consultants, experts), hired by the ministries. 7 of them hired by CONEVAL.
The evaluation includes a Summary:Strengths, Challenges and Recommendations for every program.
DESIGN Does de program identifies clearly the problem it’s
trying to solve? Is it clear that the products or services produced by
the program logically contribute to reduce the problem the program is tacking?
STRATEGIC PLANNING Does the program have updated strategic plans for
the short, middle and long-term? In the S-plan, does the final results are clearly
established? TARGET POPULATION AND COVERAGE
Does the program have a method to identify and quantify the potential and target population?
Does the program have a coverage strategy for the short, middle and long-term?
Consistency and Results Evaluation. Questions
OPERATION Are there standard and adequate procedures for the
selection of projects, beneficiaries? Are there documents showing that the procedures are
done according to the rules of operation. PERCEPTION OF BENEFICIARIES
Does the program have instruments to measure the satisfaction of the beneficiaries with the program?
FINAL RESULTS Does the program systematically collect objective
information about its Goal (Fin y Propósito)? Does the program have rigorous impact evaluations?
Consistency and Results Evaluation. Questions
Consistency and Results Evaluation. General Findings
Topic Findings
DESIGN
67%
have a Purpose and Goal that belong to the problem’s solution.
50%
of the programs defined necessary and enough outputs to achieve the Purpose.
STRATEGIC PLANNING
21%
of the plans clearly establish the results that want to accomplish.
23%
of the programs have strategic plans in the short, middle and long-term.
TARGET POPULATION AND
COVERAGE
24%
of the programs quantify the target and potential population.
23%
of the programs have a coverage strategy for the short, middle and long-term.
Consistency and Results Evaluation. General Findings
Topic Findings
OPERATION
72%
have standard and adequate procedures for the project/ beneficiaries selection.
84%
of the programs have efficacy indicators in their operation.
BENEFICIARIES PERCEPTION
50%
of the programs have instruments to measure the satisfaction level of the target population.
FINAL RESULTS
34%
of the programs collect objective information about their final results.
26%
have external evaluations that allow to measure their impact.
Consistency and Results Evaluation. Findings per Program
GOAL 5: Reduce extreme poverty and assure equal opportunities.
CORE: Equal OpportunitiesTHEME: Poverty Alleviation
Milk Supply Program
(LICONSA)
GOAL
Improve the nutritional level of poor children and pregnant women.
PURPOSEBeneficiaries have access to fortified milk.
EVALUATION / RESULTS
Topic DesignStrategic Planning
Target Population
and CoverageOperation
Final Results
Results are being
documented
Type of Results
% 100% 71% 50% 100%
Score
TOPIC ANSWER QUESTION
DESIGN
P The Outputs allow the program to achieve its PurposeP The program Purpose is unique and reflex a significant benefit in the attended populationP The Purpose contributes directly to accomplish the program GoalP The program Goal is linked with a Strategic Goal of the Ministry
STRATEGIC PLANNING
P The plans establish the results that the program seekP The program have enough indicators that reflex it PurposeP The program have pertinent goals and specific deadlines for its performance indicatorsP The evaluations results are used to improve the performance of the programÑ There are updated strategic plans in the short, middle and long-termÑ The strategic plans establish indicators, goals, politics strategies and work schedulesP There are adequate mechanisms to define goals and indicators
TARGET POPULATION AND
COVERAGE
Ñ The program has a method to quantify and determinate the potential and target populationP The coverage advance is adequate to accomplish the program Goal and PurposeÑ The program has a coverage strategy in the short, middle and long-termP The coverage strategy is adequate
OPERATION
P The support granted complied the established regulationsP There is documentary evidence that the program follow the implementation processesP There are adequate standardize processes for the project/ beneficiaries selectionP There are adequate procedures to receive and process support applicationsP There is systematized data which allows the timely monitoring of the benefits executionP The program has an organizational structure to produce the Outputs and achieve the program PurposeP The program has effectiveness indicators of the operationP There is a adequate systematization in the program management and operation
FINAL RESULTS
Impact Evaluations Results Results of the Program Beneficiaries Monitor Other Notable Results The program do not document any Final Results Adequate Final Results Enough Final Results Insufficient Final Results Final Results unknown
Milk Supply Program
(LICONSA)
GOAL 5: To reduce the extreme poverty and to assure equal opportunities.CORE: Equal OpportunitiesTHEME: Poverty Alleviation
FINAL RESULTS
Children that consume fortified milk present less anemia risk compared to the control group (10.7% vs. 15.6%)
Children’s height is half centimeter bigger compared to the control group
Milk Supply Program
(LICONSA)
GOAL 5: To reduce the extreme poverty and to assure equal opportunities.CORE: Equal OpportunitiesTHEME: Poverty Alleviation
OBSERVATIONS
The program is targeted in urban zones, it doesn’t attend localities with high levels of malnutrition
Milk Supply Program
(LICONSA)
GOAL 5: To reduce the extreme poverty and to assure equal opportunities.CORE: Equal OpportunitiesTHEME: Poverty Alleviation
Rural Housing Program
GOALTo contribute to the quality of life of indigenous and rural families in patrimony poverty that live in localities of high and very high marginalization, through housing solutions
PURPOSEMexican families living in rural and indigenous localities with high and very high levels of marginalization improve their housing conditions
EVALUATION / RESULTS
Topic DesignStrategic Planning
Target Population
and CoverageOperation
Resultados Finales
Document Results
Type of Results
% 44% 14% 0% 38%
Score
GOAL 5: Reduce extreme poverty and assure equal opportunities.
CORE: Equal OpportunitiesTHEME: Poverty Alleviation
Consistency and Results Evaluation. What did we do? EVALUATION: ACCOUNTABILITY AND PERFORMANCE
IMPROVEMENT The evaluations were sent to Congress and the
Ministry of Finance in April The complete evaluations can be found in the web
page (CONEVAL and Ministries). The scores were sent to Presidency in order to take
better budget decisions for 2009. We don’t know the results yet…
Nowadays, programs are analyzing the pertinent and feasible recommendations given in the Consistency and Results Evaluation, and they are doing a Work Plan to improve their performance.
This Work Plan will be public.
Summary
The ECR is part of the SED. It’s perhaps one of the first instruments of the SED. We were able to get a homogeneous picture of 119
programs. We’re improving the instrument. The information will be used to improve the
program The information is in the web. The President wanted the information for budget
purposes, but we don’t know if this was the case. The Work Plan to improve the program, will be a
bridge connecting Accountability and Performance improving.
2009
SED. Valuable information menu
I
II
I
I
EE
I
I
PerformanceEvaluation
(Annual Summary)
Annual Performance Evaluation
4 pages report, summarizing the information contained in the SED (to be read by Congress):
The Report will have: Name and goal Budget (last 3 years) Results (shown in SED; through indicators or
Evaluations) Coverage (shown in SED) Financial indicators (shown in SED) …. Recommendations (1 or 2) If there is no information about some of these
subjects, it must be reported
Consistency and Results Evaluation. Challenges Huge variance in the quality of external evaluators.
CONEVAL re-evaluated 47 evaluations.
Questions: A lot of questions; some duplications. We are improving the instrument.
The evaluation was highly linked to the Logical Framework. The programs (and evaluators) had just started to
know this methodology.
