MAPPMarcos C. [email protected]
Monitoring and Evaluation of Priority
Programs
Banco Interamericano
de Desenvolvimento
Management for Development Results at
Sub-national Governments
Banco Interamericano
de Desenvolvimento
RBM - MAPP
RBM – Governance StrategyRBM – Governance Strategy
MAPP – Operational MAPP – Operational InstrumentInstrument
Banco Interamericano
de Desenvolvimento
It is important to differentiate
PlanningPlanning Projects selection Projects selection
AdministrationAdministration Projects execution Projects execution
ManagementManagement Systemic Systemic coordination of projects coordination of projects based on based on resultsresults
Banco Interamericano
de Desenvolvimento
What RBM is About1.1. Do less but do Better.Do less but do Better.2.2. Outcomes prevail over outputs.Outcomes prevail over outputs.3.3. Outcomes put citizens first. Outputs put Outcomes put citizens first. Outputs put
government first.government first.4.4. Outcomes as driving force to programs Outcomes as driving force to programs
integration.integration.5.5. Outcomes to differentiate good from bad Outcomes to differentiate good from bad
programs.programs.6.6. A permanent cycle of learning: Planning-A permanent cycle of learning: Planning-
Budgeting-Monitoring-Evaluation.Budgeting-Monitoring-Evaluation.7.7. An instrument for recognition and compensation An instrument for recognition and compensation
of good performance of government employees.of good performance of government employees.
Banco Interamericano
de Desenvolvimento RBM
GOVERNMENT STRATEGIC OUTCOMES
DEPARTMENT STRATEGIC OUTCOMES
DEPARTMENT PROGRAMS
DEPARTMENT OUTPUTS
DEPARTMENT PROJECTS AND ACTIVITIES
DEZ
IGN
IMPLEM
ENTA
TION
GOVERNMENT PLANGOVERNMENT PLAN
Banco Interamericano
de Desenvolvimento
Requirements for a successful RBM
To have information To have information – outcome and output – outcome and output indicatorsindicators
To know how to use the information To know how to use the information – – information information affect budget decisionsaffect budget decisions
To have a budget based on results To have a budget based on results – RBM – RBM defines defines the budget and not the contrarythe budget and not the contrary
To know what it is and how to do RBM To know what it is and how to do RBM – – Institutional and operational capacity to implement Institutional and operational capacity to implement RBMRBM
Banco Interamericano
de Desenvolvimento
Guidelines for RBM
1.1. Operational flexibility in searching for Operational flexibility in searching for results.results.
2.2. Medium term budget. Medium term budget.
3.3. Increments in the budget are based on Increments in the budget are based on increments in the results.increments in the results.
4.4. Look for “good or better practices” and not Look for “good or better practices” and not for “best practices”.for “best practices”.
Banco Interamericano
de Desenvolvimento
MAPP
MAPP
Planning
Monitoring
Evaluation
Banco Interamericano
de Desenvolvimento
MAPP
1. PLANNING Selection of Projects Application Application CardCard
2. MONITORING Outputs MAPP MonitoringMAPP Monitoring
Macro EvaluationMacro Evaluationa) Government Outcomesb) Integration Matrix
3. EVALUATION Outcomes Micro EvaluationMicro Evaluationa) Design Evaluationb) Impact Evaluation
Banco Interamericano
de Desenvolvimento
Banco Interamericano
de Desenvolvimento
PLANNING Project Project Application CardApplication Card
1.1. The problem to be faced.The problem to be faced.2.2. The government outcomes to be targeted.The government outcomes to be targeted.3.3. The department outcomes to feed the The department outcomes to feed the
government ones.government ones.4.4. The outputs to be delivered.The outputs to be delivered.5.5. The outcomes and outputs indicators.The outcomes and outputs indicators.6.6. The targeted population or region.The targeted population or region.7.7. The time frame of expected outcomes.The time frame of expected outcomes.8.8. The sources of financing.The sources of financing.9.9. The expected efficiency (benefits-costs The expected efficiency (benefits-costs
analysis).analysis).10.10. The expected integration with others projects.The expected integration with others projects.
Banco Interamericano
de Desenvolvimento
PLANNING Project Application Project Application CardCard
Banco Interamericano
de Desenvolvimento
PLANNING Project Application Project Application CardCard
Banco Interamericano
de Desenvolvimento
PLANNING Project Application Project Application CardCard
Banco Interamericano
de Desenvolvimento
PLANNING Project Application Project Application CardCard
Banco Interamericano
de Desenvolvimento
MONITORING MAPP MAPP MonitoringMonitoring
Banco Interamericano
de Desenvolvimento
MONITORING MAPP MAPP MonitoramentoMonitoramento
Banco Interamericano
de Desenvolvimento
MONITORING MAPP MAPP MonitoramentMonitorament
oo
Banco Interamericano
de Desenvolvimento
MONITORING MAPP MAPP MonitoramentMonitorament
oo
Banco Interamericano
de Desenvolvimento
EVALUATION Macro Macro EvaluationEvaluation
Government Outcomes
Indicadores Sinalização
Nota SAEB e ESPAECE ▬Distorção Idade Série ▲Taxa Escolar Líquida: Ensino Fundamental ▼Taxa Escolar Líquida: Ensino Médio ▲Taxa de Analfabetismo
(>15 anos) ▲(de 7 a 14 anos) ▬
Taxa Mortalidade Infantil ▲Razão Mortalidade Materna ▲Taxa Internação AVC (> 40 anos) ▼Taxa de Roubos ▼Taxa de Furtos ▬Taxa de Homicídio ▼Renda Domiciliar Per Capita (R$) ▲Porcentagem Extrema Pobreza ▲Índice de GINI ▲
Indicadores Sinalização
Saldo de Empregos Formais (Novos Postos) ▲
Taxa de Crescimento do PIB (%) ▬
Fluxo de Comércio Exterior ▲
PIB Interior / PIB RMF (%) ▼
Consumo de Energia Residencial: Interior / RMF (%)
▲
Empregos Interior / Empregos RMF (%) ▬
Indicadores Sinalização
Div / PIB (%) ▲
Div / RCL (%) ▲
Resultado Primário ▲
Custeio / RCL (%) ▲
Pessoal / RCL (%) ▲
Gastos Finalísticos / RCL (%) ▬
Investimento / RCL (%) ▼
Número de Acessos às Páginas WEB do Governo ▲
Social OutcomesSocial Outcomes Economic OutcomesEconomic Outcomes Governance OutcomesGovernance Outcomes
Banco Interamericano
de Desenvolvimento
EVALUATION Macro Macro EvaluationEvaluation
Integration MatrixPrograms Integration Matrix
Government Outcome # 1 Government Outcome # 2
Government Outcome # 3
Economic Growth
Program1
Program2
Program3
Program4
Program5
Program6
Result Based Budget
Banco Interamericano
de Desenvolvimento
EVALUATION Micro Micro EvaluationEvaluation
Design Evaluation
Inputs Activities Outputs Outcomes Impacts
Logical FrameworkLogical Framework
MonitoringMonitoring Outputs deliveredOutputs deliveredXX
Planned OutcomesPlanned Outcomes
Banco Interamericano
de Desenvolvimento
EVALUATION Micro Micro EvaluationEvaluation
Design Evaluation
a) Education: . Primary education schooling rate . Secondary education schooling rate . 4th grade promotion rate.
b) Health: . Infant mortality rate . Cerebral Vascular Accident (CVA) admittance rate
c) Living Conditions: . Proportion of urban domiciles with regular water supply . Proportion of urban domiciles with sewage
d) Employment and income: . Average residential energy consumption . Formal employment quality index . Average establishments’ size
e) Rural Development:.Gross value of the agricultural production per rural establishment . Proportion of energy consumption in rural areas
OUTCOMES OUTPUTS
a) Education:. Proportion of teachers in primary education with bachelor degrees. Proportion of teachers in secondary education with bachelor degrees. Libraries, reading rooms and computer labs per public school; and. Computer equipments per public school.
b) Health:. Prop. of women assisted during the first 3 months of pregnancy. Proportion of children (2 years old or younger) assisted. Proportion of the population assisted by the Family Health Program
c) Living conditions:. Urban water supply coverage rate. Urban sewage coverage rate
d) Employment and income:. Secondary enrollment – total enrollment ratio . Health professionals per 1,000 inhabitants. Paved roads – municipality area ratio.
e) Rural development:. Mean value of rural loans. Assisted producers per rural establishment
Banco Interamericano
de Desenvolvimento
EVALUATION Micro Micro EvaluationEvaluation
Impact Evaluation
Probability (Recidivism) = F(age, gender, approach period, activity, per capita family income, guardian education, benefit)
Variables Coeficient dY/dX P Value
Age 0.0136 0.0035 0.002
Gender 0.2493 0.0619 0.000
Approach period 0.1606 0.0441 0.001
Activity 0.2590 0.0664 0.000
Per Capita Family Income -0.0050 -0.0013 0.000
Guardian Education -0.1698 -0.0013 0.000
Benefit -0.5001 -0.1344 0.000
Constant -0.8609 - 0.000
Number of observations 8397 Pseudo R2 0.0639Source: IPECE
Program Evaluation: Criança Fora da Rua Dentro da Escola
Banco Interamericano
de Desenvolvimento
LESSONS
1.1. Strategic leadership(Governor, Mayor) and Strategic leadership(Governor, Mayor) and operational leadership(line secretary) are operational leadership(line secretary) are crucial.crucial.
2.2. It is very Important a clear understanding of It is very Important a clear understanding of RBM concepts . RBM concepts .
3.3. It is important to show that RBM is about It is important to show that RBM is about opportunity and not a threat.opportunity and not a threat.
4.4. The selection of output and outcome indicators The selection of output and outcome indicators can be very tricky.can be very tricky.
Banco Interamericano
de Desenvolvimento
LESSONS
5. It is important to explain the difference 5. It is important to explain the difference between Monitoring and Evaluation:between Monitoring and Evaluation:Monitoring Monitoring Good Accounting Good Accounting
Evaluation Evaluation Good Economics Good Economics
6. Before impact evaluation, ask for design 6. Before impact evaluation, ask for design evaluation (quick evaluation) for the main evaluation (quick evaluation) for the main Program/Project.Program/Project.
7. 7. Keep the RBM policy as simple as possible.Keep the RBM policy as simple as possible.
8. 8. RBM requires 3 P’s ( a different kind of PPP): RBM requires 3 P’s ( a different kind of PPP): Patience + Persistence + PassionPatience + Persistence + Passion