Date post: | 23-Dec-2015 |
Category: |
Documents |
Upload: | robyn-rich |
View: | 213 times |
Download: | 0 times |
New frontiersEvaluation methods
Theory of changeProject cycle and risk management
Jesper Johnsøn, CMI, U4Bergen, February 4, 2014
Frontiers…
Current plateau – (cross) country-level, bribery, fraud
• Evaluations of specific reforms• Integration with risk management• Proxy indicators
Imbalances and the “missing middle” • Imbalances: – So far, resources have been invested in standardised
diagnostic and advocacy tools, not learning and evaluation• Diagnostics: follow trends – indicators, data, monitoring• Evaluation: what works – question, design and methods,
indicators, data, judgement
• Missing middle
Output Outcome Impact Individual Sector Country
Level of results
Indicator targets
When and how to evaluate corruption costs/anti-corruption effects?
Diagnose corruption risk (does corruption matter, are patterns changing?) If yes, prepare for evaluation (how and how much does corruption matter, how effective is anti-corruption?)1. Logic model: Think through how the initiative is
intended to create results by contributing to behavioral, organisational, political, or societal change, expressed in a result chain or theory of change.
2. Monitor and evaluate: Work on increasing the evaluability of the reform by shaping design, establishing indicators, systematic data collection, baselines, comparison groups
INDICATORS
EVALUATION QUESTION, DESIGN,
METHODS
PROBLEM/DIAGNOSIS
DATA
ANALYSIS, COMPARISON, JUDGEMENT,
CHANGE
Overview of types of evaluations
Impact evaluations Programme evaluations Process evaluations
Focus: cause-and-effect, attribution, impact
Methods: counterfactual, comparison groups, statistical analysis, randomisation
Can answer question: What would have happened in the absence of the programme?
Focus: effectiveness, contribution, outcomes
Methods: comparative analysis, cost-effectiveness/benefit analysis, benchmarking
Can answer question: Has the programme been effective in achieving its stated objectives?
Focus: efficiency, implementation, internal mechanisms, outputs
Methods: workflow analysis, compare performed vs planned activities and outputs.
Can answer question:Is implementation done according to plan, and do results look attainable?
Clear objectivesIndicators
Mixed methodsTriangulation
ControlsRandomisation
ComparisonsBaselines
Trends
Code of Conduct example, level of results
Code of Conduct example, building an impact story
Diagnostics, evaluation, risk management and the project cycle• Framework for holistic use of different tools• Shows interlinkages between diagnostics, risk management and
evaluation tools
Diagnostics, evaluation, risk management and the project cycle• http://prezi.com/osddirkyzzpw/?
utm_campaign=share&utm_medium=copy&rc=ex0share
The Proxy Challenge
Challenges we confront• Linked, but not inseparable:
– Measuring changes in corruption levels– Evaluating whether anti-corruption efforts are successful
• Progress on measuring bribery and financial fraud, but two overall indicator problems remaining:– How to measure other types of corruption (patronage,
conflict of interest, abuse of power, etc.)– How to present a measure of overall corruption levels in a
country, region, sector, or organisation not biased towards measurable types of corruption, and can show trajectories of change
Why proxies?
• Good, existing “direct” indicators for bribery and financial fraud (need better application of methods)
• Can capture “missing middle” – provide needed innovation for some types of corruption, to capture trajectory of performance, and measure behavioural change
“Proxy indicators are alternatives to direct indicators that more directly measure the
phenomenon under study but that may be hard to operationalize or require overly costly data
collection”
Criteria
Standard criteria for indicator development (SMART)• Reflect de facto changes and behavioral changes• Prioritize sensitivity to context over standardization• Measure specific types of corruption• Ensure that changes in corruption trends are
attributable to reforms
“To assemble a body of promising ideas…”
• Clearly define the proxy indicator (or indicators) you suggest as a good measure of anti-corruption results
• Identify the type of corruption measured by the proxy indicator • Clarify how the proxy indicator reflects changes in corrupt behaviours• Present ideas for how the validity of the proxy indicator can be tested • Describe any relevant geographic, sectoral, or institutional context• Explain how the indicator can be combined with other proxy and
nonproxy (direct) indicators to obtain a better measurement of overall anti-corruption progress
• Comment on the strengths and weaknesses of the proxy indicator and the feasibility of its use by aid agencies, governments, civil society, and others for the purposes of monitoring and reporting