Welfare Economics, Project and Programme

Appraisal and Evaluation

Lecture 2: Programme Evaluation

David Hegarty

7 October, 2011

Structure

Part 1: Definition, purpose and key issues in programme evaluation

Part 2: Overview of methodological tools

Part 3: Evaluation capacity and practice in Ireland

Part I: Definition and Purpose

No single, preferred definition of evaluation

“Evaluation” is a common, everyday activity People evaluate films, restaurants etc Firms evaluate investments

“Evaluation is an elastic word that stretches to cover judgements of many kinds” (Weiss)

Some Definitions

“The process of collecting and analysing information and reaching conclusions on specific questions” (Dept. of Finance VFM guidance manual)

“Judgement of interventions according to their results, impacts and needs they aim to satisfy” (EU Commission)

“Evaluation is the process of determining the merit, worth, and value of things, and evaluations are the products of that process” (Scriven)

“The systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy” (Weiss)

Key Features

Definitions point to some key features Evaluation can be carried at various levels: policy,

programme or project A systematic exercise based on accepted social science

research standards Involves forming a judgement to be based on certain

criteria Focus can be either on the process (operation) of the

programme or on its impacts (outcomes) Purpose of exercise is to improve the intervention under

evaluation

Evaluation Purposes: Why evaluate?

Planning Is the programme justified? Programme design Resource allocation

Implementation Is the programme working and/or how can the

programme be improved? Accountability

What was achieved?

Evaluation Purposes

Knowledge What interventions work and in what

circumstances? Does the logic of the programme and its

assumptions need to be questioned? Development

Institutional performance and strengthening Service quality

Evaluation Purposes

But… evaluation sometimes used for other, “covert”

purposesJustify decisions already made Postpone decisionsPublic relationsCompliance

“A rational exercise often undertaken for non-rational reasons” (Weiss)

Summary

Evaluation can be seen as serving an overarching learning purpose

“To learn through systematic enquiry how to better design, implement and deliver public programmes and policies” (EU Evalsed Guide)

Formative and Summative Evaluation Summative evaluation

Accountability focus What has been achieved?

Formative (“Process”) evaluation Development or learning focus How can we improve performance and delivery of programme?

“When the cook tastes the soup that’s formative evaluation. When the guest tastes it, that’s summative evaluation” (Scriven)

Most evaluations lie along this continuum Combining elements of each Both relevant and useful to public sector

Dept. of Finance VFM manual has strong summative emphasis

Key issues in programme evaluation

What's the basis for evaluation judgements?

Dept. of Finance VFM Manual refers to 5 main evaluation criteria Relevance Rationale Effectiveness Efficiency Impact

Framework originally developed for evaluation of EU-funded programmes (CSF Evaluation Unit, 1996)

Evaluation Questions

Rossi et. al (2004) identify 5 main question types in evaluation of social programmes Needs assessment Assessment of programme theory Assessment of programme process Impact assessment Efficiency assessment

DOF framework has stronger summative or economic emphasis

Relevance

Two main dimensions Policy relevance

Domestic EU

External relevance What societal needs or problems does programme

address? (needs assessment) Is the programme “fit-for purpose”? Implications of external changes for programme

(continued relevance)

Rationale

Why is the State involved? Is there a market failure?

“A necessary but not sufficient for government intervention to improve economic efficiency is that there is some form of market failure” (HM Treasury)

Could the problem be addressed through more direct means? Danger of “second-best” solutions

Types of market failure Public goods Externalities Redistribution

Effectiveness

Is the programme meeting its objectives? Generally addressed at level of

Inputs: Is the money being spent? Outputs: Results or immediate benefits

Are the above in line with expectations? If not, why not?

Almost a monitoring question

Effectiveness and impact questions often overlap A lot depends on how objectives are framed

Efficiency Some definitions:

“Efficiency in the public sector involves making the best use of resources available for the provision of public services” (Gershon UK efficiency review 2004)

“Optimising the ratios of inputs to outputs” (DOF VFM Manual)

Can be viewed in a number of ways Reduced inputs for same level of service Additional outputs for same level of inputs Improved unit cost ratio Changing mix of activities/outputs to better deliver a given objective for

same input level Using alternative delivery approaches, e.g., outsourcing to private

sector

Efficiency a core, perhaps overarching, element of value for money agenda Getting the best return from a given level of resources is the essence of

value for money

Impact

What difference has the programme made? To its beneficiaries In terms of wider socio-economic objectives

Need to consider Deadweight effects Displacement Unintended side-effects

So-called horizontal issues a sub-set of impact Rural development Poverty Gender equality

Evaluation Cycle

Evaluation cycle is a function of the wider programme and policy cycles (see diagram)

Aim should be to conduct evaluations at the right time to influence programme design and policy formulation Easier said than done!

Policy, Programme & Evaluation Cycles

ProgrammeImplementation

ProgrammeDesign

Programme Conclusions

PolicyDelivery

PolicyFormulation

Policy Review

Source: The EVALSED

Evaluation

The Evaluation Cycle

EU Structural Fund regulations require evaluations at three stages Ex ante (before) Interim or ongoing Ex post (after)

Value for Money reviews generally take form of ongoing evaluations

Ex Ante Evaluation

Focus: “to optimise the allocation of resources and improve the quality of programming” (EU Regulation) A planning purpose

Key evaluation questions What is the rationale for programme and is it

robust? Is the programme relevant or fit-for-purpose Programme design issues

Interim Evaluation

Focus Largely an implementation purpose But much depends on programme maturity

Key questions Relevance or continued relevance Effectiveness Efficiency

Ex Post

Mainly an accountability purpose what has been achieved and at what cost summative in character

Not widely practiced in Ireland except for EU programmes

Evaluation Cycle and Focus

Stage Ex ante Ongoing Ex Post

Purpose Planning Implementation Accountability

Questions

Rationale *** * *

Relevance *** *** *

Effectiveness * *** **

Efficiency * ** **

Impact * ***

25

Overview of methodological tools

Sourcing informationand data

Data analysis techniques

Tools to informevaluation judgements

26

Data Sourcing

All evaluations require data “the raw material that once collected is organised,

described, grouped, counted and manipulated by various methods and techniques” (EVALSED Guide)

Primary and secondary data Primary data are data generated as a consequence of

programme (uptake of services, data relating to beneficiaries)

Secondary data are generated for other purposes and pre-exist the programme (e.g., socio-economic and administrative data)

27

Data Types Key distinction between quantitative and qualitative approaches Quantitative used to gather “hard” data

Who, what, how many Expressed in terms of averages, ratios or ranges In practice much hard or quantitative data may be categoric or

ordinal in nature Qualitative methods used to gather “soft” data

Focus on understanding or why questions Quantitative/qualitative a continuum

Distinction stronger in terms of analytical intent Quantitative for aggregation and generalisation Qualitative to understand complexity

28

Data sourcing techniques

Main techniques/sources include Monitoring indicators Documentary analysis Administrative data Socio-economic data Beneficiary surveys Stakeholder interviews Focus groups Case studies

29

Data Analysis Techniques

Once the data is collected, how do we analyse it?

Main techniques include Statistical analysis SWOT analysis Econometric models Experimental designs Quasi-experimental designs (control groups)

30

Tools to inform evaluation judgements

Having gathered and analysed the data, how do we arrive at evaluation judgements?

Main tools Benchmarking Multi-criteria analysis Cost benefit analysis and cost effectiveness analysis (will be

addressed in project evaluation stream) Economic impact assessment

Macro Micro

Intervention logic analysis Specialist thematic tools

Gender impact assessment Strategic environmental assessment

31

Factors Affecting Choice of Method Programme type

Stage in programme/evaluation cycle

Evaluation purpose

Evaluation scope and questions

Data availability

Resources

32

Factors Affecting Choice of Method

Programme Characteristics

Stakeholder Priorities

Evaluation Questions

Mode of Enquiry

Choice of Methods

Programme Stage

Evaluation Stage

Choice of Techniques

Policy & Strategic Priorities

Data Availability

‘Evaluability’ Assessment

Part 3: Evaluation capacity and practice in Ireland

Concept of evaluation capacity

Influence and evolution of EU Structural Funds evaluation systems

Development of national programme evaluation processes Expenditure Review Initiative Value for Money and Policy Review Initiative

Where are we now?

The concept of evaluation capacity

Evaluation capacity concerns the process of setting-up the necessary systems and

infrastructures to undertake evaluation

Some definitions “the development of national or sectoral evaluation

systems” (MacKay, World Bank) “the institutional, human, resource, skill and procedural

base for conducting evaluations in public policy and public management systems” (Evalsed Guide)

Concerned with creating and sustaining factors that support evaluation in government sector

Key dimensions of evaluation capacity

Fair degree of consensus in literature as to key building-blocks

4 key dimensions generally highlighted in literature Architecture: organisation of evaluation function Demand: is there an effective demand for evaluation? Supply: evaluation resources (methods, resources, skills) Institutionalisation: building evaluation into policymaking

systems The wider, cultural factors or conditions that determine the

influence of evaluation on policy

Critical Success Factors

Key lessons as to factors needed to strengthen government evaluation systems Substantive government demand essential

Incentives important for demand Limitations of reliance on rules and regulations Need to work on demand and supply sides in parallel Need for evaluation champions Adequate evaluation resources

Including good data systems Importance of structural arrangements/architecture including

links with other functions Danger of over-engineering the system Utilisation is key A long-haul effort requiring patience, persistence and leadership

Evolution and development of EU evaluation systems Evaluation context (as of late 1980s)

Little prior tradition of programme evaluation in Ireland prior to Structural Funds

evaluation limited in scope and largely peripheral to decision making A low evaluation capacity baseline

Evaluation impetus driven by compliance considerations EU Commission pressure and support political priority attached to EU funds

Leading to …. gradual creation of evaluation structures (CSF1, 1989-1993) major expansion in evaluation capacity and output (CSF2, 1994-

1999)

Key Developments in Capacity Development

1994 to 1999 CSF saw gradual establishment of programme evaluation structures 3 internal evaluation units 6 external evaluators to other programmes By end-1996 each programme (9) had dedicated

evaluation function Central CSF Unit with a coordination and good practice

promotion remit set up in 1996

Lessons learnt influenced design of evaluation arrangements for 2000 – 2006 period

2000 to 2006 Evaluation System: Key Features

Evaluation system extended from just EU-funded elements to entire NDP (€51 bn) Applied to up to 20% of total public expenditure

Centralised system with 1 Evaluation Unit (NDP/CSF Evaluation Unit)

Main responsibilities and activities Development of performance indicators Advice on project appraisal techniques Drafting evaluation terms of reference Commissioning evaluations Doing evaluations

Extensive ongoing evaluation effort

Development of National Evaluation Systems Background/origins

Increased emphasis on public service management Public management reforms in early/mid 1990s

International developments in public management Key milestones

C&AG amendment Act, 1993 Gave C&AG mandate to carry out VFM audits And to examine adequacy of departments’ systems to evaluate

effectiveness of their operations 76 VFM audits to date

Launch of Strategic Management Initiative (1994) Delivering Better Government (1996) Public Service Management Act, 1997

Departments required to produce Statements of Strategy and Annual Reports

Expenditure Review Initiative

Expenditure Review Initiative (ERI) Non-EU evaluation system introduced in 1997

ERI influenced by Australian system A “whole of government” evaluation strategy Objectives

to provide a systematic analysis of what is actually being achieved by expenditure in each programme; and

to provide a basis on which more informed decisions can be made on priorities within and between expenditure programmes

ERI: Key Features

Aim was to review all expenditure areas every 3 years Programme of reviews agreed by each department

with Department of Finance Central Steering Committee and secretariat in

DOF Reviews undertaken by line departments and by

programme managers Department of Finance represented on steering

committees

Evolution of ERI

ERI reviewed by C&AG in 2001 Key findings

3 year target not met, significant delays Reviews focused on minor programmes Quality highly variable Limited influence on resource allocation

Number of reforms introduced Establishment of network of reviewers and training supports Independent quality review procedure Efforts to track impact of reviews and review process

generally

Evolution of ERI

Central steering committee (ERCSC) reviewed progress in October 2004

Key findings Taken time to for earlier reforms to take effect Slippage in timeframe for completion of reviews Topics selected for review relatively small scale Evaluative capacity of departments variable Process had led to improvements in approach to evaluation

and evaluation culture of departments Extent to which reviews driving resource allocation

decisions unclear

ERI Review

Series of recommendations made by ERCSC Changes to structures and reporting

arrangements in departments Independent steering committees Reporting on review results Intensify efforts to develop performance indicators Use trainee analysts and graduates from IPA

policy analysis masters course to support review process

Value for Money and Policy Review Initiative

ERI replaced in 2006 by “Value for Money and Policy Review Initiative” Somewhat wider evaluation focus Ninety reviews approved for 2006-2008 period

2 per department per year Guidance manual published 2007 Mix of internal and external reviews

Target for number of reviews does not appear to have been reached Progress rather uneven

Post 2007, EU funding very limited EU funding of just €900 mn. for 2007 to 2013 Limited evaluation requirements under EU regulations

Earlier NDP/CSF Evaluation Unit replaced by Central Expenditure Evaluation Unit in MOF Responsible for evaluation of all national programmes Main focus of Unit now on Value for Money and Policy Review

Initiative Undertaking and overseeing VFM reviews Issuing guidelines Unit also has important role in project evaluation area

Current Situation

Reflections on Irish Experience

EU requirements and external influences a key driver EU funds contributed to development of capacity

and expertise Increased awareness and understanding of

evaluation amongst policymakers Led to creation of internal evaluation structures And improved supply-side capacity in response to

evaluation demand Important long-term benefits of Structural

Funds

Reflections on Irish Experience

Ireland now in a “post Structural Funds” era Evaluation system no longer organised around EU Funds

Slow progress under ERI and VFM processes Some signs of a loss of momentum in evaluation

practice over recent years Current economic difficulties means heavy emphasis is on

expenditure control and reduction and broader expenditure review exercises McCarthy review (2009) Comprehensive Spending Review

Conclusions: Some quotes “In the age of evaluation Ireland has been encouraged or even

compelled by the pressure of very influential external forces to adopt a culture of evaluation. Despite the fact that this culture dates back over some three decades it remains a somewhat uneasy and unconvincing addition to the tools of governance”

“One thing seems clear: policy developments in the field of evaluation will continue to be largely driven by external pressures since there is very little evidence of an appetite for evidence-driven policy among senior political or public-sector leaders” (McNamara et al, Developing a Culture of Evaluation In Ireland, 2009)