EVALUATING THE PERFORMANCE OF OECD COMMITTEES 11th ANNUAL MEETING OF THE OECD SENIOR BUDGET OFFICIALS PERFORMANCE AND RESULTS NETWORK 27 NOVEMBER 2015 Kevin Williams – Head of In-depth Evaluation (SGE-EVIA)

Evaluation in the OECD

2

Integrated Management Cycle Programme Implementation Reporting (PIR)

Medium-term Orientations exercise (MTO) In-depth Evaluation of Committees (IDE)

Committee activities and work programme development and implementation (including peer reviews, evaluation

guidance, etc.)

Members and Partner countries

Organisational

focus

Substantive

focus

Integrated Management Cycle

Biennium 2013-14 Biennium 2015-16 Biennium 2017-18

PIR

MTO

PIR PIR

MTO MTO

IDE

Prospective

Restrospective

The objective of IDE is to provide a mechanism through which Council can assess whether OECD Committees are…

– conducting processes…

– delivering outputs…

– achieving impacts…

4

IDE’s overarching objective

… in line with Members’ policy

expectations and with the OECD’s

comparative advantage

5

Evaluation criteria for IDE

• the extent to which the Committee is • producing products of the requisite quality for the

resources allocated (technical efficiency) • how well it is functioning (process efficiency)

Efficiency

• the extent to which a Committee’s mandate and work programme objectives are aligned with Members’ policy needs and concerns

Relevance

• the extent to which policy impacts resulting from the Committee’s products are occurring and whether they correspond with areas of highest policy needs and concerns

Effectiveness

6

IDE Cycles and timeline

1st IDE Cycle

2005 2012

2nd IDE Cycle

2017

Example of an OECD Committee

Public Governance

Committee (PGC)

Working Party of Senior Budget

Officials (SBO)

Network on Public

Employment and

Management (PEM)

Network on Public Sector

Integrity

Network on E-Government

High Level Risk Forum

(HLRF)

Working Party of Leading

Practitioners on Public

Procurement (LPP)

Network of Senior

Officials from Centres of

Government (COG)

Network on Financial Management

Network of Parliamentary Budget Officials and Independent Fiscal Institutions

Network on Performance and Results

8

Committee policy cycle

Policy needs

Biennial Work Programme

Processes Outputs

Policy impacts

Mandate

Policy objectives

Resources

Co

mm

itte

e o

rien

tati

on

Committee functioning

Use

Awareness

Dissem

ina

tion

a

nd

tak

e up

Policy needs and impacts

• Committee mandates and work programmes aim to address Members’ policy needs, e.g.

– to deliver better, more cost-effective and user-centric public services

– to further enhance corruption resistance in risk areas at the political-administrative and public-private sector interface

– to ensure greater inclusiveness, and increased economic and social resilience through risk management

– …

Examples of policy objectives

• OECD knowledge products and instruments:

– Statistics, indicators, databases, related development work (e.g. methodological frameworks)

– Benchmarking and comparative reviews

– Reports and analyses (incl. peer reviews of Member and non-Member economies)

– Conferences, workshops, forums

– Guidelines and recommendations

– Formal agreements

10

Committee outputs

• Knowledge, information, data, guidance, recommendations, etc. embodied in a Committee output:

– substantively represent or form the basis of government policy

– are considered as the standard for policy setting

– have been raised in Parliament, been the subject of Ministerial and/or official announcements

– have been proposed to be enacted as legislation, enacted as legislation or the subject of international agreement

– have been raised in major public forums as being authoritative for policy direction

– …

Examples of policy impact

12

IDE outputs and outcomes

Conclusions and

ratings

Recommendations and Good Practices

• Reinforced transparency and accountability to Council

• More informed Council decisions on mandate appropriateness and renewal

Analyses and

findings

• Mandate and work programme development

• Meeting preparation and conducting

• Work programme oversight

• Vertical coordination

• Horizontal working

• Engagement with Partner countries

• Involvement of other international organisations and stakeholder bodies

• Communication and dissemination

Monitoring

of implementation

• Design and implementation of relevant actions

• Improved committee performance

• Planning, timing and purpose

• Support from senior management

• Involvement of stakeholders

• Dissemination of results

• Monitoring follow-up of recommendations

• Evaluation quality

• Resource availability

Factors affecting the likelihood of evaluation

use*

*Bastiaan de Laat and Kevin Williams from Enhancing Evaluation Use: Insights from Internal Evaluation Units, Marlène Läubli Loud and John Mayne (eds), Sage, 2014.

Accountability to governing bodies and awareness raising amongst stakeholders more broadly

Improving the design and implementation of ‘interventions’

Resource (re)allocation within ‘interventions’

Supporting organisational learning

Setting strategic or policy priorities

Resource (re)allocation between ‘interventions’

Different evaluation uses

• To what extent does your experience of the evaluation use tally with the previous slide?

• Why is the use of evaluation results less prevalent in priority setting and resource (re)allocation contexts?

Possible discussion questions