7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
1/35
SAI Performance Measurement
FrameworkPresented by:
Yngvild H. Arnesen, INTOSAI Development Initiative
ICGFM Winter Conference, Washington, DC, 10-12December 2012
1
http://www.wgvbs.com/Home.aspx7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
2/35
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
3/35
1. Introduction
3
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
4/35
What is a Performance Measurement
Framework? Tool for measuring performance
of an individual, system ororganization
against an agreed performancescale
accommodating localcircumstances
To enable Better management of
performance
Monitoring over time
Evaluation and lesson learning
Contributing to Understanding performance
Improving performance
4
Not everything
that matters can
be measured
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
5/35
What is the SAI Performance Measurement
Framework (PMF)?
A Performance Measurement Framework tailored to SAIs
Intended to give a high level assessment of SAI performanceagainst established INTOSAI good practice:
The International Standards of Supreme Audit Institutions (ISSAI)
framework The Framework on the Value and Benefits of SAIs
Other INTOSAI guidance material
Combination of objective measurement and qualitative
assessment
Analyzes 7 domains of SAI performance in the country context andaims at assessing the value and benefits of the SAI to society(impact)
5
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
6/35
SAI PMF How will it be used in Practice?
6
Who decides to initiate a SAI PMF?
Who decides when to do a SAI PMF?
Who decides how to do a SAI PMF?
Head of SAI
Head of SAI Consulting with INTOSAI regions & donors as appropriate
Every 3-5 years recommended
Head of SAIApproach selected to meet the SAIs objectives 3 broad approaches Quality assurance to promote objectivity and credibility
Who decides if, when and how to
publish a SAI PMF?
Head of SAI
1. Using the SAI PMF will be voluntary for the SAIs the tool will be a public
good which SAIs can make use of if it helps them achieve their objectives
2. The SAIs themselves will make the key decisions regarding the assessment
3. The SAI will also decide whether or not to publish results
4. Because it is voluntary, the SAI PMF is not intended to replace any internal
performance measurement procedures
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
7/35
7
1. SAI SelfAssessment
Trained Team
from withinthe SAI
2. INTOSAI PeerReview
Peer SAI
INTOSAIRegional Body
3. ExternalAssessment
Professional
PFM Body Donor Agency
Consultant
Externalauditor
3 Approaches to Conducting a SAI PMF Assessment
There are three main approaches to conducting an assessment using the
SAI PMF. The PMF will be equally well suited for all three:
The approach will not necessarily affect how the assessment is carried
out
but the SAI may use the results for different purposes depending on
the approach
SAI Involvement
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
8/35
2. Background and rationale for
developing a SAI Performance
Measurement Framework (PMF)
8
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
9/35
Background
9
Full set ofInternational
Standards of
Supreme Audit
Institutions (ISSAIs)
adopted by INTOSAI
in 2010
INTOSAI
Framework on the
Value and Benefits
of SAIs adopted
2010
Increased focus onuse of country
public financial
management
systems within the
international donor
community
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
10/35
Mandate: Johannesburg Accords
(INCOSAI XX, 2010)
10
Working Group on the Value and Benefits of SAIResponsibility
Develop further credible measurement tools to supportevaluations in terms of the framework for Communicating and
Promoting the Value and Benefits of SAIsTask
INTOSAI Capacity Building Committee
INTOSAI-Donor Steering CommitteeCollaborating with
Tools that are already available at the level of internationalorganizationsCognizance of
For consideration at XXI INCOSAI in 2013Timetable
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
11/35
Improve SAI approaches to capacity development through
use of performance management
Identify opportunities to strengthen SAI performance
Supreme Audit Institutions
Promote transparency and accountability through crediblepublic reporting of SAI performance (ISSAI 20)
Domestic Parliaments and Citizens
Enable better peer learning
INTOSAI Regions and INTOSAI Global Community
Demonstrate SAI performance changes to key stakeholders
International Development Community (Donorsand SAI Providers of Support)
SAI Enhanced SAI integrity andcredibility with domesticstakeholders
Stronger SAI-led capacitydevelopment programs
SAIs become modellearning organizations
Evidence-based policiesand better targeted globaland regional programs
Maintain external supportfor SAI capacitydevelopment
SAI PMF Stakeholders and Purpose Benefit to the SAI
Rationale for INTOSAI to Develop a SAIPerformance Measurement Framework
11
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
12/35
Why should SAIs use the PMF (1)
As a step towards ISSAI implementation
Increased understanding in the INTOSAI community of what goodpractice for SAIs entails
Individual SAIs can benchmark their performance against good
practice and get an indication of where there is room for
improvement They can use the assessment as a basis for more detailed ISSAI
compliance assessments
To demonstrate progress and value and benefits to society
The framework will be well suited for measuring progress overtime and demonstrating this to external stakeholders
Show to stakeholders how the SAI contributes to strengthening
public financial management
12
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
13/35
Internal performance measurement / annualreporting
Help SAIs better manage their own performance
ISSAI requirement to assess and report on own performance
lead by example and become a model organization
Inform and secure support for capacity development
initiatives
Identify strengths and weaknesses, increase understanding ofreasons for good and weak performance
Help build a culture for performance improvement
SAIs can use assessment results to get support from donors and
other external stakeholders13
Why should SAIs use the PMF (2)
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
14/35
Implement SAIDevelopment Action
Plan
High LevelPerformance Overview
Progress Monitoring
Identify Main SAIStrengths andWeaknesses
Investigate UnderlyingCauses
Recommend SAICapacity Building
Measures
Formulate or UpdateSAI Development
Action Plan
(Medium Term)
The Role of SAI PMF Assessments in
SAI Capacity Development
SAI PMF
Assessment
SAI Capacity
Building Needs
Assessment
Formulate orUpdate SAI
Strategic Plan
(Long Term)
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
15/35
SAI Performance Measurement Framework (SAI PMF)
& ISSAI Compliance Assessment Tool (ICAT)
SAI PMF
ICATs
Founding Principles LimaDeclarationLevel 1
Prerequisites for the
Functioning of SAIsLevel 2
Fundamental
Auditing PrinciplesLevel 3
AuditingGuidelines
Level 4
ISSAI Framework
Auditing
GuidelinesLevel 4
FundamentalAuditing PrinciplesLevel 3
Prerequisites for theFunctioning of SAIsLevel 2
Founding Principles LimaDeclarationLevel 1
ISSAI Framework
2. High level overview or
detailed drill downtool?
3. Assessment
against goodpractices or
against ISSAI
framework?
International
Standards of
Supreme Audit
Institutions
4. Score levels of
performance orassess compliance
with standards and
investigate causes?
ISSAIs
Human
Resources
Strategic
Planning
What needs to bein place to
implement level 4?
1. Monitor change
over time orassess capacity
building needs?
Value &
Benefits
Framework
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
16/35
3. Responsibilities and reporting
lines
16
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
17/35
17
Responsibilities and Reporting Lines
INTOSAI Working Group on
the Value and Benefits of SAIsOversight and Direction
WGVBS SAI PMF Task Team
Development of SAI PerformanceMeasurement Framework
Coordinator: INTOSAI-Donor Secretariat @ IDI
SAI PMF Reference Group
Technical Advice, Review andQuality Assurance
INTOSAI CongressApproval
INTOSAI
Capacity Building
Committee
Members
INTOSAI-Donor
Steering Committee
Members
Other Groups:
PEFA Secretariat,
International
Budget
Partnership
INTOSAI
Capacity Building
Committee
Members
INTOSAI-Donor
Steering Committee
Members
InConsultationwith:
INTOSAICapac
ityBuildingCommittee
INTOSAI-DonorSteeringComm
ittee
OtherRelevan
tINTOSAICommittees
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
18/35
18
Task Team
INTOSAI Regional Organizations:
1. African Organization of English
Speaking SAIs(AFROSAI-E)
2. African Organization of Francophone
SAIs (CREFIAF)
SAIs:
3. Bahrain
4. Brazil
5. Chile
6. Germany
7. India
8. Mexico
9. Russia
10. United KingdomDevelopment Agencies:
11. Inter American Development Bank
Coordinator:
12. INTOSAI-Donor Secretariat @ IDI
Reference Group
INTOSAI Regional Organizations:
1. Pacific Association of SAIs (PASAI)
SAIs:
2. Canada
3. Denmark
4. Estonia
5. European Court of Auditors
6. Japan
7. Netherlands
8. Norway
9. Peru
10. Sweden
Development Agencies:
11. African Development Bank12. OECD
13. European Commission
14. World Bank
Civil Society Organizations:
15. International Budget Partnership
16. Global Reporting Initiative (GRI)17. PEFA Secretariat
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
19/35
4. Indicative content and design
19
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
20/35
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
21/35
SAI PMF Task Team:
Initial Work and Decisions
21
No existing framework fully meets INTOSAI needs
Develop a new tool drawing heavily on existing ones (AFROSAI-E, PEFA, INTOSAI Peer ReviewGuide, IDI Capacity Building Needs Assessment Handbook)
Mapping of 20 Existing SAI Assessment Tools
ISSAI Framework where possiblerecognizing it will continue to evolve over time The SAIs mandate
INTOSAI guides & other generally accepted international standards or good practices
What to Measure Performance Against?
External performance delivery of core audit activities (audit coverage, submission of audit
reports, etc.) Internal enablers of performance SAI strategy, audit processes, SAI support services
SAIs operating environment (i.e. independence and legal framework) recognizing this is largelybeyond the control of the SAI
Reported in the context of the specific country governance and public financial management
environment and its impact on the SAIs performance
HOLISTIC MEASUREMENT OF THE SAI AND ITS ENVIRONMENT
What is performance? What are we going to measure?
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
22/35
22
Qualitativeassessment
(PerformanceReport)
Measurable
indicators
SAI PMF: Evidence Based
Assessment of Performance
Objective
measurement to
inform qualitative
assessment and track
progress over time
Holistic assessment of
the SAI and its
environment
including factors which
are not covered by the
indicators
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
23/35
B.
Independence
and Legal
Framework
3 indicators
A. SAIPerformance
3 indicators Contribution
to Public
Financial
Management,Governance
and Tackling
Corruption
SAI Environment SAI Capability Impact
Public Financial Management Environment
SAI
Performance
SAI Performance Measurement Framework Structure
D. Audit Standards and Methodology
3-5 indicators
SAI SUPPORT SERVICES
E. Management
& Support
Structures
2 indicators
F. Human
Resources
3 indicators
G. Communication
and Stakeholder
Management
3 indicators
C. Strategy for Organizational Development
1 indicator
SAI STRATEGY
SAI CORE BUSINESS
23
SAI P f M t F k St t
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
24/35
B. Independence and
Legal Framework
Independence of the
SAI
Independence of the
Head of the SAI and
its Officials
Mandate of the SAI
A. SAI Performance Financial Audit
Results
Compliance Audit
Results
Performance Audit
Results
SAI Value Added
Services
SAI Annual Report
SAI Contribution to
Aggregate Fiscal
Discipline
Efficient Resource
Allocation
Efficient ServiceDelivery
Good Governance,
Accountability and
Tackling Corruption
SAI Environment SAI Capability Impact
Quality and Timeliness of Annual Financial Statements (PEFA PI-25)
(Including completeness, timeliness and accounting standards used)
Legislative Scrutiny of External Audit Reports (PEFA PI-28)(Including timeliness & extent of hearings, recommendations of the legislative & implementation by the executive)
Public Financial Management EnvironmentInfluences the effectiveness of the SAI but not under direct control of the SAI.
Measured through other assessments but referred to explicitly in the narrative SAI performance report.
Foundational factors set by the
legislative and/or executive, but
which can be influenced by the
SAI in the long term
Under the control of the SAI
SAI Performance
Measured using other assessments.
Achievement of impacts can not be
wholly or directly attributed to SAI
results. However, the way in which
the SAI results are likely tocontribute to high level impacts
should be summarized in the
narrative SAI performance report.
Under the control of the SAI
SAI Performance Measurement Framework StructureNote: all indicators should try to capture both what is written in legislation and procedures (de jure) and what happens in practice (de facto)
D. Audit Standards and Methodology Audit Planning for the SAI
Financial Audit
Compliance Audit
Performance Audit
Quality Control & Quality Assurance of Audit Processes
SAI SUPPORT SERVICES
E. Management &Support Structures
Management
and Internal
Control
Support
Structures &
Infrastructure
F. Human Resources Management of
Personnel
Professional
development
Leadership
G. Communication and
Stakeholder Management
Communications
Strategy, and Internal
Communication
Communication with the
Government Branches
Communication with the
Media, the Citizens and
the Civil Society
C. Strategy for Organizational Development
Strategy for organizational development
SAI STRATEGY
SAI CORE BUSINESS
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
25/35
SAI Performance Report- end product from an assessment (30-40 pages?)
25
Enable SAI heads and senior management to
Have a high level overview of the SAIs performance
Promote and communicate the value and benefits of the SAI
Identify barriers to improved performance
Track performance changes over time
Explores how SAI external performance is constrained by
Country context and PFM environment
SAI environment
SAI internal capability
Provides a coherent and holistic assessment of SAI performance
Informed by objective and evidence based performance indicators
Analyzed within context Considered as a whole not the sum of its indicator scores
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
26/35
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
27/35
Indicator Design Each indicator provides an objective measure of current, not planned,
performance through 1-4 dimensions
Measured on a scale of 0-4
Translated into a single indicator score
27
Activity not Established
or does not Function
LEVEL 0
Founding Level
LEVEL 1
Established Level(Follows ISSAIs)
LEVEL 3
Managed Level
LEVEL 4
Development Level
LEVEL 2
Executiveagents held to
account
Accountabilityfor use of
public funds
SAI enablescontinual
improvement
Accountability forGovernmentperformance
Each levelprovides a basisfor accountability
MeasurementScale
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
28/35
Quality Assurance
Quality assurance crucial for objectivityand credibility of assessments
Enables a single assessment to meetdifferent stakeholder needs
Most appropriate approach dependson purpose of the assessment
Good practice: SAIs select QA process to meet the
assessment purpose
QA process is transparently disclosed inthe performance report
Separation between those undertakingthe assessment, and those performing QA
QA should cover: Is the report factually correct?
Has the SAI PMF tool been appliedcorrectly?
Is there sufficient evidence to justify theindicator scores?
28
Quality Assurance Statement
Who has prepared the
assessment?
Who has provided quality
assurance of the assessment?
What were their quality
assurance responsibilities?
How have matters raised in the
quality assurance process been
addressed in the final report?
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
29/35
4. Looking ahead: Responsibilities
and timelines
29
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
30/35
Potential risks related to having a SAI PMF
Revelations of performance weaknesses
It will be up to the SAI to decide if results are to be shared Openness and commitment to reform likely to encourage external
partners to support capacity development
May help draw attention to constraints outside the SAIs control
Possible negative effect of scoring: Use of performance league tables
Intention is not to aggregate scores to the level of the entire SAI as allindicators may not be of equal importance
Scores partly linked to achievements of SAIs own mandate
Narrative report provides context for scoring
Scoring is already applied in INTOSAI (AFROSAI-E)
Scoring very useful for measuring progress over time within an SAI
Profusion of Assessment Tools
Risk that the SAI PMF becomes an additional tool
Donors and SAIs continue to develop & use other tools
30
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
31/35
Identified Challenges in the Development
of the SAI PMF (1)
Different SAI models, mandates and country context
make it challenging to develop a global tool ISSAIs and other established good practice is common benchmark
Not measure at a too detailed level Assess against SAIs own mandate
Different SAI models and regions represented in Task Team and
Reference Group
Pilots to identify challenges and address them
Possible to answer N/A and to add local indicators
31
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
32/35
Identified Challenges in the Development
of the SAI PMF (2)
Scoring / objective measurement not possible Performance report will provide qualitative assessment and put
scoring into context
Scoring useful for measuring progress over time
Challenging timetable Johannesburg accords look for tool to be presented at INCOSAI
XXI, China, 2013
WGVBS looking to approve Exposure Draft for INCOSAI
Global piloting and consultations will extend beyond 2013
32
SAI PMF Timetable
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
33/35
SAI PMF Timetable
33
2011, Q3 2011, Q4 2012, Q1 2012, Q2 2012, Q3 2012, Q4 2013, Q1 2013, Q2 2013, Q3 2013, Q4
Mapping Existing SAI
Measurement Tools
Concept Note for SAI PMF
Develop SAI PMF Version 2.0
Phase 1 Pilots in 3 SAIs
& Comments from
Stakeholders
Develop
SAI PMF
Exposure
Draft
Phase 2 Piloting and Pre-Consultation
(ongoing into 2014 & 2015)
WGVBS:
Approve
way
forward
WGVBS:
ApproveExposure
Draft
WGVBS:
Set
Direction
Exposure
Draft at
INCOSAI
I l t ti d M i t
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
34/35
Implementation and Maintenance
(proposed)
Present SAI PMF at INTOSAI and other stakeholder events
Promotion and Awareness Raising of SAI PMF
Development and refinement of training material
Training of trainers within INTOSAI regional bodies
Training of SAI staff
Training of other stakeholders (donors, consultants, CSOs)
Training on Use of SAI PMF Assessments
Host and maintain guidance materials on internet
Provide Guidance and Support to Conduct of SAI PMF Assessments
Maintain and update global summary of the status of SAI PMF assessments, to enhancecoordination
Manage an e-library of published SAI PMF assessments Maintain database on (published) SAI PMF indicator scores for benchmarking, evaluation and
research
Monitor Progress on Use of SAI PMF and Maintain Consolidated Indicator Data
Decision to modify or update the SAI PMF (e.g. as ISSAI framework develops)
Develop and refine the SAI PMF
Technical Development and Refinement of SAI PMF
34
INTOSAI-Donor
Secretariat
INTOSAI-Donor Secretariat
INTOSAI-Donor Secretariat
INTOSAI-Donor Secretariat
INTOSAI-Donor SecretariatINTOSAI Regional Bodies
INTOSAI-Donor Secretariat
INTOSAI-Donor Secretariat
Database of SAIs Task Force
Database of SAIs Task Force
INTOSAI WGVBS
SAI PMF Task Team
INTOSAI
Community
Donor
Community
Approved Trainers
Approved Trainers
7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen
35/35
6. Further Information
www.idi.no
E-mail: [email protected]
35
Mapping of Tools
for Assessing
Performance of
Supreme AuditInstitutions
IDI, May 2012
Supreme Audit
Institutions
Performance
MeasurementFramework
Concept Note
IDI, May 2012
http://www.idi.no/mailto:[email protected]:[email protected]://www.idi.no/