1 Workshop: Development and Use of Outcome-based Measures in Government Planning & Reporting...

1

Workshop:Development and Use of

Outcome-based Measures in Government Planning & Reporting

September 28th, 2006 Winnipeg, Manitoba

Presented by:

Manitoba Treasury Board Secretariat ,Office of the Provincial Comptroller of Manitoba, and

The International Institute for Sustainable Development (IISD)

2

AgendaI. The Context in Manitoba [10-10:30am]

• Opening Statements and introductions• Overview of government-wide departmental planning and

reporting

II. Trends in Planning, Reporting and Performance measures [10:30-11:15]• What are outcome-based measures and why are they

important?• Innovative case examples

III. Sharing of Examples From the Department of Advanced Education and Literacy [11:15-noon]• Plenary discussions based on Pre-work assignment

IV. Analytic Tools for the Development and Use Outcome-based Measures [noon-2:15pm]• Key steps in development, use and analysis of outcome-based

measures• Group work on key steps

Closing Remarks [2:15-2:30pm]

http://www.clipsahoy.com/webgraphics/as0306.htm

http://www.clipsahoy.com/webgraphics2/as3803.htm

http://www.clipsahoy.com/webgraphics3/aw4736.htm

3

Objectives of Workshop

• Intro to developing performance measures linked to organizational goals

• Overview of current practice elsewhere

• Understanding/discussion of issues related to government performance measurement reporting

• Context for new Annual Report requirements (looking back) and new PSO requirements (looking ahead)

4

Part I. The Big Picture

• Performance Measurement in Manitoba

• Role of performance measures in annual reporting

I. The Big Picture

5

Trends and context

Trend toward reporting on changes in socio-economic and environmental conditions that matter to Manitobans

I. The Big Picture

http://www.gov.mb.ca/finance/mbperformance/perf01.html

6

2005 Discussion Document examples: Economy

I. The Big Picture

7

2005 Discussion Document examples: People

I. The Big Picture

8

2005 Discussion Document examples: Community

I. The Big Picture

9

2005 Discussion Document examples: Environment

I. The Big Picture

10

The Planning and Reporting context

• Performance Measurement is not done in isolation

• Identified as part of Priorities & Strategies Overview (PSO)

• reported upon in Annual reports

• Performance Measures show whether our plans are working

I. The Big Picture

11

Full cycle view

PSO (plan) for following

fiscal

New initiatives

for following

fiscal

Estimates for

following fiscal Budget

announcedEstimates

supplementExecution of plans during fiscal year

Annual Report

Spring Yr1 Fall Yr1 Spring Yr2 Fall Yr3Spring Yr3

I. The Big Picture

12

Full cycle view


fiscal

New initiatives

for following

fiscal

Estimates for


announcedEstimates


Annual Report


I. The Big Picture

13

Multiple cycles in play

BudgetEstimates

supplement

Annual Report

Spring 04 Fall 04 Spring 05 Fall 06Spring 06Fall 05

PSO (plan)

New initiatives

Estimates

Budget

Estimates supplement

Annual Report

PSO (plan)

New initiatives

Estimates

Budget

Estimates supplement

PSO (plan)

New initiatives

Estimates

04/

05

07/

08

05/

06

06/

07

I. The Big Picture

14

Issues/Opportunities re Annual Reports

• Previous guidelines did not request measures of progress or performance

• No specific requirement to link annual reports to larger process (Estimates Supplement or plans)

• No specific requirement for Department annual reports to be placed online

• New direction set in 2005 through Reporting to Manitobans on Performance

I. The Big Picture

15

Trends and issuesin government annual reporting

• Accountability• Who uses annual reports? (internet)• Public expectations (of results, of reporting)• Limited resources, demographic trends

(retirements)… • …governments need to do “more with less”• Trend to shared, horizontal efforts• Important to agree on how to assess, report and

use results

16

Recap: Full cycle view


fiscal

New initiatives

for following

fiscal

Estimates for


announcedEstimates


Annual Report


Annual Reports, including measures, are connected back

to plans…

I. The Big Picture

17

Part II. Trends in Government Planning and Reporting

A. What are outcome measures and why are they important

Concepts and vocabulary

B. Who is doing this really well? Oregon: Results-oriented Strategic Planning

C. What are Alberta and Saskatchewan doing? Alberta: Goal-based Budgeting Sask. Government Accountability Framework

II. Trends in Reporting

18

A. What are outcome measures and why are they important?

Our People

Our Economy

Our Environment

de•vel•op (di•vel′əp) v.t. 1. To expand or bring out the potentialities, capabilities, etc.


mailto:%[email protected]

19

A. What are outcome measures and why are they important?

Our People

Our Economy

Our Environment

de•vel•op (di•vel′əp) v.t. 1. To expand or bring out the potentialities, capabilities, etc.


Experience has shown that a pathway to sustainability cannot be charted in advance. Rather, the pathway must be navigated through processes of learning and adaptation.

National Academy of Science 1999. Our Common Journey: A Transition Toward Sustainability.

mailto:%[email protected]

20

IISD International and National-Level Perspective

• Nations starting to develop information systems to gauge societal wellbeing and sustainability

The United Kingdom

Norway


21

Vision

Priorities andObjectives

Plans

Implementation Budgets

Monitoring

Evaluation andImprovement

SD Principles

Inter-generational consideration

Systems thinking

Multi-stakeholder participation

Adaptive managementDepartmental,

Thematic

National SD Strategy Process

Vision

Priorities andObjectives

Plans

Implementation Budgets

Monitoring

Evaluation andImprovement

Accountability Principles

Transparency

Efficiency

Accountability Departmental,

Thematic

Government Accountability

SystemsSustainable

Development Principles

Accountability Principles

22

• Provinces and states using societal goal and outcome-based planning and budgeting systems

Oregon

Alberta

IISD Provincial and State Level Perspective


23

IISD Community-level Perspective

• Communities creating new forms of social infrastructure to navigate quality of life and sustainability

Orlando– Healthy Community Indicators Initiative

of Greater Orlando

Winnipeg– Quality of Life Indicators System concept

being proposed by range of community stakeholders


24

Output The result of an activity

IntermediateOutcome

Short to medium-term consequence of an output

High-levelOutcome

ActivityProcesses and Inputs of a policy, program, or project

Hybrid from three different logic models (Canadian International Development Agency, Province of Alberta, State of Oregon)

Contributing to society’s wellbeing and

sustainability objectives

External influences

Feedback

Change in wellbeing conditions (economic, social and environmental)

accountability

Key Priority Areas

Logic Model

High-level Outcomes

Actions

High-level Objectives

25

OutputThe result of an activity

IntermediateOutcome


High-levelObjectives

Change in well-being conditions (economic, social and environmental)

ActionsProcesses and Inputs of a policy, program or project

Stream water quality

(turbidity)

Rate of soil erosion by

water

Kilometers of river bank with

vegetation

% of re-vegetation programs completed

Environmental Social

Key Priority Areas Healthy, sustainable

surroundings

Quality Jobs

Example Measures

Children entering school ready to learn

% children enrolled in Pre-K

program

Demographic surveys for Head-

start program completed

Head-start program

implemented

26

Climbing the Steps toward Performance Management

Mission/GoalsObjectives

PerformanceMeasures

Analysis for Continuous

Improvement

Mission statements declare the agency’s long-range intent; its purpose. Although the goals expressed in a mission statement may help shape the agency’s values and its organizational culture, they often are imprecise and sometimes even a bit vague.

Objectives are unambiguous statements of the agency’s performance intentions, expressed in measurable terms, usually with an implied or explicit timeframe.

Performance measures indicate how much or how well the agency is doing. Ideally, they track the agency’s progress toward achieving its objectives.

Many agencies compare this month’s or this year’s performance measures to those of the past. Some are beginning to make comparisons with other agencies and to begin the process of benchmarking.

(From Gov. of Alberta 1996)


27

Who does this really well?The Oregon Shines Case Study

Oregon Shines Case Study

28

Oregon Shines

Oregon's Strategic Plan- Oregon Shines (1989)- Updated every eight years

- Encompasses the entire stateOregon Progress Board

- independent agency created to be the steward of Oregon Shines

- law mandates Board to report biennially

- chaired by governor

Oregon Shines Case StudyFrom Conrad (2005)

29

Vision – “Oregon Shines II”

Economy: Quality jobs for all Oregonians

People: Safe, caring andengaged communities

Environment: Healthy, sustainable surroundings


30

Oregon BenchmarksMeasures for how Oregon as a whole is doing.

• Quality Jobs for All Oregonians– Economy (#1-17)– Education (#18-29)

• Engaged, Safe & Caring Communities– Civic Engagement (#29-38)– Social Support (#39-60)– Public Safety (#60-67)

• Healthy, Sustainable Surroundings– Community Development (#68-74)– Environment (#75-90)



Key Priority Areas

31

High Level Outcomes

(Benchmarks)

Is work happening? Outputs

Is society benefiting?

IntermediateOutcomes

Are strategies working?

Linking Government to the Benchmarks


32 Oregon Shines Case Study

http://www.oregon.gov/DAS/OPB/




36

Goal #1: Quality Jobs for All Oregonians

Oregon Shines Goals & Benchmarks

Dept of Community Colleges and Workforce

Development


IntermediateOutcome





Key Priority Areas

Dept Goal #2: Oregon’s workforce is well trained and has access to a wide variety

of training programs

Benchmark #29

Workforce skill development

project

Percent of participants ranking WIA funded current workforce

ratings good or better

shared best practices, encouraged quality

services and conducted quality assurance review

37

Linking Government to the Benchmarks

Benchmarks

Organization’s Progress

Performance Measures

Oregon’s

Progress

External Influences


38

Towards Goal-based Budgeting in Alberta

Alberta Case Study

http://www.finance.gov.ab.ca/publications/measuring/measup06/index.html

39

Key Priority Areas for Alberta

40 Alberta Case Study

41

42

Goal #2: Albertans will be prepared for Lifelong Learning and Work

Alberta Measures Up

Department of Advanced Education


IntermediateOutcome





Key Priority Areas

Dept Goal #1: High Quality Learning

Opportunities for All

Introduced Bill 1, the Access to the

Future Act,

The learning system meets the

needs of all learners, society, and the economy.

$3 billion Access to Future endowment; a $1 billion

expansion to Alberta Heritage Scholarship Fund, $500 million

expansion to Ingenuity Fund

Employment Rates of Albertans Age 24-35 by Highest

Level of Education

43

Part III. Sharing Examples

• Sharing of Departmental examples in the use of outcome-based measures

III. Department Examples

44

Manitoba Examples:Pre-Work Discussion

• Review your pre-workshop assignment

• Brief presentations[5 minutes + 5 min questions]

• General discussion of issues

III. Department Examples

45

Part IV: Development and Use of Outcome-based Measures

A. Overview of key steps in the development and use of outcome-based measures

B. Working through an example• Groups of at least two (per

department if possible)

IV. Outcome-based Measures

http://www.clipsahoy.com/webgraphics2/as3598.htm

46

Output The result of an activity

IntermediateOutcome


High-levelOutcome

ActivityProcesses and Inputs of a policy, program, or project

Hybrid from three different logic models (Canadian International Development Agency, Province of Alberta, State of Oregon)

Contributing to society’s wellbeing and

sustainability objectives

External influences

Feedback

Change in wellbeing conditions (economic, social and environmental)

accountability

Key Priority Areas

Logic Model

High-level Objectives

Actions

47

Key Steps

1. Frame the logic model for your issue

2. Identify SMART measures for the logic model

3. Understand and articulate key external influences

4. Analyze feedback based on the SMART measures and external influences


48

Output:

________________

Intermediate Outcome:

_________________

High-level Objective:

_______________

Actions

________________

2. Measures and Targets

Measure:

Measure:

Measure:

1. Logic Model3. External Influences

Influences:

Influences:

Influences:

4. Analysis & Feedback

Measure:

Analysis:

Analysis:

Analysis:

Key Priority Areas


49 Hybrid from three different logic models (Canadian International Development Agency, Province of Alberta, State of Oregon)

Output:

River bank re-vegetation


Reduced soil erosion


Improved stream Water quality

Actions:River bank re-vegetation

program

1. Logic ModelKey Priority Area

Safe drinking water, enjoyable recreation

50

2. Identify SMART Measures

• Specific

• Measurable

• Aggressive, yet achievable targets

• Relevant

• Time-bound


51

Anatomy of a measure

Indicator

Data

Target

Source


Oregon Population Survey, a random sample telephone survey of Oregon households conducted in even-numbered years. Margin of Error +/- 1.60%.

52

Output:







program


River water quality (turbidity, nitrogen

concentration)

Rate of soil and rill erosion on farmland

Kilometers of river bank

with vegetation

1. Logic Model

% of re-vegetationPrograms complete

Key Priority Area



53

Types of Targets

Type ExamplePolicy-specific targets

Determined in a political and/or technical process taking past performance and desirable outcomes into account.

Example: official development assistance shall be 0.4 percent of national GNP.

Standards Nationally and/or internationally accepted properties for procedures or environmental qualities.

Example: water quality standards for a variety of uses.

Thresholds The value of a key variable that will elicit a fundamental and irreversible change in the behaviour of the system.

Example: maximum sustainable yield of a fishery.

Benchmark Comparison with a documented best-case performance related to the same variable within another entity or jurisdiction.

Example: highest percentage of households connected to sewage system in a comparable jurisdiction.

Principle A broadly defined and often formally accepted rule.

Example: the policy should contribute to the increase of environmental literacy.

54

STEP 3: External Influences

• Your policies, programs and projects are not the only influence on the desired outcomes and outputs

• Direct influences• Other department activities, businesses, NGOs, civil society• Other jurisdictions (provinces, countries)• Nature (e.g., weather)

• Indirect influences• Broader societal driving forces (e.g., demographics, markets,

consumption patterns)


55

Output:







program



concentration)

Rate of soil and rill Erosion on farmland


with vegetation

1. Logic Model


3. External Influences

• Quality of water flowing into MB• Treatment plant effectiveness

Zero till practices,weather, crop type

• Independent actionsfarmers• Natural growth

Key Priority Area



56

4. Analysis and Feedback

• What do the measures tell you about your activities?– Output level analysis– Intermediate outcome level analysis– High-level outcome analysis

• How does your understanding of external influences help your analysis?


57

Output:




High-level Objectives:



program



concentration)

Rate of soil and rill Erosion on farmland


with vegetation

1. Logic Model


3. External Influences

Zero till practices,weather, crop type

• Independent actionsof farmers• Natural growth/disease


Analysis: Water quality improving. Quality of water flowing into province has improved significantly.

Analysis: Could be due to less bank vegetation, change in crop or extreme series of extreme rainfall events

Analysis: Cropland expansion could be reason or natural disease. Need to research further

95%

Feedback: Water quality improving but not due to program. Other factors counteracted program such as expanded cropland and reduced zero tillage

• Quality of water flowing into MB• Treatment plant effectiveness

Key Priority Area


58

Exercise 1: Development of an Outcome-based Measure

• Form your work group [during lunch]– Find your working partner (your departmental colleague if

possible)• Select measure to focus on [during lunch]

– With your partner select a measure from your department to focus on for the exercise (from your workshop pre-assignment, or something else of mutual interest and value for your reporting cycle)

• Working with your Colleague [1:00-1:45pm]– Use the attached template as a guideline to carry out the 4

analysis steps for developing and/or using outcome-based measures

• Summarize your analysis on an overhead sheet• Plenary [1:45-2:15 pm]

– Two groups will be asked to share their results, each followed by a plenary discussion

59

Output:

________________


_________________


_______________

Action:

________________


Measure:

Measure:

Measure:

1. Logic Model3. External Influences

Influences:

Influences:

Influences:


Measure:

Analysis:

Analysis:

Analysis:

Key Priority area

60

Expectations for this past round of Department Annual Reports (2005-06)

New: Section featuring department performance reporting

New: Central review of the new section

New: Annual Reports to be consistently available online

61

What was required for the new section of the Report?

• Five progress or performance measures• The few critical indicators that illustrate

progress against desired outcomes• Ideally, those that support key Department

priorities• Could be drawn from previous PSOs,

Reporting to Manitobans on Performance document, or other sources

62

Overview of Key Questions That Needed to Be Addressed

What is Being Measured and

How? (A)

Why is it Important to Measure? (B)

What is the most Recent Available

Value for this Indicator? (C)

What is the Trend over

time for this Indicator? (D)

Comments/Recent

Actions/Report Links (E)

1.

2.

3.

4.

5.

63

Illustrative ExampleWhat is being

Measured and How?

Why is it Important

to Measure this?

What is the Most Recent Available

Value for this Indicator?

What is the Trend over time

for this Indicator?

Comments / recent actions /

report links

Access to health services by measuring the percentage of Manitobans reporting “no difficulty” in accessing health services.

One key determinant of population health is its access to quality health services. This measure assesses access only.

For 2003 (most recent survey data), % of Manitobans who reported “no difficulty” accessing: Health information or advice: 82% Immediate care: 75% Routine care: 81%

ImprovingWait times for key procedures have reduced significantly over the last five years. See page X of the report for details.The source of these data is the Statistics Canada Health Access Survey. There is no directly comparable study prior to 2003. The study will be repeated in 20xx.

See page X of the report for a discussion of recent actions addressing access to health services.

NOTE: All information above is adapted from Reporting to Manitobans on Performance 2005 Discussion Document, and is meant for illustrative purposes only.

64

Closing Statements and Next Steps

• Workshop evaluation process

• First thoughts from the group on further capacity building?

65

ReferencesGovernment of Alberta 2005-06 Annual Report – Measuring Up.

[http://www.finance.gov.ab.ca/publications/measuring/measup06/index.html]

Manitoba Provincial Sustainability Report. Government of Manitoba [http://www.gov.mb.ca/conservation/sustainabilityreport/]

Oregon Progress Board: www.oregon.gov/DAS/OPB

Rita Conrad 2005. Results Oriented Strategic Planning: A Framework for Developing Effective Performance Measurement. Presentation to the Botswana Delegation, Salem, Oregon, September 29, 2005.

Rita Conrad 2005. Oregon’s Experience with Performance Reporting. AGA’s First National Performance Management Conference. Oregon Progress Board, November 14 2005.

Reporting to Manitoba on Performance: 2005 Discussion Report. [http://www.gov.mb.ca/finance/mbperformance/index.html]

http://www.oregon.gov/DAS/OPB

Date post:	17-Dec-2015
Category:	Documents
Upload:	theresa-hensley
View:	213 times
Download:	0 times

1 Workshop: Development and Use of Outcome-based Measures in Government Planning & Reporting...

Documents