Post on 12-Jan-2016
transcript
‘Pathfinder’: Maximising Results for Communities
Dr Roger Waite (Pathfinder Project / NZ Treasury)
io.ssc.govt.nz/pathfinder
Meta-messages
• All NZ Depts: enhanced outcome focus by 2003• Outcomes: ultimate goal (Holy Grail?) • Pathfinder (collective action, individual solutions)
• Outcomes driving strategy and output mix• Evidence-based approach to enhancing outcomes /
achieving Millenium Development Goals (MDGs)• Maximising outcomes from development proposals
(prioritise, assess & assist roles for development banks?)
An Incremental Approach to Capability Building
Focus on Outcomes
PATHFINDER
Strategic Plan;Statementof Intent
Value-for-Money
TOOLBOX; DECISIONMAKING SYSTEMS
STRATEGY,CAPABILITY,
PERFORMANCE
TRADE-OFFS,TRANSFERS(Within Vote)
(Next FY)
Goal: Build NZ Capability at Agency, Sector and ‘Joined Up’ Levels
Why Pathfinder?
After 10+ yrs, output funding has delivered
Quantum gains requires modified approach
Keep what works (output mgmt), but …
Complete public sector management modelLink inputs (through outputs) to outcomes
Meet requirements of legislation, parliament, etc
Ensure we deliver to stakeholders, taxpayers
Start: Limited capability & management tools
8 Agencies, Supported by CentreBuilding capability to enhance outcomesGoal: Better results for NZ communitiesAssumes performance feedback (outcomes) is a key factor in further improving performanceDefine outcomes; Prioritise; Measure; Manage– Performance of whole agencies / business units– Maximise outcomes from the output mix / $– Accentuate the positive; eliminate the negative
Outcomes / MDGs core to strategy, prioritising funding (macro) and proposals, assessing impact
Is the approach helpful in implementing Milennium Development Goals?
PATHFINDERBetter Results, Stronger Communities
Three Basic Approachesto Improving Results
• Top Down (Pathfinder emphasis / SOI / MDG focus?)
• Hierarchical (evolutionary direction for Pathfinder)
• Bottom Up, practitioner driven (opportunity)
SOI: Statement of Intent – a forward looking strategic plan deriving outputs from outcomes, and identifying capability
needs
Pathfinder ‘Building Blocks’1. Define & measure ‘mission critical’ outcomes
(using ‘state’ or ‘situation’ outcome indicators)
2. Map causal logic linking outcomes to outputs3. Assess impact of interventions4. Assess cost-effectiveness 5. Management to maximise outcomes6. Benchmarking with outcomes (business units / nations)
7. Focus strategic / annual plans on improving o/c8. Redesign planning & operations to maximise o/c
(incl. feedback & continuous improvement)
9. Improving outcomes across agencies / sectors
Building Blocks being documented on io.ssc.govt.nz/pathfinder
First Steps towards Real Results Mission, Measures, Management (M3)
• From aspirational outcomes (‘safer communities’)
• To the ‘Vital Few’ – focus on business actions• Tight Outcome Definitions (‘% re-offending in 24 months’)
• Outcome Indicators (‘28% re-imprisonment rate for Maori’)
• Impact Measures (‘12% reduction in reimprisonment rate’)
• Outcomes as Inputs into Management Systems
Delivery-based measures can substitute for Impact Measures where local factors can be managed (e.g. drugs work, but delivery factors critical)
Levers for Improving Outcomes
Status quo
outcome (client /service groups)
Risk, Need &
Likely Change (individuals / cases)
Improvement
in outcome (‘treated’ groups)
Situation Report
Benchmarking
Crude targeting
Priority setting
Demand analysis
Risk managemnt
Continue?
Modify? Grow?
Kill?
Measure Applications
ImpactAssessment
Assessment /AllocationDecisions
State /Environment
Indicators
Outcome Type
Screening for Risk: ValidationSocial Costs (Justice + Community)
Risk of Imprisonment (decile)
Societal
Cost
Top (Squares): Community-based Offenders
Bottom (Diamonds): Prison Inmates
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
0 0.2 0.4 0.6 0.8 1
Results Chains (After OAG Canada)Simple tools for reviewing funding proposals?
Inputs
Activities
Outputs
Immediate Results
Intermediate Outcomes
End/Final Outcomes
Outputs
Activities
Inputs
End/Final Outcomes
Intermediate Outcomes
Immediate Results
O2
A2
I2
O3
A3
I3
O4
A4
I4
IO
IR
Etc
Etc
Etc
Technical Efficiency Allocative Efficiency
Core Outcomesof AgencyDefined
Agency OutcomeIndicators
ImpactMeasures
Risk-basedTargeting
Tools
StrategicPriorities &
Planning
Benchmarking& Best Practice
MaximisingBenefits from
Intervention Mix
BusinessProcess Design
ManagementApplications
Linked OutcomeMeasures
After a business model used by the Department of Corrections
MANAGEMENT SYSTEMS & CAPABILITY
Identify CoreOutcomes
Define in Measurable
Terms
Measure as Outcome Indicators
Identify AreasFor Change
Define / RefineIntervention
Logic
IdentifyIntervention
Options
PrioritiseIntervention
Options
Design & Deliver
Interventions
ImpactMeasurementFramework
Cost Effectiveness
AnalysisOptimising Performance
Strategy Formulation
Ex Ante
Focus on Systems, Not Measures(lessons from the NZ Experience)
• Build management, not o/c measurement, systems• End game is outcomes – but output results important• Conceptual approaches give ‘one-off’ gains …
• Significant technical challenge to measure impact (management stuff is easy (?) with good measures)
• Ongoing gains in outcomes needs impact feedback?• Risk-based targeting an important lever for gains
• Stretching the capability of first world nation(s) • Capability development for 3rd world required?• Outcome-based allocative tools for the funder
‘Pathfinder’: Maximising Results for Communities
Dr Roger Waite (Pathfinder Project / NZ Treasury)
io.ssc.govt.nz/pathfinder
Success is more a function of consistent Success is more a function of consistent common sense than it is of genius -common sense than it is of genius -
Success is not a place at which one Success is not a place at which one arrivesarrives,, but the spirit with which one but the spirit with which one undertakes and continues the journeyundertakes and continues the journey
Most advanced NZ agencies have several things in common: Most advanced NZ agencies have several things in common: Committed senior management; Long-term focus; Clear Committed senior management; Long-term focus; Clear
outcomes; Strong analytical capability & MIS in support; outcomes; Strong analytical capability & MIS in support; Incremental development to vision rather than ‘big bang’Incremental development to vision rather than ‘big bang’
Will require cultural & capability changes of all managers; Will require cultural & capability changes of all managers; Outcomes subversive – managers will not like all answers; Outcomes subversive – managers will not like all answers;
No quick fixes – requires effort over multiple yearsNo quick fixes – requires effort over multiple years
Office of the
Controller and Auditor-GeneralTe Mana Arotake
Robust Impact Measures: Challenging
Structured Trial
Clear, EnforcedInterventionCriteria andThresholds
RetrospectiveDesign of
Structured Trial
RandomisedDesign
Non-randomisedDesign
RegressionDiscontinuity
Impact typicallymeasured in absolute
terms. Substantialcorrection for external
factors
Output / quality measures good surrogates where outcomes not strongly influenced by local factors (e.g. Drugs; Engineering)
Accountability: High Demands
Management (SOI)• Logical story of how
outputs enhance outcomes
• Selective use of indicator and impact measures to ‘tell story’ & set priorities
• Some discretion on external reporting
Accountability• Robust model linking
outputs to outcomes
• Precise, timely and comprehensive impact measures for each output
• Mandated external reporting of outcomes
In a ‘perfect world’ managers would have all the information required by an outcome-based accountability regime
Outputs and Outcomes
After: HM Treasury (UK)
Value for Money
Better healthMore treatmentsMore nursesMore money
External influences
Budget Inputs Output Outcome
Economy Efficiency Effectiveness
Which Outcome?
F ew er V ic tim s(Ju s tice )
P erce ived S a fe ty(Ju s tice )
R eso lved
Tim es from C h arg esF iled to P rosecu tion
(C ou rts )
C ou rt H ou rsb y C rim e Typ e
(C ou rts )
P rop ortion C on vic t ion sO vertu rn ed on A p p ea l
(C ou rts )
P rosecu ted C on vic ted
C rim es S o lved(P o lice )
[P ro po rtio n; T im eliness]
R ed u cedR e-o ffen d in g
(C orrec tion s , C Y F ,P o lice D ivers ion s )
L ess R ep orted C rim e(Ju s tice / P o lice )
CO M M UNITY S AFE TY(F rom C rim e)
Example
Rehabilitation QuotientsDetermined on NZ Offender
Groups / Population
BEST USE OF REHABILITATIVE FUNDING
Full Cost (Per Head) ofIntervention as Delivered in
New Zealand
Mean Future Cost perNZ Offender within thePrimary Target Group
Risk(s) of Reconviction
Cost, if Reconvicted
Cost-Benefit Ratio(Individual Interventions)
Rank Individual Interventionson the Basis of Cost-Benefit
Determine Budget for 'Core'Rehabilitative Interventions
Determine Mix of CoreProgrammes to be Delivered
in the Next Year
Determine Mix of CoreProgrammes to be Delivered
in the Next Year
Input Government andDepartmental Priorities
RI / RQ Framework
[A] CORE PROGRAMMESAND INTERVENTIONS
[B] PILOT / EVALUATIONOF NEW INTERVENTIONS
Cost and Benefit Datafrom Successful Pilots
Cost and Benefit Datafrom Successful Pilots
Managing for Better Outcomes• Broad focus (strategy, priorities, output mix, business systems)
• Decision making, better outcomes (evaluation just a tool)
• Measure outcomes (different outcome measures & groups)
• Work with (& test) clear intervention logic• Integrate with resource allocation systems• Focus business strategy on improving outcomes• Focus business processes on improving outcomes• Improve outcomes over multiple business cycles, and
across business units / agencies / sectors
Goal: maximise social benefit from finite resources