34
DIAC Session 3, November 18 2010 Using evidence & Managing stakeholders Professor Adam Graycar Centre for Policy Innovation
| Date post: | 30-Dec-2015 |
| Category: |
Documents |
| Upload: | ralph-walters |
| View: | 215 times |
| Download: | 0 times |
DIAC Session 3, November 18 2010Using evidence & Managing stakeholders
Professor Adam Graycar
Centre for Policy Innovation
Evidence used
• To assess the nature and extent of the problem
• To assist in reforming policy• To assist in developing new policy
Policy happens faster if the knowledge base is good and if lessons from previous activities can be brought to bear
3
Evidence for
• What is the situation/ problem
• What should be done
• Is it working? Evaluating effectiveness
• What might go wrong? / unintended consequences
4
Evidence
is information that affects the existing beliefs of key people about the problem and how it might be solved
Information
comprises data that has “meaning” - can help sort things into logical or empirical categories
Data
are facts about the world - stats, observations
5
Data measures or describes
Trends
Preferences
Finance
Performance
Impacts
Evaluation
Benchmarks
Forecasts
etc etc
6
Where do you find data?
ABS
Treasury
AIHW
ABARE
AIC
Budget papers
Annual reports/ performance reports
CIA
OECD/ UNDP/ ISSA/ WHO
World Bank / IMF
Cochrane/ Campbell
etc etc etc etc
7
Collecting data
Surveys
Observations
Interviews & Focus groups
Modelling
Scenario development
etc etc
8
• Think before you collect• Review the available literature• Survey best practice• Use analogies• Talk to opponents• Construct alternatives• Start comprehensive, end up focussed
(Bardach)
9
Use existing research - commission new research?
Qualitative/ Quantitative
Cross-sectional/ Longitudinal
Prevalence /Incidence
When research contradicts itself?
10
Key evidence resources
• Research - both domestic & international• Statistics - both domestic & international• Policy evaluations• Expert knowledge• Economic modelling & forecasting
Cochrane Campbell
A HIERARCHY OF EVIDENCE OF IMPACT(UK CABINET OFFICE)
1. Systematic review – synthesis of results from various studies (preferably randomised trials)
2. Randomised policy trials – population allocated randomly to groups
3. Quasi-experimental study – similar populations compared
4. Pre-post study – results compared before and after intervention
(Higher ones more reliable; lower ones much more frequently used in Australia)
12
When evidence is equivocal?
• The political process sorts itself out• The analyst shapes the alternatives
13
Evaluation
3 questions
1. Any good?
2. Better than what we had before
3. Worth it?
15
EVALUATION TYPES
Process evaluation
Was the program delivered to the intended people? Were they satisfied? Was the administration efficient? Were the costs acceptable?
Impact evaluation
Was the program effective? Did it achieve the intended outcomes such as poverty alleviation, savings, spread of lifetime earnings? Did it have unintended effects such as on employment, earnings, savings?
PROCESS EVALUATION
It may include • whether the right people got the right
money/services and that no others did; • that the services were provided in a timely and
convenient way• that people considered the programs were
beneficial, helping when needed, being reliable etc
• that the administrative costs were low, and total costs in line with estimates
IMPACT EVALUATION
First round impact• Static picture of impact on clients circumstances (eg
income distribution, immediate education/training access, employment)
• May include surveys of the circumstances of beneficiaries (lifestyle, location, housing, health etc)
Second round impact• Dynamic picture of people moving in and out of
education, work and welfare, longer-term impact; time series
• Post-hoc examination of effects of incentives to work, to gain skills, to save; surveys of behaviour
18
IMPACT EVALUATION
Third round review
• What is the counter-factual? What might have happened anyway, or under different scenarios and policies?
• Randomised trials, dynamic modeling
20
Commissions of Inquiry• Considerable independence from
government, reports published, public consultations; may have secretariat of seconded officials to do research etc
• Government terms of reference and appointment of Inquiry head or team
21
Task Forces• Often a mix of government and external
experts• May have a ‘steering committee’ of senior
government officials• Limited independence, limited public
consultations, though task force signs off report which is usually published
22
Reviews• May involve external reviewer, or advisory
group, but usually closely controlled by government
• Reports usually published
23
examples
Inquiries• Poverty Inquiry (early 1970s)• National Superannuation Inquiry (mid 1970s)• National Compensation Inquiry (mid 1970s)• Royal Commission on Australian Government
Administration (mid 1970s)• National Health and Hospitals Reform
Commission (2007-08)
24
examples
Task Forces• Reference Group on Welfare Reform (2000)• Task Force on the Delivery of Health and Aged Care
Services (2004-05)• Review of the Child Support Scheme (2005)Reviews• Income Security Review (1970s)• Social Security Review (1980s)• Review of the Tax-Transfer System (2008-09)• Review of Australian Gov’t Administration (2009-10)
25
Who are the stakeholders? Interests?
• Clients/customers• Those subject to regulation• Provider organisations• States and local government• Advocates• Broader public
26
Stakeholders/ interests?Clients/customers
• Quality and effectiveness of services• Levels of subsidies• Limiting their obligations
Those subject to regulation• Minimum red tape, clarity, consistency• Minimum interference in their business, impact on profitability
Provider organisations• Minimum red tape, policy continuity• Long-term contracts/agreements• For-profits focus on profitability; not-for-profits more on
clients/customers
27
Stakeholders/ interests?
States and local government• Recognition of their role and expertise,
sovereignty• Maximising revenues: central agencies
wanting budget flexibility; line agencies liking conditions that guarantee they get the money
Broader public• Limited taxes• Maximum effectiveness
28
Stakeholder processes
• Consultation• Collaboration / building bridges• Co-production• Problem identification, evaluation, • Development of policy options• Post-decision processes /
implementation
29
Examples of stakeholder relations
Consultation• Includes feedback from complaints systems, service
charters, ministerial correspondence• As well as special forums, discussion paper
responses etc
Collaboration• Joint evaluations, reviews• Program arrangements allowing providers and/or
clients discretion within agreed objectives/guidelines
30
Examples cont’d
Co-production• Program requires contribution – time, effort,
not just money - from clients to be effective• Similarly, policy cannot be determined without
substantial client involvement, including evaluation
31
Processes of engagementEvaluation, problem identification, analysis
• Ongoing consultation with client and provider organisations
• Systemic feedback from program administration, including through use of new technology
• Public inquiries (range of options)• Joint reviews
Development of policy options• Informed judgments of likely views• Market research• Targeted consultations, often confidential• Use of internet communications with public, or
selected external groups• Green Papers, other public documents
32
Processes of engagement (cont)Post-decision processes
• Targeted consultations or negotiations • Targeted communications
Post-announcement, implementation • Consultations/negotiations on details• Implementation of communications strategy• Managing responses to parliamentary discussions of
legislation etc• Establishment of ongoing program monitoring with
33
Issues in stakeholder engagement• Understanding interests and conflicts of interests• Managing confidentiality• Ensuring accountability when responsibility is
shared• Respective roles of ministers, advisers and
public service• Legitimate and improper communications
management• Risks of delays, compromises not in the public
interest etc vs achieving worthwhile, sustainable results
34
Stakeholder management
High
Importance- How influential- How affected
Low
Involve Manage
Acknowledge Monitor
High Low
Support
