Date post: | 18-Dec-2015 |
Category: |
Documents |
View: | 215 times |
Download: | 0 times |
1
Systems EvaluationAn European Perspective
Erik Arnold
www.technopolis-group.com
Vienna
25 April 2006
2
‘Systems’ has become an important concept in R&D policy and evaluation
• We rely on the ‘innovation systems’ heuristic in designing and implementing policy interventions
• We increasingly justify interventions in systems failure, as well as market failure, terms
• We need to integrate evidence about systems performance with intervention logics
• We increasingly pose policy questions in terms of portfolios - most recently ‘policy mix’ - rather than individual interventions
3
While ‘innovation system’ sounds good, as currently conceptualised it tells us little more than that everything is connected to everything else
The potential reachof public policies ...
Framework ConditionsFinancial environment; taxation andincentives; propensity to innovationand entrepreneurship; mobility ...Education andResearch SystemProfessionaleducation andtrainingHigher educationand researchPublic sectorresearchIndustrial SystemLarge companiesMature SMEsNew, technology-based firms
IntermediariesResearchinstitutesBrokersConsumers (final demand)Producers (intermediate demand)Demand
Banking,venture capitalIPR andinformationInnovation andbusiness supportStandards andnormsInfrastructure
PoliticalSystemGovernmentGovernanceRTD policiesSource: Erik Arnold and Stefan Kuhlmann
4
Whatever happened to systems theory?
• General systems theory (cp Wiener, Bertalanffy…) seems to have a great future behind it, with many of the questions it raised now being tackled within disciplines (Ingelstam)
• Recent testimony came in the form of an evaluation of VINNOVA’s Complex Technological Systems programme, which centred on the absence of a theoretical vector of systems understanding among applications domains
• The ‘Limits to Growth’ episode illustrated the dangers of disconnecting mathematics from understanding and evidence - but helped set an enormously important agenda
• Nonetheless, if we don’t attempt even a verbal description of how innovation systems hang together we• Won’t learn much about how to put together our understandings in a policy-useful way
• Can’t rescue evaluation from its essentially non-cumulative, non-scientific lock-in to trying to answer impossible questions
5
Research and innovation policies are starting to overlap and to become more systemic
Multiple
Single
MultipleSingle
Development measures
MAPs and network measures
Activity promotion or subsidy measures
Linkage or ‘bridging’ measures
Measures
Actors MultipleSingle
Intra-organisational learning, capability development and
performance improvement
System strengthening
- Within actors
- Between actors
- Reducing bottlenecks
Point or step change in organisational
performance
Inter-organisational learning, network development and
strengthening
Measures
Actors
Actors
6
Funding systems and governance are complex, but moving towards the Finnish model. Horizontal co-ordination, distributed strategic
intelligence and arenas are keys
R&D Institutes
Parliament
Government Policy council
Ministry of Education
Research Councils and Academies
Universities
Other Sectoral Ministries
Producers: Firms, farms,
hospitals, etc
Ministry of Industry
Technology & Innovation Agencies
Support Programme Agencies
Programme Contractors
Instructions, resourcesAdviceResultsHorizontal co-ordination and integration
Level 1High-level cross-cutting policy
Level 2Ministry mission-centred co-ordinationLevel 3Detailed policy development, co-ordinationLevel 4Research and innovation performers
Key
7
While we claim that systems failures are especially important. They can be hard to see if you only evaluate within programmes
Market failure - mostly about basic research
• Indivisibility
• Inappropriability
• Uncertainty
Systems failure - mostly about inadequate performance
• Capability failures
• Institutional failures
• Network failures (including lock-in and transitional failures)
• Framework failures
Funding rationales
8
Five Year Assessment is a complex ritual, not easy to connect to the Realpolitik of Framework Programme design
97 98 99 00 01 02 03 04 05
FP4
FP5
FP6
Scope of 5YA in 2004
Scope of 5YA in 2000
5YA
5YA
9
Evaluating the FPs has not been easy
• The size and complexity of the FPs mean that the familiar difficulties of evaluating RTD programmes are present in large measure • Assessing dead weight, time scales, choice of methods, inadequate models of
relationships between R&D and other social variables, etc etc etc
• Some issues appear particular to the FPs• Data access
• ASIF on 5YAs prior to 2002: “Many of the recommendations drew not so much on an evaluation of past Framework activities but on the collective opinions and assessments of the panel members concerning the general structure and organisation of science,, technology and innovation in the EU”
• Peer review extended, arguably, beyond its elastic limits in past 5YAs, but panels now being better supported by studies
10
Key problems are in planning, not evaluation. FP5 is a classic case of ‘the missing middle’. (FP6 is a bit better)
• Goals that forbid little, defined more in terms of process than results
• Unclear ‘vertical’ interrelationship among goals make the intervention logic (programme theory) hard to discern and evaluate
• Lack of clarity in goals about relationship with the context leaves scope for ‘killer assumptions’
• Scale and scope issues therefore not addressed in relation to objectives
• Interplay among activities, results and purposes not well understood
• Consistency between activities and goals managed by criteria, rather than by planning
11
More comprehensive planning, overcoming the ‘missing middle’ problem would provide …
• Higher-quality, testable logics leading to an improved probability of reaching policy goals
• Evidence- and logic-based arguments to underpin the size of the budget needed by the FPs, shifting the balance of negotiation towards rationality and increasing the chances that the resources available for EU R&D policy are about the same size as the resources actually needed
• Improved evaluability, with corresponding benefits for improved processes, organisational learning and accountability to the taxpayer
12
Evaluation needs to be a component in a better articulated system of strategic intelligence and planning
Analysis of system health
Meso-level ‘bottleneck analysis’ + thematic
evaluation
Evaluating programmes and portfolios
Policy develop
ment
Evaluation and Studies
Overall Objectives
Policy Purposes
Programme / Action Goals
Projects, Activities
A successful and scientifically strong European industry and high quality of life for citizens
Policy goals of FP SPs
Other policy goals, eg creating ERA
Expected results of SP 1
Expected results of SP2
Expected results of SP3, etc
Project 1.1.1
Project 1.1.2
Project 1.1.3
FP and other ERIS-related policies
13
Swedish energy research - a response to the 1973 oil crisis
14
Performers’ ratings of Swedish research capabilities
Participants' views on Swedish research capabilities (n=322)
1.0 2.0 3.0 4.0 5.0
Hydro power
Sustainable production of biofuels,including recycling of ash
Large scale heat production
Power conversion and distribution
Heating, ventilation and controlsystems
District heating
Waste fuels, including biogas
Combustion engines
Industrial processes
Policy / system studies
Solar power systems
Industrial support systems
Electrical drives
Components and support systems
Production of transport biofuels
Wind power
Very weakBelow the
InternationalAverage
At theInternational
Average
Above theInternational
Average
Worldleading
Hydrogen based energy systems
15
Performers’ ratings of Swedish industrial capabilities
Participants' views on Swedish industry's technology development capabilities (n=267)
1.0 2.0 3.0 4.0 5.0
Power conversion and distribution
Water power
Combustion engines
Sustainable production of biofuels, including recycling of ash
Large scale heat production
Heating, ventilation and control systems
District heating
Electrical drives
Industrial processes
Waste fuels, including biogas
Industrial support systems
Components and support systems
Production of transport biofuels
Solar power systems
Wind power
Hydrogen based energy systems
Very weakCompetitive in Swedish market only
Leading in the Nordic
market
Leading in the European
market
Leading in the world
market
16
Systems issues in Sweden
• Inadequate mental models of R, D&D in the political and policy systems
• Systemic weaknesses in the Swedish research performer system, which undermine the achievement of the significant energy, environment and social goals established in the legislation for the programme
• Interference between the political and research funding systems, leading to lock-in to an undesirable balance of activities in the programme
• Poor integration between the research funding and innovation systems, leading to an ineffective expenditure pattern
• Inability of the Swedish research and innovation governance system to provide adequate co-ordination
• Key obstacles built into framework conditions, which prevent the translation of policy goals into practice, even where the needed knowledge base has been established
17
Austria - institutional challenges 2002 … (wow!)
ParliamentGovernment
BMF
Committee for science, industry and economic affairs
BMWABMVITBMBWK
AWS, ERP
FFFBIT
FWFTIGASA
Policy
Agencies
PerformersKFI
National Bank
UniversitiesFirms ARCPolytechnicsTechnologyCentres
Competence platforms, incubators
Money from primary sourcesFunding: competitive moneyin colour of primary sourcePolicy advice
ÖAWLBGCD-Lab.
AnniversaryFundCDGVTÖACR
Main thematic programmesNANOFIT-IT Sustainable D.protec2002+fFORTEGENAU
Kind, KnetKplus-CentresAplusB
Umbrella organisation
LISAA3 IBS I2
Austrian Science Council
Council for Research and Technology DevelopmentParliamentGovernment
BMF
Committee for science, industry and economic affairs
BMWABMVITBMBWK
AWS, ERP
FFFBIT
FWFTIGASA
Policy
Agencies
PerformersKFI
National Bank
UniversitiesFirms ARCPolytechnicsTechnologyCentres
Competence platforms, incubators
Money from primary sourcesFunding: competitive moneyin colour of primary sourcePolicy advice
ÖAWLBGCD-Lab.
AnniversaryFundCDGVTÖACR
Main thematic programmesNANOFIT-IT Sustainable D.protec2002+fFORTEGENAU
Kind, KnetKplus-CentresAplusB
Umbrella organisation
LISAA3 IBS I2
Austrian Science Council
Council for Research and Technology Development
18
FFF conclusions … lock-in by the stakeholders
• A brilliant performer of its 1960s job, that has not evolved as fast as it should
• Failed to exploit its strategic intelligence mission
• Deficit financing problematic
• Brings substantial benefits to beneficiaries, but is too risk-averse
• Especially good for smaller firms
• Overtaken by developments elsewhere in the funding system
19
FWF conclusions … lock-in by the stakeholders
• A brilliant performer of its 1960s job, that has not evolved as fast as it should
• Niche player
• Failed to exploit its strategic intelligence mission
• Budget too small• Compared with the GUF
• Needs to pay overheads
• Strategic intelligence, internationalisation
• Potential new roles: themes, Pasteur’s Quadrant
• Fragmentation of instruments within a narrow role
20
The Realpolitische response (text from final report) to reforms already in train - with one of the evaluators thereafter accompanying the
reform process
• FFF should be merged into a broader innovation agency. The proposed merger with TIG, BIT and ASA appears to be a reasonable option for achieving this, although other configurations would also be possible
• The Funds should be transformed into agencies and the power of their beneficiaries in the governance structures should be limited
• We interpreted the Research Promotion Act of 1967 as a vote of ‘no confidence’ in the Austrian state’s ability to govern R&D agencies in a modern manner. To reverse that vote, the ministries and political level need to demonstrate that they can• Manage by objectives and properly delegate authority to agencies, without seeking to interfere
in daily operations such as project assessment. This should include delegation of programme design as well as management
• Maintain the ‘strategic intelligence' needed to do this • Professionalise leadership and personnel decisions in the agencies, so that appointments are
made in fair and open competition• Develop reasonably standardised ways of instructing agencies, so that ministries can use
different agencies to achieve different policy objectives
21
Systems issues in the RCN evaluation focused on governance and institutional structures that prevented the new organisation from
doing the ‘crucial experiment’ of having a single council
NAVF
RS
F
RM
F
RH
F
RN
F
NM
F
RCN
KS
BF
NT IEMU
MH
NT
NF
NL
VF
NF
FR
NO
RA
S
22
It also provided testimony to the power of personalities. RCN’s three ‘steering levels’ were a battlefield in the early days but fighting stopped
when the minister sacked the protagonists
Executive Board
Division Boards (6)
Programme & Discipline Boards
Director General
Division Directors (6)
Strategy, Admin
Divisional Staff
Government
Ministries
23
Does a PART-like approach with standardised reporting help?
• PART• Programme design and purpose (20%)
• Strategic planning (10%)
• Programme management (50%)
• Programme Results/Accountability (50%) - where measurable, which they frequently aren’t
• Looks very rational - so did GOSPLAN
• Provides neither systems insight nor the comparative RoI information the finance ministries think they want
• Gives policy makers no help with bottleneck analysis and systems improvement
24
And it rather passes by the operational purpose of evaluation
The right thing
The wrong thing
Done wellDone badly
Carry on
Stop
Fix it
Aaaaaargh!!
25
Evaluate at different levels. Don’t try to do everything at once. Needs increased strategic intelligence in and for funder organisations
Analysis of system health
Meso-level ‘bottleneck analysis’ and evaluation
Evaluating programmes and portfolios
Hypotheses about bottlenecks
Hypotheses about bottlenecks
Policy developme
nt
Evaluation results
Evaluation results
26
Evaluation has a political logic
• Forget a wholly rationalistic approach to systems evaluation• Politics matter - and the more of the system you evaluate, the more the politics matter
• People matter
• Windows of opportunity come and go - useful evaluation is time-bound
• ‘Administrative shaping’ of evaluation Terms of Reference is a fact of life • We all want to speak truth to power - the question is how much truth to speak
• The overriding criterion is not absolute truth (whatever that is) but what is likely to be useful
• The MSc student question - this is not the same as saying that evaluators should be compliant - that’s a suicide mission
• Evaluation is, finally, a contribution to a debate, nothing more • (People whose egos are too large to cope with that should do research)
• We would be better served by a more articulated IS theory