Using Network Analysis to Assess

Systemic Impacts of Research

12th November 2009

AEA Annual Conference, Orlando, USA

Frank Cunningham

EvaluationDG INFSOEuropean Commission

Main Points

- EU Framework Programme Research

- The needs for new appraoches to evaluation of Research

- Evaluative Studies

- Main Findings

- March (Brainstorming) Workshop held in Brussels

- Recommendations & Next Steps

EU Framework Programmes evolutionComplementary instruments and wider mobilisation of

resources

ACTS

ESPRIT

TELEMATICS

FP5IST

FP4 FP5 FP6… FP7

FP6IST

ICT in FP7

CIP

JTIs

Art. 169

Modinis

eContent

eTEN

ETPs ETPs

eContent

eTEN

9 Technology Platforms in ICT

ARTEMIS (Embedded Systems)

Ambient-Assisted Living

ICT Policy Support Programme

ENIAC (Nano-Electronics)

3,984 m Euros + 216 m Euros

9,050 m Euros

(2003-2006) (1999-2002)(1994-1998) (1999-2002)

3,625 m Euros + 200 m Euros

EU Framework Programme 7 and CIP

• Total budget of €50.5B over 7 years (2007-13)

• Information and Communication Technologies (ICT) €9.11B

• Average annual EU investment in ICT-RTD of over €1.25B

• EU investment about 3% of total ICT-RTD in Europe

• Competitiveness and Innovation Framework Programme (CIP) - €3.625 over 7 years (2007-13) ICT-PSP ~ €700M

Making the best possible use of the evaluation findings

Current Evaluation and Impact Assessment Methodologies

• Commission has considerable experience in evaluation and monitoring of effectiveness of EU RTD investments over the last 15 years.

• Focused mainly on ‘outputs’ rather than impacts.

• Focused mainly on ‘project’ level and on the ‘private-return’ to RTD participants.

• Now increased attention is given to results-based assessment and macro-economic and social impacts of EU RTD.

New Approaches to Evaluation of Research

• Aho Report; present evaluation techniques tend to undervalue the contribution of R&D to innovation and wealth creation.

• New Evaluation methodologies and tools are required to identify the systemic effects on increased R&D spending

• A higher quality evidence base and more focus on ‘systemic’ effects on research-innovation-competitiveness links

Knowledge is Networked

• Innovation requires combining scientific, design, engineering and operational knowledge from different sources

• An individual or organization rarely has all the knowledge needed for the whole process of innovation

• The types of skills and knowledge of research activity is unknown at the start

• The most efficient way to create knowledge is enable researchers themselves to identify what is needed, to find each other, link up, and share resources

• The knowledge-creating process can be facilitated by understanding how this happens

Innovation Systems and Research

• Innovative firms do not innovate in isolation but in interaction with other ‘actors’

• In ‘Innovation Systems’ the links between innovation actors are crucial

• Collaborative R&D and Deployment Programmes are one important way of strengthening linkages

• Partnerships through collaboration networks helps to achieve ‘critical mass’ of dispersed EU R&D and Deployment resources

From Linear to Open/Systemic InnovationFrom linear innovation

IST-FPs eTEN Modinis

RTD

Deployment Progress Measurement

To Open/Systemic innovation on Value Chains - CIP

Demonstrated European Value Added. But increased potentials from further integration)

Best Use

Pro

gres

s

Rep

ortIST-FPs

RTD + =ICT PSP

Responding to the i2010 Challenge

• Minimising risks of uptake (esp. for SMEs)

• Pooling resources and technical requirements (e.g. for information space – interoperability)

• Fostering inclusion and Sustainability

• Creating “similarity”/”replicability” effects in Member States

eGov.nt eContent

Deployment

Series of Evaluative Studies

• DG INFSO Evaluation Unit commissioned a series of evaluative studies to help us develop our understanding of:

• The structuring effects of FP ICT networks (RAND Europe – 2004/5)

• Their international reach (CESPRI – Bocconi Univ. - 2005)

• The linkages between EU research and deployment and regional innovation systems (CESPRI - 2006)

RTD-Innovation Linkage in EuropeObstacles

Our evaluative analysis revealed two structural obstacles:

- The low participation of the most innovative SMEs in the EU FP research

- Poor linkage between the new knowledge and those able to exploit it at the regional level

..and that

- While FP ICT RTD is highly networked , most regional and deployment innovation networks are not

RTD-Innovation Linkage in EuropeOvercoming Obstacles

- Regional innovation networks are considerably strengthened by links to EU ICT research networks.

- The most important hubs in the regional deployment and innovation are private companies, notably SMES

- A key linkage is between SMEs at regional level and Global ICT Knowledge Hubs

Potential Leverages

• ~ €54B on EU RTD and Deployment

• ERDF Funding ~€320B (2007-13) – 20% (~€65B) spend on innovation

actions

• Innovation in Public Procurement– ~ 16% of European GDP - ~ €2Trillion

Network Analysis Brainstorming Workshop

Brussels on 5th March 2009

Workshop Objectives

a) To Formulate of next steps in the development and deepening our understanding of the linkages between RTD & deployment programmes and innovation systems…

..through evaluative studies based on network analysis

b) To make recommendations on how to make further contributions to capacity building in this area of evaluative analysis

Questions/Issues 1

- What is the relationship between research networks and the overall innovation diffusion and deployment system?

- What is the interaction between the various types of networks? (connections between network layers)

- Research Networks- Deployment Networks- Policy Networks- Regional/Innovation Networks

Questions/Issues 2

- How can we develop a better understanding both of linkages and the nature of ‘hubs’ (as gatekeepers, tech transfer actors, regional administration etc)?

- Can ‘gatekeeper hubs’ become over-dominant?

- How can the links between EU research & deployment and systems of innovation be strengthened at EU, MS and Regional Levels?

- Can we identify where and how the leverage of EU research on regional innovation initiatives can be strengthened?

Questions/Issues 3

Time Dimension– How do we best take into account that networks,

hubs and clusters are not ‘born’ fully formed but emerge over time?

..taking into account the structural evolution and the overall innovation network and the evolution over time of regional networks

Directionality of links

- Links are not static, - some are symmetric and bi-directional,biare mostly

bidirectional- In others knowledge flow are mainly in one

direction

How do we take this into consideration in our analysis?

Data Requirements

- Node (mainly participant) data

- Programme link data (project data, incl. possibly financial)

- External link data (patents, publications, institutional links)

March ’09 Workshop Recommendations

1) Policy Impact Simulation

2) Monitor Networking effects over time

3) Knowledge Production vs Diffusion

4) Development of existing Network Analysis Tools

Next Steps

• Assessment of trends and evolution of the EU ICT Research and Deployment Landscape (Study 2009)

• Significant part of the evidence base for interim evaluations of CIP and FP7 ICT Research

• Continued improvements in data acquisition and archiving

• Build-up of network analysis skills and tools ‘in-house’.

Further Information

Overview of Innovation Systems Evaluative Work

• http://ec.europa.eu/dgs/information_society/evaluation/studies/index_en.htm

Contact• Frank Cunningham,• Evaluation Unit,• DG INFSO

• Email: frank.cunningham@ec.europa.eu• Tel + 32-2-296-8119