A Methodology for Evaluating Ex Ante the Regional Benefits of R&D Projects

Stephen Roper, Nola Hewitt-Dundas and James H Love

Aston Business School, Birminghams.roper@aston.ac.uk

Why is ex ante evaluation of R&D projects particularly difficult?

• Uncertain outcomes or success, i.e. risk• Uncertain timing of outcomes or benefit

stream• Difficulty of assessing chance of technology

developed being exploited locally by the R&D performer

• And, crucially, difficulty of assessing the extent of local spillovers through:– technology or knowledge transfer– supply chains

– development of research trained labour

The Task

• Key questions:– What is best practice in ex ante evaluation of

publicly funded R&D projects?– Can we create a framework/tool to enable us to

better evaluate the returns to R&D projects? • Elements of study

– Discussions with agencies in Scotland, Wales and DTI

– International consultation – Finland, Israel, US– Extensive review of evaluation/academic

literature– Development of tools for Invest NI – Calibration and revision of tools in seven case

studies of existing R&D Centres

Best Practice?

• Best practice/ international/ lit survey - no existing systems for predicting returns (or ex ante evaluation) – No ‘silver bullet’

• Instead we found different strategies:– Avoidance of issue (Israel)– Peer review of projects (US, Finland, Scotland)– Strategic rather than economic evaluation (DTI,

Wales)

• But: evaluation studies do provide rich knowledge base of ex post effects of different types of R&D projects.

Key Questions

• So .. can we use existing knowledge base to draw inferences ex ante about the likely regional benefits of different types of R&D project.

• RQ1: What (global) benefits stem from the establishment of a new R&D centre or facility?

• RQ2: What determines the share of these benefits which accrue to the host region?

• How do we codify the answers into a ‘tool’ usable by a regional development agency?

RQ1: What global benefits stem from the establishment of a new R&D centre or facility?

- An Inventory Approach

A Private Benefits 1. Increments to Knowledge 1.1 Basic R&D Results 1.2 Applied R&D results 1.3 Developmental R&D results 1.4 Other research results 2. Benefits to future research and research use 2.1 Better targeting of future research 2.2 Staff development and education 2.3 Better research management skills 2.4 Increased capacity to use existing research findings 2.3 Reputational and halo effects 3. Private Benefits from Commercial Application 3.1 Cost reduction in existing products/services 3.2 New or improved products or services 3.3 Process or organisational improvements 3.4 Reputational or strategic benefits 3.5 Revenues gained from Intellectual Property Rights

RQ1: What global benefits stem from the establishment of a new R&D centre or facility?

- An Inventory Approach

7. Wider Social or Public Benefits – Market Transactions (Rent spillovers) 7.1 Benefits of cost reductions in other organisations 7.2 Benefits of new improved products or services to other

organisations 7.3 Availability of pool of trained staff etc. 7.4 Partnership or network gains 7.5 Fiscal benefits of research and other activity 8. Wider Social or Public Benefits - (Pure) Knowledge Spillovers 8.1 Intentional knowledge spillovers 8.2 Unintentional knowledge spillovers 8.3 Spin-outs etc 8.4 Reputational, image or halo effects 8.5 Demonstration effects 8.6 Agglomeration or informational advantages

RQ2: What determines the share of these benefits which accrue to the host

region?

• The Profile of the R&D Centre or Project– Type of R&D– Institutional or Organisational Setting– Applicability – Business Ownership

• Embeddedness of the R&D centre– Parallels with high-tech plants?

– Effects of region size • Absorptive capacity of the RIS

– Industrial composition– Absorptive capability of local firms – Public and private knowledge mediating institutions – Networks

Key Conclusions from Research Phase

• Ex ante impact assessment needs– benefits inventory– knowledge base from prior evaluations/studies– profile/landscape/synergy consideration

• The bottom line?– Private benefits depend crucially on type of

business– Rent based spillovers depend on embeddedness

(labour may be stronger than product market)– Pure knowledge spillovers depend on

synergy/RIS

Tool Building and Validation

• Spreadsheet based scoring tool for agency staff

• Developed using inventory x regional share approach

• ‘Validated’ using 7 existing Centres and comparison of outcomes to ex ante predictions

• Tool developed can – at best- give a broad indication of expected effects

• So, use alongside a narrative template developed to reflect same structure

End Notes

• As yet no way of making reliable ex ante predictions of social rate of return from individual R&D projects

• But we can use our framework to identify a comprehensive list of potential benefits and a general idea of their likely importance to host region.

• Individual case-specific factors will still be important, however.

Spreadsheet ModelStep 1: Global benefits of R&D

Project

5

3

1

1

4

5

4

5

5

0

0

0

3

5

0 1 2 3 4 5

1. Increments to Knowledge

1.1 Basic R&D Results

1.2 Applied R&D results

1.3 Developmental R&D results

1.4 Other research results

2. Benefits to future research

2.1 Targeting of future research

2.2 Staff development

2.3 Better research mment.

2.2 Increased learning capacity

2.3 Reputational and halo effects

3. From Commercial Application

3.1 Cost reduction

3.2 New or improved products

3.3 Process change

3.4 Reputation/strategic benefits

3.5 Revenues gained from IPR

ExpeimentalDevelopment

6

AppliedResearch

4 5

BasicResearch

1 2 3

University Collaborative BusinessOnly Only

Exhibit 1: Position of R&D Centre in Typology

Spreadsheet ModelStep 2: Regional Share of Benefits

GenericTechnologies 7 8 9

Sectoral Technologies 4 5 6

SpecificTechnologies 1 2 3

Locally GB Multi-national Owned Owned Group

Exhibit 4: Type of Exploiting Business and Technology

0

20

40

60

80

100

private wider total

Narrative Template – Private Benefits

Narrative Template – Wider Benefits