Stakeholder Involvement and the Choice of Science & Technology Policy Outcome Evaluation Methods

Presenter ： Shan-Shan LiSTPI, NARL

Presentation on 2009 AEA conference 2009.11.14

2

Outline

IntroductionMethodologyAnalysis Research FundingsConclusions

3

IntroductionIntroduction

4

The importance of stakeholder

• Stakeholder involvement plays an indispensable part in science & technology management.

• Stakeholder involvement can…– help to clarify the root causes of problems and to

develop effective strategies for addressing them during different phrases of the S&T management process

• Evaluation methods are the means for solving such problems or meeting stakeholders’ requirements.

5

Comparison of 10 globally used policy outcome evaluation methods

Sources: Summarized and organized from Fahrenkrog, Gustavo, Polt, Wolfgang, Rojo, Jaime, Tubke, Alexander, & Zinocker, Klaus. (eds.) 2002

6

Research questions

Stakeholders’ demand-side requirements

Stakeholders’ supply-side issues

which S&T policy outcome evaluation methods are most suited for measuring stakeholders’ interests and in assessing their impacts in each phase of publicly funded S&T programs?

which S&T policy outcome evaluation methods are most appropriate to use in measuring their interests and in assessing their impacts in each phrase of publicly funded S&T programs?

7

Stakeholders in S&T management

ProfitOrganization

S&T Funding Organization

Internal Environment

Competitors

Contractors

End user

Reviewers

Mass media

Government/decision makers

Consultants

Customers/community

Industrial strategic Partners

Research team

Administrative personnel

Researchers

External environment

The internal environment: which is the environment that is related to S&T management and assignment, used to describe individuals or a group of stakeholders that participate directly in the internal S&T environment, such as customers, contractors, etc.

The external environment: This is formed by species elements and is related to the S&T management including technology, law, economy, politics, physics, labor supply and demand, culture and environments, etc.

8

MethodologyMethodology

9

Methodology

funders A

funders B

S&T executive institutions A

cooperative institutions A

cooperative institutions C

cooperative institutions B

cooperative institutions D

the end-users utilizing the final products or services of S&T

research A

the end-users utilizing the final products or services of S&T

research C

the end-users utilizing the final products or services of S&T

research B

the end-users utilizing the final products or services of

S&T research D

Allocation of funding Science community’s research output, Outcomes and impact

Research fields’ collaboration societal impact (impact of research utilization)

the end-users utilizing the final products or services of

S&T research E

S&T executive institutions B

S&T executive institutions C

Sources: Academy of Finland (2006)

we will use the 4 phases explained in the Academy of Finland’s report to further derive the issues that the stakeholders are concerned with in order to draw a strategic plan for the future

10

The properties of the research funding impact phases

Phase Concerns Properties

Allocation of public funds

Can be seen from the point of view of a nation, sponsoring organization itself, applicants and impact

Understand the impact caused by the approval of the funding and the allocation ratio

Outputs and outcomes of S&T communities

The impact of the modality caused by scientific research

Number of research output and impact elementsNumber of research output provided by the research community to non-academic users for utilization and developmentThe impact of the research executor’s participation on the research program

Cooperation of research communities

Cooperative system between research organizations or between research organization and research users

Scientific community in the form of knowledge spillovers effectThe formation of international cooperation networkThe level of knowledge flows between research communities and research usersSimple calculation of cooperation agreementVisualization of network frameworkInvestigate formation of cooperative relations

Impact of research utilization

The impact of research results is often not formed by a single program, institution, enterprise etc. but rather, it’s the stakeholders that cause the impact, such as social, economic etc.

Evaluation on relevance of the users’ researchesTarget creation of the research result commercializationInvestigation on the change in users systemGreat impact caused by the evaluation research results

Sources: Academy of Finland (2006)

11

The 4 groups of stakeholders within S&T management

•Academic system•Industrial strategic partner

•Government•Investigator•Contractor•Competitor•Decision-maker

•Research team•Administrators•Consultants•Profit Organization (e.g. insurance)

•End users•Customers•Academic staff•Community

Sponsors

S&T R&D management environment

S&T R&D implementation

system+

Cooperative network system

S&T R&D result users

12

Stakeholders’ supply-side and demand-side

PhasesStakeholders Main concerns

Supply-side Demand-side Supply-side Demand-side

Allocation of public funds

FundersS&T research and development implementation system

S&T research and development management processS&T research and development planning implementation detailReasonability of the funds allocationComparison between actual expenditure and budgeted amountExternality problemAcademic research institute financial enhancementSupport and promote high-quality research

Investigation of system fairness and transparencyCommunication process

Outputs and outcomes of S&T

communities

S&T executive institutions

Cooperation network systemCustomers/ community/ end-users

Resource allocationExternal network connectionImplementation process efficiencySchedule arrangement and planningFuture expected outcomeOutput produced meet expectation

Cooperation directionUtilization of research results

Academic staffS&T research and development energy development

Cooperation of research

communities

Cooperation network system

Academic staff/ end-users

Cooperation direction and systemThe role of the systemThe level of relevance between industry, government, academic and research

Energy upgrade

Impact of research utilization

Cooperation network system Academic staff/end users

Community

Increase productionEnhance economic growthCreate legal environmentLevel of result commercializationThe impact that result commercialization has on the industrySpillover effectProducts sold at a profit and impact on employmentBenefits of technologyOutstanding innovation procedures provide feedback to the S&T research and development result

Research utilization

13

AnalysisAnalysis

14

• Through the analysis of the two dimensional matrix of ：– stakeholders’ interests

– evaluation methods

• Comparisons (demand-side and supply-side) are shown in p.16, p.17.

Analysis

15

Direction of discussion : example

Phases Sides Meanings Methodspublic fund allocation

supply-side Funders (including governments, decision makers, contractors, competitors, etc.) are the main stakeholders who provide S&T related funds for some publicly funded executors (including S&T institutions and end-users). Under the circumstance, they have own selective criteria on who is more appropriate institution performing publicly funded S&T programs, including program management efficiency, appropriateness and relevance of programs, rationality of funds allocation, whether the funds additionally improve S&T institutions’ financial situation or not, or whether the funds support and promote excellent research.

Take expert review, a qualitative method

public fund allocation

demand-side S&T institutions (including team members, administrators, program officers/managers, consultants, etc.) are the main stakeholders who want to apply for publicly funds. Their concerns include the efficiency of communication between managers and members, and the transparency and fairness of the auditing mechanism.

Take expert review as an example also to discuss these issues.

16

Comparisons between stakeholders, and evaluation methods (Demand-side)

Issue form Allocation of public funds Outputs and outcomes of S&T communitiesCooperation of

research communities

Impact of research

utilization

Stakeholders S&T research and development implementation system

Cooperation network system

Academic staff Research result users Community

Concerning issues Investigation of system fairness and

transparency

Communication process

Cooperation direction

S&T research and development

energy development

Research utilization/

safety/ function/ quality

Energy upgrade Research development utilization/

safety/ function/ quality

Evaluation methods

Macroeconomic Modeling

& Simulation*

Micro-econometric Models

* * *

Cost-benefit Analysis * * *

Case Studies * * * * * *

Benchmarking * *

Expert Review * * * * *

Control Group * * *

Innovation Survey * * *

Network Review * * * *

Foresight/Technology Assessment

* *

17

Comparisons between stakeholders, and evaluation methods (Supply-side)

18

Research findingsResearch findings

19

Macroeconomic modeling and simulation

demand sidedemand sidesupply sidesupply side

The funders’ concern about the externalities problem created by research resources (e.g. knowledge spillover effect or industrial innovation spillover effect etc.)

The community concern about impact of R&D input on the economy (e.g. economic output and employment creation etc.)

It can be applied together to measure policy’s direct outcome and between research fields, inter-industries spillover effects’ direct outcome.

20

Micro econometric model

This method can directly measure the outcome of public policies towards company or individual project.

supply sidesupply side demand sidedemand side

In order to answer questions about rationality of fund allocation and external outcome etc. and also help S&T research and development implementation system into comparing similar projects implementation results, for S&T research and development results users, the model can evaluate researches for instance R&D activities’ achievements (e.g. relationship between R&D budget and company’s ability to earn profit) or spillover effect of individual project’s R&D activity towards industries etc

As for the community, evaluation of the effect caused by government’s science and technology policy measures can be done using the method (e.g. implementation and authentication results of scientific projects)

21

Cost benefit analysis

It calculates the socio-economic impacts when the implementation plan is carried out, and through the cost benefit analysis result, suitable project, case or policy can be selected. It helps the manager to think about the cost and benefit in a more structural and systematic way.

supply sidesupply side demand sidedemand side

It is suitable in measuring the appropriateness of the resources and funds allocation and the efficiency and the outcome of the project implementation

It can evaluate academic staff’s S&T research and development energy development, research output quality and also S&T research and development result users and the community in the level of research output utilization

22

Expert panels/ Peer review

They are able to not only carry out ex-post measurement, but also ex-anti evaluation.

supply sidesupply side demand sidedemand side

experts can use the rationality of the expected budget allocation to further analyze the ex-anti evaluation section. It can play an important part in terms of S&T research and development implementation system’s investigation fairness & transparency, communication process and the utilization of the energy development.

with research result development and utilization, they can also provide credible and effective suggestions.

23

Control group approaches

supply sidesupply side demand sidedemand side

This method is mainly under the control of other variables, explore the effect single manipulated variable has on different samples; the main manipulated variables are sponsorship funds and applicants.

It is appropriate to apply control group approaches on measuring and evaluating private enterprises economic activities for example: the impact of governmental department’s S&T research and development grants on private enterprises R&D, the impact of government fiscal measures (lowering taxes), and the effectiveness of loaning between emerging companies and banks etc.

It is able to help provide analytical information on the demand-side’s fund allocation issues and also additionality effect, such that academic staff’s energy development can be further analyzed.

24

Innovation surveys

supply sidesupply side demand sidedemand side

It mainly focus on the evaluation of the corporations’ innovation activities.

Funders can monitor whether policy tools can help to achieve product innovation target or not through funds grant formula or rationality of funds allocation

Stakeholders will be very much concern about economic and industrial growth; level of commercialization and product sales due to innovation etc

25

Network analysis

supply sidesupply side demand sidedemand side

It is mainly a relationship structure between each node and also the behavioral change result at each node through the end of the decision-making process.

Cooperation network system’s stakeholders have always been the target of the network analyze explore. As a result, necessary information can be obtained through network analysis; also it can explore the condition of knowledge distribution

It can fully perform its function when looking at the fund allocation communication process and network system cooperation direction on the demand-side. It explores the interaction and state between each node; therefore the academic staff’s S&T research and development energy development and users’ energy enhancement on the demand-side can also be

analyzed and explored

26

Foresight/ technology assessment

supply sidesupply side demand sidedemand side

The transfer of foresight and technology onto the evaluation is mainly a comprehensive outline of the whole fund allocation direction and S&T research and development project content.

The stakeholders on the supply-side will care most about the S&T research and development planned implementation procedures and rationality of the allocation of the public funds

It would be more interested in knowing the safety of the research and development result and whether or not future demand is met etc

27

Field/ Case study

supply sidesupply side demand sidedemand side

The survey method is qualitative and is suitable for those innovative projects that haven’t been analyzed in depth yet and is also suitable on the policy measures that can have complicated socio-economic impacts, though it is not very appropriate to use it to test the causal relationship.

Due to explore the details of S&T implementation. It is suitable for some issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality

In order to respond the S&T impact. It is the application of these issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality

28

Benchmarking

supply sidesupply side demand sidedemand side

It can apply on product, service, procedure, method, framework, and the whole organization etc. In the process of benchmarking, target selection is very important.

The choices of indicators can be applied in some issues, Investigation of system fairness and transparency, S&T research and development energy development

The choices of indicators can be applied in some issues, Investigation of system fairness and transparency, S&T research and development energy development

29

Conclusions

• Stakeholders are a determinate element in the choice of outcome evaluation methods. The level of attention paid towards each issue will depend on each stakeholder ultimate goals causing a constant need of coordination in conflicted S&T policy decision makings and evaluation systems. A combination of multiple S&T policy outcome evaluation methods is necessary.

• Government organizations can carefully formulate a suitable policy through interacting with the stakeholder ; resolving concerning issues and planning future development of the policies together, in order to achieve a realm of ‘mutual benefit’.

• The identification of stakeholders indeed have influence on the whole process of evaluation.

• When we make the choices of evaluation methods, we need consider the indispensable element, the stakeholders.

Li, Shan Shanssli@mail.stpi.org.tw

http://www.stpi.org.tw/STPI/index.htm

31

Thanks For Your Attention!Thanks For Your Attention!