New Direction of National R&D Evaluation System in

Korea

2011. 11. 3

Changwhan MaDirector of Performance Policy Division

EVALUATION 2011 @ Anaheim, CA, U.S.A.

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

Contents 13 slides

1

3

5

13

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

1

3

5

13

R&D Program Management Process

R&D Bud-get Survey/Analy-

sisEvaluation

Implementation(Programs/Projects) In-depth

RecommendationFeedback

Evaluation strategy & Data collection

Self → Meta

Evaluation processUtilization process

Feasibility Study (Ex-ante)

Program (Plan-ning)

Policy (Planning)

1

National R&D Evaluation System in Korea 2

(Guidelines)

Self Eval.

Results (Grades, Recommendations)

Meta EvalProgram

Evaluation

GRIEvaluation

Ex-post

In-depth Eval

Ex-post/postFollow-up

SurveyFollow-up Eval

Management& Research Self Eval. Meta Eval

Ministries NSTC/(KISTEP)

Feedback

Report to NSTC

Efficiency/Effectiveness Of R&D Investment ↑

Preliminary Study

FeasibilityStudy

FeasibilityStudy

Results (Budget Size,Recommendation)Ex-ante Program

Planning

Ex-post

NSTC/MOSF/(KISTEP)

*NSTC: National Science & Technology Commission

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

1

3

5

13

Background of New Direction 3

Continuous increase of R&D investment

3rd on R&D investment / GDP (3.74%, 2010), 14.89 billion USD in 2011

Quantitative Efficiency of R&D Performance has been continuously increased

(Publication and Patent) Near top 10 in the world

Qualitative Excellence is stalled around 30th in the world

S&T and Socioeconomic Competitiveness depend on

the qualitative excellence of researchers and their performance

Therefore, evaluation system should meet the direction of S&T advancement

And new NSTC has recently been launched to respond to the recent change.

New NSTC 4

8

ce

삭제

Ministries, GRIs etc.Planning

ResourceAllocation

Evaluation

NSTC

MoSF

Suggestion about direction

of Budget allocation

Allocation · Coordi-nation · Formation

Self evaluation

Meta & Specific evaluation

AfterBefore

Constructing overall R&D System

MoSF Budget Forma-tion

Self evalua-tion

Min-istries

Min-istries

MoSF NSTC

NSTC

Meta & Specific evaluation

Suggestion about direction of budget alloca-

tion & Budget allocation / coordination of

main National R&D program

NSTC (Multi-departmental R&D pro-gram, etc.)

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

Contents

1

3

5

13

1. Open Evaluation System

10

① More Openness in Performance Information & Expert Participation ① More Openness in Performance Information & Expert Participation

To ensure variety of evaluation perspectives by increasing

Openness of Performance Information and Various Opinions

to the information

To promote expert review activity by establishing compre-

hensive expert community pool

To introduce on specific evaluation first and escalated to

other evaluation types

To ensure variety of evaluation perspectives by increasing

Openness of Performance Information and Various Opinions

to the information

To promote expert review activity by establishing compre-

hensive expert community pool

To introduce on specific evaluation first and escalated to

other evaluation types

② Establishing Online Open Evaluation System ② Establishing Online Open Evaluation System

To share and communicate more by establishing user-friendly

online open evaluation system (OOES)

To develop model and manual for OOES

To share and communicate more by establishing user-friendly

online open evaluation system (OOES)

To develop model and manual for OOES

5

2. Mission-oriented GRI Evaluation System

11

① Introducing Absolute Rating System ① Introducing Absolute Rating System

To introduce absolute rating system considering characteris-

tics of each GRI (government-supported research institutes)

To expand the autonomy of GRI during evaluation process

To focus on accomplishment of GRI mission

To encourage to set challengeable objective to reach world

class technology level

To introduce absolute rating system considering characteris-

tics of each GRI (government-supported research institutes)

To expand the autonomy of GRI during evaluation process

To focus on accomplishment of GRI mission

To encourage to set challengeable objective to reach world

class technology level

② Alleviating Burden on Evaluation ② Alleviating Burden on Evaluation

To reduce the portion of management evaluation gradually

and integrated into performance evaluation

(depends on revision of act on R&D performance evaluation)

To reduce the portion of management evaluation gradually

and integrated into performance evaluation

(depends on revision of act on R&D performance evaluation)

6

3. Rationality and Autonomy of Self Evaluation

12

① Stringent Pre-check of Performance Plan ① Stringent Pre-check of Performance Plan

To ensure the quality of self evaluation by pre-check the ap-

propriateness of performance objective and indicator

To operate official committee on reviewing performance indica-

tor

To ensure the quality of self evaluation by pre-check the ap-

propriateness of performance objective and indicator

To operate official committee on reviewing performance indica-

tor

② More Autonomy on Self Evaluation ② More Autonomy on Self Evaluation

To set up the evaluation schedule (or cycle) according to

milestones of each programs by budget-spending ministry

To carry out meta evaluation in general and re-evaluation on

specific programs

To set up the evaluation schedule (or cycle) according to

milestones of each programs by budget-spending ministry

To carry out meta evaluation in general and re-evaluation on

specific programs

7

4. Evaluation of Qualitative Excellence

13

① Qualitative Evaluation considering Impact ① Qualitative Evaluation considering Impact

To carry out comprehensive evaluation considering scientific/

technological and socioeconomic impact and sustainability

To encourage to set challengeable performance objective by

using qualitative performance indicator

To carry out comprehensive evaluation considering scientific/

technological and socioeconomic impact and sustainability

To encourage to set challengeable performance objective by

using qualitative performance indicator

② Guideline on R&D Performance Evaluation ② Guideline on R&D Performance Evaluation

To develop and disseminate R&D performance evaluation

guideline for better understanding of qualitative evaluation

and for strengthening evaluation capacity of ministries

To develop and disseminate R&D performance evaluation

guideline for better understanding of qualitative evaluation

and for strengthening evaluation capacity of ministries

8

5. R&D Policy Evaluation and Cross-cutting Re-view

14

① Evaluation of R&D Policy and Issue ① Evaluation of R&D Policy and Issue

To introduce R&D policy evaluation for better coordination of

R&D programs and R&D environment

To introduce R&D policy evaluation for better coordination of

R&D programs and R&D environment

② Cross-cutting Review ② Cross-cutting Review

To expand the scope of specific evaluation from individual

program to fields of technology and similar program groups

To ensure optimize and coordinate the delivery system the

similar, overlapping programs

To expand the scope of specific evaluation from individual

program to fields of technology and similar program groups

To ensure optimize and coordinate the delivery system the

similar, overlapping programs

9

6. Strategic Performance Budgeting & Program Improvement

15

① Strategic Performance Budgeting ① Strategic Performance Budgeting

To set priority and allocate budget of the similar programs ac-

cording to evaluation program selection and results

To carry out integrated performance budgeting by reviewing

mid/long-term direction of R&D investment

To keep maintaining performance budgeting with the evalua-

tion results

To set priority and allocate budget of the similar programs ac-

cording to evaluation program selection and results

To carry out integrated performance budgeting by reviewing

mid/long-term direction of R&D investment

To keep maintaining performance budgeting with the evalua-

tion results

② Monitoring System of Performance Information ② Monitoring System of Performance Information

To establish the DB for evaluation history of the program in-

cluding evaluation results, recommendation and management

action plan, etc.

To ensure the continuous monitoring evaluation feedback ef-

ficiently

To establish the DB for evaluation history of the program in-

cluding evaluation results, recommendation and management

action plan, etc.

To ensure the continuous monitoring evaluation feedback ef-

ficiently 10

7. Customized Evaluation

16

① Evaluation according to Program Type ① Evaluation according to Program Type

To apply differentiated evaluation perspective according to

types, size of the program

To develop evaluation model and checklists with common or

specialized indicators

To apply differentiated evaluation perspective according to

types, size of the program

To develop evaluation model and checklists with common or

specialized indicators

② Milestone-based Performance Evaluation ② Milestone-based Performance Evaluation

To select the program after considering the cycle or time for

producing key performance (creative period; 3 yr, 5 yr, ex-

empted, etc.)

To encourage to carry out objective and comprehensive per-

formance analysis for self monitoring

To select the program after considering the cycle or time for

producing key performance (creative period; 3 yr, 5 yr, ex-

empted, etc.)

To encourage to carry out objective and comprehensive per-

formance analysis for self monitoring

11

8. Infrastructure and Network

17

① Infrastructure of R&D Performance Evaluation ① Infrastructure of R&D Performance Evaluation

To establish DB for major performance and budget information

as a program management platform

To serve as an active channel for the exchange of information

on evaluation trend and various opinions

To carry out R&D performance evaluation capacity building

To establish DB for major performance and budget information

as a program management platform

To serve as an active channel for the exchange of information

on evaluation trend and various opinions

To carry out R&D performance evaluation capacity building

② Network of R&D Evaluation ② Network of R&D Evaluation

To hold the evaluation forum regularly (global R&D evaluation

network)

To carry out collaborative research on evaluation

To participate the conference on evaluation regularly

To hold the evaluation forum regularly (global R&D evaluation

network)

To carry out collaborative research on evaluation

To participate the conference on evaluation regularly

12

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

1

3

5

13

Future Direction 13

Professional Evaluation

Feasible Policy

Optimal In-vestment

National R&D Program

Qualitatively Excellent Researchers with Perfor-mance

Strong (S&T → National) Competitiveness

Open system

Customized

Mission

…

Thank you!