A Study of the Model and Methodology for A Study of the Model and Methodology for

Institute Evaluation in CASInstitute Evaluation in CAS

Xiaoxuan Li， Bing Shi， Jianzhong ZhouInstitute of Policy and Management， CAS

Management Innovation and Evaluation Research Center, CAS

ContentsContents1.1. Brief introduction of CAS and Evaluation Brief introduction of CAS and Evaluation

Practice in CAS InstitutesPractice in CAS Institutes

2.2. Three-hierarchy Evaluation ModelThree-hierarchy Evaluation Model

3.3. ConclusionsConclusions

Brief Introduction of CAS and Brief Introduction of CAS and Evaluation Practice in CAS InstitutesEvaluation Practice in CAS Institutes

Brief introduction of CASBrief introduction of CAS The Chinese Academy of Sciences (CAS) was founded in Beijing on

November 1, 1949. Consisting of the Academic Divisions and various subordinate institutions, it is the lead national academic institution in natural sciences, a major advisory body to the government on science and technology related issues, and a national comprehensive research and development center in natural sciences and high technology areas in China.

The general objectives of CAS are to develop into a base for scientific research, for training high caliber scientific talent and for incubating high-tech industries in China; to become a national scientific think tank and to evolve into a national research institution that boasts “first-class achievements, first-class efficiency, first-class management and first-class talent.”

CASCAS

Academic DivisionsAcademic Divisions CAS HeadquartersCAS Headquarters

Main InfrastructureMain Infrastructure

－ － 17 Large-scale Scientific Resear17 Large-scale Scientific Resear

ch Facilitiesch Facilities

－ － 7 National Labs7 National Labs

－ － 5 Field Stations Networks5 Field Stations Networks

－ － 36 National Engineering Center36 National Engineering Center

ss

－ － 273 Technology Transfer Center273 Technology Transfer Center

ss

－ － 317 Journals 317 Journals

－ － 46 National Associations and So46 National Associations and So

cieties cieties

–Division of Mathematics and Division of Mathematics and PhysicsPhysics

–Division of ChemistryDivision of Chemistry–Division of Life Sciences and Division of Life Sciences and

MedicineMedicine–Division of Earth Sciences Division of Earth Sciences –Division of Information Technical Division of Information Technical

SciencesSciences–Division of Technological SciencesDivision of Technological Sciences

Members 709Members 709

Foreign Members53 Foreign Members53

Committee for Consultation Committee for Consultation

and Review and Review

Committee on Scientific EthicsCommittee on Scientific Ethics

Committee for Science Committee for Science

Popularization and PublicationPopularization and Publication

Institutions Directly Institutions Directly under CASunder CAS

－－ 94 Research Institutes94 Research Institutes

－－ 2 Universities and Schools2 Universities and Schools

－－ 2 Supporting Units 2 Supporting Units

－－ 3 Botanical Gardens 3 Botanical Gardens

－－ 12 Branches12 Branches

－－ 2 Press and Publication Comp2 Press and Publication Comp

aniesanies

－－ 1 Assets Management Compa1 Assets Management Compa

nyny

－－ 22 Holding Enterprises 22 Holding Enterprises

Distributed in 22 Distributed in 22 provinces and cities provinces and cities

over China.over China.

Overview of CASOverview of CAS

In 1998, with the approval of the Chinese government, the CAS launched the Knowledge Innovation Program (KIP), and has since turned over a brand new page in its history of development.

In the process of conducting the KIP, in recognition of the national strategic requirements and the world trend in science and technology, the CAS has made the most profound and extensive adjustments in its disciplinary deployment and organizational structure since its founding in 1949. Scientific restructuring and management innovation has led to the preliminary formation of a structure and mechanism that are both geared to a socialist market economy and aligned with international standards.

Changes of Concepts in Institutes Changes of Concepts in Institutes Evaluation in CASEvaluation in CAS

Changes of Concepts in Institutes Changes of Concepts in Institutes Evaluation in CASEvaluation in CAS 15Years/3Period15Years/3Period

From 1993 to 1998, the institutes evaluation in CAS was

focused on research outputs ,it is called output evaluation.

From 1998 to 2004, the CAS launched the Knowledge

Innovation Program (KIP), the institutes evaluation in CAS was

focused on achievements and level of accomplishment, it is called

performance evaluation.

From 2005 to now, the institutes evaluation in CAS was focused

on S&T creativity and concerned performance in integrity, it is

called quality-orientated comprehensive evaluation.

Three-hierarchy Evaluation ModelThree-hierarchy Evaluation Model

Background and Tasks

• Comprehensive Complementary Reform Pilot Project

– In 2007, seven pilot institutes have been selected to investigate modern R&D institutions system

– Four thematic issues haven been launched in the following aspects respectively: Resource Allocation, Human Resources, Organizational model of R&D, Evaluation.

– Basic Tasks of Evaluation Issue: To study evaluation models for different types of institutes.

Categories of 7 Pilot Institutes

Institute of Institute of High Energy High Energy PhysicsPhysics

Cold and Arid Cold and Arid Regions Regions Environmental Environmental & Engineering & Engineering Research Research InstituteInstitute

Shanghai Shanghai Institute of Institute of Technical Technical PhysicsPhysics

Institute of Institute of Computing Computing TechnologyTechnology

Dalian Dalian Institute of Institute of Chemical Chemical PhysicsPhysics

Institute of Institute of MicrobioloMicrobiologygy

Institute of Institute of Physics Physics

• Large scale scientific Large scale scientific facilitiesfacilities

• Resources, Environment Resources, Environment and institute in western and institute in western ChinaChina

• Engineering Engineering and big teamsand big teams

• Basic Basic researchresearch

• High-tech High-tech ResearchResearch

• Comprehensive Comprehensive instituteinstitute

• Biological Biological researchresearch

Yearly quantitative and qualitative monitoring

Peer review

At the bottom hierarchy are indicators that can reflect common characters of the institutes, including yearly quantitative monitoring and yearly communication review

At the second hierarchy are the key indicators and benchmarks, which reflects the characters of certain category of institute. It can be used in both diagnosing evaluation for individual institute and comparison between institutes of the same category.

At the top hierarchy is qualitative evaluation by experts review, and it is also a hierarchy of individual evaluation. Experts review include two stages: diagnosing evaluation for individual institutes and comparing evaluation among the institutes of the same category.

Quantitative Qualitative

Three-hierarchy Evaluation Model

Key indicators

Key indicators and benchmark

The selection of key indicators: Based on the orientation and charac

teristics of institutes, we selected no more than 6 key indicators for e

ach pilot institute. Totally, there are 3 common key indicators and 8

specific key indicators.

There are four methods to set the benchmark in our research : 1) th

e more the better, for example the more awards, the better; 2)differ

ent standards will be given based on international comparison; 3)be

chmark will be set according to the development level of institutes;

4)judged by experts experience.

key indicators of 7 pilot institutes

共性指标：

不同研究所的共性因素

Common key indicators

S&T Talents

per capita funding

Awards

Specific key indicators

Invited reports of important conference or workshop

Accomplishment of Major Tasks

Construction and operation of large research facilities

Platform of resource and data

Intellectual Property

High quality papers

Engineering Application and Demonstration

Advisory Reports

The methods to set common key indicators of benchmark

CountryInstitutio

nper capita funding (ten

thousand dollars)per capita funding for researc

her (ten thousand dollars)

USA NIH 15.4 46.2

USA NIST NA 31.8

German MPG 18.8 51.9

German HFG 12.4 39.6

German FhG 13.8 NA

France CNRS 12.5 34.6

Japan RIKEN 32.3 41.4

Japan AIST 28.7 36.9

Korea KIST 17.7 27.5

Australia CSIRO 13.9 15.6

indicators The method to set benchmark

S&T Talents Make use of the definition of quantitative monitoring on innovation capacity about the S&T Talents

per capita funding

With reference to international research institutions and the real per capita budget of this year.

Awards More the better

The methods to set the specific indicators of benchmarks

Specific key indicators are determined by separately by

the 7 pilot institutes.

The study is expected to be accomplished by the end of

this year.

Conclusions

Conclusions

——Through the study of 7 pilot institutes, key indicators and bench

marks regarding different types of institutes were proposed, includin

g key quantitative indicators and benchmarks as well as key qualitati

ve indicators and anchoring method. Thereby, three hierarchy evalu

ation model has been set up.

——the new model puts more emphases on the character of evaluati

on by category and individual features of institutes.

—— In summary, this model has developed Quality-orientated Comp

rehensive Evaluation system in methodology.

ThanksThanks

Contact information:

www.casipm.ac.cn

jzzhou@casipm.ac.cn