Peer Review in CAS Institute Evaluation

Zhou Jianzhong, Zhou Changhai, Li Xiaoxuan

Institute of Policy and Management, Chinese Academy of Sciences

Assessment Office, Bureau of Planning Strategy, Chinese Academy of Sciences

2012.10.24-27 Minneapolis

OutlinesOutlines

1. A brief introduction of CAS

2. The practices of peer review in the evaluation of

institute of CAS

3. A New model of Peer review in CAS institutes:

“ONE-THREE-FIVE” Expert Diagnosis Assessment

4. Discussion and conclusion

1949 2010

History & Position

CAS was Founded on Nov. 1, 1949◆ Highest academic institution in natural sciences in China

◆ A national comprehensive research and development center in natural sciences and high technology areas in China

◆ A major advisory body to the government on science and technology

2011Staff ： 60700Graduate Students ： 49000

CAS

Academic Divisions

CAS Headquarters

Main Infrastructure

－ 17 Large-scale Scientific Research Facilities

－ 7 National Labs

－ 5 Field Stations Networks

－ 36 National Engineering Centers

－ 273 Knowledge Transformation Centers

－ 317 Journals

－ 46 National Associations and Societies

–Division of Mathematics and Physics

–Division of Chemistry–Division of Life Sciences and Medicine

–Division of Earth Sciences –Division of Information Technical Sciences

–Division of Technological Sciences

Members 709

Foreign Members53

Committee for Consultation and Review Committee on Scientific EthicsCommittee for Science Popularization and Publication

Institutions Directly under CAS

－ 100 Research Institutes

－ 2 Universities and Schools

－ 2 Supporting Units

－ 3 Botanical Gardens

－ 12 Branches

－ 2 Press and Publication Companies

－ 1 Assets Management Company

－ 22 Holding Enterprises

Overview of CAS

Jilin （ 3 ）

Liaoning （ 4）

Shandong （ 3）

Shanghai （ 11）

Jiangsu （ 6 个）

Anhui（ 1 ）

Hubei（ 5 ）

Shanxi（ 1 ）

Shanxi（ 3 ）Sichuan

（ 3 ）

Yunnan（ 2 ）

Guizhou（ 1 ） Hunan

（ 1 ） Fujian（ 2 ）

Zhejiang （ 1 ）

Guangdong（ 5 ）

Qinghai （ 2 ）Gansu（ 3 ）

Xinjiang （ 2） Beijing

（ 40 ）12 branches

Beijing Branch

Shenyang Branch

Changchun Branch

Shanghai Branch

Nanjing Branch

Wuhan Branch

Guangzhou Branch

Chengdu Branch

Kunming Branch

Xi‘an Branch

Lanzhou Branch

Xinjiang Branch

Landscape of CAS affiliates （ 2010 ）

Tianjin（ 1 ）

• CAS affiliates could be found in 27 cities in 22 provinces, autonomous regions and municipalities directly under the central government

Layout of the main subject areas and research fields

Four Aspects

Coverage of the natural science and high technology of the main research fields

· Basic Research· Life Sciences and Biotechnology· Resources and the Environment· High Technology

Material Science

and Technology

Life Science and

Technology

Space Science and

Technology

Earth Science and Technology

Information Science and

Technology

Math and Big Science

Facilities

Changes of Concepts in EvaluationChanges of Concepts in Evaluation

1993 ～ 1998 1999 ～ 2004 2005 ～ 2010

Output Evaluation

Focus on research

outputs

Output Evaluation

Focus on research

outputs

Performance Evaluation Focus on achievements

Focus on level of completion

Performance Evaluation Focus on achievements

Focus on level of completion

Comprehensive Quality Evaluation

Focus on S&T creativity

Care performance in integrity

Comprehensive Quality Evaluation

Focus on S&T creativity

Care performance in integrity

15Years/3Period

Analyzing toolPolicy tool

Incarnates guidance &

inspiring

Compare & construe states of development

Managing tool

Control & management

of process

Encouraging competition,

enhance innovation

capability and efficiency by

performance evaluation

and process control.

The development of institute evaluation in CAS

Give priority to quantitative evaluation

Give priority to quantitative evaluation Qualitative evaluation Qualitative evaluation

Ranking of institutesRanking of institutes Group by sortGroup by sort

AnnuallyAnnually Evaluate every five yearEvaluate every five year

Isolated quantitative evaluation and peer-review

Isolated quantitative evaluation and peer-review

Integration of quantitative quality assessment and peer-review

Integration of quantitative quality assessment and peer-review

Combination of self-evaluation & focus group discussion

Combination of self-evaluation & focus group discussionSimple external evaluationSimple external evaluation

Comprehensive evaluation of Performance, management process, future development

Comprehensive evaluation of Performance, management process, future development

Performance evaluation only

Performance evaluation only

Changes of Evaluation Concepts and Methods of Institutes in CAS

Changes of Evaluation Concepts and Methods of Institutes in CAS

The practice of peer review in the evaluation of institute of CAS

Stage Object and content arrangements Experts sources and types

1998-2001: Extent of completion of objectives

The completion of scientific and technological objectives entered into between the institute and CAS

Communication assessment

Peers: domestic experts, including academic leaders and renowned experts inside and outside CAS; the proportion of experts inside CAS to those outside CAS was 1:2

2002-2004: Major results of scientific and technological innovation

Major results of scientific and technological innovation, including the innovativeness, influence and the level of academic leaders and their teams (1-5 items)

Communication assessment

Peers: mainly experts outside CAS, accounting for 73% of the total of experts, and academicians accounting for 23%

2005-2010: Comprehensive Quality Evaluation

the overall quality and level of the institute, and the international status and level of research units and fields

On-site assessment 33 international experts

Communication assessment

67 international experts

Meeting assessment

221 Domestic peers and user experts

Innovation outcome talents Communication assessment

criteria : Have considerable popularity in certain discipline Have generally experience in research management Currently in the first-line of research

mechanism : The institute recommends candidates of about two times the number of review

experts wanted The profession office of CAS authorities reviewed the list of experts Inviting experts in the name of the president of CAS to participate in the review

Experts selection criteria and mechanism

Organization and procedural arrangements

Communication assessment:

The Evaluation Research Center(ERC) contacts the assessment experts that

have been determined, sends evaluation materials and questionnaires

Assessment experts give assessment feedback within the specified period

The assessment center organizes the review results to form the assessment

report

On-Site assessment:

CAS leaders introduce the basic situation of CAS and the background and

requirements of assessment to international assessment experts

The experts group hear reports of the institute director and of a number of

academic leaders

Visit labs and discuss with scientists and graduate students

The experts group carries out a closed meeting to communicate with the

leading body

Issues

The nature of scientific research activities of different research need

different experts

The peer review system is suit to evaluate the basic research and the

experts is easy to select from international.

However, the peer review isn’t suit to evaluate the applied research and

high-tech research, which meets national strategic needs and national

economic development.

Perhaps some research work is low levels by experts’ opinion, but for

China's economic and social development is of great significance, such

research work should not invite international experts.

How to select the experts and provide the criteria of evaluation

The selection of experts, as well as the evaluation criteria of scientific and technical work have been always plagued us. The International criteria accepted peer-reviewed in the scientific community is mostly innovative in the science and technology. This is the base for compare the different research work.

However, The general objectives of CAS are to develop into a base for scientific research, for training high caliber scientific talent and for incubating high-tech industries in China; to become a national scientific think tank and to evolve into a national research institution that boasts “first-class achievements, first-class efficiency, first-class management and first-class talent.” a lot of work of CAS are to meet national needs. Therefore, evaluation criteria can’t only be just the scientific research criteria, the evaluation experts can not only be scientists.

How to make use of the results of evaluation

The results of experts evaluation are not comparable because different experts use

the different evaluation criteria. Some experts will give the lower evaluation that

refer to the highest international research level criteria . Some expert will give the

higher evaluation with reference to the relevant domestic research level criteria.

The discrepancy of expert evaluation results was less and was difficult to

distinguish grade. Because the experts was recommended by institute and have

the better relations with the institute. So these experts are easy to give a good

evaluation result. So the management is difficultly to makes use of the result of

the peer review.

In 2010, the innovation capacity of CAS institutes had been greatly

improved, the level of the staff was advanced and the institute entered a

new stage and class in the CAS. The 105th executive meeting of the State

Council fully affirmed the achievements made by KIP and adopted the

“Innovation 2020” of CAS.

Innovation 2020 aims at solving major scientific and technological

problems in basic, strategic and prospective research fields that

concerning the overall and long-term development of the country.

In order to fulfill the goal of “Innovation 2020”, CAS put forward new

strategies in the new developing era. That is “one-Three-Five” strategy ： One Positioning, Three Major Breakthroughs, Five Cultivating Directions.

A New model of Peer review in CAS institutes: “ONE-THREE-FIVE” Expert Diagnosis Assessment

The Twelfth Five-Year Development Plan

During the Twelfth Five-Year (2011~2015), required as the strategic

planning called “One-Three-Five”, each institute of CAS will promotes its

develop plan ， identify its position, and define its research priorities with

potential breakthroughs and its prospective layout.

One

One Positioning indicates each CAS institute should specify its major research areas, unique features, core competitiveness, and anticipated position in international circles and should avoid homogenization with other CAS research institutes.

Three

Three Major Breakthroughs indicate major basic, strategic and prospective S&T innovative achievements to be made in the next five to ten years; generally each institute shall raise no more than 3 breakthroughs.

Five

Five Potential Directions indicate research priorities with unique features, a future competitive advantage and potential breakthroughs; generally each institute shall set no more than 5 priorities.

During the 12th five-year plan ， CAS has developed a major R&D

outcome-oriented evaluation systemClarifying the value guidance ： 6 types of major innovation contribution and innovation in talents and mechanism

Designing reasonable assessment process

Judging performance

Diagnosing and monitoring key element which is

the benefit of performance

Key Scientific Issues Key Scientific Issues

Key Core Technology Key Core Technology

Open new directions Open new directions

Significant Social and Economic Benefits

Significant Social and Economic Benefits

Systematic Solution Systematic Solution

Great influence on decision making

Great influence on decision making

2 sections

1 base

structure Sections Aim

Supporting platformTheory tank 、 tank of methods and tools 、 expert

tank 、 database ， network research organization……

• Based on the assignment book

• To judge the target completion

• Emphasizing key breakthrough

Checking target

completion of key

breakthrough

During 2015

• Diagnosing key areas and factors• Guiding the development of the institutes• Driving the implementation of “1,3,5”

“1-3-5”

Expert diagnosis

assessment

During 2013-2015

• Holding the changing and development of core competitiveness

• Serving expert diagnosis assessment

Monitoring key

indicators of institutes

yearly

yearly

The major research outcome-oriented evaluation system of CAS

Objective and role of “One-Three-Five” expert diagnosis assessment

Inviting high level experts domestic and overseas

Diagnosing the positioning, advantages and disadvantages of the institutes

Diagnosing the research quality and technical value of the key fields, research

significance and impact, talents, resources and technology platform etc.

Diagnosing the positioning, core competitiveness of the institutes and their position

among relative institutes around the world

Diagnosing the positioning, advantages and disadvantages of the institutes

Diagnosing the research quality and technical value of the key fields, research

significance and impact, talents, resources and technology platform etc.

Diagnosing the positioning, core competitiveness of the institutes and their position

among relative institutes around the world

Through the diagnosis assessment

Help the institutions to improve management ， clarify core

advantage 、 avoid homogenization

Lay a foundation for the future major innovation contribution

Help the institutions to improve management ， clarify core

advantage 、 avoid homogenization

Lay a foundation for the future major innovation contribution

As a national research institution, each institute in CAS has its own mission and position. What kinds of expert can evaluate not only research quality but also the research work that meets the national strategic needs? So it needs the peer has different type. The peer of the institute evaluation is not only to judge the research quality based on international level, but also from the perspective of users invited experts to evaluate their capacity of meeting the national need.

Some institutes in CAS are the comprehensive research institutes which include basic research and applied research. That leads to institutions in CAS covering many disciplines and fields. So, how to ensure the peers can evaluate every discipline and field in the institute while controlling the expert group proper scale?

There are so many institutes in CAS and in different development levels. Some institutes in high level can be evaluated and measured by international benchmark. However, when some institutes are still in a relatively low level, it is difficult for making a judgment.

Discussion and conclusion

The object and content of review are more focusedIn the institute peer review of the major research outcome-oriented evaluation system of CAS, the diagnostic assessment is carried out for each institute every five years by domestic and foreign high-level peer experts and user experts mainly from the international perspective and national needs. The experts do not assess the entire institute or diagnose the entire research staff, but focus on the advantages and core competencies of the institute and judge the advancement of the work of major fields.

The assessment results focus on diagnosis but are not linked with resource allocation

In CAS’s new peer review, the assessment results are not linked with the allocation of resources but focus on diagnosis and help the institute to identify problems, grasp the direction, make a rational layout, improve management and enhance the efficiency of achieving significant achievements output. At the same time, the results are used to promote the exchanges between research staff and high-level experts in the same field and form the academic atmosphere of pursuing excellence and daring to innovate. Therefore, the institute has small pressure in the process of evaluation without too much burden and can organize the evaluation easily. This is a large difference from the previous peer review.

Summary of the “1-3-5”Expert diagnosis assessment

Challenges

How to determine the criteria of

Major Outcome?

How to select proper experts?

Can expert judge the Major

Breakthroughs?

……

Thanks ！

Contact:jzzhou@casipm.ac.cn