32
Evaluation Procedures in the Max Planck Society Workshop „Identification of applicable approaches for the evaluation of the National Academy of Sciences of Ukraine“ Kyiv, 24-25 March 2015
Evaluation Procedures in the
Max Planck Society
Workshop „Identification of applicable approaches for the
evaluation of the National Academy of Sciences of Ukraine“
Kyiv, 24-25 March 2015
Mission and Guiding Principles of the Max Planck Society
Basic Research at cutting-edge, strictly
curiosity-driven and quality oriented
Autonomy, where scientists
decide upon science
„Harnack Principle“:
People not programs
Flexible, dynamic,
interdisciplinary MPIs
Long-term trust systems with
significant core funding for high-risk
projects
Quality assurance by peers
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 2
„Insight must precede Application.“
Max Planck, 1858 -1947
Founder of the Quantum Theory, Nobel Prize 1918
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 3
Max Planck Society – Facts and Figures
21,600 employees, including 280 directors, 4,600 grant holders and guest scientists
Annual budget: approx. € 1.6 billion plus additional project funding
Non-profit organization under private law in the form of a (private) registered association
Established in 1948 (successor of the Kaiser Wilhelm Society, founded 1911)
President: Professor Dr. Martin STRATMANN
82 Max Planck Institutes and research facilities, including 5 Institutes abroad
Professor Dr. Martin Stratmann
18 Nobel Prize Laureates since 1948 32 Nobel Prize Laureates since our inception in 1911
Governance of the Max Planck Society
Administrative Headquarters
General Meeting
of Members of MPG
Secretary General
Scientific Council
CPTS BMS GSHS
ele
cts
ele
cts
Executive Committee
P, VP CPTS, VP BMS, VP GSHS, VP Industry Secretary General, Treasurer, 2 Senators
Senate
President
32 elected and 15 “ex-officio” Senators representing German science, politics, industry, media and other sectors
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 4
Inter – Institutional Projects and Networks
evaluates scientific
performance and
advises the institute.
Scientific
Advisory
Board
Board of
Trustees
supports institute in its
local and public
relations.
Director
Dept. D
Director
Dept. C
Director
Dept. B
Director
Dept. A
Scientific Management
Rotating management duties
Managing Director (temporary)
Head of
MP
research
group
Head of
MP
research
group
Administration Library, Laboratories, IT
Service facilities
Structure of Max Planck Institutes
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 5
Max Planck Principles
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 6
Evaluation Procedures of the Max Planck Society
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 7
Scientific Autonomy of Institutes/
Scientific Directors
Long-term funding without ties to
specific applications
High degree of trust placed by funding providers
(i.e. Federal Government, Federal States)
Evaluation for • Measuring of excellence • Quality assurance • Justification of trust
Evaluation Procedures of the Max Planck Society
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 8
Ex-ante evaluation
Ex-post evaluation
Others
• Establishing institutes
• Appointments
• Program concepts
• Concepts for individual scientific proposals
• Regular evaluation every 2/3 yrs. by the SAB of the institute • Extended evaluation every 6 yrs. by the extended SAB of the institute
• IMPRS
• System evaluations commissioned by donors and granting agencies
• Structure oriented presidential committee • Internal analysis of the activities and the performance
Peer Review
The Max Planck Society (MPG) is convinced that the major part of its
productivity and performance is based on successful appointments.
Experience shows that a „wrong“ appointment cannot be built up to
a top performer afterwards.
That is why the MPG attaches maximum importance to the ex-ante
evaluation of its future scientists and devotes considerable effort to
screening the applicants with complex assessment procedures.
Ex-ante evaluation
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 9
Ex-ante evaluation in appointments
Institute With a vacancy for a Director
Proposal for an appointment
Approval for appointment procedure to begin Expert opinions
obtained
President
Senate Endorsement on 1st and 2nd reading
Recommen- dation
Appointment
Written opinions
Section
External Experts
Scientific concept
Scientist as an outstanding figure
Appointment Committee
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 10
Ex-ante evaluation in appointments
The Appointment Committee consists of Scientific Members and
external experts
The Committee examines:
The scientific concept for the appointment
The long-term perspectives of the research area
The unique feature of the research area
Originality, international standing an leadership qualities of the
potential candidate
Does the candidate „fit“ to the MPG?
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 11
Evaluation Procedures of the Max Planck Society
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 12
Ex-ante evaluation
Ex-post evaluation
Others
• Establishing institutes
• Appointments
• Program concepts
• Concepts for individual scientific proposals
• Regular evaluation every 2/3 yrs. by the SAB of the institute • Extended evaluation every 6 yrs. by the extended SAB of the institute
• IMPRS
• System evaluations commissioned by donors and granting agencies
• Structure oriented presidential committee • Internal analysis of the activities and the performance
Peer Review
Ex-post evaluation
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 13
Ex-post evaluation through Scientific Advisory Boards (SAB)
SAB at every Max Planck Institute
SABs
• both advise and evaluate
• Peer Review System
• Members are internationally visible scientists from all over the
world, experts in the field of the institute
• have 6 to 12 members, depending the size of the institute
• convene every 2 or 3 years for 2-3 days on site
• base assessment on a status report and an on-site visit
Status Report of the Institute
Basic Data / Ressources Structure and organization of the institute Research Program of the institute and its departments Budget Material resources, equipment and premises
Scientific Activities Publications Knowledge transfer activities; relations with industry, politics
and society Appointments, scientific awards and memberships
National and international visibility
Visiting scientists Relations with research institutions in Germany and abroad Symposia and conferences Committee activities Public relations Promotion of junior scientists/ Diversity / Equality …
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 14
On-site Visit
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 15
Procedure
Given Talks
Poster Presentations
Discussions between SAB Members and Scientific Members, Max
Planck Research Group Leaders, junior scientists
Closed sessions of the SAB
Oral report by the SAB to the President
Present
SAB, Competent Vice President, President (for the concluding
discussion), Institute Liaison Officer
Significance of the institute /department in national and
international contexts of the research field
Quality of research program and research performance of the
institute/ the individual department/the Research Group
Quality of personnel structure
Prospects of research fields
Efficency of ressource allocation
Collaboration at the institute, with other MPIs, with external
partners
Promotion of junior scientists
…
Evaluation Criteria
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 16
The SAB reports orally to the President at the end of the visit. A
written report follows, if neccessary they might add a confidential
letter.
SABs deliver the President detailed, differentiated and topical
information on the institutes‘ performances.
The President sends the report to the institute and asks for
comments and answers.
The SAB meetings have direct consequences, and thus serve the
internal optimization directly.
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 17
Ex-Post-Evaluation
President Institute
Members of the Scientific Advisory Board (SAB)
responds to the report by the SAB
requests comments/ intervention
report by the SAB
appoints SAB members
proposes SAB members
status report information to the SAB
Institute inspection
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 18
Extensive and thorough search for appropriate candidates ex-ante.
Every newly appointed Scientific Member is met with high
expectations and high trust.
On-going and ex-post evaluation by peers
Major features: science-driven, international, highly confidential
Conclusion
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 19
Evaluation Procedures of the Max Planck Society
Thank you for very much for your attention!
Dr. Ralph Meiers
Evaluation Procedures of the Max Planck Society
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 31
Ways of searching for candidates are manifold and heterogeneous.
Publications of a candidate are an important indicator for his or her
achievement potential for the Appointment Committee
No bibliometrical analysis
Commission members are expected to directly deal with the
candidate‘s publications. (Discussion of the most important
publications.)
Ex-ante evaluation in appointments
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 16
Governance of Max Planck Institutes Scientific Management‘s Rights and Duties
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 4
Structure of the Max Planck Society (Part A)
President • represents the Society
• drafts basic aspects of scientific policy
• ensures Society‘s work is carried out in a spirit of trust
• chairman of the Executive Committee, the Senate and the General Meeting
• supported by the Vice Presidents
Executive Committee
President, four Vice Presidents, Treasurer, two additional Senators
• advises the President
• prepares important decisions for the Society
• draws up overall budget, annual report, and annual accounts
• through the President, carries out supervision of the Administrative Headquarters
• together with the Secretary General, makes up the Board of Directors
Senate
max. 32 elected Senators / 15 ex-officio Senators
• elects the President decides on establishing Institutes
• appoints the Secretary General decides on involvement in other institutions
• decides on appointment of scientific members adopts the overall budget and the annual report
• adopts statutes resolves to accept supporting members
• resolves to give awards draws up annual accounts
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 6
Structure of the Max Planck Society (Part B)
General Meeting
Supporting, Ex-officio,
and Honorary Members
Scientific Members
• elects the Senate • resolves to amend the Statutes • receives the annual report • examines and passes the annual accounts • discharges the Board of Directors
Scientific Council / Sections
Scientific Members
Representatives of the science staff
Discusses and recommends • inter-institutional matters • founding (and closing) of Institutes • appointments
Max Planck Institutes
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 7
Evaluation Procedures of the Max Planck Society
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 25
Ex-Ante-Evaluation
Ex-Post-Evaluation
Others
• Establishing institutes
• Appointments
• Program concepts
• Concepts for individual scientific proposals
• Regular evaluation every 2 yrs. by the SAB of the institute • Extended evaluation every 6 yrs. by the extended SAB of the institute
• IMPRS
• System evaluations commissioned by donors and granting agencies
• Structure oriented presidential committee • Internal analysis of the activities and the performance
Peer Review
Extended Evaluation
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 29
Rapporteurs take part in meetings every 6 years
are members of Research Field Committee
Meetings of the SABs
Status reports
Research field
Institutes dedicated to related subject
Strategic Planning Reallocation of resources Reorientation
SAB reports
Research Field Committee •SAB chairs •Rapporteurs •President, •Vice President •Section Chair
Comparison between Institutes in the Research Field
MPS bodies
President
Evaluation/ Recommendation
Extended Evaluation
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 26
Research field evaluation
5-8 Institutes grouped into research fields
e.g. Research Field I of the CPTS: Astronomy,
Astrophysics, Gravitational Physics, Extraterrestrial
Physics and Radio Astronomy
Every six years 2-3 rapporteurs supplement SAB
Rapporteurs are international experts with ability to assess
research field as a whole
Rapporteurs take part in meetings of SAB in research field
Extended Evaluation
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 27
Meeting of a Research field Committee when all institutes of
the research field are evaluated
Research Field Committee is composed of
Rapporteurs
Chairs of the respective Scientific Advisory Boards
President
Vice President
Competent Section chair
Cross-institute, comparative evaluation of institute status reports
budget
evaluation standards applied by the boards
Extended Evaluation
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 28
Meeting of a Research field Committee
Committee discusses prospects for development
Committee presents results of extended evaluations
to Senate
Results of extended evaluation are necessary for
Strategic planning
Reallocation of resources
Reorientation
82 Institutes and facilities 280 Directors
Nijmegen
Rome
Florence
Jupiter
Max Planck Society : At home in Germany – present all over the world
Luxembourg
M A X - P L A N C K - G E S E L L S C H A F T | Dr. Ralph Meiers | PAGE 12
