Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

MODULE 5: MODULE 5:

PEER REVIEWPEER REVIEW

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

OverviewOverview

This module highlights the importance of peer review in IEA

In particular, it emphasizes the need to ensure scientific credibility, policy relevance and legitimacy

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

The module also outlines:

what constitutes peer review in the context of the AEO process

the objectives of peer review

why peer review is critical to IEA

how to select peer reviewers

Overview (cont’d)Overview (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Peer review definition in AEO

Peer review is a documented critical look at the AEO report, conducted by qualified individuals and organizations.

experts are considered equivalent in technical capacity (thus peers) to the experts who draft the contents of the report.

The peer review is important for ensuring that research and conclusions of the report are technically adequate, competently performed, properly documented, and satisfy established quality requirements.

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

The AEO peer review is an in-depth assessment of the assumptions, calculations, extrapolations, alternate interpretations, methodology, acceptance criteria, and conclusions of AEO reports and documentation supporting them.

The process is characterized by a limited number of interactions by peer reviewers.

It is undertaken during the whole AEO process, including selection of methods, research and drafting.

Peer review… (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Extent of peer review for the AEO Scientific, technical and policy review has

always been a key component of the AEO and other environmental assessment reporting processes

It involves hundreds of stakeholders both inside and outside government structures

The peer review process takes different forms – ranging from informal to formal

activities include sub-regional and regional consultations, collaborating centre network meetings, and targeted expert input.

AEO draft sections are sent to scientists and government experts for review and comprehensive comment

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Objectives of peer review

The objectives of the peer review are to check, among others, for:

adherence by the authors to the TORs provided by the coordinating agency at the start of the assignment

reliability and appropriateness of the scientific basis of the analyses, as well as the data and information used

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

reliability of the sources of information and citations, and if these are up-to-date

regional and sub-regional coverage of the issues

appropriateness of the conclusions/findings of the analyses

relevance and soundness of the indicators used in the analyses

Objectives of peer review (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

The peer review process

comments provided by the expert peer reviewers are documented, with every input being logged in a comprehensive database for follow-up, and assigned to different authors and staff to address

where conflict between/among review comments arises, coordinators contact the experts to discuss and determine an acceptable compromise

peer review builds upon previous activities by widening the scientific input

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

The peer review process (cont’d)The peer review process (cont’d)

the process aims to harmonize methodologies (and) to improve and expand the peer review system and UNEP networks - in line with the recommendations of the Scientific and Technical Meeting on strengthening the scientific base of UNEP (January 2004)

The peer review process is, therefore, systematic and adapts experiences from other processes

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Peer reviewers

have technical expertise in the subject matter for which they have been selected; they are expected to participate fully in the process and read all materials

should also protect confidentiality - by not sharing any AEO material with third parties

should respect the set deadlines, be objective and avoid bias

should report any problems encountered in finalizing the peer review process

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Some peer review activities

Advising on the treatment of the issues across the chapters

Commenting on the key issues as highlighted

Pointing out other key issues which have not been highlighted but would be considered a serious omission if they were left out in the final report.

Where such omissions are highlighted, the reviewer should present. in a paragraph or two, the issues, including full relevant references.

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt Where such omissions are

highlighted, the reviewer should present within a paragraph or two the issues, including the relevant full references

Indicating additional sources of relevant data

Ensuring that trend data have been presented in all chapters.

Ensuring that key indicators on the state of the environment and environmental policy performance are properly tracked

Some peer review… (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt Paying special attention to the

interlinkages between trends and policies (both specific to the issue under each chapter and more general policies that impinge on the issue)

demonstrating the environment as a driver of policy and vice versa, and evaluating policy in terms of environmental impact (effective, ineffective, unexpected, etc.)

Providing additional information and data, including boxes and other illustrations to enrich the sections, if necessary

Some peer review… (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt Pointing out inconsistencies and/or

contradictions of facts/data within and across the chapters and regional perspectives

Highlighting the major messages from the state of the environment both at the regional and sub-regional levels

Providing a list of the major messages emanating from the state of the environment both at the regional and sub-regional levels

Highlighting the major policy weaknesses clearly evident from both the regional and sub-regional thematic sections

Some peer review… (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt Highlighting the major policy strengths

clearly evident from both the regional and sub-regional thematic sections

Providing a list of both the strengths and weaknesses emanating from the state of the environment both at the regional and sub-regional levels

Advising on possible alternative policy initiatives that should be considered to ensure effective environmental management at both regional and sub-regional levels

Indicating any priority and emerging issues, or areas of outstanding vulnerability related to the issue as analysed

Some peer review… (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Providing an assessment of the treatment of IEA, both between the chapters and the regional sections, as well as across them.

Providing detailed comments on all the points listed above. Reviewers should avoid sending questions on the sections but rather provide substantive comments, which can be considered during revision of the sections.

Some peer review… (cont’d)

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Finding peer reviewers

Candidates may be recommended by:

government ministries/departments

scientific institutions

universities, collaborating centres

other stakeholders

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Determining the Peer Review schedule

The Peer Review schedule is a critical feature of the AEO process and should take the following into account:

availability of quality draft material

availability of appropriate experts

time available to use the review comments

deadlines for the AEO report and logistics

Div

isi o

n O

f E

ar ly

Wa

rnin

g A

nd

As s

ess

me

nt

Documents to send peer reviewers

Most recent and quality copy of the draft environmental assessment report - this could be the whole or section of the report

Clear TORs, including specific activities and deadlines and a clear reporting hierarchy

Clear guidelines on how comments should be provided