Research and Science Information Guidelines for Australian ... Projects/2014-009 - A.pdf · not...

Research and Science Information Guidelines for Australian Fisheries

A. Penney, D. Bromhead, G. Begg, I. Stobutzki, S. Clarke, R. Little, T. Saunders, J. Martin

September 2016

FRDC Project No 2014-009

DRAFT

FRDC 2014-009 Research and Science Information Guidelines for Australian Fisheries

Final v4.0 i

© 2016 Fisheries Research and Development Corporation.

All rights reserved.

ISBN 978-0-9577587-1-1

Development of guidelines for quality assurance of Australian fisheries research and science information FRDC 2014-009

Ownership of Intellectual property rights

Unless otherwise noted, copyright (and any other intellectual property rights, if any) in this publication is owned by the Fisheries Research and Development Corporation.

This publication and any information sourced from it should be attributed to:

Penney AJ, D. Bromhead, G. Begg, I. Stobutzki, S. Clarke, R. Little, T. Saunders and J. Martin (2016) Research and science information guidelines for Australian Fisheries. FRDC Project 2014-009, 17 pp.

Creative Commons licence

All material in this publication is licensed under a Creative Commons Attribution 3.0 Australia Licence, save for content supplied by third parties, logos and the Commonwealth Coat of Arms.

Creative Commons Attribution 3.0 Australia Licence is a standard form licence agreement that allows you to copy, distribute, transmit and adapt this publication provided you attribute the work. A summary of the licence terms is available from creativecommons.org/licenses/by/3.0/au/deed.en. The full licence terms are

available from creativecommons.org/licenses/by/3.0/au/legalcode.

Inquiries regarding the licence and any use of this document should be sent to: [email protected]

Disclaimer

The authors do not warrant that the information in this document is free from errors or omissions. The authors do not accept any form of liability, be it contractual, tortious, or otherwise, for the contents of this document or for any consequences arising from its use or any reliance placed upon it. The information, opinions and advice contained in this document may not relate, or be relevant, to a readers particular circumstances. Opinions expressed by the authors are the individual opinions expressed by those persons and are not necessarily those of the publisher, research provider or the FRDC.

The Fisheries Research and Development Corporation plans, invests in and manages fisheries research and development throughout Australia. It is a statutory authority within the portfolio of the federal Minister for Agriculture, Fisheries and Forestry, jointly funded by the Australian Government and the fishing industry.

Researcher Contact Details FRDC Contact Details

Name:

Address:

Phone:

Email:

A. Penney

PO Box 997, Belconnen ACT2616

0401-788289

[email protected]

Address:

Phone:

Fax:

Email:

Web:

25 Geils Court

Deakin ACT 2600

02 6285 0400

02 6285 0499

[email protected]

www.frdc.com.au

In submitting this report, the researcher has agreed to FRDC publishing this material in its edited form.

https://www.myidentifiers.com.au/myaccount_manageisbns_titlereg?isbn=978-0-9577587-1-1&icon_type=new

mailto:[email protected]

DRAFT


Final v4.0 ii

Contents

1. Research and Science Information Guidelines for Australian Fisheries ............................. 1

1.1. Purpose ................................................................................................................................ 1

1.1.1. Scope ........................................................................................................................... 1

1.1.2. Application .................................................................................................................. 2

1.2. Key Principles for Scientific Information Quality ............................................................... 4

1.3. Responsibilities for Scientific Quality Assurance ............................................................... 4

1.3.1. Responsibilities: Research Purchasers ......................................................................... 4

1.3.2. Responsibilities: Research Providers ........................................................................... 5

1.4. Criteria for Effective Peer Review ...................................................................................... 7

1.4.1. Peer Review Criteria .................................................................................................... 7

1.4.2. Stages and Forms of Peer Review ..............................................................................10

1.4.3. Peer Review Terms of Reference ...............................................................................13

1.5. Data Retention and Management .......................................................................................13

1.5.1. Retention of Data and Primary Materials ...................................................................13

1.5.2. Provision of Access to Data ........................................................................................14

1.5.3. Confidentiality of Information and Data ....................................................................14

1.6. Implementation and Reporting ...........................................................................................14

1.7. Appendix A: Definition of Terms .....................................................................................15

Figures Figure 1. Components of the Research and Science Information Guidelines for Australian Fisheries

showing linkages between components. ..................................................................................... 2

Figure 2. Decision tree to guide decisions regarding the stages and forms of peer review that should be

applied to scientific research of increasing complexity and expected influence. ......................11

DRAFT


Final v4.0 1

1. Research and Science Information Guidelines for Australian Fisheries

1.1. Purpose

Fisheries research and scientific information is used to inform and underpin fisheries management

decisions and the formulation of fisheries and fisheries-related environmental policy.1 Government

Ministers and decision-makers, stakeholders and the public need to have confidence and trust in the

research and scientific information used to inform fisheries management decisions. To help achieve

this, key principles for ensuring quality of science need to be adhered to, and effective science quality

assurance processes need to be put in place, to:

• ensure the quality and integrity of research and scientific information, irrespective of the source

of that information;

• require research providers, relevant advisory committees and advisory processes to meet

sufficient Guidelines for ensuring the quality of scientific information; and

• ensure that peer review processes, the primary mechanism for ensuring the quality of scientific

information, are cost-effective and efficient.

These Guidelines provide guidance as to what constitutes high quality and reliable scientific

information, and on best practice in relation to the quality assurance of research and scientific

information intended or likely to inform management decisions for wild capture fisheries, regardless of

the source of that information. The Guidelines set out key principles for research and scientific

information quality, identify key responsibilities, and describe requirements for peer review processes,

evaluation of scientific information quality, storage and management of data and documentation and

communication of science results. The footnotes and the definitions of terms contained in Appendix A

are an integral part of these Guidelines and must be read in conjunction with the Guidelines. The

components and inter-relationships of the Guidelines are shown in Figure 1.

1.1.1. Scope

These Guidelines are intended to apply to all research and scientific information intended or likely to

inform management decisions relating to wild capture fisheries and their impact on the marine

environment. Scientific methods strive to produce objective and reliable information, and document

how that information has been derived, such that the results can be validated and checked for

reproducibility.

Scientific methods and quality assurance processes can be applied to any research project. Much of the

research and scientific information used to inform fisheries management decisions relates to fisheries

characterisations, biological studies, stock assessments, resource abundance surveys and evaluations of

fishery impacts on associated or dependent species. Other disciplines using scientific methods also

produce information that is used in fisheries management decisions, including broader ecosystem,

social science and economic studies. The principles and quality assurance processes in these

Guidelines can be applied to any such information, if derived using scientific methods.

1 Scientific information: means any knowledge, facts or data that have been generated, tested and verified using

scientific methods. Scientific information includes, but is not limited to, factual input, data, models, analyses,

technical information, or scientific assessments, whether conveyed through data compiled directly from surveys

or sampling programmes, or through statistical analyses and models that are mathematical representations of

reality constructed using primary data. In the context of these Guidelines, relevant fields of research and science

include, but are not limited to biology, ecology, oceanography, economics and sociology.

DRAFT


Final v4.0 2

Figure 1. Components of the Research and Science Information Guidelines for Australian Fisheries showing

linkages between components.

The provisions of these Guidelines are intended to be applicable to:

• Fisheries research projects contracted or conducted by research purchasers and/or providers

such as the Fisheries Research and Development Corporation (FRDC), Australian Fisheries

Management Authority (AFMA) and Commonwealth or State government departments

purchasing or providing research intended or likely to inform fisheries management decisions

relating to wild capture fisheries and their impact on the marine environment.

• Fisheries research projects conducted or contracted by the seafood industry or other stakeholder

organisations as part of established research programmes intended to inform fisheries

management decisions relating to wild capture fisheries and their impact on the marine

environment.

• Any other research conducted or contracted by other organisations, if that research is intended

or likely to inform fisheries management decisions relating to wild capture fisheries and their

impact on the marine environment.

1.1.2. Application

The provisions of these Guidelines are intended to apply to all stages of the research process, including

aspects of research planning processes related to evaluating the relevance of proposed research and the

appropriateness of proposed methodology, to ensure the reliability and objectivity of resulting

scientific information. Processes related to how research and scientific information is subsequently

combined at policy or management level with other sources of information to inform fisheries

management decisions, are not within scope of these Guidelines.

Stages and forms of peer review

Australian Fisheries Research and Science Information GuidelinesComponents & Linkages

ResearchPurchasers Research

Providers

Criteria for Effective Peer ReviewExpertise Independence Inclusiveness Timeliness

Transparency

Responsibilities

Key Principles for Research & Science Information Quality

Relevance Reliability Objectivity Integrity Peer Review

Definitions of Terms

Evaluation of information quality

Communication of information

DRAFT


Final v4.0 3

The Guidelines are not intended to apply to subjective information such as anecdotal information,

opinions and impressions of individuals, or observations for which there are no quantifiable data or

verifiable evidence beyond their testimony. Such information does not meet the definition of scientific

information and cannot be objectively evaluated against the principles in these Guidelines. Decision-

makers may nonetheless take such information into account, and it may provide important context

against which to review the potential impact of alternative fisheries management decisions.

Depending on the extent to which research projects fall directly or indirectly under the control of the

FRDC or government departments, different classes of research projects may be subject to the

provisions of these Guidelines in different ways:

i. There will be an expectation that all relevant research provided to the FRDC will meet the

requirements of these Guidelines. Such requirements will be incorporated into research contracts

issued by the FRDC.

ii. For other research projects conducted under fisheries research programmes and intended to

inform fisheries management decisions, research purchasers and/or providers should ensure that

the Guidelines are applied.

iii. For research projects not covered by the above two categories, and that have not been subject to

the requirements of these Guidelines during the research process, research purchasers and users

should determine how to assess their quality on a case-by-case basis. Such research may include:

Research emanating from other government agencies or other organisations such as

regional fisheries management organisations;

Industry-purchased research conducted outside of established research programmes; or

Other research including academic studies not originally intended to inform fisheries

management decisions, but which is subsequently considered to be useful for that purpose.

There are several reasons why information might not be required to undergo further quality assurance

and peer review before being used in fisheries management decisions:

• The information may already have been subjected to adequate peer review considered to be

compatible with the provisions of these Guidelines. This may include peer review associated

with publication in scientific journals or other formal scientific publications. It must nonetheless

be determined that that such peer review meets the requirements of these Guidelines for

scientific information intended to inform fisheries management decisions.

• The information may not be particularly influential on the fisheries management decision

concerned, or it may be supported by other reliable information, such that the time and cost of

further peer review is not justified.

• The information may emanate from a usually reliable source, or already been subject to some

degree of peer review, and time constraints may require the information to be used to inform an

important fisheries management decision before further peer review can be conducted. Under

such circumstances, the risks associated with using such information without further peer review

should be acknowledged and communicated.

Where there is uncertainty regarding the adequacy of previous peer review processes, or uncertainty as

to the quality of the information, such that the information is determined to require further peer review,

research purchasers or users should specify and arrange for the necessary additional peer review.

Where such information has been subjected to comparable quality assurance processes outside of these

Guidelines, the information may be determined by scientific reviewers, scientific working groups, peer

review panels or other appropriate peer review processes as meeting the quality requirements for

research and scientific information under these Guidelines, as a result of such previous review.

DRAFT


Final v4.0 4

1.2. Key Principles for Scientific Information Quality

The quality of research and scientific information relates primarily to relevance, reliability, objectivity

and integrity. The primary, internationally-accepted mechanism for evaluating the quality of research

and scientific information is peer review. These key principles should underpin all quality assurance

processes for research and scientific information.

Relevance – research and scientific information must be relevant to the fisheries management

objectives and associated key questions for the fishery concerned, contributing directly to answering

those questions and addressing management objectives for that fishery. Whether information is

likely to be relevant to a fisheries management objective or question should be determined and

documented as part of the peer review of research proposals.

Reliability – relates to the accuracy and reproducibility of information. Research and scientific

information must be accurate, reflecting the true value of the results being reported, within an

acceptable level of precision or uncertainty appropriate to the data and analytical methods used.

Information should not be statistically biased or suffer from such a high level of imprecision that the

results and conclusions are rendered unreliable. Methods and models used to produce scientific

information must be verified and validated to the extent necessary to demonstrate that results may be

reliably reproduced by an independent scientific expert using the same data and analytical methods.

Objectivity – refers to whether the information presented is impartial and free from personal bias.

Objective interpretations or conclusions do not depend upon the personal assumptions, prejudices,

viewpoints or values of the person presenting or reviewing the information.

Integrity – refers to the security of information, and to the protection of information from

inappropriate alteration, selective interpretation or selective presentation, including with regard to

uncertainty in that information. Scientific information should remain complete throughout the

science-to-decision process. It must be ensured that the information and associated uncertainty is not

selectively reported in a way that introduces bias into the interpretation of such information. Where

such information is required to be summarised for the purposes of reporting, such summary should

not be biased with respect to the complete information.

Peer Review - is a process of evaluation of research or scientific information by one or more experts

in the appropriate field, either with similar competence or in the same occupation, profession or

industry to the producers of the work. Peer review methods are employed to ensure that the work

meets appropriate or applicable standards of quality. Peer review usually emphasises the importance

of independence of the reviewers in order to obtain an unbiased evaluation, recognising that a larger

and more diverse group of people will usually find more weaknesses and errors in research, and will

make a more impartial evaluation of it, than the person or group responsible for that research. There

are many options for conducting effective peer review, depending on the novelty and complexity of

information.

1.3. Responsibilities for Scientific Quality Assurance

Implementation of scientific quality assurance practices to meet the requirements of these Guidelines

will primarily be the responsibility of those who contract, purchase or otherwise require research to be

conducted ('Research Purchasers'), and those who subsequently conduct and supply the contracted

research ('Research Providers'). These roles may resort under a single organisation or entity that both

contracts and conducts research, or may resort under separate research contracting and research

conducting organisations. The responsibilities outlined below relate to the respective components of

the process, irrespective of whether these resort under single or multiple organisations.

1.3.1. Responsibilities: Research Purchasers

All purchasers of research and scientific information that is intended or likely to inform management

decisions for wild capture fisheries and their impact on the marine environment should implement

DRAFT


Final v4.0 5

processes and procedures to ensure that the provisions and requirements of these Guidelines are

implemented and adhered to. Research purchasers should:

• Establish, maintain or support appropriate quality assurance and peer review processes, and

ensure that research and scientific information is subjected to effective peer review against the

provisions of these Guidelines.

• Ensure that research proposals are evaluated against the requirements for research and scientific

information quality established by these Guidelines relating to relevance, project design and

proposed methodology.

• Where necessary to ensure the quality of scientific information produced by substantial or

complex projects, provide for staged technical guidance or peer review at appropriate stages in

the project, ensuring that such peer review is appropriate to the cost, novelty, complexity, or

contentiousness of research and scientific information.

• Ensure that research providers comply with relevant provisions of these Guidelines, including

requirements relating to scientific expertise, data management procedures, project management

and research quality assurance systems.

• Establish, maintain or support, or require research providers to establish, maintain or support,

databases to manage and securely store any required raw data sets and relevant final data sets,

analyses and research reports emanating from relevant research projects, to enable subsequent

verification of the repeatability and reliability of the results.

• Ensure that the quality of research and scientific information provided to decision-makers is

evaluated against the key principles for scientific quality in these Guidelines, and that the

integrity of research and scientific information provided to decision-makers is protected.

1.3.2. Responsibilities: Research Providers

Research providers providing research and scientific information intended or likely to inform

management decisions for wild capture fisheries and their impact on the marine environment should

meet requirements relating to the following aspects of the research process:

Qualifications and Capabilities

• Research should be conducted by reputable research providers and designed, overseen and

conducted by research staff with appropriate science qualifications and expertise.

Project Management and Quality Management

• Research providers should demonstrate that they implement and maintain effective in-house

project management, research quality assurance and data management systems. Research project

leaders are to be designated to be responsible for project management and quality assurance.

Data Management and Provision

• Research providers are to establish and routinely follow effective data management and data

processing procedures, to ensure the accuracy and integrity of research data. Such systems

should include processes for error checking, data validation, data-filtering and error correction.

Research data are to be stored in appropriately designed databases with adequate cataloguing,

documentation and metadata. Data backup and disaster recovery systems are to be implemented

and maintained.

• Subject to any applicable confidentiality and privacy requirements, relevant datasets and

analyses associated with research and scientific information intended or likely to inform

management decisions for wild capture fisheries and their impact on the marine environment

should be made available, if required, for independent peer review and possible validation or re-

analysis. Where relevant, the computer code developed to analyse data should also be made

available.

DRAFT


Final v4.0 6

Certification of Laboratories and Equipment

Where research will involve laboratory analyses or the use of equipment that requires calibration or

operation in accordance with applicable technical protocols, research providers will be required to:

• Demonstrate that laboratories meet any applicable certification requirements, where required.

• Demonstrate that all equipment has been calibrated and certified in accordance with applicable

technical protocols for the equipment concerned.

• Maintain the equipment according to manufacturer’s instructions or specifications.

Data Collection

• Data should be collected according to documented procedures and in a manner that reflects

Guidelines or best practices generally accepted by the relevant science and technical

communities. Data and information sources should be identified.

• Data collection methods, systems, instruments and statistical sampling designs must meet the

requirements and objectives of the research projects concerned, and should be validated before

use. Instruments used to collect data should be calibrated using applicable standards or

fundamental engineering and scientific methods.

• Data should undergo quality assurance and any necessary data filtering and error correction

prior to being used. Data filtering and error correction processes should be documented and the

error-corrected data should be securely stored in a database.

Data Analysis and Synthesis

• Routine data analyses should be conducted using methods that are documented in published

methods manuals or agreed technical protocols, or methods otherwise published and generally

accepted by the relevant science and technical communities.

• Routine analytical methods, including statistical procedures, models and other analytical tools

and resulting analyses, should be periodically reviewed by suitably qualified internal and/or

external experts to ensure their validity.

• Analyses that are novel, complex or contentious should be submitted for appropriate peer

review, and reviewed by appropriately qualified independent experts.

• Analytical methods used should be documented, particularly where new methods are developed,

and information on methods used should be included when analyses are disseminated. Details of

analytical methods used should be included in final research reports.

• Data requirements and assumptions associated with any statistical or analytical model used

should be appropriate to the resolution and accuracy of the available primary data.

Experimental Studies

• The theory and details of experimental designs and methods should be documented, including

details of assumptions made, hypotheses established or tested, experimental design,

experimental data and results, analytical methods and the statistical procedures employed.

• Novel, complex or contentious experimental studies should be peer reviewed by appropriate

independent experts. For such experimental studies, results of any initial experimental trials

conducted should be subject to staged technical guidance prior to conducting subsequent stages

in the study.

• Where it is intended that new experimental methods and approaches should move towards

regular or production use, such approaches should be subjected to rigorous scientific peer

review before being transferred into general research use.

DRAFT


Final v4.0 7

Technical Protocols

• Research provider organisations should implement and follow any established or adopted

technical protocols and established best practices relevant to the research field(s) concerned.

• Where research tools, techniques, methods or processes represent a significant development,

advance, innovation or improvement in the research approach used, technical protocols should

be drawn up that describe the tools, techniques and processes used. Such protocols should be

subjected to independent scientific peer review.

Internal and External Peer Review

• Research providers should implement appropriate internal or external quality assurance and peer

review processes relevant to each of the above requirements, including the production of final

research reports.

• Research providers may also be required to submit data, analyses, results, conclusions and

reports intended or likely to inform fisheries management decisions to external peer review

processes specified by research purchasers, as or when requested to do so. Depending on the

cost, size and complexity of the research project, this may require submission of project

proposals, initial data, interim analyses and results, and final results and conclusions to several

stages of technical guidance and peer review.

Research Reports

• All research projects should be written up in a format appropriate to the intended use of such

information. Subject to applicable confidentiality arrangements, such reports should be made

publically available. Where the research represents a significant advance in the research field

concerned, research providers are encouraged to write up and submit the results to an

appropriate peer reviewed science journal.

1.4. Criteria for Effective Peer Review Peer review is the accepted and most reliable process for assessment of the quality of research and

scientific information. Peer review processes designed to ensure that research and scientific

information meets the key principles for scientific information quality specified in these Guidelines

should be established and implemented for all research and scientific information that is intended or

likely to inform management decisions for wild capture fisheries and their impact on the marine

environment. Peer review may be conducted using a range of alternative processes, and at various

stages, depending on the complexity, novelty, contentiousness or likely influence of the scientific

information.

1.4.1. Peer Review Criteria

Irrespective of the chosen process, peer review should be designed and conducted to meet the criteria

described below, as appropriate to the relevance and expected influence of the research and scientific

information concerned. The degree to which each criterion can be met will differ for alternative forms

of peer review. Trade-offs may be required, for example, between the independence of peer reviewers

and the inclusiveness of stakeholder representatives with expert knowledge; or between the need for

timely research and scientific information and the time required to conduct additional independent

expert peer review.

Independence and Expertise – One of the prerequisites for trust and credibility of research and

scientific information is that it must be seen as being provided by impartial processes that operate

independently of politics, financial interests and advocacy. Peer reviews should be conducted by

science experts who:

were not responsible for conducting the research and analyses under review;

DRAFT


Final v4.0 8

have the appropriate expertise and experience to review the research and scientific

information and analyses concerned; and

are able to provide an impartial and objective review.

• Peer reviewers should primarily be selected on the basis of scientific expertise and experience

relevant to the disciplines and subject matter to be reviewed.

• Participants in peer review processes are expected to act in an independent and expert manner

during peer review processes. They should not act as advocates for any interest group, and are

expected to step aside from their sector affiliations and participate as expert individuals

primarily interested in producing objective, unbiased science.

• For peer review of research projects that are novel, complex, or contentious, a greater degree of

independence may be necessary to ensure objectivity and credibility of the peer review process.

In such cases, reviewers should not be affiliated or associated with affected stakeholder groups,

or with the research providers involved in the research under review.

• Peer review processes should be designed and conducted in ways that are not adversarial, but

participants should be prepared to have their contributions challenged in constructive ways.

• Intentional involvement of interested stakeholders with relevant knowledge or experience can be

beneficial to increasing trust and acceptance of research results. In this case, potential conflicts

of interest must be identified and managed during peer review processes to ensure that they do

not result in bias in the information and conclusions.

Balance of Expertise – Where peer review is to be conducted by a panel, committee or advisory

group, these groups should incorporate an appropriate range and variety of scientific expertise

relevant to review of the information concerned.

• Selection of scientific experts should match the nature of the information under review and the

level of technical expertise required, be sufficiently diverse to represent the range of scientific

and technical fields of knowledge under review, and be sufficiently balanced to reflect the

potential diversity of opinion amongst experts.

• In the context of peer review participation, the term ‘balance’ does not refer to balancing of

stakeholder or political interests, but rather to diverse representation of alternative scientific

perspectives and intellectual views.

Inclusiveness – Where relevant and useful to the interpretation and objective evaluation of the

information under review, seafood industry and other stakeholders with knowledge and experience

can be included in peer review processes.

• Provided potential conflicts of interest are identified and managed, the presence of stakeholder

representatives at peer review meetings can facilitate transparency and openness without

compromising objectivity. Constraints on stakeholder representatives or observers may include

not participating in the scientific evaluation of information, analyses and conclusions, or not

contributing to the achievement of consensus regarding scientific conclusions.

• The knowledge and expertise of representatives from the different stakeholder or interest groups

that is used to inform the scientific debate should be identified as such when reflected in the

scientific reports and advice provided.

Transparency and Openness – To facilitate trust and credibility of research and scientific

information, science processes should be transparent and open to public scrutiny, particularly

regarding the peer review processes followed, the results of peer review, and when reporting

information.

• Subject to relevant confidentiality requirements and privacy legislation, the public should have

access to all final research reports. results and conclusions.

DRAFT


Final v4.0 9

• Such reports should provide adequate detail on data collection, analysis and modelling methods,

results, conclusions and scientific advice, to facilitate understanding and trust in the scientific

research being reported.

• The integrity of research and scientific information must be protected when making the

information available to ensure that such information is not inappropriately altered, selectively

presented or selectively interpreted.

Timeliness – Practical and efficient fisheries management decisions often require rapid review and

provision of research and scientific information to fisheries managers.

• Science quality assurance processes need to be efficient, balancing the need to maximise the

quality of research and scientific information with the requirement for cost-effectiveness and

timely provision of information. Peer review processes should be appropriate to the

requirements, particularly for research that follows established and well-tested methodology.

• The need for timeliness of research and scientific information can mean that preliminary results

of scientific research or monitoring programmes may need to be presented before the study is

complete, or before rigorous peer review can occur. Uncertainties and risks that arise from

interim results, or from insufficient time to subject the information to independent peer review,

should be acknowledged and communicated.

Impartiality and Management of Conflicts of Interest – Conflicts of interest arise when there is a

divergence between the individual interests of a person and their role in a peer review process. Such

conflicts can impair, or be perceived to impair, the participant’s objectivity and impartiality in peer

review processes, and contribute to bias in scientific conclusions or advice. Actual or potential

conflicts of interest must be identified and actively managed so that the impartiality of the peer

review processes is not called into question.

• Conflicts of interest may include, but are not limited to:

personal financial interests and investments;

employer affiliations;

consulting arrangements;

grants or contracts held by, or anticipated by, an individual or research provider; or

commercial or personal relationships with others who have material interests in related

businesses or stakeholder organisations.

• Peer reviewers should not have conflicts of interest that may seriously constrain their ability to

provide impartial, objective advice. In particular, Chairs of peer review working groups,

workshops, or panels must be impartial, and should not have any direct affiliation with research

providers whose research is being reviewed, or with seafood industry or other stakeholder

groups that may be affected by management decisions based on the research and scientific

information under review.

• While the existence of conflicts of interest need not preclude participation in peer review

processes, all actual and potential conflicts of interest need to be identified and managed.

• Were peer review is to be conducted by a panel, management of conflicts of interest should

primarily be the responsibility of the Chair of the peer review working group, workshop, or

panel concerned. Procedural rules should be established for ensuring that conflicts of interest do

not jeopardise the objectivity of the peer review process.

• Terms of Reference for peer review processes should include requirements for declaring and

managing conflicts of interest. Participants should be required to:

declare all interests relating to any of the scientific research under review;

endeavour to provide their expert advice impartially, free from any undue influence by the

seafood industry, fisheries managers, stakeholder organisations or other interest groups;

declare any actual or potential conflicts of interest that arise during discussion of the

research, scientific information or resulting scientific advice; and

DRAFT


Final v4.0 10

work with the Chair of the peer review processes to manage any actual or potential

conflicts of interest that arise.

Reporting of Uncertainty and Risk – Peer review processes must ensure that presentation of research

and scientific information includes the appropriate evaluation and reporting of uncertainty and risk.

Research reports should identify and explain known or likely sources of uncertainty, evaluate levels

of uncertainty in results, and assess the relevant risks that arise as a result of those uncertainties.

• Stock assessments, environmental assessments, risk assessments and other research and

scientific information products should describe data collection methods, state major

assumptions, report sources and ranges of uncertainty or statistical error of analytical models,

evaluate data limitations and, where appropriate, identify studies or analyses that could assist in

reducing those uncertainties.

• Scientific conclusions must be appropriate to the reported evaluation of uncertainty. Attention

must be paid to not over-emphasising or under-emphasising uncertainties in the information or

analyses presented.

Staged Technical Guidance – The more costly, novel, complex, or contentious that research and

scientific information is, the more rigorous and robust the science quality assurance requirements

need to be. Irrespective of the peer review process used, early engagement of peer reviewers in the

research process will enable technical problems to be identified, prevent wastage of resources on

invalid or suboptimal methods, and improve the quality and reliability of results. This is best

achieved by staged technical guidance.

Particularly where research projects are costly, novel, complex, or contentious, peer review and

technical guidance should be conducted at the following stages in the research process (Figure 2):

• Review of the research project design to evaluate whether the proposed research methods are

appropriate, and whether key fisheries management questions will be answered and research

objectives will be met;

• Evaluation of the quality, representativeness and adequacy of data generated by the project, and

consideration of the most appropriate analytical methods to use for those data;

• Review of the analyses, results, conclusions, summary documents and final research reports,

including evaluation of the uncertainties of the research results and the associated risks for

fisheries management.

1.4.2. Stages and Forms of Peer Review

There are many options for conducting effective peer review and the most appropriate form of review

should be chosen to be cost-effective and appropriate to the information under review. The choice

depends on factors such as: the need for timeliness; preferences for inclusiveness to facilitate buy-in

and mitigate later objections to scientific results and advice; the novelty, complexity or

contentiousness of the research and scientific information under review; and other relevant

circumstances, requirements or limitations relating to the review process.

A variety of peer review processes may be undertaken by research purchasers and/or research

providers. Research purchasers should primarily be responsible for specifying the preferred or most

appropriate form of peer review, and the membership and terms of reference for peer review working

groups, workshops or panels, following consultation with relevant stakeholders, and for ensuring that

criteria for effective peer review criteria are appropriately met at each stage of the process.

An overview of the optional stages and components in a peer review process and some of the

alternative forms of peer review is shown in Figure 2. This provides a decision tree to assist in

determining the most appropriate stage and form of peer review for different research projects or

reports.

DRAFT


Final v4.0 11

Figure 2. Decision tree to guide decisions regarding the stages and forms of peer review that should be applied

to scientific research of increasing complexity and expected influence.

In addition to the stages during a research project at which peer review and staged technical guidance

can contribute to ensuring the quality of scientific information, there are a number of alternative forms

of peer review. Selection of the most appropriate form depends on characteristics of the research under

review, particularly the complexity, contentiousness and expected influence of the research, and the

urgency of decisions that need to be informed by resulting scientific information.

Review by Individual scientist(s) – if a research project is relatively uncomplicated or simply an

update of previous work, or has already been peer reviewed elsewhere by processes that meet the

requirements of these Guidelines, peer review of the final research report by one or more qualified

scientists with the appropriate expertise may be adequate. Depending on the level of independence

required, such scientific experts may be chosen from within the research provider organisation, or be

contracted external experts.

Scientific Working Groups – where there is a requirement for regular and timely review and

provision of scientific advice, peer review can most effectively be conducted by existing scientific

working groups or advisory committees, or by establishing such working groups or committees.

Scientific working groups are particularly suitable for review of regular fishery characterisations,

updated biological studies, stock assessments, fisheries abundance surveys and regular evaluations of

impacts of fisheries. For such research, where there is a long history of addressing similar questions,

and technical protocols or agreed methods for reliable science have already been established and

tested, the accumulated experience of members of established scientific working groups can result in

efficient and reliable review of research results.

• Membership of established scientific working groups should tend towards being more inclusive,

and should include an adequate range of scientific expertise and stakeholder experience in the

Stages and Forms of Peer Review

Stage 1:Review of project

design

Stage 2:Review of initial results

and analytical methods

Stage 3:Review of final results

and conclusions

Submit to staged peer review

Research Report or Proposal

Review of final report by individual scientist(s)

Q1 – Is the research relatively uncomplicated, or a simple update of previous work?

or - Is it unlikely to strongly influence fisheries management decisions?

or – Has the research already been reviewed elsewhere by processes that meet the requirements of this Standard?

Q2 – Is the research novel, complex, or contentious? or – Will the validity of the results be strongly dependent on research design?

Q3 – Will the characteristics of initial data or results affect or determine the final analytical methods to be used?

Q1

N

Peer Review Stages

Q2

Q3

Q3

NY

Y

Y

Scientific working groups

Iteration as needed

Final quality assurance report

Y

NSpecialist technical review workshops

N

Alternative formsof Peer Review

Q5

Q5 –Does the research require review beyond the capabilities of established scientific work groups? If so, then the most appropriate alternative form of review should be determined.

Y

N

Independent expert review

Review by individual scientist/s

DRAFT


Final v4.0 12

range of research and scientific information referred to each working group. Such working

groups will benefit from the experience of members familiar with the techniques used in the

work being reviewed, and from established working relationships where conflicts of interest

have been resolved.

• Where adequate expertise does not exist within working groups, but where members of other

working groups or advisory committees possess expertise, experience and institutional

knowledge relevant to the information to be reviewed, inclusion of additional invited experts as

needed will provide broader perspectives and reduce the risk of inadequate peer review that

might result from limited knowledge or fixed views of existing participants.

Specialist Technical Review Workshops – are more appropriate where the questions to be addressed,

and the information to be reviewed, relate less to providing immediate science advice for fisheries

management decisions, and more to reviewing novel, complex, or contentious research approaches

in order to provide technical guidance to future peer review processes.

• Specialist technical review workshops should be led by, and primarily consist of, recognised

technical experts in the scientific fields or methodologies being reviewed. Inclusion of

additional experts in related fields, and of experienced individuals or stakeholder

representatives, may be beneficial to fully identifying the benefits and shortcomings of

alternative technical approaches to particular research questions.

• Emphasis in specialist technical workshops should be on technical expertise, wide canvassing of

expert opinion and technical information, consideration of diverse expert perspectives and

exploration of new ideas. Such workshops might include review and planning exercises for new

data collection or survey methodologies, or technical workshops to reconsider old, and develop

new, analytical methods.

Independent Expert Peer Review – may be required:

where the research is novel, complex, or contentious;

when there are strong conflicts of interest relating to potential impacts of fisheries

management decisions on organisations, industries or groups with whom some participants

in regular peer review processes are affiliated; or

where attempts at peer review using existing committees or panels have resulted in

adversarial debate and irreconcilable opposing views.

• It may be adequate to commission one or more subject matter experts, rather than a panel, to

provide independent expert peer review. This is particularly relevant to periodic reviews of

research programmes and assessment methodologies to ensure their balance, efficiency and

effectiveness in addressing specified management objectives and questions. Establishing the

range and priority of questions to be asked, and the appropriate balance of research projects to

address these, is as important as ensuring that individual projects are conducted correctly.

Reviews of research programmes should be conducted by independent science experts who were

not involved in the original design or development of those programmes or methodologies.

• Fully-independent ad hoc expert peer review panels can be constituted as and when necessary to

provide the highest level of independent peer review under situations when one or a

combination of the following circumstances applies:

questions exceed the technical expertise of the existing science working groups;

there is substantial uncertainty and a range of conflicting scientific opinions regarding the

interpretation of results;

the findings are controversial; or

implications for fisheries management decisions are substantial.

• Fully-independent expert peer review panels should be facilitated and managed by a suitably

qualified independent expert, with primary responsibility for the review residing with

recognised and independent experts in the research field concerned, who are not directly

affiliated with anyone involved in, or affected by, consequent fisheries management decisions.

DRAFT


Final v4.0 13

• Government, industry-affiliated or other experts may be requested to provide input to the

deliberations of an independent peer review panel, but the peer review report should be

produced by the appointed independent experts, free from undue non-scientific influences and

considerations.

1.4.3. Peer Review Terms of Reference

Irrespective of the chosen form of peer reviews, the scope of work and terms of reference for any peer

review must be determined in advance of the selection of reviewers. Terms of Reference must:

• Specify the mandate, roles and responsibilities of the participants.

• Require all participants to be familiar with, and to adhere to, requirements for scientific quality

assurance and effective peer review specified in these Guidelines,

• Identify the research projects or issues to be dealt with, including technical questions required to

guide the peer review process.

• Allow peer reviewers the opportunity to express their views on the range of research and

scientific information under review.

• Require that uncertainties and associated risks for fisheries management are clearly identified

and appropriately and objectively characterised and documented.

• Specify expectations regarding peer review processes and reporting of peer review outcomes.

1.5. Data Retention and Management

Retention, secure storage and provision of access to data and information used in scientific analyses to

inform management decisions for wild capture fisheries and their impact on the marine environment is

required to allow for validation, verification and evaluation of reproducibility, accuracy and objectivity

of the methodology and research results. Retention of such data and information ensures that it is

available for future re-analysis, if this is required, allowing for the cumulative process of building on

reviews and revisions of knowledge. Provision of access to data facilitates transparency of the research

process, contributing to increased trust in analyses and advice produced using these data.

1.5.1. Retention of Data and Primary Materials

• Each research provider or research purchaser (where research purchasers retain ownership of

fisheries research data) should have a policy on the retention of primary materials (such as

research samples from which data are derived) and research data produced as a result of any

research project that contributes scientific information used to inform fisheries management

decisions.

• Research providers or purchasers should support or provide storage facilities, databases and

archives for the secure storage of research data. All data used to inform fisheries management

decisions should be stored in such facilities and retained for future verification or use, subject to

applicable confidentiality requirements.

• In projects that span several institutions, an agreement should be developed at the outset

covering responsibilities for the ongoing storage of research data and primary materials within

each institution.

• Accurate and clear records should be kept of where research data are stored and a catalogue of

research data should be maintained in an accessible form. Fisheries databases should include

descriptive metadata for each relevant data set.

• Catalogues of stored data, and metadata for fisheries research databases, should include details

of data ownership, identification of data sources, access arrangements, confidentiality

requirements and contact details relating to data access and use. Adequate descriptions of data

DRAFT


Final v4.0 14

characteristics and data collection methods should be provided, to allow prospective users to

understand possibilities and limitations relating to data analysis.

1.5.2. Provision of Access to Data

• Each research provider and research purchaser should have a policy on the ownership of

research materials and data during and following the conducting of fisheries research projects.

• Research data should generally be made available for wider use unless this is prevented by

privacy or confidentiality requirements. Subject to contractual arrangements, confidentiality

requirements or privacy legislation, research providers should provide access upon request to

relevant datasets and analyses. Where relevant, this includes computer code used in filtering and

error correction of those data.

1.5.3. Confidentiality of Information and Data

• Arrangements for access to fisheries data and information must be consistent with applicable

confidentiality requirements, legislation, privacy legislation and other relevant guidelines.

• Where required to protect the commercial sensitivity of certain data, appropriate confidentiality

arrangements and agreements must be developed regarding access to these datasets. Release of

these data will be governed by these confidentiality arrangements.

• Where commercially sensitive data sets are protected by confidentiality agreements, but access

to the data is necessary for the purpose of further analysis, non-sensitive data sets may be

prepared upon request by, for example, aggregating data to a non-sensitive level.

1.6. Implementation and Reporting

Research purchasers and research providers intending to implement the provisions of these Guidelines

to ensure the quality of scientific information used or produced by them should develop and maintain

implementation plans appropriate to their particular circumstances, documenting how this will be done

within their organisation.

Implementation plans should include:

• A statement of intention to implement these Guidelines for the purposes of ensuring the quality

of scientific information used to inform management decisions for wild capture fisheries and

their impact on the marine environment.

• Identification of roles and responsibilities within the organisation for implementation of

processes relating to implementation of science quality assurance and peer review requirements

under these Guidelines.

• Description of peer review processes that will be implemented, specifying:

requirement that scientific information to be submitted for peer review;

provisions for establishment of scientific working groups or peer review panels or other

appropriate peer review process;

provisions for independent expert peer review and circumstances under which this would

occur;

requirements for documentation and reporting on the deliberations and outcomes of peer

review processes relating to quality of scientific information reviewed by them;

supporting documentation, including terms of reference for peer review processes.

• Annual reporting requirements on the implementation of peer review processes to evaluate the

quality of scientific information used to inform fisheries management decisions.

Public reporting on the details and results of implementation of scientific quality assurance and peer

review processes is important for ensuring transparency and increasing government, stakeholder and

public trust in the quality of scientific information used to inform fisheries management decisions.

DRAFT


Final v4.0 15

Research purchasers and research providers implementing science quality assurance and peer review

processes under these Guidelines should document:

• Measures taken to implement processes relating to scientific quality assurance and peer review

under these Guidelines, including the implementation plan.

• Details of peer review processes implemented, including composition of any scientific working

groups, peer review panels or independent expert peer review processes used.

• Summary of scientific information submitted to these peer review processes for review, and

outcomes of peer review relating to evaluation of the quality of this information.

• Overview of how the outcomes of peer review processes were taken into consideration during

the development of fisheries policy and fisheries management decisions.

1.7. Appendix A: Definition of Terms

For the purposes of interpretation and implementation of these guidelines, the following terms are

defined to have the following meanings.

Accuracy – the accuracy of data or analyses is a measure of the proximity of those data or results to

the actual (true) values. As such, accuracy is a core component of information quality, but one that

is impossible to measure directly when the true value is unknown. The processes of science quality

assurance set out in these Guidelines provide the means to indirectly assess accuracy by checking at

each stage of the scientific process for sources of statistical bias and imprecision, which are key

factors that degrade accuracy.

Bias – may result from statistical bias, personal bias or a combination of the two. Statistical bias

results from non-representative data collection methods or the use of inappropriate analytical

methods by which data are reviewed or analysed, interpreted, or published, such that results and

conclusions deviate systematically from the truth. Personal bias is an inclination or prejudice in

favour of a particular viewpoint or conclusion. Both statistical and personal bias may contribute to

the selective interpretation or presentation of results and uncertainties in a manner that influences

subsequent interpretation of the most likely outcome of a scientific analysis.

Data filtering and error correction – is any process whereby data are checked for accuracy using

objective rules, and data that are known or likely to be incorrect are corrected, deleted or replaced

with appropriate estimated values derived from accurate data. This may initially be a data analysis

stage rather than a data management process. However, where data filtering and error correction

procedures become routine, they should be incorporated into established data management

processes to avoid variation in processes or duplication of effort.

Impartiality – requires that decisions be based on objective criteria, and not on the basis of personal

bias or prejudice towards or against any particular party or viewpoint. In the context of peer review,

impartiality requires that a participant not act as an advocate for any particular group or

organisation, and that conflicts of interest do not result in selective or biased interpretation of

scientific information.

Independence – as it relates to science quality assurance and peer review processes, means that the

evaluation of the quality of research and scientific information is conducted by persons who were

not involved in producing the information being reviewed, and who do not have conflicts of

interest.

Integrity – refers to the security of information, and to the protection of information from

inappropriate alteration, selective interpretation or selective presentation. It must be ensured that

the information is not compromised or biased, particularly with regards to presenting uncertainty in

that information, to ensure that information remains complete throughout the science-to-decision

process.

DRAFT


Final v4.0 16

Objectivity – refers to whether the information presented is accurate, impartial and unbiased.

Objective interpretations or conclusions do not depend upon the personal assumptions, prejudices,

viewpoints or values of the person presenting or reviewing the information. Objectivity includes

whether the information is presented within a proper context. Sources of information should be

documented, so that the public can assess for itself whether there may be some reason to question

the accuracy of the data sources.

Peer Review – is a process of evaluation of research or scientific information by one or more

experts in the appropriate field, either with similar competence or in the same occupation,

profession or industry to the producers of the work. Peer review methods are employed to ensure

that the work meets appropriate or applicable standards of quality. Peer review usually emphasises

the importance of independence of the reviewers in order to obtain an unbiased evaluation,

recognising that a larger and more diverse group of people will usually find more weaknesses and

errors in research, and will make a more impartial evaluation of it, than the person or group

responsible for that research. There are many options for conducting effective peer review,

depending on the novelty and complexity of information.

Precision – the precision of a measurement system is the degree to which repeated measurements

under unchanged conditions show the same results. Precision does not necessarily imply accuracy:

a method may be precise, but may not be providing an accurate (true) measure. Measurements that

exhibit an unacceptably high level of imprecision are considered unreliable.

Quality – in relation to research and scientific information, is an encompassing term comprising

peer review, relevance, integrity, objectivity and reliability. Scientific information that meets these

requirements is considered to be robust and of high quality.

Relevance – refers to the usefulness of the information to its intended users, including government

decision-makers, , stakeholders and the public. Scientific research must be relevant to the fisheries

management question(s) being addressed, contributing directly to answering those questions and

addressing fisheries management objectives for the fishery of concern.

Reliability – relates to the accuracy and reproducibility of information. Research and scientific

information must be accurate, reflecting the true value of the results being reported within an

acceptable level of imprecision or uncertainty appropriate to the data and analytical methods used.

Information should not be biased and should not suffer from such a high level of imprecision that

the results and conclusions are rendered meaningless. Methods and models used to produce

scientific information must be verified and validated to the extent necessary to demonstrate that

results may be reliably reproduced by an independent scientific expert using the same data and

analytical methods.

Reproducibility – means that the scientific information is capable of being substantially

reproduced, subject to an acceptable degree of imprecision or error, by another expert working

independently from the expert who originally presented the information. With respect to analyses,

‘capable of being substantially reproduced’ means that independent analysis of the supporting data

using identical methods would generate similar results, subject to an acceptable degree of

imprecision or error.

Research – is a process of organised and systematic investigation or inquiry to find answers to

specific questions by establishing facts or principles. When research is conducted using scientific

methods, the resulting research results can be termed to be scientific information.

Scientific information – means any knowledge, facts or data that have been generated, tested and

verified using scientific methods. Scientific information includes, but is not limited to, factual

input, data, models, analyses, technical information, or scientific assessments, whether conveyed

through data compiled directly from surveys or sampling programmes, or through statistical

analyses and models that are mathematical representations of reality constructed using primary

data. In the context of these Guidelines, relevant fields of research and science include, but are not

limited to biology, ecology, oceanography, economics and sociology.

DRAFT


Final v4.0 17

Scientific method – is a systematic and cumulative process, employing a range of techniques to

acquire new knowledge, or to integrate or correct previous knowledge, by gathering observable,

empirical and verifiable evidence that is used in the formulation and testing of hypotheses.

Scientific methods must be objective to reduce biased interpretations of the results, and

methodological process steps must be reproducible. All data and methodologies must be

documented, archived and shared so that they are available for verification by other scientists, to

confirm the reproducibility of results, and to allow statistical measures of the precision or reliability

of these data to be established.

Transparency – a transparent peer review process is one that allows the public access to the results

of peer review working group, workshop or panel meetings, background documents and reports,

subject to relevant confidentiality requirements or agreements. Transparency also requires the

communication to the public in plain language of how decisions were reached, the presentation of

policies in open forums, and public access to the findings and advice of scientists as early as

possible.

• Validation – refers to the testing of analytical methods to ensure they perform as intended.

Validation should include evaluation of whether:

the analytical method has been programmed correctly in the computer software;

the accuracy of the estimates is adequate for the intended use;

the precision of the estimates is adequate; and

the estimates are robust to model assumptions.

• Verification – is the process of determining that the same results can be obtained from the

application of the same methods to the same data. Providing for verification requires that the

results, data and procedures used to produce the research and scientific information are

documented in sufficient detail to allow the reproducibility of the results to be tested by others,

within an acceptable degree of precision.

Date post:	30-Apr-2018
Category:	Documents
Upload:	trinhkien
View:	215 times
Download:	2 times

Research and Science Information Guidelines for Australian ... Projects/2014-009 - A.pdf · not...

Documents