The Future of Peer Review
Irene Hames, PhD, FSB @irenehames Editorial and Publishing Consultant
Council Member, COPE (Committee on Publication Ethics)
ORCID: http://orcid.org/0000-0002-3806-8786
EASE/ISMTE Conference, Blankenberge, 23 September 2013
Irene Hames, EASE/ISMTE, 23 Sept 2013 3
What is (editorial) peer review?
Peer review in scholarly publishing is the process by which
research output is subjected to scrutiny and critical
assessment by individuals who are experts in those areas.
(Hames, 2012, in Academic and Professional Publishing, Chandos Publishing, Eds
Campbell, Pentz and Borthwick, p.16)
and
…the critical assessment of manuscripts submitted to journals by
experts who are not usually part of the editorial staff
(ICMJE, International Committee of Medical Journal Editors, http://www.icmje.org/)
Enormous scale
~28,100 active scholarly peer-reviewed journals
publishing ~1.7-1.8 million articles a year
The STM Report: an overview of scientific & scholarly journal publishing, Nov
2012, M Ware & M Mabe
http://www.stm-assoc.org/2012_12_11_STM_Report_2012.pdf
Irene Hames, EASE/ISMTE, 23 Sept 2013 4
.
*Good practice and quality in peer review is
system and access- and business-model
independent*
Irene Hames, EASE/ISMTE, 23 Sept 2013 5
What do people think of peer review?
Editors … value it
The research community … does too Peer review survey: Ware and Monkman, 2007 data; Sense
About Science, 2009 data
85% & 82% - peer review greatly helps scientific communication
83% & 84% - without peer review would be no control in scientific
communication
accuracy and quality of work not peer reviewed cannot be trusted
89% & 91% felt own last accepted paper improved by peer review
Open access survey: Taylor & Francis Group 2012-13 data
authors rated ‘rigorous peer review’ most important service expected
when pay to publish OA (above rapid peer review & publication)
Irene Hames, EASE/ISMTE, 23 Sept 2013 6
“Peer review in scholarly publishing, in one form
or another, is crucial to the reputation and
reliability of scientific research” (Para 277)
Irene Hames, EASE/ISMTE, 23 Sept 2013 7
… but is dissatisfaction
12% (Ware & Monkman) and 9% (SAS) in the
two surveys
Only about a third in both surveys think current
system of peer review is best that can be
achieved
Researchers want to improve peer review, not
replace it
Irene Hames, EASE/ISMTE, 23 Sept 2013 8
Criticisms of peer review
Irene Hames, EASE/ISMTE, 23 Sept 2013 9
“Peer review is in
crisis”
“Publish all, filter
later”
Unreliable and unfair
No clear standards, idiosyncratic
Open to abuse and bias
Stifles innovation
Slow, causes delays in publication
Expensive and labour intensive
Reviewers overloaded, working ‘for free’
Almost useless at detecting fraud and misconduct
Can ‘fail’ in even the best-run systems [Image, Gideon Burton, Utah, USA (CC BY-SA 2.0)]
Critical role of the Editor
“…[peer review] works as well as can be expected. The critical feature
that makes the system work is the skill and insight of the editor. Astute
editors can use the system well, the less able who follow reviewer
comments uncritically bring the system into disrepute.”
(a respondent, Ware & Monkman, 2008, PRC peer review survey)
“Unfortunately, all too often editors relinquish their responsibilities and
treat the peer review process as a vote … the real problem is editors …
increasingly, one sees editors who don’t use any judgement at all, but
just keep going back to reviewers until there is agreement.”
(Dorothy Bishop, Professor of Developmental Neuropsychology, Oxford University, ‘In
defence of peer review’, comment 4 Jan 2011, to R Smith (2010) Breast Cancer
Research, 12(Suppl 4):S13 )
Irene Hames, EASE/ISMTE, 23 Sept 2013 10
Editors have to act as editors
Being an editor is:
not just moving manuscripts automatically
through the peer-review process
not just ‘counting votes’
not passing on editor responsibilities to
reviewers
making critical judgements (‘reviewers advise,
editors decide’)
Irene Hames, EASE/ISMTE, 23 Sept 2013 11
Some problems due to
Variable quality of peer review
Inconsistency in decision making
Lack of training for new editors
Perceived gaming by journals
Unethical behaviour
Irene Hames, EASE/ISMTE, 23 Sept 2013 12
Ethics and integrity
Important in research communication/publication
whatever the model
Lack of knowledge and training
COPE
Guidelines and resources
Irene Hames, EASE/ISMTE, 23 Sept 2013 14
.
‘COPE’s new Ethical Guidelines for Peer Reviewers: background, issues, and evolution’,
ISMTE, EON May 2013, Vol6, issue4, http://www.ismte.org/Shared_Articles-
COPEs_new_Ethical_Guidelines_for_Peer_Reviewers_background_issues_and_evolution/
Irene Hames, EASE/ISMTE, 23 Sept 2013 15
Two functions of peer review separated
Publication based on ‘soundness’ - research methodology,
results and reporting - not novelty, interest or potential impact
Evaluation of interest/impact left for post-publication
Irene Hames, EASE/ISMTE, 23 Sept 2013 17
Launched December 2006, now a ‘mega’ journal
Published 23,468 articles in 2012 (14,000 in 2011)
(~1.4% of world’s scholarly literature)
Used >60,000 reviewers in 2012 (38,400 in 2011) from
154 countries
Impact Factor ~4
‘PLOS ONE clones’ being launched (BMJ Open, Sage
Open, Scientific Reports, Biology Open, AIP Advances,
SpringerPLus)
18 Irene Hames, EASE/ISMTE, 23 Sept 2013
Eliminating ‘wastage’ of reviews
Rejected manuscripts can go from journal to
journal, fresh reviews at each
‘Cascading’ submissions and reviews
Within publishers and societies
Between publishers
• Neuroscience Peer Review Consortium
(http://nprc.incf.org/)
• eLife, BMC, PLOS and EMBO – ‘portable peer review’
Irene Hames, EASE/ISMTE, 23 Sept 2013 19
More transparent approaches
Publishing reviews, ms versions and editorial
correspondence, reviewers’ names may or may not be
revealed
BMC series medical journals – ‘pre-publication history’
The EMBO journals – ‘peer review process file’
BMJ Open – ‘peer review history’
eLife – decision letter + author response (have doi’s)
Reviewer interaction: pre-decision at The EMBO Journal
(‘cross-peer review’) and eLife
Reviewer + author + editor interaction: Frontiers
‘Open’ peer review
Irene Hames, EASE/ISMTE, 23 Sept 2013 20
TPJ anonymous reviewer
“That you would consider making a reviewer’s identity
known to the submitting authors is alarming in the extreme.
How can such a practice not but undermine the peer review
process and lower standards? ... I would hold the journal,
not the reviewers, responsible for the evaporation of your
journal’s credibility, and for an inexcusable erosion of the
philosophical framework of modern plant biology.”
Irene Hames, EASE/ISMTE, 23 Sept 2013 22
‘Peer review process goes meta via blogs and Twitter’
‘Ultra-open’ peer review at GigaScience July 2013
Reviewer blogged about a manuscript under peer-review
Shock, followed by, “This is fantastic!!”
“As we are promoting and encouraging more transparent
review and the use of pre-print servers, if the authors
and reviewers consent then we do allow open discussion
of the work prior to publication.”
http://blogs.biomedcentral.com/bmcblog/2013/07/23/ultra-open-peer-
review/
Irene Hames, EASE/ISMTE, 23 Sept 2013 23
Peer review doesn’t stop at publication
When real peer review starts?
Post-publication review and evaluation
Increasing opportunities for innovation
Challenges and problems
Increasing importance of blogs, twitter and
other social media (#arseniclife)
Irene Hames, EASE/ISMTE, 23 Sept 2013 26
.
“Online scientific interaction outside the traditional
journal space is becoming more and more
important to academic communication”
Mark Hahnel, founder, FigShare (http://figshare.com/)
Irene Hames, EASE/ISMTE, 23 Sept 2013 27
A big challenge - data
Massive amounts being generated
Recognition for producing, making usable by
others and curating
Where to put?
Dryad http://datadryad.org/ - international
repository of data underlying peer-reviewed
articles in basic and applied sciences; can be
made securely available for peer review
Irene Hames, EASE/ISMTE, 23 Sept 2013 28
Irene Hames, EASE/ISMTE, 23 Sept 2013 29
A reviewer’s point of view
‘Reviewers peering from under a pile of ‘omics’ data’ J.K.
Nicholson, Nature (2006), 440, 992
“The scientific community needs to reassess the
way it addresses the peer-review problem,
taking into account that referees are only human
and are now being asked to do a superhuman
task on a near-daily basis.”
Will peer review survive?
Mark Ware: ‘far from being in crisis, peer review remains
widely supported and diversely innovative’ Ware M (2011) New Review of Information Networking, 16(1): 23-53
Fiona Godlee: (BMJ Editor): ‘At its best I think we would
all agree that it does improve the quality of scientific
reporting and that it can improve, through the pressure of
the journal, the quality of the science itself and how it is
performed.’ Godlee F (2011) Evidence given to the UK House of Commons Science and
Technology Inquiry into Peer Review, 11 May 2001. Transcript of oral evidence,
HC856, Q97.
http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf
Irene Hames, EASE/ISMTE, 23 Sept 2013 30
In 2013 …
Irene Hames, EASE/ISMTE, 23 Sept 2013 31
2007: Peer review … at its best a very powerful and sophisticated tool
… since so much hinges on it, it is essential that it is carried out well
and professionally, and that it is viewed with confidence and respect.
But …
… what will it look like?
… who will be running it?
… how will it cope with the increasing volume
and range of research output?
Irene Hames, EASE/ISMTE, 23 Sept 2013 32
Irene Hames, EASE/ISMTE, 23 Sept 2013 33
Thank you!
Dr Irene Hames
email: [email protected]
twitter: @irenehames