ISSN 1946-7664. MCFNS 2015 AVAILABLE ONLINE AT HTTP://MCFNS.COM Submitted: Sep. 10, 2015Vol. 7, Issue 2, pp. 81–92 Accepted: Sep. 24, 2015Mathematical and Computational Published: Sep. 30, 2015Forestry&Natural-Resource Sciences Last Correction: Sep. 25, 2015

EVALUATING TRADITIONAL PEER-REVIEW PROCESSES ANDTHEIR ALTERNATIVES: AN OPINIONATED DISCUSSION

Aaron WeiskittelUniversity of Maine, School of Forest Resources, Orono, ME 04469 USA

Abstract. The advancement of science requires the timely and effective communication of importantfindings, which often takes the form of peer-reviewed journal articles. In the past decade, there has beensignificant changes in the world of scientific publishing with the rise of e-journals, open-access articles, anda greater volume of manuscript submissions. However, the overwhelming majority of journals (particularlyin forestry) rely on a traditional peer-review model, which is often inefficient and ineffective. In thisdiscussion, I evaluate the core assumptions of traditional peer-review processes, assess current alternativesto traditional peer-review, and provide recommendations for authors, reviewers, Associate Editors, andEditors. Overall, the intent of the discussion is to raise the importance of this issue and provide somesuggestions for change.

Keywords: Scientific publications and journals, pre- and post-publication review, high volumeand independent review

1 Background

Peer-reviewed publications have long been the ’goldstandard’ for communicating important scientific find-ings and are one of the primary metrics of a researcher’sproductivity. In the last two decades, there havebeen dramatic changes in how scientific informationis published, particularly with the tremendous rise inopen-access articles and purely electronic journals likeMathematical and Computational Forestry & Natural-Resource Sciences (MCFNS). In fact, MCFNS waslargely founded to improve how scientific findings wereevaluated and presented (Cieszewski and Strub 2009).This significant rise in new journals has coincided withan ever increasing volume of both manuscript submis-sions and resulting publications. For example, ForestEcology and Management transitioned from publishing31 articles in 1979 to over 545 articles in 2014, whichis a trend consistent (though not so dramatically) withother major journals in forestry (Figure 1).

Given an average acceptance rate of 20–40%, this sug-gests that most forestry journals are handling betweenapproximately 300 to 1,500 manuscript submissions peryear. Assuming that 30% of these submissions are re-jected without review, this means each forestry journalneeds an annual pool of 500 to 2,000 reviewers plusan editorial board of 20–30. Despite this tremendouschange in volume, the process of obtaining peer-review

evaluations has largely remained the same since its ori-gins in 17th century Europe and is relatively undiscussedin the literature (Lee et al. 2013). Recently, Kangas andHujala (2015) provided a succinct overview of currenttrends in scientific publishing including the need to re-assess peer-review.

In traditional peer-review, a submitted manuscriptis generally evaluated by the Editor and assigned toan Associate Editor (AE) whom is tasked with find-ing two or more independent reviewers and making arecommendation on acceptance or rejection. Althoughthere are alternatives to this model (discussed below),nearly all journals in forestry and natural resources usea traditional peer-review process. In contrast, MCFNSuses a hybrid system that combines the advantages ofthe traditional peer-review process and alternative ap-proaches including open public peer-review (Cieszewskiand Strub 2009). Given the recent changes in the scien-tific publishing process, I believe it is time to reconsiderhow peer-review is conducted and acknowledged (e.g.MCFNS ).

In this discussion, I highlight the primary assump-tions of traditional peer-review processes, evaluate alter-natives to it, and provide recommendations to authors,reviewers, AEs, and Editors. This discussion is basedlargely on my experiences as an author, reviewer, andAE for several major forestry journals as well as on the

Copyright c© 2015 Publisher of theMathematical and Computational Forestry & Natural-Resource SciencesWeiskittel (2015) (MCFNS 7(2):81–92). Manuscript Editor: MCFNS Editor

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 82

current scientific literature on the topic. For example,I have been an author/co-author, reviewer, and AE for60, 139, and 140 manuscripts since 2008, respectively.

0

200

400

600

2000 2005 2010

Year

No. of C

itable

Ite

ms / N

o. of F

ore

str

y P

hD

Aw

ard

ed

Journal

AFSCJFRFORECOFSTREE

Figure 1: Number of citable items by year from 1997to 2014 for the major forestry journals including An-nals of Forest Science (AFS), Canadian Journal of For-est Research (CJFR), Forest Ecology and Management(FORECO), Forest Science (FS), and Trees-Structure &Function (TREE). The black line is a linear smoothedtrend line, while the red dashed line is the numberof forestry PhD degrees awarded by National Associa-tion of University Forest Resource Programs (NAUFRP)from 2004 to 2014. The citable item data was obtainedfrom InCitesTM Journal Citation Reports provided byThompson Reuters and the degree data was from theUSDA Food and Agriculture Education InformationSystem (FAEIS; http://faeis.ag.vt.edu/faeisrpt.cfm)

2 Traditional Peer-Review

Whether the peer-review process is open (authors andreviewers are disclosed), single-blind (authors disclosedand reviewers anonymous), or double-blind (both au-thors and reviewers are anonymous), the process is of-ten inefficient and many times, ineffective. This is be-cause traditional peer-review processes are based on sev-eral key assumptions including: (1) there is willing andqualified peer-reviewers to call upon for each submit-ted manuscript; (2) reviewers are internally incentivizedto do quality work; (3) the process is unbiased; and (4)double-blind reviews with multiple reviewers is best. Os-

terloh and Frey (2015) also identified several other fun-damental flaws of traditional peer-review processes in-cluding low prognostic quality, inconsistency over time,a lengthy and expensive ordeal with potential for signif-icant delays, but only the four primary assumptions arediscussed in detail below.

2.1 Willing and Able Reviewers Despite thetremendous rise in manuscript volume, there has notbeen a similar increase in the number of qualifiedpeer-reviewers. In fact, the number of qualified peer-reviewers has likely significantly decreased given ob-served trends in undergraduate (Sharik et al. 2015) andgraduate (Sharik and Lilieholm 2012) student enroll-ment in forestry and related natural resources fields,which have both steadily declined since 1980 in theUnited States. In addition, the US Forest Service hasseen a 50% reduction in research scientists over the last25 years (FRAC 2012). Although this trend might bespecific to the United States, it does suggest a significantshift in research capacity in forestry. Consequently, thisshift has made finding willing reviewers for submittedmanuscripts rather difficult. Based on a sample of 64manuscripts I have handled and sent out for peer-reviewas an AE for 4 international forestry journals (Annals ofForest Research, European Journal of Forest Research,Forest Science, and New Zealand Journal of ForestrySciences) between 2011–2015, I have experienced an av-erage rejection or no response rate by potential reviewersof 47.8 ± 25.6% (mean ± SD). Based on this figure, Iwould need to contact 4.8 and 7.2 individuals to obtaintwo or three willing reviewers, respectively. In reality,these numbers are likely higher as I am quite selectivein the individuals I invite to review and make the besteffort to align authors and reviewers. Regardless, giventhe size of the forestry profession and particularly sub-disciplines like forest biometrics, these figures are highlyunsustainable and not likely to improve.

Although these figures are poorer than the 20% poten-tial reviewer rejection rate previously reported by a sur-vey of scientists (Ware 2008), they are much better thanwhat has been given for other fields. For example, thejournal Ecology recently reported an average potentialreviewer rejection rate of 70% (Brotons 2015). In fact, arecent manuscript submitted to a prestigious ecologicaljournal was rejected because 15 potential reviewers in arow had refused to review it (Brotons 2015). This posesthe interesting dilemma of whether manuscripts shouldbe evaluated by the willingness of reviewers to evaluateit (Brotons 2015). For several manuscripts I have han-dled in the past few years, it is not uncommon to havecontacted over ten individuals to find two willing review-ers and the manuscripts have often been rejected in the

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 83

end, which suggests that reviewer willingness to reviewmight be one indicator of manuscript quality.

In my own experience as an AE, many of the will-ing reviewers are graduate students and other youngresearchers who have the time and desire to con-tribute, which is an observation that is supported bythe ’principal-agent problem’ (agent agrees to work infavor of another party in return for some incentives)in economics (Garcıa et al. 2015). This also createsan interesting dilemma as should relatively inexperi-enced researchers be given the critical duty of evaluat-ing other potential scientific contributions that they maynot fully understand or be confident enough to criticize?Recently, the publisher Springer introduced the PeerReviewer Academy (http://academy.springer.com/peer-review-academy) that instructs individuals how to con-duct and write a review, likely as means for increasingthe pool of qualified reviewers. Despite the potentiallimitations of using graduate students and other earlycareer scientists as reviewers, peer reviewing is a criti-cal part of their education and training. However, forit to be truly effective, students and other early careerscientists should be well mentored and supervised fortheir first few reviews as outlined by Browman (2004).This includes discussing the role of reviewers, the ethicsinvolved, and the details of specific guidelines (Brow-man 2004). In fact, the majority (68%) of respondentsin a large international survey felt that formal trainingwould improve the quality of reviews and over 89% of therespondents less than 36 years in age indicated enjoyingthe review process (Mulligan et al. 2013).

Identifying and selecting suitable peer reviewers is adifficult task and one that many AEs struggle with. Infact, the majority (58%) of respondents in a large in-ternational survey indicated the primary reason for de-clining a review request was that the submission wasoutside of their area of expertise (Mulligan et al. 2013),which suggests that there is a potential issue with align-ing reviewers with submitted manuscripts. Regardless,potential reviewers are often identified based on thosesuggested by the manuscript authors or determined bythe AE as they were either cited in the submittedmanuscript and/or have previously published researchon the topic. However, rarely are the full credentialsof the individual taken into consideration before beinginvited to review a manuscript. This flaw has been ex-ploited as some authors have been caught reviewing theirown papers with the creation of multiple false onlineidentities (Ferguson et al. 2015). This highlights thelengths that some individuals are willing to go in orderto be ’peer-reviewed’, and places an extra burden onAEs to not use reviewers suggested by the manuscriptauthors, particularly when handling a manuscript that isoutside of their expertise area (which is often the case).

Resolving this limitation will likely require moving toan alternative method of peer-review or finding a betterway to align reviewers with manuscript submissions.

Overall, manuscript submissions are numerous andthe number of available/willing reviewers are not. Infact, the most productive reviewers are often overloaded(Figure 2; Ware 2008), which is a condition I commonlyrefer to as ’peer reviewer fatigue’ and is something I havepersonally reached in my own short career.

Year Year

19901990 20102010

10

20

30

40

200

400

600

Requested

Accepted

Status

Cu

mu

lativ

e N

o o

f Ma

nu

scrip

ts

No

of M

an

uscrip

ts

Figure 2: Observed annual (left) and cumulative (right)trends in review requests (blue) and accepted (red) as-signments of a prominent North American forest biome-trician by year from 1991 to 2015.

2.2 Incentivized Reviewers Like many natural re-sources issues, peer-review has been described asthe ’tragedy of the reviewer commons’ (Hochberg etal. 2009) as a core assumption is that a pool of willingand able reviewers is available to evaluate each and ev-ery submitted manuscript with limited acknowledgmentof actual reviewer contributions. Currently, credit is pri-marily given to number of publications (i.e. ’publish orperish’) and not the number of completed manuscriptreviews. This highlights a primary issue with the tradi-tional peer-review process; i.e. what incentivizes review-ers to do quality work, given that their contributions arelargely unacknowledged (i.e., the ’principal-agent prob-lem’; Garcıa et al. 2015).

Currently, many journals offer incentives to poten-tial reviewers such as the Canadian Journal of ForestResearch’s offer to provide a ’free reproduction of onecolour plate (a value of $950) in the next article you pub-lish in the Canadian Journal of Forest Research, if youreturn your completed review within 2 weeks.’ However,these incentives are largely based on time to respondand not the actual quality of the review. Journals gen-erally acknowledge reviewer contributions by providingthe names of all reviewers at the end of year and have re-cently started giving outstanding reviewer awards, but

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 84

professionally there is very limited acknowledgment ofthe time and effort required to conduct a quality review.

Ensuring quality is a difficult predicament for journalsbecause reviewers are generally volunteers and most ofthe journals have not adapted methods for incentiviz-ing volunteers. For example, one of the key elementsof ensuring higher-quality volunteer contributions is tohave systematic quality feedback systems (e.g., Moonand Sproull 2008), which most journals do not provide.However, the common practice of blind-carbon copyingreviewers on the journal decision letter to the authorsdoes allow reviewers to compare their recommendationsto those of their peer(s). Interestingly, MCFNS andmany of the Ecological Society of America journals cur-rently allow AEs to rate individual reviewers based ontheir contributions. Although this is largely an internalsystem, it does prevent the use of non-useful or non-responsive reviewers in the future. In addition, the ex-istence of websites like publons (https://publons.com/)does offer the potential for verifying the contributionsof reviewers and providing reviewer merit rankings thatare publicly available, which may also help to improveprofessional acknowledgment. Others have suggested in-centivizing reviewers by establishing a reviewer exchangesystem in that authors must have accumulate credits fordoing reviews before being able to submit manuscriptsfor publication (Mutz 2015). However, both publonsand a reviewer exchange system focus primarily againon quantity rather than quality.

The current anonymity of reviewers and lack of ac-countability creates the potential for unprofessional sit-uations. Recently, a reviewer accused a former PhD stu-dent of mine of being a ’liar’ for accidentally using thewrong citation style. I have seen and heard about othertype of unnecessary and unprofessional behavior duringthe review process. Although there has probably beenan element of that type of behavior over time, I believethat it has potentially increased in recent years due tothe pressure to publish, higher submission volumes, andgreater demands on reviewers. Interestingly, other fieldshave also identified unruly and unprofessional reviewerbehavior as a current issue in the peer-review process(e.g. Glen 2014). This type of behavior is unneces-sary and creates a positive feedback loop, particularlyfor early career scientists, in that people believe that be-ing harsh and condensing are acceptable practices duringpeer-review.

Reviewers differ primarily in their thoroughness assome give rather brief assessments of the manuscript,while others provide multiple pages of detailed feedback.A survey of researchers indicated that the average reviewtakes 9 hours (median of 5 hours) to complete and theywere primarily motivated by altruistic reasons ratherthan self-interested ones (Ware 2008), which was re-

cently confirmed with an empirical analysis by Sugimotoand Cronin (2013) (i.e. no ’ego’ bias in reviewing). Al-though most manuscripts are between 10,000 and 15,000words, reviews are often 556.2 ± 528.9 (mean ± SD)words with no statistical difference in average reviewword counts between accepted and rejected manuscripts(Sugimoto and Cronin 2013). However, the maximumobserved review word count went from 843 words for ’ac-cept’ recommendations, to over 4,800 words for ’reject’recommendations (Sugimoto and Cronin 2013). Thishighlights the large range in quality and quantity of re-viewer feedback that can make decisions on manuscriptsquite difficult. Consequently, a review should be longenough to provide the authors with enough direction toimprove the manuscript, but concise enough in that theycan fully comprehend the recommendations.

In short, reviewers and AEs are largely volunteerswith limited incentives to do reviews, let alone doingquality reviews. Most reviewers and AEs are simplytrying to maximize expected return minus the cost ofeffort (Garcıa et al. 2015).

2.3 Unbiased Process Most judicial systems havean ’innocent until proven guilty’ philosophy, while manypeer-review situations have a ’guilty until proven inno-cent’ stance (Huntoon 2009). Consequently, peer re-viewers are assumed to be unbiased and correct in theirassessments, while authors generally must address andrefute each and every one of a reviewer’s comments. Of-ten it only takes one potential flaw raised by a reviewer,whether right or wrong, to lead to manuscript rejection.However, Lee et al. (2013) identified several sources ofpotential bias in the peer-review process with some ofthe most important being: (1) prestige, affiliation, na-tionality, language, and gender of the manuscript au-thor; (2) content-based (manuscript is consistent withcurrent standards); (3) ego (manuscript cites the jour-nal or reviewer); (4) confirmation (manuscript findingsalign with current beliefs); and (5) publication (generalpreference towards positive rather negative outcomes).

Based on a review of the existing literature, Lee etal. (2013) did not find strong evidence of bias during thepeer-review process, but did question whether impar-tially should be upheld as the ideal for peer-review andcalled for the need to further evaluate the peer-reviewprocess. Recently, Walker et al. (2015) found strong sup-port for biases caused by author gender and their institu-tion, while previous studies have supported the existenceof publication bias (e.g., Emerson et al. 2010). Con-sequently, some have considered publishing in a peer-review framework as a form of ’prostitution’ betweenauthors and reviewers (Frey 2003) as 25% of the au-thors in a survey of 173 published articles revised their

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 85

manuscript based on reviewer feedback, even if they feltthe changes were incorrect (Bedeian 2003).

Recently, Siler et al. (2015) found that many highlycited articles were surprisingly initially rejected and thenpublished elsewhere in a case study of 1,008 manuscriptsto three elite medical journals, which illustrates thepotential influences of the biases outlined by Lee etal. (2013). Siler et al. (2015) concluded that peer-reviewimproved the quality of publications and was effectiveat identifying the ’good’ contributions, but was not suc-cessful at identifying the outstanding or breakthroughcontributions. Hence, peer-review has the tendency tomaintain the ’status quo’.

2.4 Multiple, Double-blind Reviewers A stan-dard for most journals is to obtain at least two double-blind reviewers to reduce potential biases and ensure ef-fectiveness of the process, which is in agreement with themajority of researchers’ current beliefs (Ware 2008). Incontrast, most forestry journals are single-blind and gen-erally prefer two reviewers (but one reviewer has beenincreasingly used). The Society of American ForestersForest Science journal is one of the few forestry journalwith a double-blind approach and has a general policyof requiring three reviewers. Unfortunately, I believethat double-blind has limited effectiveness (and likely adetrimental effect) in small fields like forestry as it isrelatively easy to identify authors and can consequentlylead to flawed reviews, which is consistent with findingsin other fields (Hill and Provost 2003).

In addition, the policy of requiring three reviewerslike Forest Science burdens the system and creates sig-nificant delays, while offering limited value. Interest-ingly, Sugimoto et al. (2013) found rather high varia-tion in journal acceptance rates based on the numberof qualified reviewers, and although there were severalsignificant differences, there was no statistically signif-icant difference between the acceptance rate for no re-viewers (AE makes decision) and those with two review-ers at p >0.01. The observed median acceptance rateswere approximately 0.28, 0.40, 0.35, 0.22, and 0.25 formanuscripts with zero, one, two, three, and four or moreexternal reviewers, respectively (Sugimoto et al. 2013)Moreover, Sugimoto et al. (2013) found a weak correla-tion between journal metrics like impact factor and ar-ticle influence score and average acceptance rate, whichsuggests the limitation of using acceptance rate as a mea-sure of journal quality. When the pool of available re-viewers is relatively small, it appears there is a ratherlimited benefit to using more than two reviewers, par-ticularly in a double-blind framework.

One of the issues with having multiple reviewers is thegeneral low degree of consensus among reviewers on thesame manuscript (Lee et al. 2013) with default generally

being rejection in situations like this. Osterloh and Frey(2015) highlight several studies that evaluated reviewerreliability, and it has generally been found to be quitelow. In fact, Bornmann and Daniel (2009) highlight thekey element of luck in peer-review as they found 23%of the manuscripts in an international chemistry journal(Angewandte Chemie International Edition) would havehad a different outcome if a third reviewer was includedin the decision. Clearly, this highlights the imperfectionsof the traditional peer-review model and suggests someimprovements are necessary, which is in agreement themajority of scientists (Mulligan et al. 2013).

3 Alternatives to Traditional Peer-review

As discussed by Lee et al. (2013), several alternativesto the traditional peer-review process exist and include:(1) pre-publication; (2) post-publication; (3) high vol-ume; and (4) independent peer-review. Each is brieflydescribed below and their advantages and disadvantagesdiscussed (Table 1).

3.1 Pre-publication Peer-review In this model,a manuscript (pre-print) is generally posted onlineto a community of peers and feedback is provided.The revised manuscript can either be simply sub-mitted to an Editor who decides on publication, orsubmitted to a more traditional peer-review process.Examples of pre-publication peer-review are ArXiv(http://arxiv.org/) and the University of California’seScholarship (https://escholarship.org/). The advan-tages of this approach are that it relies on a commu-nity of engaged researchers and is more transparent asthe reviews are often posted online, while the disadvan-tages are that the same community gate-keepers coulddrive the process and a diverse as well as active com-munity is required. In addition, an excessive amountof reviewer feedback could be obtained making revisionsdifficult and tedious.

3.2 Post-publication Peer-review In this model(also known as open or public review), documents aremade available online and ratings and comments arepermitted (non-anonymous and/or anonymous), whichallows for a dialog between the authors and review-ers. The submitted documents can either be static ordynamic (e.g., Wikipedia) and can be retracted. Theadvantages of this approach are that there are no in-formation gate-keepers, it is interactive, and open toall. The disadvantages are that it requires an engagedcommunity, has limited filtering, and is prone to abuse.Examples include Faculty of 1000 (http://f1000.com/),PubPeer (http://pubpeer.com/), and journals such as

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 86

Table 1: Advantages and disadvantages of the various methods of peer-review.

Method Advantages DisadvantagesTraditional Widely accepted and regarded; Effective at de-

termining ’good’ papers; Generally leads to im-proved manuscripts

Time-intensive; Often maintains the status quo;Limited interaction between authors and re-viewers; Not effective at identifying outstandingor breakthrough papers; Heavily reliant on 2–3peer reviewers

Pre-publication

Allows vetting from a community of reviewers;Reviews are generally open and available; Re-viewers are genuinely interested in the work;Not limited to 2–3 peer reviewers

Community needs to be engaged; Manuscriptsoften submitted to a traditional peer-reviewoutlet; Reviewers may limit criticisms if feed-back is done openly; An excessive number ofreviews could be obtained

Post-publication

No delays or information gate-keeping; Mini-mizes potential biases; Allows for a greater di-alog between authors and reviewers; Open toall potential participants and not just scientists;Not limited to 2–3 peer reviewers

No filtering and significant potential for abuse;Requires an engaged community; reviewers maylimit criticisms if feedback and ratings are notanonymous

High-volume Focuses on methodological and presentation offindings rather than originality and uniqueness;Emphasis on efficiency

Limited interaction between authors and re-viewers; Primarily relies on 2–3 reviewers

Independent Reviews are independent of journal’s scope andmission; Relies on a community of engaged re-viewers; Not limited to 2–3 peer reviewers

Requires an engaged community of reviewers;Manuscripts often submitted to a traditionalpeer-review outlet; An excessive number of re-views could be obtained

Open Medicine (http://www.openmedicine.ca/). Al-though Kangas and Hujala (2015) supported this idea,no examples currently exist in forestry to my knowl-edge. At the University of Maine, the Center forResearch on Sustainable Forests is currently experi-menting with a post-publication review process. Thewebsite NorthEast Forest Information Source (NEFIS;http://www.nefismembers.org/) allows users to uploadboth previously published and unpublished documentsas well as rate and comment on documents relevant toforest managers in the region.

3.3 High Volume Peer-review The goal of thisapproach is to efficiently evaluate and publish a highvolume of scientific contributions. This is done byusing a professional editorial board who make aninitial evaluation and then assign it to an appropriateAcademic Editor if they feel the submission has merit.The Academic Editor then assigns reviewers much liketraditional peer-review, but the primary difference isthat the reviewers are instructed to evaluate solelyon scientific and methodological validity rather thanperceived impact or uniqueness. The advantages of thissystem are that it is built on efficiency and attemptsto minimize information gate-keepers, but it still

relies on invited peer reviewers and provides limitedinteractivity between reviewers and authors. Examplesinclude PLOS ONE (http://www.plosone.org/),Peer J (https://peerj.com/), and SAGE Open(http://sgo.sagepub.com/).

3.4 Independent Peer-review Although similar topre-publication peer-review, independent peer-review isslightly different as it is generally not associated with aparticular publisher or even journal. Like pre-publicationpeer-review, a submission is posted to a community ofpotential reviewers, comments on the submission are so-licited, and revisions to the submission made. Thesereviews can then be transferred to a traditional peer-review journal. The advantages are that it is an openprocess (like pre- and post-publication review), relieson a community of reviewers, the reviewers can selectmanuscripts that are of greatest interest to them, andreviews are independent of a particular journal’s missionand scope, while the disadvantages are that the commu-nity of reviewers must be relatively large and engaged,it assumes journals are receptive to this type of model,and limited incentives for reviewer participation still re-main. In addition, like pre-publication review, an ex-cessive amount of reviewer feedback could be obtained,

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 87

which can make reasonable revisions difficult. Examplesinclude Rubriq (http://www.rubriq.com/) and Peerage(https://www.peerageofscience.org/).

In particular, Peerage employs a rather unique modelin that both manuscript authors and fellow reviewersrate and critique other reviewers on the quality of theirassessment. This helps to ensure a quality review andprovides reviewers with much-needed evaluation on theirperformance as well as recognition for their efforts.

4 Recommendations

Based on my experience as an author, reviewer, andAE as well as my assessment of the existing literatureon peer-review, the following recommendations are pro-vided for authors, reviewers, AEs, and Editors.

4.1 Authors

• Be your own worst critic. Given the plethora ofjournals that exist today, some authors adapt a’submit and see’ attitude where they’ll simply ’shop’their manuscript around until it is finally accepted.This burdens the system and requires the involve-ment of multiple reviewers, so it is best to just sub-mit to the journal with the ’best fit’. In today’sdigital world, publishing in a ’top-tier’ journal doesnot mean the paper will be well cited. If the paperis meaningful, it will be found and cited, which isrelatively easy to track and document today withwebsites like Google Scholar. Obtaining ’friendly-reviews’ is a helpful and effective method that canhelp improve manuscript quality with minimal in-vestment, but is often skipped in the rush to ’sub-mit’ (Hochberg et al. 2009). Many organizationslike the US Forest Service require an ’internal’ eval-uation prior to submission and this should likely beadapted by other organizations too.

• Don’t simply recycle rejected papers. If a manuscriptis rejected, take the time to revise as suggested bythe reviewers and/or AE rather than simply submitto another journal. On multiple occasions as bothan AE and reviewer, I have received the exact samemanuscript I had recommended for rejection at an-other journal with absolutely no consideration ofmy original suggestions. This is truly unacceptable.In fact, some journal like Marine Ecology ProgressSeries require authors to upload any and all for-mer reviews on the manuscript and their replies tothe comments to avoid this type of situation (Ri-isgard 2003).

• Suggest multiple reviewers that are actually likelyto review your paper. Finding qualified reviewers

is difficult and a significant contributor to the de-lay in the traditional peer-review process. This isbecause AEs must literally ’hunt down’ potential re-viewers and often nag them to complete the review.A lot of this could be resolved if manuscript au-thors suggested reviewers that were actually likelyto review their manuscript. Senior researchers andother prominent figures in the field are unlikely toreview the manuscript and are probably not worthsuggesting. In addition, it is important to suggestindividuals that are relatively independent of theauthors to ensure a ’fair yet critical’ review and notsimply a ’friendly’ review.

• Push back when you feel a reviewer or AE is wrong.People make mistakes, but this often is not fullyacknowledged during the peer-review process as re-viewers often act as rulers not partial to criticalfeedback (Tsang and Frey 2007). Too often, a re-jected manuscript is simply submitted to anotherjournal and the process starts over until successful.Based on current journal rejection rates, Hochberget al. (2009) estimated that between 5 to 10 review-ers were required for every published article. In-stead, it would be more efficient to make AEs andEditors reconsider something if there were errorsin the original assessment. Alternatively, journalsshould be willing to assess revised manuscripts ifthe authors provide previous reviewer feedback andthe changes made rather than start the peer-reviewprocess anew.

• Review in proportion to your publications. Ageneral rule of thumb has been to review twicethe number of your publications, but some sug-gest this should be much higher. Tracking theratio of publications to reviews on websites likeResearchGate (https://www.researchgate.net/)would help shift the focus away from just pub-lications and allow a fuller acknowledgement ofreviewer contributions. Elsevier has recentlycreated the Reviewer Recognition Platform(http://www.reviewerrecognition.elsevier.com/),which like publons (https://publons.com/) recordsand awards merit badges for reviewers. Thesereviewer achievements should be listed on CVs andacknowledged by scientist evaluation committees.

4.2 Reviewers

• Adapt the Golden Rule. As suggested by McPeeket al. (2009), reviewers should ’review for others asyou would have others review for you.’ Glen (2014)indicated that this be updated to ’if you wouldn’tsay it in person, don’t say it in an anonymous re-

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 88

view.’ This includes refraining from insulting lan-guage, being constructive, and limiting insistence(Glen 2014). This is particularly important whenhandling manuscript submissions from early careerscientists (given that the mentors have fully vettedthe contribution). The goal of a review should beto help improve a manuscript and assess its generalmerit, which does not require belittling or other in-sulting forms of language.

• Respond to review requests. An important delayin the peer-review process is waiting to hear backfrom potential reviewers before inviting additionalreviewers. A colleague of mine has said repeat-edly that he can’t delete those review requests fastenough despite typically publishing four to six arti-cles per year. Even if you are unable to do the re-view, a quick reply helps to keep the process movingalong. Suggesting additional potential reviewers ishelpful too, as most AEs are handling manuscriptsthat are outside of their expertise.

• Complete the review if you agree to do it. Multipletimes a reviewer is quick to agree to do the review,but then becomes non-responsive when the reviewbecomes due. This is even more disruptive to thereview process than the last item listed above asgenerally three to four weeks have elapsed since thereviewer agreed and the AE now must either findsomeone else to do the review or make a recom-mendation based on a single reviewer’s feedback.Also, I don’t consider a simple ’looks fine’ or evenno comment as an acceptable review.

4.3 Associate Editors

• Do your job. The role of the AE is to interface be-tween the authors and journal Editor. To do this,you must find qualified reviewers with some exper-tise on the topic, ensure they provide a quality re-view, and make a recommendation based on theirfeedback. This generally includes synthesizing thefindings of the reviewers and helping the authors toidentify the key changes that are needed given thatreviewers may often provide conflicting recommen-dations. Too often, the AEs rely solely on the re-viewer feedback and increasingly, I have found my-self being the sole reviewer on a manuscript with nofeedback from the AE, which I don’t believe is fairto the authors or the peer-review process.

• Do not fear to reject without review. AEs generallyearn their appointment by demonstrating their abil-ity to produce high-quality publications and shouldknow what is publishable or not. Given that vol-ume is relatively high, reviewer availability low, and

manuscript quality varying widely, I do not believethat since a manuscript was submitted that it de-serves to be reviewed. Although I tend to give earlycareer authors the benefit of the doubt and obtainreviews when possible, I am quite careful to makesure that each and every manuscript is ready forreview before sending it out.

Recently, I reviewed two separate manuscripts for’top-tier’ forestry journals that had a non-nativeEnglish-speaking first author and a ’prominent’ co-author, but both manuscripts probably should nothave gone out for review due to significant language,presentation, and methodological issues, which im-plies that the co-authors never read the manuscript.However, if an AE rejects without review, theyshould at least provide sufficient justification fortheir decision and include several suggestions forimprovement rather than simply claiming poor fitfor journal or other meaningless feedback. Schimelet al. (2014) provided an interesting discussion onthe reject without review debate.

• Do not request reviews from fellow AEs. By the na-ture of the duty, most AEs are rather busy peopleand not likely to be available as reviewers, particu-larly in a pinch. Serving as an AE is a demandingand time-consuming assignment, which leaves lit-tle time to do additional reviews given that mostAEs have other full-time jobs. In particular, I be-lieve it is important to avoid inviting AEs to be re-viewers, particularly for the same journal they arecurrently serving. A journal I formerly served foras an AE has a tendency to request reviews fromfellow AEs and had no internal system for desig-nating members of its own editorial board. De-spite highlighting the need to change this, I re-ceived a flawed review from a fellow AE who wasserving as last minute stand-in for a reviewer thathad failed to deliver. I fear this happens all toooften. To prevent this, a journal should identify ed-itorial board members, notify AEs when they selecta fellow AE as a reviewer, and allow across-journalidentification of both reviewers as well as AEs forjournals that share a common online platform (e.g.http://www.editorialmanager.com/).

• Remove inappropriate or insulting reviewer feed-back. Glen (2014) highlights the detrimental effectsof harsh reviews, particularly on young researchers.This is often unnecessary and unprofessional, whileits removal often doesn’t alter the review. As men-tioned above, this type of language is particularlydetrimental for early career scientists as it demor-alizes them and creates a positive feedback in that

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 89

individuals believe it is acceptable to write harshlanguage in reviews.

4.4 Editors

• Be open to change. The world of scientific pub-lishing is currently changing rapidly and new waysof doing things are much needed. AEs have apretty good sense of current issues and their feed-back should be solicited regularly on what is and isnot working. When I became an AE for a certainforestry journal, I sent the Editor a reasonable listof potential ideas to consider and I received a longemail essentially detailing why most of the ideaswere not sound or unreasonable. Hence, I am nolonger an AE at that journal.

• Provide AEs with feedback. Just like reviewers, AEsneed to be incentivized and it is easy to simply letthem do their thing with limited interaction, exceptassignment of manuscripts. In the last five yearsI have been serving as an AE, I have never oncereceived any positive or negative feedback from theEditor, which makes me wonder if one or the otherof us is not doing their job. The type of feedback canrange from a simple note of appreciation to morequantitative assessments like response time, averagedays to decision, and workload.

• Consider new ways of reviewer evaluation. Cur-rently, most journals require reviewers to submita decision on manuscript acceptance. Instead, itmight be more effective to encourage reviewers tosimply provide suggestions for improvement, withthe decision placed on the AE or Editor (Armstrong1997). Alternatively, journals could move to an ’as-is’ review process where the recommendation is ei-ther accept or reject based on one round of revi-sions (journals today are requesting 2–4 rounds ofrevisions by reviewers) (Tsang and Frey 2007). Toensure coherent and concise reviews, Peerage usesa standard review template and limits reviews to1,000 words.

• Allow AEs and authors to rate reviewers. Asnoted by Garcıa et al. (2015), keeping detailedrecords on reviewers allows a better assessmentof their capabilities to provide quality reviews.Most journals just keep information on the num-ber of reviews completed, date of last review in-vitation/acceptance, and average days to respond,which are not particularly helpful in determiningif the individual will provide a quality review. Asstated above, Ecological Society of America jour-nals and MCFNS allow AEs to rate reviewers basedon the quality of their reviews, which is helpful in

selecting individuals who can make useful contri-butions to authors. To my knowledge, Peerage re-mains one of the few peer-review systems that al-lows both reviewers and authors to rate each other.

• Identify the handling AE and the reviewers on eachpublication. Although handling AEs are currentlyidentified in several forestry journals like Annalsof Forest Science, European Journal of Forest Re-search, and MCFNS, this should become standardpractice and be expanded to include the reviewers.This practice acknowledges the contributions of in-dividuals involved during the review process and al-lows accountability in the system. Also, if reviewerswere aware of this practice, they may limit unprofes-sional behavior. In a recent survey, 45% indicatedthat having your name published alongside the pa-per as a reviewer would make them much less or lesslikely to review a manuscript, while 18% respondedthat it would make them much more or more likelyto review a manuscript (Mulligan et al. 2013).

• Diversify editorial boards. Scientific publishing isa truly international practice now with a growingnumber of manuscript submissions and reviewersfrom non-Western countries. However, most edi-torial boards in forestry are primarily from West-ern countries. For example, the editorial boardsof Forest Science and European Journal of For-est Research are 92% North American and Euro-pean, respectively, with no representation of non-Western countries. In 2014, a prominent inter-national forestry journal had at least 35% of itssubmissions from non-Western countries. Diver-sifying editorial boards, particularly with greaterrepresentation of non-Western countries, can helphandle the greater number of manuscript submis-sions coming from non-Western countries. Cur-rently, MCFNS has one of the most diverse edi-torial boards in forestry with representation fromnon-Western countries, which should be replicatedby other journals in the field.

• Enact and enforce term limits on editorial boards.Editorial boards should be dynamic and reflectiveof the current composition of incoming manuscriptboth in terms of geography and topic. In contrast,most editorial boards are static and change onlyoccurs when someone resigns. Allowing for moreturn-over in editorial boards prevents stagnation,increases innovation, and maintains diversity. Inaddition, this promotes AEs with an ’end in sight’perspective rather than simple frustration with theendless and largely unacknowledged drudgery.

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 90

• Maintain and provide data on your journal. Al-though many journals are willing to report (andeven promote) their impact factor, rejection rates,and editorial boards, fewer provide data on averagereviewer response rates, time to decision, AE consis-tency, or country of origin for manuscripts. In fact,I requested this information from several forestryjournals in preparing this discussion and either re-ceived no response or a response suggesting thatthey did not track this information.

• Do not ’penalize’ efficient AEs with more assign-ments. AEs differ greatly in their responsivenessand most journals have a goal of providing authorswith a decision within 60 days of submission. Basedon data provided from a prominent forestry journalin 2014, the average time to first decision was nearly75 days with a range between 0 to well over 200 daysand relatively large variation between AEs. Conse-quently, Editors might avoid using ’slow’ AEs infavor of ’fast’ ones. For example, I have always no-tified the Editor that I would handle a set number ofmanuscripts per year (e.g. 10–15) before acceptingan AE position, but most Editors have exceededthat requested number by 100–200% with limitedability to decline potential assignments. This sug-gests that there is a need to increase the size of cur-rent editorial boards and better balance the work-load between AEs.

• Consider moving to an alternative method of peer-review. Despite the prevalence of traditional peer-review, there are advantages to the alternativemethods described above for both authors and re-viewers. Even hybridizing traditional peer-reviewwith an alternative method is a positive step for-ward. This can be as simple as allowing registeredusers to provide comments on a publication (e.g.,PubPeer) or publishing the un-edited reviews alongwith the final, accepted paper. This recommenda-tion was echoed in the review of Kangas and Hujala(2015), but nearly all journals in forestry (exceptMFCNS ) rely on a traditional peer-review process.

5 Conclusions

Peer-review has been likened to a mythology (Mac-donald 2014) and a ’sacred cow’ (e.g., Osterloh andKieser 2015), but still remains the most trusted sourceof scientific findings even in the digital era (Nicholaset al. 2015). The survey of Ware (2008) indicated that90% of the surveyed researchers believe that peer-reviewimproves the quality of the published paper and most(64%) were satisfied with the current system, but only32% of the respondents in a large international survey

across disciplines believe that the current system is thebest we can achieve (Mulligan et al. 2013).

As identified above, there are several reasons of con-cern with the current system and the situation is notlikely to improve with the increasing demand for peer-reviewed publications. Improving the situation canrange from relatively simple solutions like modifying howreviewers provide feedback (e.g., Tsang and Frey 2007)to changing the entire process with the use of an alterna-tive system. Likely, requiring reviewers to sign an oath(e.g., Aleksic et al 2014) is unnecessary, but might nothurt.

Ultimately, I think a key solution to the issue is in-centivizing authors that quality rather than quantitycounts. This is possible today because a variety of cita-tion metrics and altmetrics can be used to assess the rel-ative influence of publications (Sud and Thelwall 2014).Technology will continue to make tracking these metricspossible and provide addition assessment of ’impact’,but real change will only likely happen when scientistevaluation committees begin modifying their standardsand recognizing the importance of reviewing. This mightjust move the bar from being simply ’published’ to ’pub-lished and useful’ with a more limited burden for peerreviewers.

Acknowledgments

Thanks to all the reviewers who have helped me toimprove my own work (particularly the two anonymousreviewers on a previous draft of this manuscript) or makerecommendations on the work of others. This paperwas a product of a presentation at a Western Mensura-tionist meeting inspired by an invitation from Dr. KimIles. Assistance with the awarded PhD data for Figure 1was provided by Wanda Lindquist. An earlier draft wasimproved with feedback from Drs. Anthony D’Amato,Arne Pommerening, Chris Cieszewski, Chris Woodall,John Kershaw, John Paul McTague, Kim Iles, RobertLilieholm, and Robert Wagner. Funding was providedwith the University of Maine, School of Forest Resourcesand Maine Agricultural and Forest Experiment Station.

This work is also based upon research supported, inpart, by McIntire-Stennis Grant (ME041516) from theUSDA National Institute of Food and Agriculture. Thisis Scientific Contribution no. 3448 of the Maine Agri-cultural and Forest Experimentation Station.

References

Aleksic, J., Alexa, A., Attwood, T.K., Hong, N.C.,Dahlo, M., Davey, R., Dinkel, H., Forstner, K.U.,Grigorov, I., Heriche, J.K., Lahti, L., MacLean, D.,Markie, M.L., Molloy, J., Schneider, M.V., Scott, C.,

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 91

Smith-Unna, R., Vieira, B.M. 2014. The open sciencepeer review oath. F1000Res 3, 271.

Armstrong, J.S., 1997. Peer review for journals: Evi-dence on quality control, fairness, and innovation. Sci.Eng. Ethics 3, 63–84.

Bedeian, A.G. 2003. The manuscript review process:The proper roles of authors, referees, and editors. J.Manag. Inq. 12, 331–338.

Bornmann, L., Daniel, H.D. 2009. The luck of the refereedraw: the effect of exchanging reviews. Learn. Publ.22, 117–125.

Brotons, L. 2015. Peer-review warning: system error,reviewers not found. Front. Ecol. Environ. 13, 241–242.

Browman, H.I. 2004. The rules of the game in sciencepublishing. Mar. Ecol. Prog. Ser. 270, 267–268.

Cieszewski, C.J., Strub, M.R. 2009. The foundation ofMathematical and Computational Forestry & Natu-ral Resources Sciences (MCFNS). Math. Comput. For.Nat.-Res. Sci. 1, 41–46.

Emerson, G.B., Warme, W.J., Wolf, F.M., Heckman,J.D., Brand, R.A., Leopold, S.S. 2010. Testing for thepresence of positive-outcome bias in peer review: Arandomized controlled trial. Arch. Intern. Med. 21,1934–1939.

Ferguson, C., Marcus, A., Oransky, I. 2015. The peer-review scam. Nature 515, 480–482.

Forestry Research Advisory Council (FRAC)2012. Report to the Secretary of Agriculture.http://www.fs.fed.us/research/docs/forestry-research-council/reports/2012-report.pdf.

Frey, B.S. 2003. Publishing as prostitution? — Choosingbetween ones own ideas and academic success. PublicChoice 116 205–223.

Garcıa, J.A., Rodriguez-Sanchez, R., Fdez-Valdivia, J.2015. The principal-agent problem in peer review. J.Assoc. Inf. Sci. Technol. 66, 297–308.

Glen, A.S. 2014. A New “Golden Rule” for Peer Re-view?. Bulletin of the Ecological Society of America,95(4), 431–434.

Hill, S., Provost, F. 2003. The myth of the double-blindreview?: Author identification using only citations.ACM SIGKDD Explor. Newsl. 5, 179–184.

Hochberg, M.E., Chase, J.M., Gotelli, N.J., Hastings,A., Naeem, S. 2009. The tragedy of the reviewer com-mons. Ecol. Lett. 12, 2–4.

Huntoon, L.R. 2009. Tactics characteristic of sham peerreview. J. Am. Physicians Surg. 14, 64–66.

Lee, C.J., Sugimoto, C.R., Zhang, G., Cronin, B. 2013.Bias in peer review. J. Am. Soc. Inf. Sci. Technol. 64,2–17.

Kangas, A., Hujala, T. 2015. Challenges in publishing:producing, assuring, and communicating quality. SilvaFennica 49: 1304.

Macdonald, S. 2014. Emperor’s new clothes: The rein-vention of peer review as myth. J. Manag. Inq.doi:10.1177/1056492614554773

McPeek, M.A., DeAngelis, D.L., Shaw, R.G., Moore,A.J., Rausher, M.D., Strong, D.R., Ellison, A.M.,Barrett, L., Rieseberg, L., Breed, M.D., Sullivan, J.,Osenberg, C.W., Holyoak, M., Elgar, M.A. 2009. Thegolden rule of reviewing. The American Naturalist,173(5), E155–E158.

Moon, J.Y., Sproull, L.S. 2008. The role of feedback inmanaging the internet-based volunteer work force. Inf.Syst. Res. 19, 494–515.

Mulligan, A., Hall, L., Raphael, E. 2013. Peer review in achanging world: An international study measuring theattitudes of researchers. J. Am. Soc. Inf. Sci. Technol.64, 132–161.

Mutz, D.C. 2015. Incentivizing the manuscript-reviewsystem using REX. Polit. Sci. Polit. 48, 73–77.

Nicholas, D., Watkinson, A., Jamali, H.R., Herman, E.,Tenopir, C., Volentine, R., Allard, S., Levine, K. 2015.Peer review: still king in the digital age. Learn. Publ.28, 15–21.

Osterloh, M., Frey, B.S. 2015. Ranking games.Eval. Rev. 39, 102–129.

Osterloh, M., Kieser, A. 2015. Double-blind peer review:How to slaughter a sacred cow. In: Incentives andPerformance. Springer International Publishing, pp.307–321.

Riisgard, H.U. 2003. Misuse of the peer-review system:time for countermeasures? Mar. Ecol. Prog. Ser. 258,297–309.

Schimel, D., Strong, D.R., Ellison, A.M., Peters, D.P.,Silver, S., Johnson, E.A., Belnap, J., Classen, A.T.,Essington, T.E., Finely, A.O., Inouye, B.D., Stanley,E.H. 2014. Editors are editors, not oracles. Bulletin ofthe Ecological Society of America, 95, 342–346.

Sharik, T., Lilieholm, R. 2012. A national perspectiveon forestry education. West. For. 57, 1–5.

Weiskittel (2015)/Math.Comput. For.Nat.-Res. Sci. Vol. 7, Issue 2, pp. 81–92/http://mcfns.com 92

Sharik, T.L., Lilieholm, R.J., Lindquist, W., Richard-son, W.W. 2015. Undergraduate enrollment in natu-ral resource programs in the United States: Trends,drivers, and implications for the future of nat-ural resource professions. J. For. in press. doi:http://dx.doi.org/10.5849/jof.14–146.

Siler, K., Lee, K., Bero, L. 2015. Measuring the effec-tiveness of scientific gatekeeping. Proc. Natl. Acad.Sci. United States Am. 112, 360–365.

Sud, P., Thelwall, M. 2014. Evaluating altmetrics. Sci-entometrics 98, 1131–1143.

Sugimoto, C.R., Cronin, B. 2013. Citation gamesman-ship: Testing for evidence of ego bias in peer review.Scientometrics 95, 851–862.

Sugimoto, C.R., Lariviere, V., Ni, C., Cronin,B. 2013. Journal acceptance rates: A cross-disciplinary analysis of variability and relationshipswith journal measures. J. Informetr. 7, 897–906.doi:10.1016/j.joi.2013.08.007

Tsang, E., Frey, B.S. 2007. The as–is journal review pro-cess: let authors own their ideas. Acad. Manag. Learn.Educ. 6, 128–136.

Walker, R., Barros, B., Conejo, R., Neumann, K., Tele-font, M. 2015. Bias in peer review: a case study. F1000Res. 4, 21.

Ware, M. 2008. Peer review in scholarly journals: Per-spective of the scholarly community – Results from aninternational study. Inf. Serv. Use 28, 109–112.