Revision Process Does it have a real effect on an article’s impact? Faculdade de Medicina da...

Revision ProcessDoes it have a real

effect on an article’s impact?

Faculdade de Medicina da Universidade do Porto

May, 2012

VAZ ANA| MANÉ FERNANDO | MARRANA FRANCISCO | SOARES JOANA | OLIVEIRA JOANA | PEREIRA JOÃO | SOUTO JOÃO| REBOLO JOSÉ | FERNANDES LUÍS | MACHADO MAFALDA | DIAS

SOFIA

[email protected]

Adviser: Pedro Pereira Rodrigues

Revision Process: Does it have a real impact on an article's impact?

2

SUMMARY

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION CONCLUSIONS

PHASE 1

PHASE 2

PHASE 3


3

INTRODUCTION

The idea of redefine FMUP’s

journal

4 Classes contributing with a

specific subject

Background – Revision process

Literature Research

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS


4

INTRODUCTION - MOTIVATION

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


5


Perfect one?

Different ways to manage a journal

Journals

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


6


NATURE developed a study about an open peer to peer review.

Evaluation of on-line comments

Submission

Standard review Open review

E-mails were sent to subscribers

Evaluation of standard comment

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


7


NATURE developed a study about an open peer to peer review.

Results High number of visitorsX Low number of commentsX Low tecnical value in comments

Authors interviewX Afraid of having their ideas stolen Some thought it was useful

ConclusionNature decides NOT to implement this method… yet!

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


8


THE NEW ATLANTIS developed a study about the effects of knowing the provenance of the article.

Author associated with prestigious institution

Approval

Same article submitted to the same journal using peer review system

Independent author

Disapproval

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


9


THE NEW ATLANTIS developed a study about the effects of knowing the provenance of the article.

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY

ConclusionThere can be a huge bias introduced in the process in knowing the provenience of an article.


10

RESEARCH QUESTION

Revision Process: Does it have a real effect on an

article’s impact?

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


11

AIMS

Literature Research

To know what has already been studied and is described in the available literature about the revision processes and its relation with the impact factor;

Find out if the REVISION PROCESS can be

associated with the IMPACT FACTOR of a scientific article. Contribute to FMUP’s journal

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


12

METHODSWe set out to find any information we could regarding this subject in the available bibliography.

QueryTOPIC ("scientific articles" OR articles OR papers OR "journal

articles“)AND

TOPIC ("process of revision" OR "review process" OR "revision process" OR "process of review" OR "peer review" OR "peer

revision“)AND

TOPIC (impact factor" OR "number of citations" OR impact OR citation)

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


13

METHODSLiterature Analysis Q

uery

816 articles

Analysis

Exclusion criteria• Not mencioning a revison process;• Not mencioning the impact factor;• Irrelevant main topic;• Not acessible;• Not being written in english or portuguese.

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


14

Study 1Literature Analysis – Phase 1


15

Study 1Literature Analysis – Phase 2


16

METHODS

Final included articles were submited to a qualitative and quantitative analysis, concernig crucial information to find out if there are any coherent conclusions in the available

literature. AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYPhase 3


17

METHODS

Quantitative analysis

Characterize the included articles Publication journal; Publication journal’s Impact factor factor (in

the year the article was published); Year of publication; Journal’s classification in medical or non-

medical (following ISI Web of Knowledge criterions).

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYPhase 3


18

METHODS

Does it mention peer review? Does it criticize/council peer review? Does it suggests improvement paths? Empirical/opinion article?

Does it concludes any association between the impact factor and the review process?

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYPhase 3



19

METHODS

Data was analyzed using IBM SPSS Statistics 20.AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYPhase 3



20

RESULTS


21

RESULTS – QUANTITATIVE ANALYSIS

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY

n MedianPercentile

5Percentile

95

Year of publication 29 2008 1995,50 2011,50Impact factor of the publication journal 29 2,167 0,515 15,994

Yes (n(%))Is it from a medical journal?

29 9 (31)

Does it mention peer review?

29 27 (93)

Does it criticize peer review?

29 14 (48)

Does it council peer review?

17 7 (41)

Does it suggest any improvement paths?

29 14 (48)

Is it an opinion article? 29 11 (38)

Is there empirical data? 29 24 (83)

Is there any conclusions relating IF and the RP?

27 15 (56)

Table 1 – Frequency table.


22

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYPublication journal n (%)

Scientometrics 5 (17)

PlOS ONE 3 (10)

JAMA 2 (7)

Archivum immunologiae et therapiae

experimentalis2 (7)

BMC Med Res Methodol 2 (7)

Others 15 (52)

Table 2 - Distribution of the included articles per publication journal. Others - journals with only one article published.

Author of the article n (%)

Bornmann L. 6 (21)

Rowland F. 2 (7)

Others 21 (72)

Table 3 – Distribution of the included articles per author.

Table 3 – Distribution of the included articles per author.



23

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


Graph 1 - Distribution of the included articles per publication year.


24

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


Graph 2 - Distribution of the included articles per impact factor of the publication journal.


25

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


Does it mention peer review?

Is there any conclusions

relating the IF and the revision process?

Fischer’s exact test

No (n(%)) Yes (n(%))

n Exact. Sig. (2

tailed)

No 2 (100) 0 (0) 27 0,188

Yes 10 (40) 15 (60)


No 5 (56) 4 (44) 16 0,034

Yes 0 (0) 7 (100)


No 6 (40) 9 (60) 27 0,707

Yes 6 (50) 6 (50)

Is it from a medical journal?

No 5 (28) 13 (72) 27 0,037

Yes 7 (78) 2 (22)

Table 4 - Comparison between different articles’ characteristics within the groups concluding or not an association between the impact factor and the revision process. Fischer’s exact test - significance level of 0.05.


26

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


Does it suggest improvement paths?

Is there any conclusions

relating the IF and the revision process?


No (n(%)) Yes (n(%))

n Exact. Sig. (2

tailed)

No 7 (47) 8 (53) 27 1,00

Yes 5 (42) 7(58)

Is it an opinion article?

No 7 (41) 10 (59) 27 0,706

Yes 5 (50) 5 (50)

Does it have

empirical data?

No 2 (50) 2 (50) 27 1,000

Yes 10 (44) 13 (57)

Table 4 - Comparison between different articles’ characteristics within the groups concluding or not an association between the impact factor and the revision process. Fischer’s exact test - significance level of 0.05.


27

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY



Is there any conclusions relating the IF and the revision

process?


No (n(%)) Yes (n(%))

n Exact. Sig. (2 tailed)

No 6 (40) 9 (60) 23 0,685

Yes 4 (50) 4 (50)


No 4 (57) 3 (43)

13 0,070 Yes

0 (0) 6 (100)

Table 5 - Comparison between different articles’ characteristics within the groups concluding or not an association between the impact factor and the revision process, considering only articles with empirical data. Fischer’s exact test - significance level of 0.05.


28

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY




Does it council peer review? No (n(%)) Yes (n(%))

n Exact. Sig. (2

tailed)

No 5 (50) 5 (50) 17 1,00

Yes 4 (57) 3 (43)

Table 6 – Comparison between the variables: council or not peer review and critize or not peer review. Fischer’s exact test - significance level of 0.05.


29

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


Table 7 - Comparison between some characteristics of the articles and articles published or not in a medical journal.Fischer’s exact test - significance level of 0.05.

Does it counsil peer review?

Is it from a medical

journal?


No (n (%))

Yes(n (%))

n Exact Sig. (2-

tailed)

No5 (50) 5 (50)

17 0,044Yes

7 (100) 0 (0)


No11 (73) 4 (27)

29 0,700Yes 9 (64) 5 (36)


30

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYQuantitative analysis – Significant results

Articles that council peer review, are more likely to conclude some relation between the IF and the revision process. Articles from medical journals are more likely to conclude that there is no association between the IF and the revision process. Articles from medical journals will more likely not to council peer review.



31

RESULTS– QUALITATIVE ANALYSIS

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY

14 articles had suggestions to improve the revision process…


32

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYPossible paths of improvement

Use an editor or editorial board to prescreen and remove manuscripts of low suitability; Use a three-of-three or four-of-four decision rule when deciding on the acceptance of papers; Use a stricter decision rule for repeat submissions; Conceal the author’s name and afiliation; Reduce the length of the manuscript review periods as well as publication lags.

RESULTS– QUALITATIVE ANALYSIS


33

CONCLUSIONS

Revision process: does it have a real influence on an article’s impact?

It certainly affects the quality of the article... 56% articles agreed that there is, in fact, a tight relation between these two concepts.

The problem with this conclusion is in the wide spread of opinions on HOW it affects the article and HOW it can be improved to maximize the quality of the published articles.

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


34

CONCLUSIONS

Study LimitationsInaccessible articles (1,22%);Limited time;The concept of “Impact factor”.

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY

Several articles criticized the impact factor as a way of measuring the quality of an article. The authors often say that it is only a reflection of the visibility of the article. So, it is highly dependent on the visibility of the publication journal.


35

CONCLUSIONS

Suggestions for future researches Journal’s research: evaluate if what journal’s

with a high impact factor range are doing about the revision process is in anyway different between high and low impact factor journals and if it is concordant with what is described in the actual literature.

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


36

ACKOWLEDGEMENTS

“We would like to thanks our Professor Pedro Pereira Rodrigues for his guidance and help and also to

Professor Liliana Leite.Our ackowledgements are also directed

to Professor Altamiro Pereira,

whose advices were precious during our

research planning.”

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY


37

REFERENCES

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARY

Allen, L., C. Jones, et al. (2009). "Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs."B. D. Neff and J. D. Olden “Is peer review a game of chance?”Bornmann, L. (2010). "Does the Journal Peer Review Select the "Best" from the Work Submitted The State of Empirical Research."Bornmann, L. and H.-D. Daniel (2010). "The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal."Bornmann, L. and H.-D. Daniel (2010). "The validity of staff editors' initial evaluations of manuscripts: a case study of Angewandte Chemie International Edition."Bornmann, L., H. Schier, et al. (2011). "Is Interactive Open Access Publishing Able to Identify High-Impact Submissions? A Study on the Predictive Validity of Atmospheric Chemistry and Physics by Using Percentile Rank Classes."Bornmann, L., R. Mutz, et al. (2011). "A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication?"Bornmann, L., W. Marx, et al. (2010). "From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics."Citrome, Leslie (2007). Impact factor? Shmimpact factor!: the journal impact factor, modern day literature searching, and the publication processClarke, M, 2006, 20th Dec., Report of Nature's peer review trialCMAJ, [Consult in 27-10-2011],Dittrich, J. (2010). "Paper Bricks: an Alternative to Complete-Story Peer Reviewing."Floyd, R. G., K. M. Cooley, et al. (2011). "An overview and analysis of journal operations, journal publication patterns, and journal impact in school psychology and related fields." Journal of school psychology 49(6): 617-647.Haeffner-Cavaillon, N. and C. Graillot-Gak (2009). "The use of bibliometric indicators to help peer-review assessment." Archivum Immunologiae Et Therapiae Experimentalis 57(1): 33-38.Honekopp, J. and J. Kleber (2008). "Sometimes the impact factor outshines the H index."Jackson, J. L., M. Srinivasan, et al. (2011). "The Validity of Peer Review in a General Medicine Journal." Plos One 6(7).Khosrowjerdi, M., N. Zeraatkar, et al. (2011). "Publication Delay in Iranian Scholarly Journals." Serials Review 37(4): 262-266.Laband, D. N. and M. J. Piette (1994). "A CITATION ANALYSIS OF THE IMPACT OF BLINDED PEER-REVIEW." Jama-Journal of the American Medical Association 272(2): 147-149.Lee, C. (2012). "Open peer review by a selected-papers network." Frontiers in Computational Neuroscience 6.Lee, C. (2012). "Open peer review by a selected-papers network." Frontiers in Computational Neuroscience 6.M. S. Patterson and S. Harris. The relationship between reviewers' quality-scores and number of citations for papers published in the journal Physics in Medicine and Biology from 2003-2005Marcus A., Oransky I. (2011) “Science publishing: The paper is not sacred” Nature 480, 449–450

http://www.nature.com/nature/journal/v480/n7378/full/480449a.html




38

REFERENCES

AIMS

METHODS

RESULTS

ACKNOLEDGEMENTS

REFERENCES

SUMMARY

INTRODUCTION

RESEARCH QUESTION

CONCLUSIONS

SUMMARYMarotti M., (2011) “Primum non nocere: authors or reviewers?” Rev. Assoc. Med. Bras. vol.57 no.5 São PauloMoed, Henk F. "New developments in the use of citation analysis in research evaluation"Moret B., (1997). “ACM’s Journal of Experimental Algorithmics Bridging the Gap Between Theory and Practice.” JEP - the Journal of Electronic Publishing, 3(1)Nadasky, Z. (1997). “. A Truly All-Electronic Journal: Let Democracy Replace Peer Review” JEP - the Journal of Electronic Publishing, 3(1)Robertson M. (2011) “Pit-bull reviewing, the pursuit of perfection and the victims of success.” Rowland, F. (2002). "The peer-review process." Learned Publishing 15(4): 247-258.Rowland, F. (2003). Peer review of electronic journals.Starbuck, W. H. (2005). "How much better are the most-prestigious journals? The statistics of academic publication." Organization Science 16(2): 180-200.Szklo, Moyses. (2006). "Quality of scientific articles"T. Montini, C. Mangurian and L. A. Bero. "Assessing the evidence submitted in the development of a workplace smoking regulation: The case of Maryland"The Editors of The New Atlantis (2006). "Rethinking Peer Review," The New Atlantis, Number 13, pp. 106-110Van Raan, A. F. J. (2006). "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups."Varian, H, 1997. “The Future of Electronic Journals. Scholarly Communication and Technology”, Emory University (EUA), Wager, E., E. C. Parkin, et al. (2006). "Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study."Weber, E. J., P. P. Katz, et al. (2002). "Author perception of peer review - Impact of review quality and acceptance on satisfaction."Wets, K., D. Weedon, et al. (2003). "Post-publication filtering and evaluation: Faculty of 1000."Wood, M., M. Roberts, et al. (2004). "The reliability of peer reviews of papers on information systems."Wouters, P. (1997). "Citation cycles and peer review cycles."Yousefi-Nooraie, R., B. Shakiba, et al. (2006). "Country development and manuscript selection bias: a review of published studies."

http://www.ncbi.nlm.nih.gov/pubmed?term=%22Robertson%20M%22%5BAuthor%5D

Date post:	02-Jan-2016
Category:	Documents
Upload:	alexis-allen
View:	222 times
Download:	0 times

Revision Process Does it have a real effect on an article’s impact? Faculdade de Medicina da...

Documents