What's wrong with our scholarly infrastructure?

transcript

What’s Wrong with Us?Or:

How accountability is killing science

Björn BrembsUniversität Regensburg

http://brembs.net - @brembs

SCHOLARSHIP

Scientists produce publications, data and code

CROWN JEWELS

PRECIOUS

PROBLEM I

Dysfunctional scholarly literature

Literature• Limited access

Literature• Limited access• No global search

Literature• Limited access• No global search• No functional hyperlinks

First demonstration: 1968 WWW: 1989

Stanford Research Institute: NLS Tim Berners-Lee: CERN

Literature• Limited access• No global search• No functional hyperlinks• No data visualization

Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards

Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics

Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics• No text/data-mining

Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics• No text/data-mining• No effective way to sort, filter and

discover

Antiquated Functionality

…it’s like the web in 1995!

• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics• No text/data-mining• No effective way to sort, filter and

discover• No scientific impact analysis• Lousy peer-review • No networking feature• Etc.

PROBLEM II

Scientific data in peril

Small Data – Long Tail

Report on Integration of Data and Publications, ODE Report 2011http://www.alliancepermanentaccess.org/wp-content/plugins/download-monitor/download.php?id=ODE+Report+on+Integration+of+Data+and+Publications

October 1-16, 2013

PROBLEM III

Non-existent software archives

UNIVAC (A-2) 1953

Dysfunctional Infrastructure

• Institutional email• Institutional webspace• Institutional blog• Library access card• Open access repository

• Publications?• Code?• Data?

HISTORICAL LESSON

Don‘t let someone with orthogonal interests touch your precious

They don‘t know science

They don‘t understand elementary science

http://www.sciencedirect.com/science/article/pii/S0092867415002421

They don‘t understand elementary science

http://science.sciencemag.org/content/351/6274/737.full

They support arms trade

They support collusion in torture• 2013: US$86m of US$126m annual APA budget comes from publishing

They fake journals

“This was an unacceptable practice, and we regret that it took place.”

Michael Hansen, CEO Of Elsevier's Health Sciences Division

They hold science hostage

They detest the scientific method

#researchparasites

They promote sexism

They pay politicians to make OA illegal

Your tax dollars at work!

They parasitize public funds

Modified from ARL: http://www.arl.org/bm~doc/arlstats06.pdf, http://www.arl.org/bm~doc/arlstat08.pdf

400Subscription pricesCPI/inflationJournals purchased

(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)

rticle

Legacy SciELO

rticle

Legacy SciELO(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)

“The decision, based on market and competitor analysis, will bring Emerald’s APC pricing in line with the wider market, taking a mid-point position amongst its competitors.”

Emerald spokesperson

MONEY FOR NOTHING

Wasting billions on a parasitic industry

They sell bogus journal rankings

• Thomson Reuters: Impact Factor• Eigenfactor (now Thomson Reuters)• ScImago JournalRank (SJR)• Scopus: SNIP, SJR

Source Normalized Impact per Paper

Main Problems with the IF• Negotiable • Irreproducible • Mathematically

unsound

Negotiable• PLoS Medicine, IF 2-11 (8.4)

(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. http://www.plosmedicine.org/article/info:doi/10.1371%2Fjournal.pmed.0030291)

• Current Biology IF from 7 to 11 in 2003– Bought by Cell Press (Elsevier) in 2001…

Negotiable

https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/

June, 2014 (19 months)

Not Reproducible• Rockefeller University Press bought their data from Thomson Reuters• Up to 19% deviation from published records• Second dataset still not correct

Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091

Not Mathematically Sound• Left-skewed distributions• Weak correlation of individual

article citation rate with journal IF

Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497http://www.bmj.com/cgi/content/full/314/7079/497

Not Mathematically Sound

https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/

‘QUALITY’

Is journal rank like astrology?

Selectivity

http://blog.frontiersin.org/2015/12/21/4782/

Citations

The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vincent Lariviere, Yves Gingras arXiv:1205.4328

Methodology I

Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1002273

Methodology II

Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291

Methodology III

Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77

‘Quality’

Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847

Quality

Journal Rank

Berghmans et al. (2002): doi: 10.1093/annonc/mdg203, Tressoldi et al. (2013) doi:10.1371/journal.pone.0056180, Brembs et al. (2013) doi:10.3389/fnhum.2013.00291, Fraley & Vazire (2014) doi:10.1371/journal.pone.0109019, Macleod et al. (2015) doi:10.1371/journal.pbio.1002273

Journal Rank and Fraud/Error

Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033

Journal Rank and Fraud/Error

Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033

Journal Rank and Retractions

Data from: Fang, F., & Casadevall, A. (2011). RETRACTED SCIENCE AND THE RETRACTION INDEX Infection and Immunity DOI: 10.1128/IAI.05661-11

Credit: Scott Edmund

INCENTIVES

“High-Impact” journals attract the most unreliable research

Productivity

Research questions:True:False:Significant:

100505021

44222219

PUBLISH OR PERISH

Quality & Productivity: Selecting the sloppy scientists

Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002165

Irreproducibility

61%(n=100)

Open Science Collaboration

Replication in Psychology

Open Science Collaboration Science 2015;349:aac4716

Replication in Psychology

Open Science Collaboration Science 2015;349:aac4716

Retractions on the Rise

Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291

“Do you trust scientists?”

Status Quoan obscenely expensive anachronism

Beyond Science

If we cannot even trust the scientific literature,what information is trustworthy?