A replication crisis in the making: how we reward

unreliable science

Björn BrembsUniversität Regensburg - Neurogenetics

http://brembs.net - @brembs

15th century: Adoration of the lamb

21st century: Adoration of the Glam

More and Better!

Application Instructions

Application Instructions

Publikationstätigkeit(vollständige Publikationsliste, darunter Originalarbeiten als Erstautor/in, Seniorautor/in, Impact-Punkte insgesamt und in den letzten 5 Jahren, darunter jeweils gesondert ausgewiesen als Erst- und Seniorautor/in, persönlicher Scientific Citations Index (SCI, h-Index nach Web of Science) über alle Arbeiten)

Publications:Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications.

Main Problems with the IF• Negotiable • Irreproducible • Mathematically

unsound

Negotiable

https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/

June, 2014 (19 months)

Not Reproducible• Rockefeller University Press bought their data from Thomson Reuters• Up to 19% deviation from published records• Second dataset still not correct

Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091

Not Mathematically Sound• Left-skewed distributions• Weak correlation of individual

article citation rate with journal IF

Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497http://www.bmj.com/cgi/content/full/314/7079/497

Not Mathematically Sound

https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/

‘QUALITY’

Is journal rank like astrology?

Methodology I

Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1002273

Methodology II

Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291

Methodology III

Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77

‘Quality’

Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847

Excel Errors

DOI: 10.1186/s13059-016-1044-7 -omics studies

p-Value Errors

Cog. Neurosci & PsychDOI: 10.1101/071530

QUALITY?

“High-Impact” journals attract the most unreliable research

Statistical Power and Sample Size

Source: Daniel Lakens DOI: 10.1177/1745691614528520

Statistical Power and Sample Size

Source: Daniel Lakens DOI: 10.1177/1745691614528520

Statistical Power and Sample Size

Source: Daniel Lakens DOI: 10.1177/1745691614528520

Statistical Power and Sample Size

Productivity

Research questions:True:False:Significant:

20010010040

88444437

Models

Selecting for ‘productivity’

Selecting for ‘novelty’

PRODUCTIVITY?

“Publish-or-Perish” disadvantages meticulous scientists

Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002165

Irreproducibility

61%(n=100)

Open Science Collaboration

INCENTIVES

Counting ‘Quality’ & Productivity => Selecting the sloppy scientists

Scholarly Infrastructurean obscenely expensive anachronism

PROBLEM I

Dysfunctional scholarly literature

Antiquated Functionality• Limited access• Link-rot• No scientific impact analysis• Lousy peer-review • No global search• No functional hyperlinks• Useless data visualization• No submission standards• (Almost) no statistics• No content-mining• No effective way to sort,

filter and discover• No semantic enrichment• No networking feature• etc.

…it’s like the web in 1995!

PROBLEM II

Scientific data in peril

Small Data – Long Tail

Report on Integration of Data and Publications, ODE Report 2011http://www.alliancepermanentaccess.org/wp-content/plugins/download-monitor/download.php?id=ODE+Report+on+Integration+of+Data+and+Publications

PROBLEM III

Non-existent software archives

UNIVAC (A-2) 1953

INFRASTRUCTURE

Antiquated and missing functionality

“Publishers” parasitize public funds

Cost

s [th

ousa

nd U

S$/a

rticle

]

Legacy Modern(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)

(SciELOUbiquityScholasticaScienceOpenPeerJF1000ResearchFrontiersetc.)

MONEY FOR NOTHING

Wasting billions on a parasitic industry

SO MUCH FOR THAT

The disaster that is our scholarly infrastructure

OPEN ACCESS ACTIVISM“Pretty please be open!”

GitHub

Scientific Code with Persistent Identifiers

openknowledgemaps.org

575+ such solutions and counting…

Institutions are trying…

The Department of Psychology embraces the values of open science and strives for replicable and reproducible research. For this goal we support transparent research with open data, open material, and pre-registrations. Candidates are asked to describe in what way they already pursued and plan to pursue these goals.

Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications.

versus

APC-OA?

+70%

APC-OA?

“The decision, based on market and competitor analysis, will bring Emerald’s APC pricing in line with the wider market, taking a mid-point position amongst its competitors.”

Emerald spokesperson

WHAT NOW?

Save time and money (and make science open by default as an added benefit)

Potential for Innovation

(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. doi:10.1038/495426a, Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)

Potential for innovation: 9.8b p.a.

Cost

s [th

ousa

nd U

S$/a

rticle

]

Legacy SciELO

1. International Coordination

LEGAL

2. Cancel all subscriptions

3. Implement current technology

The square traversal process has been the foundation of scholarly

communication for nearly 400 years!