Post on 23-Aug-2014
description
transcript
Next-generation Metrics for Scholarly Performance
William Gunn, Ph.D. Head of Academic Outreach
Mendeley
@mrgunn
https://orcid.org/0000-0002-3555-2054
Open Access Policies
Open Access Has Benefits
• The perception that there were tradeoffs when choosing open access slowed adoption
• Only Open Access can fully provide the benefits of the web – interoperability, accessibility, discovery
• An expanded definition of impact brings new kinds of content and new metrics
King, Christopher (2012) Thomson Reuters Annual Report
http://ar.thomsonreuters.com/_files/pdf/MultiauthorPapers_ChrisKing.pdf
Adams, Jonathan. "Collaborations: the fourth age of research."
Nature 497.7451 (2013): 557-560.
Amgen: 47 of 53 “landmark” oncology publications could not be reproduced
Bayer: 43 of 67 oncology & cardiovascular projects were based on contradictory results
Dr. John Ioannidis: 432 publications purporting sex differences in hypertension, multiple sclerosis, or lung cancer. Only one data set was reproducible
There is no gold standard
Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph
publishing." new media & society 15.3 (2013): 359-378.
Authors Publishers
Printers
Shippers
Booksellers
Readers
Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph
publishing." new media & society 15.3 (2013): 359-378.
Authors Publishers
Printers
Shippers
Booksellers
Readers
1770
Authors Readers Internet
Research is changing
Publishers
Repositories
Professional
Websites
Challenges for Repositories
• The world is leaning green
• Repositories thus have an essential role
• Repositories aren’t in the workflow of a researcher, neither in search for new ideas nor in presentation of their work
The Ouroboros problem
• repositories need content deposited to get usage, and they need usage to get deposits
• Discovery tools can address usage and usage will promote content
– scholarly outputs are papers, datasets, code, and more.
New forms of discovery
• personalized recommendations based on reading history
• related articles
• recommender frameworks
–implement recommendations as a service
• third-party recommender services
–serve niche audiences
research impacts more than authors
http://altmetrics.org/manifesto/
...and aggregates research data in the cloud
Mendeley extracts research data…
Install Mendeley Desktop
Collecting rich signals from domain experts.
Home Work Mobile
Cloud Library
Shared Folder
Mendeley Research Catalog
Mendeley Research Catalog
Read papers + keep track of notes
470M documents
Taking some misery out of writing
Information Extraction
We are publishing this data to the LOD cloud
http://code-research.eu/
Defining readership
• Each document addition is a “read”
• stamped with metadata describing the context of the read event
• a read is like a citation, but faster and captures more
research impacts more than authors
http://altmetrics.org/manifesto/
Readership vs. citations
• it comes with a payload of metadata
• it accrues faster
• it illuminates previously hidden impact
http
://arx
iv.org
/htm
l/1203.4
745v1
altmetrics show broader impact
htt
p://d
x.d
oi.org
/10.3
789/isqv25no2.2
01
3.0
4
http://www.asis.org/Bulletin/Apr-13/AprMay13_Lin_Fenner.html
altmetrics show broader impact
Issues To Be Addressed
• Identity
• Privacy
• Attribution
• Gaming
• Filtration
standards/ best practice
Amgen: 47 of 53 “landmark” oncology publications could not be reproduced
Bayer: 43 of 67 oncology & cardiovascular projects were based on contradictory results
Dr. John Ioannidis: 432 publications purporting sex differences in hypertension, multiple sclerosis, or lung cancer. Only one data set was reproducible
There is no gold standard
Cultural skew is important
South America is weak on N.A. social media, strong on Mendeley
What would people build if they could get the data?
• Impact Story – get credit for all your work
• PLOS ALM – article-level metrics for papers
• Plum Analytics – bespoke analytics for libraries (EBSCO)
• Altmetric.com – altmetrics for publishers. (Digital Science)