Ranking
Journals
Using
Altmetrics
© Imperial College LondonPage 1
15th International Society of Scientometrics and Informetrics
Conference, Istanbul
30th June 2015
Tim Evans
Centre for
Complexity Science
Work done with:
Tamar Loach
© Imperial College LondonPage 2
Ranking Journals Using Altmetrics
• Traditional Journal Metrics
• The Altmetric Data
• Journal Ranking from Altmetric Data
• Results
• Visualisation
• Conclusion
Paper Citation Network OR Journal Citation Network
© Imperial College LondonPage 3
Paper
Journal A Journal B
A
B
3
2
3
2
Journal or Paper Citation Network?
• Measure for each paper then aggregate
over papers in journal
– e.g. Impact Factor
• Aggregate papers in journal, then analyse
citations between journals e.g.
– e.g. EigenFactor
Order makes a difference when distributions
are fat-tailed and methods are non-linear.
© Imperial College LondonPage 4
Traditional Journal Metrics Used Here
We used
• Impact Factor 2013
– Number of citations in 2013 to articles
published in journal in 2011 & 2012,
divided by number of articles published in
‘11 and ‘12.
• EigenFactor 2013 [Bergstrom 2007]
– PageRank on Journal citation network
© Imperial College LondonPage 5
Note both based on the same
Web of Science database
Traditional Journal Metrics
• Aim to measure attention received by a
journal
• Based on citations
• Simple Citation Counts per paper
• Count citations weighted by the
importance of source (self-consistent,
PageRank style)
© Imperial College LondonPage 6
© Imperial College LondonPage 7
• Traditional Journal Metrics
• The Altmetric Data
• Journal Ranking from Altmetric Data
• Results
• Visualisation
• Conclusion
Altmetric Data
• Data from altmetric.com
• 20 months ending June 2013
• Sources include
– BBC
– Blogs
– …
© Imperial College LondonPage 8
Information in our altmetric.com data
• An account can be one or many people
• One person may contribute to several accounts
© Imperial College LondonPage 9
A1
P1
J1 Journal identifier
Paper identifier
Author (account and source)
© Imperial College LondonPage 10
• Traditional Journal Metrics
• The Altmetric Data
• Journal Ranking from Altmetric Data
• Results
• Visualisation
• Conclusion
Simple Altmetric Counts
Rate Journals by counting altmetric mentions
Example: CA = Combined altmetric.com
Counts number of mentions of articles
published in a journal weighted by source
– Blog or News item worth more than Twitter mention
© Imperial College LondonPage 11
>,
Pairwise Comparison by each Author
• No statement
about a journal if
author does not
read its articles
© Imperial College LondonPage 12
A1
P1
J1
P2 P3
J2
P4
J3
Authors Define their own Field
Pairwise comparison
by each Author
means comparisons
are within their field
© Imperial College LondonPage 13
A1
P1
J1
P2 P3
J2
P4
J3 J4
J5
Count Mentions
© Imperial College LondonPage 14
A1
J1 J2 J3
2
1
1
0
0
Jaj=
j
a-th column of matrix J
counts mentions by author a
a=A1,
j=J1,J2,J3,J4,J5
J4
J5
Combine Counts using Author Comparison
© Imperial College LondonPage 15
Score for journal j
when compared to
journal l
number of mentions
of journal j by
authors who mention
BOTH j and l,
divided by number of
such authors
𝑆𝑗𝑙 =1
𝐴𝑗𝑙
𝑎∈𝐴𝑗𝑙
𝑆𝑎𝑗
Sjl =
Sjl =
, Ajl = set of authors who
mention papers in
both journal j and l
1
j l
32
Combine Counts using Author Comparison: Example
© Imperial College LondonPage 16
• Author 1 reads journal j
twice but journal l once.
• Author 2 reads both
journals once.
• Author 3 does not read
journal j.
𝑆𝑗𝑙 =1
𝐴𝑗𝑙
𝑎∈𝐴𝑗𝑙
𝐽𝑎𝑗
a=2 a=3
a=1
j l
Sjl = ½(2+1) = 1.5
Slj = ½(1+1) = 1.0
Combine Counts using Author Comparison: Example
© Imperial College LondonPage 17
• Author 1 reads journal j
twice but journal l once.
• Author 2 reads both
journals once.
• Author 3 does not read
journal j.
𝑆𝑗𝑙 =1
𝐴𝑗𝑙
𝑎∈𝐴𝑗𝑙
𝐽𝑎𝑗
a=2 a=3
a=1
j l
Sjl = ½(2+1) = 1.5
Slj = ½(1+1) = 1.0
Combine Counts using Author Comparison: Other
© Imperial College LondonPage 18𝑃𝑗𝑙 =
1
𝐴𝑗𝑙
𝑎∈𝐴𝑗𝑙
Θ 𝐽𝑎𝑗 − 𝐽𝑎𝑙
a=2 a=3
a=1
j l
Pjl = ½(1+0.5) =0.75
Plj = ½(0+1) =0.25
Many, many alternatives[Langville & Meyer, 2012]
e.g. 1 point for a ‘win’,
0 for a ‘loss’,
½ for a ‘draw’
Produce Journal Ranking and Rating
Once you have a journal-journal
score matrix, Sjl, Pjl, etc etc
apply use centrality measures
• PageRank
• HITS
• PSR = Points Spread Rating
• etc, etc …
© Imperial College LondonPage 19
[Langville & Meyer, 2012]
© Imperial College LondonPage 20
• Traditional Journal Metrics
• The Altmetric Data
• Journal Ranking from Altmetric Data
• Results
• Visualisation
• Conclusion
Overall Comparisons
• Basic Multivariate Statistics for overview of
behaviour of different ratings
– Hierarchical Clustering
– Principle Component Analysis
• Look at top forty or so journals by eye
© Imperial College LondonPage 21
© Imperial College LondonPage 22
EF=EigenFactor
IF=Impact
Factor
CA=Combined
altmetric.com
Most rating
measures
similar
except
PSR
(Points
Spread
Rating)
One source, several rating variations
[Spearman Correlation Matrix]
Different Rating Methods
Most
sources
reasonably
similar
but they do
show
interesting
variations
© Imperial College LondonPage 23
[Spearman Correlation Matrix]
Different Sources
Blogs
© Imperial College LondonPage 24
EF=EigenFactor
IF=Impact
Factor
[Spearman Correlation Matrix]
Blogs & Other Sources
Blogs are
closest to
traditional
standard
rankings:
Impact
Factor
&
Eigen-
Factor
Blogs
© Imperial College LondonPage 25
EF=EigenFactor
IF=Impact
Factor
[Spearman Correlation Matrix]
CA = Combined altmetric.com
Blogs
CA(combined
altmetric.com)
weighted
altmetric
count
balances
source
differences
nicely,
much
closer to
IF & EF
CA=Combined
altmetric.com
© Imperial College LondonPage 26
EF=EigenFactor
IF=Impact
Factor
[Spearman Correlation Matrix]
Simple Alt Media Counts
CA=Combined
altmetric.com
Simple
counts of
alt. media
are often
similar to IF
& EF:
CA (2),
Blind Count
(1),
Paper
Count (7),
Social
Media
Count (14)
BC (1) &
SMC (14)
Paper
Count (7)
© Imperial College LondonPage 27
Q, H, news Q, H,Blogs S,PR,blogs IF 20131 Nature Nature Nature N.E.J.Medicine2 PNAS PNAS Science Rev.Mod.Phys.3 PLoS ONE Science PNAS Chemical Rev.4 Science PLoS ONE PLoS ONE Nat.Rev.Genetics
5 N.E.J.Medicine N.E.J.Medicine N.E.J.Medicine The Lancet6 B.M.J. B.M.J. B.M.J. Nature7 Nature Comm.s The Lancet The Lancet Nature Rev.Mol.Cell Bio.
8 JAMA JAMA JAMA Ann.Rev.Immunology
9 The Lancet Proc.R.Soc.B: Bio.Sci PLoS Biology Nature Materials
10 Pediatrics Current Biology Current BiologyNature Genetics11 Psychological Sci. Pediatrics PLoS Medicine Nat.Rev.Cancer12 Current Biology Psychological Sci. Proc.R.Soc.B: Bio.Sci Adv. in Physics13 Sci. Reports J.Neuroscience Psychological Sci. Nat.Rev.Immunology
14 Cell Am.J.Clinical Nutrition Pediatrics Nat.Rev.Drug Discovery
15 Nature Medicine PLoS Medicine Cell Nat.Biotechnology
© Imperial College LondonPage 28
• Traditional Journal Metrics
• The Altmetric Data
• Journal Ranking from Altmetric Data
• Results
• Visualisation
• Conclusion
Cube Box Space
• Each journal has D
rankings
• Connect
from journal j to
journal l
iff
𝑗𝑟 < 𝑙𝑟 ∀ 𝑟 = 1,2,⋯ , 𝐷
© Imperial College LondonPage 29
D=2
[Bollobás & Brightwell 1991]
Ranking 1
Rankin
g 2
Journal Top Trumps: Nature vs. PRL
© Imperial College LondonPage 30
Nature
Nature is a weekly, international, interdisciplinary journal of science, published by MacMillan Publishing.
Physical Review
Letters
The American Physical Society publishes Physical Review Letters with short article
on fundamental and interdisciplinary physics research.
Connect as Nature higher than PRL on all three ratings
Journal Top Trumps: Nature vs. NEJM
© Imperial College LondonPage 31
The New England
Journal of Medicine
The mission of NEJM has been to bring physicians the best research at
the intersection of biomedical science and clinical practice.
Nature
Nature is a weekly, international, interdisciplinary journal of science, published by MacMillan Publishing.
No connection as Nature beats (loses) NEJM on EF & CA (IF)
Box Space Representation of
Journal Rankings
© Imperial College LondonPage 32
• Node = Journal
• Directed Edge
if journal has
better IF, EF
& CA
• Height
=
minimum
longest
path
distance
to root
• Top 1000
journals
used
• Top 40
shown
wors
e
Hasse diagram technique[Brüggeman et al. 1994]
© Imperial College LondonPage 33
• Traditional Journal Metrics
• The Altmetric Data
• Journal Ranking from Altmetric Data
• Results
• Visualisation
• Conclusion
Conclusions
• Shown how to extract journal ranking from
mentions of papers in alternative media
sources
• Shown that most ranking schemes are
reasonable
– But not Point Ranking Scheme
• Some interesting differences
– To understand why need some bibliometric input
• Which scheme is best?
– Axiomatic, Social science criteria?
© Imperial College LondonPage 34
Tim Evans, Imperial College London
http://netplexity.org
Work done with: Tamar Loach
Thanks
Work done in
collaboration with Tamar Loach
• Euan Adie (altmetric.com)
– provided and explained the altmetric data
• Daniel Hook (Digital Science)
– provided financial support and useful discussions.
• Jonathan Adams (Digital Science)
– for useful discussions
© Imperial College LondonPage 35
© Imperial College LondonPage 36
Bibliography
• Archambault, É., & Lariviére, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.
• Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College and Research Libraries News, 68, 314-316.
• Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of
Informetrics, 8, 895-903.
• Brin, S. & Page, L. (1998). The anatomy of a large-scale hypertextual Web search engine. Computer networks and ISDN systems, 30, 107-117.
• Bruggemann, R.; Halfon, E.; Welzl, G.; Voigt, K. & Steinberg, C. Applying the Concept of Partially Ordered Sets on the Ranking of Near-Shore
Sediments by a Battery of Tests J. Chem. Inf. Model., American Chemical Society (ACS), 2001, 41, 918-925.
• Bruggemann, R., M¨unzer, B. and Halfon, E., An algebraic/graphical tool to compare ecosystems with respect to their pollution – the German
river “Elbe” as an example - I: Hasse-diagrams, Chemosphere, 28 (1994) 863–872.
• Garfield, E. (1955). Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas, Science, 122 (3159), 108-
111.
• Garfield, E. (1990). Journal citation studies 52. The multifaceted structure of crystallography research. Part 1. Core journals, high impact
papers, and current research fronts. Essays of an Information Scientist: Journalology, KeyWords Plus, and other Essays, 13, 327-336
• Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. JACM, 46, 604-632.
• Langville, A. N. & Meyer, C. D. (2012) Who’s #1?: The science of rating and ranking, Princeton: Princeton University Press.
• Clough, J. R.; Gollings, J.; Loach, T. V. & Evans, T. S., Transitive reduction of citation networks, J.Complex Networks, 2015, 3, 189-203
[10.1093/comnet/cnu039 arXiv1310.8224]
• Clough, J.R. & Evans, T.S. What is the dimension of citation space? 2014 arXiv:1408.1274
• Evans, T.S., Complex Networks, Contemporary Physics, 2004, 45, 455-474 [10.1080/00107510412331283531 arXiv:cond-mat/0405123]
• Evans, T.S., Clique Graphs and Overlapping Communities, J.Stat.Mech, 2010, P12037 [10.1088/1742-5468/2010/12/P12037 arXiv:1009.0638]
• Evans, T.S. & Lambiotte, R., Line Graphs, Link Partitions and Overlapping Communities, Phys.Rev.E, 2009, 80, 016105
[10.1103/PhysRevE.80.016105 arXiv:0903.2181]
• Evans, T.S. & Lambiotte, R., Line Graphs of Weighted Networks for Overlapping Communities, Eur.Phys.J. B 2010, 77, 265–272
[10.1140/epjb/e2010-00261-8 arXiv:0912.4389]
• Expert, P.; Evans, T. S.; Blondel, V. D. & Lambiotte, R. Uncovering space-independent communities in spatial networks, PNAS, 2011, 108,
7663-7668 10.1073/pnas.1018962108 [10.1073/pnas.1018962108 arXiv:1012.3409]
• Garfield, E.; Sher, I. H. & Torpie, R. J., The use of citation data in writing the history of science, DTIC Document, 1964.
• Holme, P. & Saramäki, J., Temporal Networks, Physics Reports, 2012, 519, 97-125.
• Hummon, N. P. & Dereian, P., Connectivity in a citation network: The development of DNA theory, Social Networks, 1989, 11, 39-63.
© Imperial College LondonPage 37
Tim Evans
Centre for Complexity Science
http://netplexity.org