Altmetrics for large, multidisciplinary research groups

Alexandra Jobmann (IPN) & Isabella Peters (ZBW)

Anita Eppelin (ZB MED), Christian Hoffmann (Universität St. Gallen), Sylvia Künne

(IfW), & Gabriele Wollnik-Korn (ZB MED)

A case study of the Leibniz Association

Bibliometrics

Seite 2

3

footprints

impact

Altmetrics

Seite 3

12

1

1

1

footprints

impact

Motivation for study

1. Initiatives that demand for new approaches in research evaluation (e.g.,

DORA)

2. Leibniz Association’s evaluation guidelines ask for appropriate public

outreach and engagement in public discourse how to measure?

3. Research showed significant disciplinary differences: coverage and

impact (Haustein & Siebenlist, 2011; Haustein et al., 2013; Holmberg & Thelwall, 2013; Mohammadi & Thelwall, 2013)

Seite 4

Research Questions

1. Where and to what extent are the publications of the institutions of

the Leibniz Association covered on social media platforms?

2. What impact do publications of the members of the Leibniz

Association have on users (i.e., altmetrics)?

3. What tools can be used to assess research impact? What

challenges might occur?

Seite 5

Methods

1. Webometric Analyst

for the collection of missing

DOIs via Crossref

2. Checked DOIs and retrieved DOIs

3. ImpactStory

for the collection of DOI-based

altmetrics data (e.g., Twitter

mentions, Mendeley readers)

Seite 6

• Disciplines of the Leibniz Association1. humanities and educational research2. economics, social sciences, spatial

research3. life sciences4. mathematics, natural sciences,

engineering5. environmental sciences

• 2-3 institutes of each discipline• Articles in conferences/ journals and

book chapters• Publication years: 2011, 2012

Data

Seite 7

Articles of 12 institutes 2.834

Correct DOIs 1.762 (62%)

Altmetrics 1.739 (99%)

Results

• Mendeley attracts readers across disciplines• Enviromental Science reluctantly uses Twitter

Seite 8

A: Humanities and Educational Re-search (n=128)

B: Economics, Social Sciences, Spatial Research (n=387)

C: Life Sciences (n=454)

D: Mathematics, Natural Sciences,

Engineering (n=429)

E: Environmental Sciences (n=341)

0

1000

2000

3000

4000

5000

6000

0

50

100

150

200

250

300

350

Mendeley readers tweets (Altmetrics.com)

Me

nd

ele

y r

ea

de

rs

twe

ets

Results

• Social media use is discipline-specific

Seite 9

A: Humanities and Educa-tional Research (n=128)

B: Economics, Social Sciences, Spatial Research (n=387)

C: Life Sciences (n=454) D: Mathematics, Natural Sciences, Engineering

(n=429)

E: Environmental Sciences (n=341)

0

2

4

6

8

10

12

14

16

blog Facebook Google+ f1000

Results

• Where do disciplines find their readers?

Seite 10

blog

Facebook

Google+

tweets (Altmetrics.com)

Mendeley readers

f1000

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

A: Humanities and Educational Research (n=128) B: Economics, Social Sciences, Spatial Research (n=387)

C: Life Sciences (n=454) D: Mathematics, Natural Sciences, Engineering (n=429)

E: Environmental Sciences (n=341)

Results

• Altmetrics can complement missing data (e.g., life sciences)

Seite 11

0

10

20

30

40

50

60

70

0

20

40

60

80

100

120

140

160

180

PubMed Central citations Mendeley readersarticles

cit

ati

on

s

rea

de

rs

0

2

4

6

8

10

12

14

16

18

20

0

20

40

60

80

100

120

PubMed Central citations Mendeley readersarticles

cit

ati

on

s

rea

de

rs

institute 1 institute 2

Results

• Institutes from the same discipline (e.g., life sciences) find readers on

different platforms

Seite 12

C1 C20

1

2

3

4

5

6

7

8

9

0

50

100

150

200

250

300

blog Facebook Google + tweets (Altmetrics.com)

twe

ets

Lessons Learned

• Chosen tools determine quality of data• Tools and altmetrics providers change settings

• Chosen identifiers affect data• PubMedID is more popular than DOI

• Missing or erroneous identfiers in social media

• Multiple identifiers for one publication

Underestimation of real numbers

· Collection of publication data• Missing DOIs on institutes‘ websites• Double-entry of publication on websites

• Carry out data download at the same time

Seite 13

Lessons Learned

• Aggregated numbers may give wrong picture (e.g., discipline basis)• Sum html views: 2,447 (n=2) - Sum readers: 921 (n=76)

Seite 14

0

10

20

30

40

50

60

0

200

400

600

800

1000

1200

1400

1600

1800

Mendeley readers PLoS html viewsarticles of institute A1 (humanities and educational research)

read

ers

Htm

l vi

ews

Lessons Learned

• Mendeley is the platform which covers a substantial amount of

papers and shows reasonable user activity

• Look for good coverage/ usage ratio

• However, some disciplines prefer other platforms• Get to know the community preferences

• Respect reader/ community choices

• Altmetrics should not substitute, but can complement citation data

•Comparability of altmetrics not given – same situation as in traditional

citation analysis

Seite 15

Thank you!

Seite 16

Alexandra Jobmann, IPNjobmann@ipn.uni-kiel.de

Prof. Dr. Isabella Peters, ZBWi.peters@zbw.eu

References

• Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457.

• Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2013). Coverage and adoption of altmetrics sources in the bibliometric community. In Proceedings of the 14th International Society of Scientometrics and Informetrics Conference, Vienna, Austria, Vol. 1 (pp. 468-483). Retrieved from http://www.issi2013.org/Images/ISSI_Proceedings_Volume_I.pdf

• Holmberg, K., & Thelwall, M. (2013). Disciplinary differences in Twitter scholarly communication. In Proceedings of the 14th International Society of Scientometrics and Informetrics Conference, Vienna, Austria, Vol. 1 (pp. 567-582). Retrieved from http://www.issi2013.org/Images/ISSI_Proceedings_Volume_I.pdf

• Mohammadi, E. & Thelwall, M. (2013). Assessing the Mendeley readership of social sciences and humanities research. In Proceedings of the 14th International Society of Scientometrics and Informetrics Conference, Vienna, Austria, Vol. 1 (pp. 200-2014). Retrieved from http://www.issi2013.org/Images/ISSI_Proceedings_Volume_I.pdf

Seite 17