Eric T. Meyer & Kathryn EcclesOxford Internet Institute, University of Oxford

Digital Humanities@Oxford Summer School29th July 2011

Impact as a process: considering the reach of resources from the

start

@etmeyer#tidsr#oess#dhox

What is impact and why consider it?

What do we mean by impact?

•Reaching intended audience•Reaching new audiences•Attracting users•Attracting new usage•Enabling new research questions•Enabling new approaches to education

Where to begin?

• Identify your audience and key stakeholders

• Set your goals. What types of impact do you envisage your resource having?

• What steps are you taking to ensure these types of impact?

• Identify connections

• What resources do you see as successful in terms of audience and impact?

• Do you see your resource as part of a network of connected resources?

Digitisation and Impact

Measuring usage and impact

What to measure?

• Users

• Types of use

• Awareness

• Citation practices

• Marketing strategies

• Embedding

• JISC funded project

• July 2008-April 2009

• Looked at five specific JISC-funded resources

• Designed to test the TIDSR methods and review them for the TIDSR toolkit

TIDSR: The first usage and impact study

Methods

Quantitative methods

• Webometrics

• Web Analytics

• Log file analysis

• Scientometrics / bibliometrics

• Content Analysis

Qualitative methods

• Interviews

• Focus groups

• User feedback

• Referrer analysis

• Content Analysis

Project 1 – Online Historical Population Reports (OHPR/Histpop)

Survey: Low Awareness

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

88%

63%

39%

63%69%

7%

27%

35%

28% 22%

3% 7%

18%

7% 7%2% 2%

7%1% 2%

Use it regularly or frequently

Use it on occasion

Have seen it, but don't use it

I haven't heard of it

Survey: High Importance to Users

HistPop BOPCRIS BL News BL SoundsMed Backfiles0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

71%

51% 49%

36%

61%

29%

35% 34%

40%

33%

79%

69%72%

60%

76%

96%

84%

90%86%

82%

Important to my research

Important to my teaching

Important to field

Would recommend

Log Files: Non UK Activity

Log File Analysis

Top Search Phrases: Histpop

Histpop: User Communities

Perception: Specific niche community Well known by target audience Transforming access and usage patterns

User surveys: Embedded in educational resources Enhanced access to primary sources

▪ ‘Histpop made it possible to do a completely different project’

Continuing education, online resources, non-traditional learners

Project 2 – British Library 19th Century Newspapers

Project 2 – British Library 19th Century Newspapers

Citation Habits

HistPop

BOPCRIS

BL News

BL Sounds

Med Back-files

9%

36%

53%

38%

43%

55%

36%

38%

50%

48%

36%

21%

6%

13%

10%

0%

7%

2%

0%

0%

Original version Original + URL Online version Other

0%10%20%30%40%50%

46%

29%

35%

20%

43%

Have you ever published a piece based on your work in this collection?

If so, how did you cite the collection?

Webometric results

• Highest numbers for original British Library resource (analogue)• 19th Century British Library Newspapers registers strong links for a project page• Note: Importance of comparator sites when using webometrics

Blog Evidence

Project 3 – British Library Archival Sound Recordings

Interviews, Group Interviews, Focus Groups

Time intensive, but productive if you are careful about what you ask!

Different stakeholders: Project team: Positive view of the work

only Broader stakeholders: While the digital

project was good, it also introduced tensions in the broader setting of the library

New kinds of serendipity, wide range of users

News

Engagement officer

Project 4 – British Official Publications Collaborative Reader Information Service (BOPCRIS): 18th Century Official Parliamentary Publications Portal 1688-1834

Project 4 – British Official Publications Collaborative Reader Information Service (BOPCRIS): 18th Century Official Parliamentary Publications Portal 1688-1834

Webometrics

• Some resources are available through multiple outlets• Webometrics can capture comparative awareness• These results show how powerful known resources and/or publishers can be

Project 5 – Wellcome Medical Journals: the backfiles project

Project 5 – Wellcome Medical Journals: the backfiles project

Webometrics

Wellcome Medical Journals Backfiles project page records strong links, links to Pub Med for WMJB material impossible to trace

Knowing the Users

Historians? (would be looking at older articles) Not typical PubMed users Search interface issues / limited search

Clinicians? (would be looking at newer articles) Not typically reading 100 year old articles

Other users? Paths of discovery?

New uses?

Majority of downloads targeted more recent material – opening up of new resources to clinicians

More thorough and comprehensive searches Historians reported more comprehensive

search results (quantitative results) Also reported increased browsing,

greater serendipity, due to time saved finding articles

7%

12%

16%

17%

31%

34%

45%

47%

51%

54%

58%

62%

71%

77%

83%

13%

11%

9%

30%

18%

29%

32%

38%

27%

24%

32%

36%

56%

50%

48%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Cornell Animal Sounds

Sciper

Histpop

Chronicling America

Fine Rolls

Wellcome Medical Journal Backfiles

Historical Directories

Internet Lib of Early Journals

BOPCRIS

Archival Sounds

Imperial War Museum

Old Bailey Online

British Periodicals

British Library Newspapers

House of Commons Parliamentary Papers

Non-UK Awareness

UK Awareness

Awareness of Resource by Country

How did you find this resource?

0%5%

10%15%20%25%30%35%40%45%50%

HistPopBOPCRISBL NewsBL SoundsMed Back-files

http://microsites.oii.ox.ac.uk/tidsr/

University of Oxford Podcasts

Proceedings of the Old Bailey Online

British History Online

Siobhan Davies RePlay

http://www.rin.ac.uk/humanities-case-studies

Bulger, M., Meyer, E.T., de la Flor, G., Terras, M., Wyatt, S., Jirotka, M., Eccles, K., Madsen, C.

The Case Studies

Browsing and Searching

Libraries

Journals

Peers

79%66%

Google

Google Scholar

59%

55%

62%

83%

48%

76%

95%

Visit the library

Browse library materials online

Search library materials online

Citation chaining

Browse printed journals

Browse online journals

Consult peers and experts

Reconfiguring Resources

“ Old Bailey Online hasn’t replaced anything for me or displaced anything for me, but it is part of this general transformation of how I do what I do.

“The amount of time I now spend doing the very mechanical, laborious, time-consuming work is much smaller. You can now do things in 5 seconds which it took you 3 months to do a few years ago.

Transformations in Use

It’s a huge change. You can do things much more quickly, read much more widely, find connections…it’s very, very important. “

With something like the Burney Collection, 5 years ago for writing an article I would need to review the newspapers, I would have gone into the British Library and done it on microfilm.

20 years ago, I would have gone into the British Library and done it with the actual paper in front of me. Now I sit at home and I do a keyword search.

“

Asking new questions?“

I’m not sure all of this raises the quality of anybody’s work. I think it would be quite daft to pretend that all of this makes us better scholars, or makes our books or papers of higher quality. I don’t know if that is true by any means, but it certainly makes it easier and I suppose makes the quantity of stuff that you can produce greater.

“

What might take you several months if not years of research, you could do in hours, days, a week. So I think that means that it makes the nature of your research different because it allows you quantitative information much more quickly, which then allows you to maybe think about how you might use that information differently, because you’ve got so much more time.

“

Eric T. Meyereric.meyer@oii.ox.ac.uk

http://www.oii.ox.ac.uk/people/?id=120

Kathryn Eccleskathryn.eccles@oii.ox.ac.uk

http://www.oii.ox.ac.uk/people/?id=138

Oxford e-Social Science Project

Project work funded by: