Kristi Holmes, PhD 2AM - Amsterdam 7 October 2015

Understanding impact through alternative metrics Library-based assessment services

Northwestern University Clinical and Translational Sciences (NUCATS) Institute

Mission: Speeding transformative research discoveries to patients

and the community

http://nucats.northwestern.edu/

What is Translational Science?

Translation form “Bench to Bedside”

• Circa 2000: 18 years from discovery to dissemination... -  for successful therapies

• Circa 2015: 7 years... -  and we are smarter about what should go

through the pipeline • And we understand that translation has to go beyond the bedside

CTSAs – Mission and Structure

10/7/15 5

Emphasis on local/institutional support for

Preclinical developmental therapeutics Enhanced clinical trial infrastructure Community-engaged research

Incorporated aspects of training and career development of translational scientists (K/T awards)

http://www.academicdepartments.musc.edu_sebin_b_o_Translational_Science.jpg

Translational Medicine

Northwestern University Clinical and Translational Sciences (NUCATS) Institute Evaluation and Continuous Improvement Program

Mission: Speeding transformative research discoveries to patients and the community

http://nucats.northwestern.edu/

Some sample metrics…

OUTPUT Metrics

Time to Publication or other output

Number of technology transfer products

ROI of pilot awards

ROI of TL1, T32, KL2 scholars

Time from publication to research synthesis

IMPACT Metrics

Influence of a research output

Researcher and institutional collaborations

Career development and path/trajectory

PROCESS Metrics

Time from IRB submission to approval

Time from grant award to start of study

Volume of investigators who use services, take training, and other activities

Principles to guide ECI

• Open and collaborative processes • Good data (quality and quantity) • Look for opportunities for data reuse • Limited human interaction with data whenever possible

• Leverage common workflows, look for opportunities

Desperately seeking data

What do we want? a better perspective of both productivity and impact

•  Multiple sources covering multiple subjects and disciplines; authoritative sources

•  Disambiguated •  De-duplicated

1.  quality metadata 2.  a perspective of attention 3.  a way to tie these outputs to institutional efforts, systems,

and workflows in a meaningful way grants and to our IR

9

Papers published by NU-affiliated authors 1950-present (as indexed in Scopus)

10

Lots of missing data here: •  Citations to these works •  Valuable bibliographic metadata (collaborators, journal

titles, affiliations, keywords and indexing, etc.) •  Non-NU affiliated papers, but written by NU people (e.g.,

before they arrived) •  Reliable links to funding •  Open access status? Compliance status? •  Attention metrics •  Only traditional outputs are accounted for…so several

other outputs are missed! Low hanging fruit – already addressing this with the Galter Metric and Impact Core and other key initiatives.

What is research impact?

A Definition of Research Impact Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia.

Academic impact The demonstrable contribution that excellent research makes to academic advances, across and within disciplines, including significant advances in understanding, methods, theory and application.

Economic and societal impacts The demonstrable contribution that excellent research makes to society and the economy. Economic and societal impacts embrace all the extremely diverse ways in which research-related knowledge and skills benefit individuals, organizations and nations

Research Impact Principles and Framework, Australian Research Council http://www.arc.gov.au/general/impact.htm

Pathways to Impacts, Research Councils UK http://www.rcuk.ac.uk/ke/impacts/

Context is everything!

•  Measurement  instruments  •  Con.nuing  educa.on  materials  •  Quality  measure  guidelines  •  Cost-­‐effec.ve  interven.on  •  Consensus  development  conferences    •  American  Medical  Associa.on  Current  

Procedural  Terminology  (CPT)  codes  •  Change  in  delivery  of  healthcare  services  •  Gray  literature  

Going beyond the counts to find evidence of meaningful impact

•  New  experimental  methods,  data  models,  databases,  or  soFware  tools  

•  New  diagnos.c  criteria    •  New  standards  of  care  •  Biological  materials  or  animal  models  •  Consent  documents  •  Clinical/prac.ce  guidelines  •  Quality  measure  guidelines  

https://becker.wustl.edu/impact-assessment http://nucats.northwestern.edu/

Diverse outputs Diverse impacts

Diverse roles

Each a critical component of the research process

Building an ecosystem

Library as Partner

Opportunity!

Why a library--based support model?

•  Are a trusted, neutral space •  Have a tradition of service and support •  Strive to serve all missions of the institution •  Are technology centers and have IT and data expertise

•  Have skills—information organization, instruction, usability, subject expertise, ontologies and controlled vocabularies

•  Have close relationships with their clients (buy in) •  Understand user needs •  Understand the importance of collaboration and know

how to bring people together •  Have knowledge of institution, research, education,

clinical landscape

Library  Staff:  

Libraries:  

Galter Library’s Metrics and Impact Core

The Metrics and Impact Core (MIC), housed in Galter Library, has expertise in bibliometrics, data visualization, continuous improvement, information systems and alternative metrics. The core provides extensive advisory services for researchers, groups or departments on topics such as:

•  developing successful publishing strategies •  managing or tracking publications •  maintaining an impactful online identity •  measuring or assessing research impact by discipline •  communicating research impact to audiences

MIC uses a wide collection of resources, including Scopus, Web of Science, Essential Science Indicators, Google Scholar, NU Scholars, Journal Citation Reports, and more, to provide services and reports for: -  Researchers or clinicians to demonstrate impact of published works to promotion or tenure committees, or

the impact of research studies to funding agencies when applying for funding -  Research groups/institutions/departments to discover how research findings are being used to promote

science, or an overall view of research publications and outputs by a specific group

So why did you name it …CORE?

NUCATS Evaluation Metrics and Approach

19

Workflows, dashboards, system integrations, feedback loops

20

7 steps for success

1.  Identify stakeholders, audience, and champions 2.  Pay attention to local and national/global issues 3.  What’s missing? What’s needed? 4.  What can you do today? Next week? In a year? 5.  What do you have in hand? What do you need? 6.  Communicate – develop a communication plan to

encourage engagement during development and launch

7.  Don’t wait until you (fill in the blank)! You can do something terrific NOW!

…and CONNECT!

• res-impact@googlegroups.com -  Librarians and information scientists doing assessment and

visualizations -  Very practical group – sharing ideas, strategies, etc.

• Mosaic: Be Part of the Big Picture joint meeting – to be held May 13–18, 2016, in Toronto, ON. -  Medical Library Association (MLA) -  Canadian Health Libraries Association/Association des

bibliothéques de la santé du Canada (CHLA/ABSC)

-  International Clinical Librarian Conference (ICLC)

Thank you! Kristi Holmes @kristiholmes kristi.holmes@northwestern.edu orcid.org/0000-0001-8420-5254