Post on 19-May-2015
description
transcript
The Infrastructure for Alternative Metrics
What do we need to compare digital apples to digital apples?
!Todd Carpenter, Executive Director, NISO
AAAS Annual Meeting Chicago, IL | February 16, 2014
We hate discussing infrastructure
Photo: Minneapolis Collegeof Art and Design Library
• Non-profit industry trade association accredited by ANSI with 190+ members
• Mission of developing and maintaining technical standards related to information, documentation, discovery and distribution of published materials and media
• Volunteer driven organization: 400+ spread out across the world
• Responsible in various wasy for standards like ISSN, DOI, Dublin Core metadata, DAISY digital talking books, OpenURL, MARC records, and ISBN
About
Photo by dumbledad on Flickr http://www.flickr.com/photos/dumbledad/298650884/!
But what is the “infrastructure” for
assessment?
We stand on the shoulders of giants...
The Mighty Citation
Impact factor
Thomson-Reuters IMPACT FACTOR
Two Stanford Students
Two Stanford Students
They wrote a paper
They started a company in someone’s garage
Photo: AP
Citations are awesome, aren’t they?
A perfect solution to all the world’s problems, no?
A simple conclusion can be drawn
Back to Sir Isaac Newton“What Des-Cartes did
was a good step. You have added much several ways, & especially in
taking ye colours of thin plates into philosophical consideration. If I have seen further it is by
standing on ye shoulders of Giants."
Citations aren’t always
what we think they
are
Caveat emptor
Another problem with citations
So where to from here?
Define altmetrics, please
Social Media Metrics
Image: Danny Brown, The State of Social Media Marketing 2012
Usage Based Metrics
Click Map Analysis
Clickstream data yields high-resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009
Network Analysis
Clickstream data yields high-resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009
Network Analysis
Behavioral Metrics
Sentiment Analysis
Image: Domenico, Caron, Davis, et al.
Non-traditional content
Assessing impact of new forms of communication isn’t as simple as one might think.
Image: Domenico, Caron, Davis, et al.
Non-traditional content
Assessing impact of new forms of communication isn’t as simple as one might think.
We’ve got what, now how?
How do we measure things?
At what granularity do we measure things?
Consistency in what is counted
Source: Scott Chamberlain, Consuming Article-‐Level Metrics: Observations And Lessons From Comparing Aggregator Provider Data, Information Standards Quarterly, Summer 2013, Vol 25, Issue 2.
Which measurements are valuable?
How do we ensure the validity of those measurements?
How to (or even if we should) deal with other issues
such as gaming?
What infrastructure do we need for
alternative metrics?
Basic Definitions !
(So we are all talking about the same thing)
Comparison across providers
Source: Scott Chamberlain, Consuming Article-‐Level Metrics: Observations And Lessons From Comparing Aggregator Provider Data, Information Standards Quarterly, Summer 2013, Vol 25, Issue 2.
Authorship -----
Disambiguation &
Contributor Roles
Element Identification
Open exchange of component data
TRUST
=
Where does NISO fit in?
Alterna(ve Assessment Ini(a(ve
!
Phase 1 Mee(ngs October 9, 2013 -‐ San Francisco, CA December 11, 2013 -‐ Washington, DC
January 23-‐24 -‐ Philadelphia, PA !
Phase 1 report expected in May 2014
Just a few ideas from in-‐person mee(ngs Disambigua)on problems
Create a central repository for data Define provenance for metrics
Foster culture of DOI use in mass media Establish defini)ons for metric types Define data extract/gathering methods
Standardize units of measure Define levels of transparency/privacy
!
!
Alterna(ve Assessment Ini(a(ve !
Phase 2 Presenta(ons of report (June 2014) Priori(za(on Effort (June -‐ Aug, 2014)
Project approval (Sept 2014) Working group forma(on (Oct 2014)
Consensus Development (Nov 2014 -‐ Dec 2015) Trial Use Period (Dec 15 -‐ Mar 16)
Publica(on of final recommenda(ons (Jun 16)
May 15, 2013 74
Thank you! Todd Carpenter, Executive Director
tcarpenter@niso.org @TAC_NISO
!National Information Standards Organization (NISO)
3600 Clipper Mill Road, Suite 302 Baltimore, MD 21211 USA
+1 (301) 654-2512 www.niso.org