No Advanced User Left Behind - Discovery for All

Beth Namei & Caroline Muglia University of Southern California Libraries

ProQuest User Summit | Chicago, IL | October 8, 2015

What we’ll cover today

● Timeline of USC’s implementation of Summon● Portrait(s) of Advanced Users● Results & Snapshots of 3 user studies and search query

analysis conducted from 2010-2015● Pain Points● Recommendations● Q&A

Why USC launched Summon

● Better discoverability

● Relevant results to our users

● Richer user experience with the library’s website

Novice researchers are happy with Summon!Freshman: “For my research papers, I always come to the USC website and Quick Search is really helpful…”Freshman:“The first thing I would do is probably Quick Search it… that’s my first step, that’s the easiest, it’s right there.”Freshman:“I use Google Scholar, but I find that it’s easier to use Quick Search because Google Scholar doesn’t link you to USC LIbraries all the time.”

Source: https://docs.google.com/document/d/1ajpryZbNmmFFceimG1yHgBnO5F6Icl5lr9eXb0d29SE/edit?ts=5612e99d

Why discovery vendors and academic librarians (should) care about advanced users?

Who are advanced users?They have…

● Specialized subject knowledge ● Familiarity with library's collections and resources● Experience researching in their discipline● Commitment to in-depth research● Sophisticated searching skills● Mastered and adapted to less than ideal search tools/systems

They want…● Precision search tools that allow them to hone in on sources● To search for known items using objective info. (author name, projects, article, book or journal titles, etc.)

Sources: “Not in Love, or Not in the Know? Graduate Student and Faculty Use (and Non-Use) of E-Books” - doi:10.1016/j.acalib.2012.08.005; http://ejournals.library.ualberta.ca.libproxy2.usc.edu/index.php/EBLIP/article/view/1802/3366 and http://www.emeraldinsight.com.libproxy1.usc.edu/doi/pdfplus/10.1108/07378830710840509

Advanced users are the majority

http://clipgid.com/professor-teaching-clipart.html http://www.supercollege.com/images/grad_shadows_white_2.jpg

http://img11.nnm.me/9/0/b/6/d/d096d47ae3b91b239668df781ea.jpg

Advanced users are embedded Quality Control

Image from: http://qainsight.net/category/qa-comics/

Advanced users influence novice users

Image from: http://blog.ocad.ca/wordpress/fcdc/files/2014/10/college-clip-art-9iRq8p4ie.gif

Advanced users are decision makers

Image from: http://agilevietnam.com/2012/12/28/understanding-decision-making/

Discovery-centric Library Homepages

2010-2015

2010 - Pre-Summon homepage

July 2010 - Summon is launched as “Quick Search” - becomes the default tab

July 2012 - Summon-centric homepage (catalog tab is removed)

From 2012-2015, the catalog was still a visible option, just less prominentvia a drop-down menu

an icon below the main Summon search box

at the same time we also launched SIRSI Enterprise - a next-generation catalog

July 2015: Complete website redesign & rebranding of our Summon instance

Catalog is less visible than ever before

Catalog vs. Summon Usage @ USC (2008-2015)

3 user studies (2013-2015)

Usability testing of homepageFall 2013

● Interviewed 13 users○ 4 undergraduates (3 freshman, 1 senior)○ 4 faculty○ 5 graduate students

● Accessed current homepage● Accessed mock-up of new homepage

Trust

Image from: http://www.gratis-malvorlagen.de/science-fiction/wissenschaftler-2/

“I don’t trust Quick Search. I only use it if it is absolutely necessary….In order for this site to be successful, you have to have success after success after success because people...aren’t going to waste their time…I will admit to you that the odds are slim and none that Quick Search will get me anywhere close to what I want….If I need to search something I don’t go to any USC search engine….I go to Google where I can get things ten times as fast.”

Advanced user #1: Religion Professor

Reliability

Advanced user #2: English Professor

“I could throw an author and title in [Summon] but I know that works very erratically….I often get inundated by irrelevant things…”

Image from: http://www.flickriver.com/photos/mitopencourseware/sets/72157621971051805/

Annual InSync Library Survey Fall 2013

2,736 survey responses

● 2,256 undergraduate and graduate students

● 235 faculty

“Journal articles are SO much easier to find using google if you already know the citation title.”

Need for Precision Tools

Advanced user #3: Cinematic Arts Graduate Student

“Your catalog seems to have problems returning accurate results. For example, searching the exact title of a book sometimes shows that you do not own the book, when you actually do have it.”

Advanced user #4: Law Professor

image from: http://famonthelam.com/wp-content/uploads/2015/02/dc8oKMgce.jpeg image from: http://blogs.bedfordstmartins.com/bits/files/2012/02/digi-liz.jpg

Problems finding known items“Please, please, please fix the search function for books….The results are awful. I'll type in a book's title, and the book itself will be on page two after pages of reviews. Or I'll be looking for a journal article and it'll be impossible to find in a swarm of totally unrelated results.” Advanced user #5:

Undergraduate, Junior

“Make it easier to search for a book by author through the website. Searching by title is easy. It's difficult to search by author.”

Advanced user #6: Social Work Graduate Student

Image from: http://jeannelking.com/wp-content/uploads/2014/08/Closeup-of-good-enough-suspicious-facial-expression.jpg

Image from http://www.huffingtonpost.com/wait-but-why/generation-y-unhappy_b_3930620.html

Search Log Analysis (2014)

http://bit.ly/Summon-relevance

2014 ACRL Study of Known Item Search Queries in Summon Objectives: To compare the relevancy of results when searching for known items in Summon to Google and Google Scholar.

To better understand user behavior and experience when searching for known items via our main search option.

Sample Size: 384 known item queries were examined → in order to compare the relevancy across 3 search tools we reduced these to 278 (by eliminating non-scholarly items--videos, sound recordings, etc.--and items USC did not own)

What kinds of known items are USC users searching for?

n= 384

How are they searching for these known items?

n= 384

Defining Relevant Results● Relevant = a match for the known

item shows up 1st or 2nd in the list of results.

● Partially Relevant = a match for the known item is found 3rd-10th in the list of results.

● Not relevant = known item is not listed in the first 10 results or no results were returned.

Image from: http://relevantinfo.org/wp-content/uploads/2013/09/good-o-meter.jpg

Relevancy of Results for Scholarly Known Item Searches - March 2015

28%

n= 278

Relevancy of Results for Scholarly Known Item Searches - June 2015

21%

n= 278

in other words...

between 177,500 - 288,450 known item searches did

not lead to relevant results in 2014!

Image from: http://cache1.asset-cache.net/xc/538990835.jpg?v=2&c=IWSAsset&k=2&d=69V2Nb-aNPGQep7gMBCUQ2J-GKIQ445o7LyHHB91Fdj6ANhQcooFq0hMe_YiRJd10

failed results = dead ends for users

Image from: https://gigaom.com/2011/03/28/opinion-preaching-to-the-green-choir-a-dead-end/

Causes of “failed” searches After April 2015 Summon Release

n=57

2 Pain Points (Dead Ends): Linking Problems & User Errors

n=57

360Link 2.0We will be launching the new 360Link 2.0 Sidebar to provide more intuitive and visible help when users are faced with error messages or broken links.

Not a perfect solution though...

360Link Sidebar Usability Study September/October 2015

Graduate student: When faced with an error message after clicking a Full Text Online link from Summon:

“When this happens I assume I did something wrong….I would do a Google search to make sure I have the correct article information”

Instead of reporting a problem when faced with these linking problems, she would request the item via interlibrary loan.

User errors● 17% of all queries had 1-2

(47/278) user input errors○ 36% (17/47) of these led

to failed searches ● Types of errors:

○ Typos (transpositions, substitutions, omissions, insertions)

○ incorrect use of boolean Image from: http://human-error.sarkisozlerik.com/human-error/a-lifetime-by-design.html

Help wanted!“Did you mean” suggestions only showed up 8 times (3% - 8/278) for all searches in our study. ● And only for 4 of the (47)

searches with user errors. ● And of these, only 2 of the

suggestions actually led to the known item.

Image from: http://www.debrafine.com/wp-content/uploads/2015/01/image004.gif

Recommendations for Librarians● Decide whether or not to include ALL

databases/vendors in your discovery system○ Especially those that do not support direct linking or

have persistent linking problems○ Be a squeaky wheel! Make sure vendors know that

addressing these problems is a priority.

● Upgrade to 360Link 2.0 and do usability testing to make sure wording is actually helpful to ALL users

Recommendations for ProQuest● Improve query

reformulation suggestions○ consider automatically

redirecting errors (like Google)?

● Continue to improve the relevancy of results for both known item and topic searches

Other Recommendations for ProQuest● Accommodate copy and pasted and formatted citation queries

○ 4% (17/384) of the queries copied and pasted 3 or more citation elements (not necessarily something advanced users do but other studies have found this happening more and more)

● Offer more browsing options● Improve discovery for non-textual known item searches (videos,

sound recordings, etc.) ○ 78% (7/9) searches for videos USC owns did not find relevant

results (related to average # of words in film titles and minimal/poor metadata)

“digital abundance [is] making it harder for [researchers] to feel confident that they had found everything they would need to find in order to reach and defend their research conclusions….Would it be possible...to develop a discovery tool that is designed not to find the best items but rather to provide some assurance that you hadn’t missed something?”

Keep imagining the future of discovery

Roger Schonfeld, June 1, 2015 The Scholarly Kitchen blog

Questions? Comments?

Image from: http://www.opencolleges.edu.au/informed/features/student-driven-learning/

Thank you!Caroline Muglia, Collection Assessmentmuglia@usc.edu

Beth Namei, Reference & Instruction Librarian namei@usc.edu