User-Centered Evaluation of a Discovery Layer System with Google Scholar
Tao ZhangDigital User Experience SpecialistPurdue University [email protected]
Thursday, July 25, 13
Library Website is still Important
Breadth of scholarly resourcesCredible, highly relevantGuaranteed full text access
Thursday, July 25, 13
User Expectation
“The problem with the databases is that you can’t content yourself with just searching one place, like you can with Google. It is hard for the students to differentiate between the different databases.”
Akselbo et al., The Hybrid Library: From the Users’ Perspective, 2006
Thursday, July 25, 13
Thursday, July 25, 13
Thursday, July 25, 13
Discovery Layer System - Ex Libris Primo®• Uni"ed search interface for (almost) all library resources
• Catalogs, databases, e-journals, digital collections, and repositories
• Increased discoverability
• Easier, faster, and more efficient search interface
• Faceted browsing/"ltering of search results (e.g., by format, date, subject, topic, etc.)
• Record grouping (e.g., multiple editions or versions of the same material)
• Consistent “Get it” option
Thursday, July 25, 13
Thursday, July 25, 13
Study Goals• Compare users’ subjective assessment of search
results from a discovery layer system (Ex Libris Primo) and Google Scholar
• Evaluate the discovery layer system for usability issues
Thursday, July 25, 13
Method• 12 graduate students (mean age = 26.3, SD = 3.2)
• Search tasks on Primo and Google Scholar
• Given topic for "rst three tasks: classroom assessment techniques, food addition, natural language processing
• Participant own keywords for fourth search task
• Rate "rst ten results from 1 (highly irrelevant) to 7 (highly relevant)
• Within-subject design, balanced order of search systems, randomized tasks
• After search tasks: System Usability Scale (SUS) ratings, preference ratings, open questions
Thursday, July 25, 13
Relevancy Ratings
Search 1: Article 9 and 10 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"Ra,ng"
Ar,cle"2"Ra,ng"
Ar,cle"3"Ra,ng"
Ar,cle"4"Ra,ng"
Ar,cle"5"Ra,ng"
Ar,cle"6"Ra,ng"
Ar,cle"7"Ra,ng"
Ar,cle"8"Ra,ng"
Ar,cle"9"Ra,ng"
Ar,cle"10"Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
Relevancy Ratings
Search 2: Article 10 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"Ra,ng"
Ar,cle"2"Ra,ng"
Ar,cle"3"Ra,ng"
Ar,cle"4"Ra,ng"
Ar,cle"5"Ra,ng"
Ar,cle"6"Ra,ng"
Ar,cle"7"Ra,ng"
Ar,cle"8"Ra,ng"
Ar,cle"9"Ra,ng"
Ar,cle"10"Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
Relevancy Ratings
Search 3: Article 3 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
9"
Ar-cle"1"Ra-ng"
Ar-cle"2"Ra-ng"
Ar-cle"3"Ra-ng"
Ar-cle"4"Ra-ng"
Ar-cle"5"Ra-ng"
Ar-cle"6"Ra-ng"
Ar-cle"7"Ra-ng"
Ar-cle"8"Ra-ng"
Ar-cle"9"Ra-ng"
Ar-cle"10"Ra-ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
Relevancy Ratings
Search 4: Article 3 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"Ra,ng"
Ar,cle"2"Ra,ng"
Ar,cle"3"Ra,ng"
Ar,cle"4"Ra,ng"
Ar,cle"5"Ra,ng"
Ar,cle"6"Ra,ng"
Ar,cle"7"Ra,ng"
Ar,cle"8"Ra,ng"
Ar,cle"9"Ra,ng"
Ar,cle"10"Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
Results• No signi!cant difference of relevancy ratings between
given and participant chosen searches
• Usability ratings and preference of Google Scholar signi!cantly higher than Primo
• SUS: 4.27 (SD = 0.29) vs. 3.59 (SD = 0.43)
• Preference: 8.33 (SD = 0.65) vs. 7.08 (SD = 1.38)
• Usability correlated to preference ratings (r = 0.59, p < 0.01)
Thursday, July 25, 13
Usability Issues of Primo - Work$owSearch Results List
Identify Article
Link Resolver
Article Page
Thursday, July 25, 13
Usability Issues of Primo
!
Thursday, July 25, 13
Usability Issues - Consistency
Thursday, July 25, 13
ConclusionPrimo’s search relevancy may be on par with Google Scholar, but usability issues have affected users’ perceived usability and preference.
User interface of discovery layer systems needs to address work$ow and consistency issues.
Thursday, July 25, 13
What’s More• “Blind” test on relevancy ratings
• Test large sample of search keywords
• Server log analysis on keywords, facets, session length, etc.
• Better (Google Scholar-like?) link resolver
• UI improvement
Thursday, July 25, 13