Discovery Tool Evaluation: A Three-Pronged Approach

Discovery Tool Evaluation: A Three-Pronged Approach

9th Annual AMICAL Meeting and ConferenceSharjah, UAE

Background• AUS implemented Encore Discovery in 2009• Other products like Serials Solutions Summon and

EBSCO’s EDS are actively marketing in the Middle East• Decision in 2010 to evaluate all products• Having Encore installed gave us opportunity to evaluate

products against the same databases, and think about how an ILS-integrated discovery solution compares to a less integrated solution

Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

Methodology• Setting up trials of Summon and EDS• Scheduling vendor presentations of Summon and EDS• Testing all 3 products for ability to produce useful,

relevant and accessible results• Reviewing the features of each of the products• Performing usability testing with all products• Assessing the costs of implementing each product• Assessing the impact of implementing each product on

other library systems and operations


Usability Testing


Informal Usability Testing• Think Aloud Protocol

• 5 participants per product

• Participants only told they would be testing a new discovery tool to find library resources

• Participants used their own research topic to complete realistic tasks

• Observers did not interact with participant except to give tasks and encourage thinking aloud


Testing Environment• 1 test administrator • 1 observer• Laptop with built in microphone for recording voice• Camtasia Studio with Powerpoint plug-in to capture

screen and voice• Browser of choice• Other software, websites as requested

– Google and Microsoft Word for spelling


TasksBased on common assignments requirements seen at the Research Help Desk, participants were asked to find:

• an article• an article from a peer reviewed/scholarly/academic

journal• a newspaper article• a reference book• a general collection book• a book published in 2008 or 2009


Observations• Limit / facet confusion

– “Format” or “Content Type” vs “Location” – Books, e-books, e-journal

• E-journal format was used to find articles

• Limits reset for each new search

• Click through to article– Expect to get article, but given a WebBridge or 360 Link page


System Usability Scale• Participants filled out System Usability Scale

questionnaire for perceived ease-of-use score

• 10 item Likert scale questionnaire used to determine user’s perception of the system for– Effectiveness– Efficiency– Satisfaction

• Resulting scores range from 0 to 100– Average SUS score across 500+ studies is 68


System Usability ScoresSystem Usability Scale Encore EDS Summon

1. I think that I would like to use this system frequently 4 3.2 4.4

2. I found the system unnecessarily complex 2.4 3 2.4

3. I thought the system was easy to use 4 3.4 4.2

4. I think that I would need the support of a technical person to be able to use this system 2.4 2.2 1.8

5. I found the various functions in this system were well integrated 4.4 3.4 3.8

6. I thought there was too much inconsistency in this system 2 2.6 1.4

7. I would imagine that most people would learn to use this system very quickly 4 3.2 4

8. I found the system very cumbersome to use 2.2 2.4 1.8

9. I felt very confident using the system 4 3.6 4

10. I needed to learn a lot of things before I could get going with this system 2.2 3.2 2

Average System Usability Score 73.5 58 77.5


Overall Usability• Summon

– Pop-up abstract and number of results for limit in brackets were very popular

• Encore – Worked very well for books– Article search interface (Synergy) not well understood or used

• EDS – Limits not intuitively named, extra “Update” click to refine– Can set mutually exclusive limits giving 0 results

• Preference to use web for finding newspaper articles


Relevancy and Functionality


Relevancy Test• Testers - AUS public service librarians• Searches - Actual searches done this year in Encore• Scoring - Scale 1 - 5 + comments• Scorecards - One scorecard for each search in each

discovery tool. Extensive comments• Discovery tools tested

– Encore – Summon– EDS


Actual SearchesSelected from Google Analytics




Encore URLs• /iii/encore/search/C|

Scarbon+fiber+reinforced+polymer|Orightresult|U1?lang=eng&suite=pearl

• /iii/encore/search/C|Sglobalization|Orightresult|U1?lang=eng&suite=pearl


Searches• arranged+marriage• carbon+fiber+reinforced+polymer• Electronics+All-In-One+Desk+Reference+For+Dummies• electronics+for+dummies• globalization • jean+luc+godard• sex+education• uae+federal+laws• walt+disney• "Artificial+intelligence"+And+"Manufacturing"


Types of Results• Top 5 results

– Number of: • Books • Articles• Other (e-books, e-journals, DVDs, e-resources,

print journals, videos, etc)


Types of ResultsBooks Articles Other

EDS 1 2 3 0EDS 2 0 0 5 reportsEDS 3 1 2 2 e-booksEDS 4 1 4 0EDS 5 0 5 0

Summon 1 0 5 0Summon 2 0 3 2 newspapersSummon 3 3 2 0Summon 4 1 3 1 e-journalSummon 5 0 5 0

Encore 1 3 2 0Encore 2 3 2 0Encore 3 1 2 2 e-booksEncore 4 2 2 1 e-journalEncore 5 3 2 0


Relevancy AssessmentOverall reaction to top 5 results:

5 Excellent (just what I was looking for)4 Very good (on target and useful)3 Fair (acceptable)2 Poor (not useful)1 Bad (unrelated to topic)


Test Searches Scorecard

Discovery Tool

Test Search

Top 5 results

Number of:

Books

Articles

Other (e-books, e-journals, DVDs, resources, print journals, videos, etc)

Relevancy assessment – overall reaction to top 5 results:

5 Excellent (just what I was looking for) 4 Very good (on target and useful) x 3 Fair (acceptable) 2 Poor (not useful) 1 Bad (unrelated to topic)

Comments:

Books were excellent, articles were poor.

Encore Synergy

3

2

arranged marriage


ResultsTester 1 Tester 2 Tester 3 Tester 4 Tester 5 AVG Total Score

Summon 1 1 1 4 1 1 1.6 8Summon 2 3 3 3 4 3 3.2 16Summon 3 5 5 5 5 4 4.8 24summon 4 4 5 5 5 3 4.4 22Summon 5 4 5 4 5 3 4.2 21SUMMON TOTAL 17 19 21 20 14 18.2 91

EDS 1 4 4 4 4 3 3.8 19EDS 2 2 3 2 3 2 2.4 12EDS 3 4 4 4 4 4 4 20EDS 4 4 4 3 3 3 3.4 17EDS 5 4 4 4 4 3 3.8 19EDS TOTAL 18 19 17 18 15 17.4 87

Synergy 1 2 3 1 3 3 2.4 12Synergy 2 3 3.5 2 3 5 3.3 16.5Synergy 3 2 3 3 5 2 3 15Synergy 4 2 3 3 2 3 2.6 13Synergy 5 4 4 2 4 4 3.6 18SYNERGY TOTAL 13 16.5 11 17 17 14.9 74.5


Functionality


Functionality ConclusionsBasically, all 3 systems offer similar user functionality – sorting, limiting, emailing and “did you mean” are available. There is no superior feature that would make one system preferable to another, just as there is no missing feature that would eliminate a system.


Impact Assessment


Pieces & Configurations

A-Z List Link Resolver Discovery Federated Search

1 Existing 360 Core WebBridge Encore Synergy Research Pro

2 Summon 360 Core 360 Link Summon na

3 Summon 360 Core WebBridge Summon na

4 EDS EH A-Z LinkSource EDS

5 EDS 360 Core WebBridge EDS

New


Functional Equalization• EDS Federated Search• EDS & Summon – Millennium Integration Pieces• III Mobile Interface (free if Airpac)• III Encore Harvesting• III Query API – in development


Costs• Sunk Costs to write off

– Encore Server– WebBridge License

• Ongoing Costs– Quotes for each component


Work Needed for any Change• Implementation

– Profiling of resources and catalog records– Interface customization where possible– Changes to website– Changes to electronic resources– Staff training– Modify instruction materials and curriculum

• Ongoing Work– Bibliographic Data Loads– Changes to Holdings Loads for EDS


Potential Lost Functionality• Tag Clouds & Authorities for related searching• Availability Facet• WebBridge/Pathfinder for Collection Maps• User tags• Relevancy modification• Millennium integration

– Automatic update/syncing– Hold placement– Collection map links– My Lists– Saving RP Groups in My Millennium


Potential Added Functionality• Administrative interface for searching & user interface• Mobile Interface bundled in• Repository Integration bundled in• API’s bundled in• 1-Click in 360 Link


Final Scoring

Factor Summon EDS Encore Weight Summon EDS Encore

Relevance 3 2 1 0.3 0.9 0.6 0.3

Usability 3 1 2 0.3 0.9 0.3 0.6

Financial Impact 3 1 2 0.2 0.6 0.2 0.4Implementation Work Impact 1 2 3 0.1 0.1 0.2 0.3

Ongoing Work Impact 2 1 3 0.1 0.2 0.1 0.3

12 7 11 2.7 1.4 1.9


Thank you

Thomas Hodge [email protected] Botter [email protected] Ann Nash [email protected]

Date post:	23-Feb-2016
Category:	Documents
Upload:	gitano
View:	56 times
Download:	0 times

Discovery Tool Evaluation: A Three-Pronged Approach

Documents