1. Quick Search It Is Not: Testing Response Times of Traditional and NextGen Catalogs
Nina McHale
Margaret Brown-Sica
LITA Forum 2010

2. Esteemed Researchers
3. Our Research
Forthcoming:
Margaret Brown-Sica, Jeffrey Beall, and Nina McHale, Next-Generation Library Catalogs and the Problem of Slow Response Time, Information Technology and Libraries, Volume 29/4,December 2010, 207-216.
4. Not-So-Quick-Search
5. Our Research Questions
Are NextGencatalogsor traditional catalogs that add NextGen contenttoo slow?
Do 2.0/NextGen features slow them down too much?
6. Our Conclusions
Yup.
Features such as cover art, reviews, tagging, etc., can significantly increase the amount of data, and therefore time, required to return a catalog record page.
Performance factors, particularly speed, should be required criteria for librarians and vendors evaluating and designing products.
7. Speed Standards?
W3C does not set forth standards
Jakob Nielsen
0.1 of a second: feels instantaneous
1.0 second: feels uninterrupted
10 seconds:
About the limit for keeping the users attention focused on the dialogue.
Give the user time-remaining feedback
Jakob Nielsen, Usability Engineering (San Francisco: Morgan Kaufmann, 1994) 135.
8. Our Method
During a busy time during the semester, we recorded response times in seconds of permalinks for three catalog records
Tested our classic/NextGencatalogs and three others
3 books, 5 catalogs, 3 times per day for 13 days=585 data points
Collecting several data points in this way using www.websitepulse.com ensured that data was consistent
9. Additional Catalogs Tested
Library of Congress Catalog
Voyager
Traditional catalog
University of Texas at Austin
Innovative Interfaces
Traditional catalog with added NextGen elements
University of Southern California
Sirsi/Dynix
Traditional catalog with added NextGen elements
10. Books Used
Hard Lessons: The Iraq Reconstruction Experience. Washington, DC: Special Inspector General, Iraq Reconstruction, 2009. (OCLC number 302189848)
Ehrenreich, Barbara. Nickel and Dimed: On (Not) Getting by in America. 1st ed. New York: Metropolitan Books, 2001. (OCLC number 256770509)
Langley, Lester D. Simn Bolvar: Venezuelan Rebel, American Revolutionary. Lanham: Rowman & Littlefield Publishers, c2009. (OCLC number 256770509)
11. Permalink Examples
http://aurarialibrary.worldcat.org/oclc/302189848
http://skyline.cudenver.edu/record=b2433301~S0
http://lccn.loc.gov/2009366172
http://catalog.lib.utexas.edu/record=b7195737~S29
http://library.usc.edu/uhtbin/cgisirsi/x/0/0/5?searchdata1=2770895{CKEY}
12. Testing Tools
WebSitePulse
www.websitepulse.com
Allows testing on any web page/site; does not require server installation
Similar services:
websiteoptimization.com
browsermob.com
tools.pingdom.com
your favorite?
13. WebSitePulse
14. WebSitePulse Results
Horizontal bar:
Gives visual representation of load time for each item (image files, javascript files, style sheets, etc.)
Provides quick indication of sticking points
Table:
provides specifics about file size and delivery time for each
DNS, Connect, Redirect, First Byte, Last Byte, Error
15. Numbers Crunched: Average Response Time in Seconds
Aurarias Skyline: 1.2930
Aurarias WCL: 11.5734
Library of Congress: 2.1530
University of Texas at Austin: 3.4997
University of Southern California: 4.1085
16. Individual Catalog Test Results
After data was analyzed, we took a closer look at each individual catalog, using the Hard Lessons catalog record
WebSitePulse allowed us to take a glimpse at the inner workings of each catalog
Findings confirmed that extra data and load times were from 2.0/NextGen content
17. Skyline, Auraria Library
18. Skyline Test Results: Graph
19. Skyline Test Results: Table
20. Skyline Findings
Missing favicon (item 4)
0.9172 seconds uninterrupted per Nielsen
14 items, for a total of 84.64 K:
9 GIFs
2 CSS
1 JavaScript
Good performance, but an interface that only a librarian could love
21. WorldCat@Auraria
22. WorldCat@Auraria Results: Graph
23. WorldCat@Auraria Results: Table
24. WorldCat@Auraria Findings
Reference & Instruction librarians observations corroborated
10.3615 seconds
31 items, for a total of 633.09 K, to load:
10 CSS files
10 JavaScript files
8 GIFs/PNGs
No single NextGen feature slowed down load time, but multitude of files created unacceptable delay
25. Library of Congress Catalog
26. Library of Congress Catalog Results: Graph
27. Library of Congress Catalog Results: Table
28. Library of Congress Catalog Findings
Overall, second fastest of all five catalogs tested
1.2900 seconds
Only six items and 19.27 K to load:
2 CSS files
3 GIFs
Like Skyline, fast, but has that legacy look
29. University of Texas at Austin
30. UT Austin Results: Graph
31. UT Austin Results: Table
32. University of Texas at Austin Findings
Added NextGen features:
Cover art
LibraryThings Catalog Enhancement
Supports recommendations, tag browsing, alternate editions/translations
2.4674 seconds: user experience interrupted
19 items, 138.84 K
Cover art nearly doubles response time
Item 14: script on ILS that queries Amazon for art
33. University of Southern California: HOMER
34. USC Results: Graph
35. USC Results: Table
36. USC Findings
Slowest among traditional catalogs; Sirsi/Dynix takes longer to make initial connection (Item 1 on graph)
8.7295 seconds (though average was 4.1085 seconds)
16 items, 148.47 K
While attractive and well-integrated, Syndetic Solutions content (cover art, summary, author biography, and table of contents) adds 1.2 seconds to load time
37. Isthe ContentWorth the Wait?
The new database seems based on Amazon.com. I dont need suggestions, and poor ones at that, of related books when I use the library. I dont need to see what other borrowers thought of the book. The information I need is poorly displayed. It is hard to cut and paste. It takes several screens to scan through, instead of the much quicker scroll in the traditional format. It supplies distracting, if not useless information (a picture of the cover, the distance to other librariesas if I need to know how far Provo is).
-Auraria Campus Faculty Member
38. Our Conclusions
Make performance testing part of evaluation process for vendor products
Adhere to industry standards for acceptable response times when testing
Optimize delivery of 2.0/NextGen features as much as possible
Conduct user testing to ensure that the content is worth the wait to their minds
39. Questions? Comments?
Nina McHale
@ninermac
milehighbrarian.net