VIUS Reports 8The UW interface has the same features as the VIUS --keyword box to search across all...

8.4 Think Aloud Protocol Study of CONTENTdm ™ Interface Page 1 of 29

VIUS Reports 8.4 Think Aloud Protocol Study of CONTENTdm™ Interfaces

By James Frost

July 3, 2003

This user protocol analysis examines interfaces for CONTENTdm™ created by DiMeMa, by Penn State, and by the University of Washington. User Protocol Analysis has been defined as:

A user protocol is a structured, exploratory observation of clearly defined aspects of the behavior of an individual performing one or more designated tasks. The purpose of the protocol is to gather in-depth insight into the behavior and experience of a person using a particular tool or product. User protocol studies include multiple research subjects to identify trends or patterns of behavior and experience. Data gathered from protocols provide insight into what different individuals do or want to do to perform specific tasks. (Denise Troll author of Usage and Usability Assessment: Library Practices and Concerns, Digital Library Federation and Council on Library and Information Resources, January 2002. (available online at: http://www.clir.org/pubs/reports/pub105/contents.html))


CONTENTdm™ User Protocol Study: Executive Summary Subjects: 25 subjects with 3 different interfaces: 10 Custom VIUS interface, 10 with the University of Washington’s (UW) Custom interface, and 5 with DiMeMa’s out-of-the-box interface. Interfaces The VIUS interface is very simple, almost like Google, with a single search box plus the option to restrict searches by keyword, fields, and changing the Boolean relationship between search terms. The default is to search across all fields in all collections. The UW interface has the same features as the VIUS--keyword box to search across all fields in all collections plus the ability to limit the keyword search by subject areas. This took some extra programming and metadata. The UW database offers the out-of-the-box interface as the “advance search” option which gave the study more data about that interface. The DiMeMa’s out-of-the-box interface for CONTENTdm™ has four search boxes where each box can be limited by field. There is also “Browse Terms” option for search by terms appearing in each field. This can be set by the system administrator to include any term that appears in the selected field or for only authorized terms, names, and phrases. Results: Most results of a user protocol are about minute details such as confusing text or buttons and bottlenecks in the interface that make finding relevant records and using database features more difficult. See the complete report for examples of these. Sometimes it is possible to get to a few larger picture issues. In this case, the following were found: 1. For both the VIUS and UW custom interfaces users found what they were looking for

99% of the time without any help. One “worst case scenario subject” who never used an online database before had a tough start but the learning curve was fast enough that he became proficient in 15 minutes. There were almost no searches with zero results. On the other hand, with DiMeMa’s interface users frequently needed help and were repeatedly stymied by obtaining zero search results.

2. The simple Google like VIUS interface worked but only because the task list was

focused on architecture images and the VIUS database is essentially, at this point, an architectural image database. The UW database contains a much more diverse collection of pictures and when users tried to find architectural images in this more diverse subject setting using the equivalent of the VIUS standard search (keywords across all fields and all collections) the users often retrieved numerous irrelevant


search results. In fact, they received so many irrelevant search results they were forced to find out how to limit their searches by subject area which was extremely effective in eliminating irrelevant results. When in the proper subject area the UW interface worked as effectively as the VIUS interface. This does not highlight a weakness of the UW interface. Quite the contrary, this illustrates the necessity for developing specialized subject area search pages as the diversity of pictures in the database grows. The UW site effectively handles this as users had no problem finding that specialized search page once they understood their need for it. This is a valuable lesson for the VIUS database since as of this writing it is currently an architectural database but will become more like UW database as the diversity of collections is expected to increase.

3. The DiMeMa’s out-of-the-box interface was very difficult for the users to use. While

theoretically offering the ability to precisely limit the search the actual real-world results were that “No matches found” was the common result. Users generally did not know which fields from the long list of possible fields were appropriate for the keywords they had selected. Frequently the wrong field was selected. Additionally, users often felt compelled to fill in more than one box since four were available. This over specificity also tended to result with “No matches found.” Users who found their way to this interface via the “advanced search” feature in the UW site invariably returned to the custom interface to find their images. Those who were testing DiMeMa’s interface alone generally had a more difficult time finding the images as they experienced frustration in getting their search terms and proper fields coordinated. A greater percentage of users needed help in accomplishing the tasks whereas almost no users needed help when using the two custom interfaces. The “Browse Terms” box was completely unused. Since neither UW or VIUS used the controlled vocabulary the result was that the “Browse Terms” contained a long list of seemingly random words (including articles and prepositions) and numbers. This made the feature unusable and most subjects could not even figure out its purpose.

4. The “My Favorites” features, while not as user-friendly as they could be, were found

usable by nearly all subjects. The feature set and format of “My Favorites” did not vary by interface as this part of CONTENTdm™ is not customizable. All subjects could perform all tasks related to this section. Some subjects experienced frustration in accomplishing these tasks and a few changes would eliminate these problems. On the plus side, there were no “show stoppers” in terms of design problems. On the other hand, many users mentioned that in real world usage they would most likely “just download” the images and use them with other software.


Introduction This user study is one of the final steps in the assessment portion of the Visual Image User Study. Previous steps have included surveys, focus groups, and personal interviews as a means to gain a general understanding of digital image use, to select and refine a digital image delivery system for the University Libraries, and, finally, to gauge the effectiveness of that system. Methodology Subjects were a mix of Penn State students that included both graduates and undergraduates from a variety of majors. Each subject’s session was scheduled for a maximum of one-hour. Generally, it took nearly the full hour to complete all components of the session. Each session included; an introduction to the study; explanation of the think-aloud procedure and recording methods; signing the consent forms; completing the task list; debriefing of their experience using the database and obtaining any suggestions they may have; completion of the two page survey; and, paying the subject $20. Each subject was asked to work on only one interface. The VIUS custom interface, UW interface, or DiMeMa’s out-of-box interface for CONTENTdm™ was pre-selected before their arrival so it would be a true first-time experience with the selected interface. The only exceptions to this were several subjects who were assigned the UW interface and during the course of completing the tasks selected “Advanced Search” which is DiMeMa’s interface. There were a total of 25 subjects. Ten subjects were assigned to both the VIUS and UW custom interface and 5 were assigned to DiMeMa’s interface. The subjects took between 10 and 40 minutes to complete a ll the tasks on the list with an average of around 20 minutes. Subjects were given identical task lists regardless of which interface they were asked to use. Indeed, this list was devised in advance so that one list could be accomplished in both databases. There were nine tasks on the list and they carefully designed so that they would incorporate all the features of a CONTENTdm™ driven database. The actual task list can be found in the Appendix. The 25 sessions were conducted over a two month period from February 24 to March 25, 2003. There were extremely minimal, if any, changes to either database during this timeframe. Each session was observed by two researchers who took notes. Each subject’s verbalizations were recorded along with the image displayed on the computer monitor using the Camtasia™ recording software. This report is being written by the researcher who attended all 25 sessions.


Detailed Results Introduction Three interfaces were tested—DiMeMa’s out-of-the-box interface, and the VIUS and UW’s custom interfaces. For a more detailed understanding of each interface please be sure to check the appendices which contain in-depth descriptions as well as screen shots of all interfaces discussed in this report. Comparing the VIUS interface to the University of Washington’s interface was pretty interesting. The VIUS database is essentially an architectural database at the time of this writing but the subject areas of the pictures is expected to diversify over time. In this sense, the University of Washington, with its more diverse collection of pictures, represents something that the VIUS database may grow into over time. Is the UW model a good one to follow? It is not too surprising that subjects who were asked to find architectural images in a strictly architectural (VIUS) database tended not to experience many problems. However, how do subjects fare when they are asked to find the same architectural images in a database that contains not only architectural images but many other subject areas as well? This is a more difficult task with some potential pitfalls. As this report will detail, the design of the UW site effectively deals with these potential problems and suggests that the approach taken by UW is a good model to follow. Also included in the comparison is DiMeMa’s out-of-the-box interface. This is the interface that is included with the CONTENTdm™ software and is very different from either of the custom interfaces. Is this different approach worthwhile? Results suggest that it is not. Some areas of the CONTENTdm™ databases are fully customizable and, hence, varied more between the VIUS, UW, and DiMeMa’s interfaces. These more variable sections received different reactions from the users. For this reason, the DiMeMa, VIUS and UW search pages sections are kept separate in this report since they are the most customizable. On the other hand, other areas are minimally or not customizable at all. As a result, the search results, item display, and My Favorites sections did not vary much between the three interfaces and tended to receive similar comments. In these cases, the results between the three interfaces are combined in order to avoid repetition. This report will look at the following in this order: search pages, search results, image and metadata display, My Favorites, and saving pictures to the hard drive.


Detailed Look at the Customized Search Pages

VIUS Main Search Page The vast majority of the subjects who used the VIUS custom interface were able to complete all the tasks without any help. There were a number of comments about the nice, simple search page. While the users generally did not limit the search by fields or collections a large number of them did look at the pull down menus and commented on how seeing the fields would help them pick their search terms. Further, the same was true of the Boolean radio buttons immediately under the keyword search box. No users changed the setting from the default “AND” setting but many commented on how seeing how the search operated helped them pick their terms. In contrast, the UW site did not have either of these features (field listing and Boolean radio buttons) on the main keyword search page and there was some confusion about how the search was going to be conducted. This was not a serious issue on the UW interface but the simple inclusion of these on the VIUS interface reassured a portion of the users. On the down side, the names of the collections under the “Image Collections” scroll box did not match the collection names under the “Browse” on the top banner. This caused confusion when users were asked to search a specific collection that showed up in one place but not another. Separate terms should be used (ie. do not use “collections” for both) in order to distinguish were to look. UW Main Search Page—Searching Across All Collections The subjects were started out at the main, cross-collection UW search page (see appendix) where they are presented with the option to search across all fields in all collections or they can click on a specific subject to search. The subjects were nearly evenly split between whether they did their initial search on this cross-collection search page or whether they selected a specific subject area to search within. Using the keyword search box on UW’s main search page was functionally equivalent to the VIUS standard search of searching across all fields in all collections. For those subjects who tried the cross-collection search page they frequently encountered a problem that was not present with the VIUS interface. The search terms that they used would frequently pull up irrelevant results. It soon became apparent that the VIUS standard search worked so well mainly because in the VIUS database the subjects were searching for architectural images in a strictly architectural database which virtually eliminated irrelevant results. The fact that this did not work so well in the more diverse UW interface was an important lesson learned in this user study. This prevalence of irrelevant results initially confused subjects--especially when the object was not one they were already familiar with. A good example of this was the frequent search for “San Pietro” which called up 34 matches but the first two screens of search results were wholly irrelevant. While this proved confusing to the subjects initially, in general, they realized they needed to limit their search somehow. About half soon found the “Arts and Architecture” subject area discussed below. Other subjects


tried adding more keywords such as a search for “San Pietro Rome” which worked equally as well. There were no cases in which the irrelevant results prevented subjects from finding the images they needed. It is important to note that this problem of irrelevant results did not afflict all searches. For example, another common search for this study was “Florence cathedral” which retrieved no irrelevant results at all. In this sense, the tasks we chose dictated whether the subjects would be subjected to irrelevant results or not. Tasks were chosen on the basis that they could be performed in both the VIUS and UW database but this problem of irrelevant results was not a selection criteria and not something we tested for in advance. In this light, it is reasonable to assume that irrelevant results will grow as a problem as an image database diversifies and that this problem will affect users differently depending on their specific searches. In UW’s database, their structure of both cross-collection and subject specific search pages effectively dealt with this problem when it arose. The only other problem on the cross-collection search page was the pull-down menu containing a list of the various collections. This proved confusing since the collections did not match the subject areas. The relationship between collections and subjects in this database was not clear to first-time users and they were generally puzzled momentarily pondering using “collections” versus “subjects.” Since the pull-down menu was both harder to read than the larger subject area icons and since the collection names were generally not as easy to interpret as the subject areas the end result was that almost no subjects used this pull-down menu despite the fact that approximately a third to half looked at the pull-down menu to see what it contained. A few did use it when asked to find the specific “Ancient Near East Photographs” collection however more simply found it in the “Arts and Architecture” subject search page. Perhaps the pull-down menu would be useful for repeat users who are already aware of the specific collection they need and are looking for a shortcut to it. However, exploring this was beyond the one-time nature of this user-study. As users worked their way into the subject specific search pages which listed various collections the relationship between subjects and collections seemed to become clearer. As indicated, many subjects liked the highly visible subject icons and made use of the subject specific searching as discussed below. UW Subject Specific Searching Please note the following terms for this discussion: “collections” are the base division in this database. “Subject area” is a higher level division that is comprised of more than one collection. Further, collections can reside in more than one subject area depending upon their contents. Each subject area has its own search interface which will search for the keywords across all fields within the various collections that fall under the subject area. The subject area page also contains detailed descriptions of each specific collection in that subject and links to those collections. Additionally, each image collection has its own search interface.


The subject specific search screens really aided the users and nearly all users eventually made use of this feature. Due to the nature of the tasks we requested of the subjects, those that chose a specific subject area all selected “Arts and Architecture.” About half selected the “Cities and Building” image collection within the “Arts and Architecture” subject area. Functionally, “Cities and Buildings” is nearly identical to “Arts and Architecture” since the former collection comprises 86% of the latter subject area. Many users either selected the subject specific search interface either initially upon seeing the cross-collection interface thanks to the highly visible subject area icons or after a cross-collection search yielded irrelevant search results. Either way, the number of irrelevant results was lower in the subject specific areas and user confusion was lessened. UW Collection Specific Searching There are also search screens for searching within specific collections. A number of the image collections have pull-down menus to aid in searching. The predefined search options in the “Cities and Buildings” collection is a pull-down list of cities and countries. As with the pull-down field selection on the VIUS main search page, the subjects tended to look at this pull down menu, gained valuable information from its contents but did not use it. The main information they gathered was that this was a collection which contained images for the country (Italy mainly) that they needed. The main reason they ended up not using this pull-down menu was that it did not yield a specific enough search to suit their needs. For example, using “Italy” from the pull-down menu will yield 249 results spread out across 17 pages. At this point, subjects who did try the pull-down menu suggested a “search within these results” option on the search result pages echoing subjects who used the VIUS interface. Subjects who did try the pull-down menu generally went back to the keyword search box and included the word “Italy” along with additional terms and found their images that way. Based on the close proximity of the keyword box and pull-down menu, a number of subjects thought it would be possible to combine the two. For example, they thought it would be possible to type in “cathedral” in the keyword box and “Italy” in the pull-down menu to get a combined search for “Italy AND cathedral.” When they found that this did not work several subjects expressed that being able to combine searches in this manner would be helpful. Even though the pull-down menu proved to be ineffectual for searching it did reassure subjects that they were in the right area. UW Search Page Summary All in all, the subjects had a number of favorable comments about the UW interface arrangement that employed subject area search icons in combination with subject and collection specific search pages. Beyond the technical reasons of having the subject areas to reduce irrelevant results, the highly visible subject icons served other purposes the subjects appreciated such as: identifying the general subject areas included in the database; leading to subject specific search screens that provided much more specific detail about what was in each collection than any cross-collection screen could; and, generally served to reassure the user that what they were searching for was, in fact, in


the database and that they were searching in the correct area. In short, they felt it was a nice search environment that assured them what the database contained and where to find it. This environment was created by everything from the large subject icons to the ever increasing detail as one progressed from the all collections search page to subject area search pages and, finally, to the collection specific search pages. DiMeMa’s Search Page Users came across DiMeMa’s interface by one of two ways. They were either assigned to it in which they used the interface that was set to use the same database of images as the VIUS custom interface. These users had no option of getting out o f the DiMeMa interface. Other users came across DiMeMa’s when they selected “Advanced Search” in the UW interface. Both sets of users had very similar problems with using DiMeMa’s interface. Without exception, all users who had come across it in the UW custom interface exited DiMeMa’s interface to return to UW’s interface. An important note before discussing the findings. As described above, the DiMeMa interface was used on both the contents of the VIUS and UW databases. It is important for the reader to note the distinction between using the DiMeMa interface on either the VIUS or UW databases as opposed to using either custom interface on their respective databases. The first step of using DiMeMa’s interface is to select the collection or combination of collections to search across. This is vitally important since DiMeMa’s default setting is to only search across the first collection listed in a listing of all collections . Despite being vitally important the DiMeMa interface does not tell the user this. There is a top banner which indicates the specific collection that is being searched or whether multiple collections are being searched. However, most users did not notice this right away. Since they were first-time users they were unfamiliar with the arrangement of collections and just went straight to searching without checking this. UW had an introductory page about using the DiMeMa interface that warned about this but it went largely unread by users. When using the DiMeMa interface on the VIUS database this was not a problem since the architectural collection was the default collection. However, it was a major problem when using the DiMeMa interface on the UW database since the default is the “Alaskan Yukon Pacific Expositions Photographs” which did not contain any images relevant to our task list. After many searches with zero results and mounting frustration, most users abandoned the DiMeMa interface and went back to the UW interface. However, some did determine the source of the problem and went about selecting the collections. A better default option for the DiMeMa interface would be to search across all the collections and then let users remove collections as needed. In terms of actually selecting the collections, the DiMeMa interface was theoretically more flexible in terms of allowing the user to precisely select which collection or combination of collections to search across. The custom UW interface only allowed searching across one collection, all collections, or a predefined subset of collections. The VIUS custom interface allowed those options plus the ability to search across any


user defined subset of the collections just like the DiMeMa interface. However, unlike the DiMeMa interface, the VIUS custom interface default was to search across all collections and this became the de facto searching method. This proved to work rather better than the DiMeMa default described in the preceding paragraph since the majority of users did not pay attention to the arrangement of collections. It should also be pointed out that the ability to search across any user defined subset of collections was not missed in the UW custom interface since the subject areas (the predefined subsets of collections) were so useful. As stated above, the issue of selecting the collection to search across was not such a big problem using the DiMeMa interface on the VIUS database. The VIUS database was comprised of only 3 or 4 collections and the default collection turned out to be the proper collection for all but one of the tasks. However, the UW database contains 45 collections for which many of the names do not give a good idea of the contents. On top of this, there apparently is no way to add descriptions of these collections to the collection selection page itself or to create important subsets of collections. There was no way to use UW’s subject areas with the DiMeMa interface. This illustrates a problem with DiMeMa’s interface when the number of collections expands. This was all very confusing to the UW users attempting to use the “Advanced Search” (DiMeMa’s interface). Many had returned to the custom interface at this point but those who went on faced further problems. Like their approach to selecting collections, DiMeMa’s approach to searching was also very different than those taken by the two custom interfaces. With the possibility to limit multiple keywords by multiple fields, the DiMeMa interface theoretically offered the greatest ability to precisely limit the search compared to the other two interfaces. The actual real-world result of this over-precision was that the interface was very difficult for the users to use. The two major problems were selecting the proper fields and over-specificity. Users generally did not know which fields from the long list of possible fields were appropriate for the keywords they had selected. Frequently the wrong field was selected and, hence, there would be no match. The bottom of the four boxes had the default setting of “Across all fields” which would have helped them drastically. However, in all cases this escaped the notice of the users as this option was never used even once by any of the subjects. As mentioned in other parts of this report, users tend to not look at the bottom of pages since they assume the most important items are at the top. Selection the inappropriate fields for keywords frequently produced “No matches found.” Additionally, users often felt compelled to fill in more than one box since four were available. Most users did not use all four boxes they almost always used more than one. Instead of using “search across all fields” (as discussed above) they would select a specific field for each keyword. The subjects generally had a very difficult time getting their search terms and proper fields coordinated. This over specificity also tended to result with “No matches found.”


The extremely common result of “No matches found” produce much irritation in the users of the DiMeMa interface. Users who found their way to this interface via the “advanced search” feature in the UW site invariably returned to the custom interface to find their images. Those who were testing DiMeMa’s interface alone on the VIUS database frequently asked for help from the researcher. The researcher wanted to avoid providing help as much as possible however it was frequently necessary when subjects used the DiMeMa interface. In contrast, no users needed help when using the two custom interfaces. Recommendations for Searching in DiMeMa’s interface There are two major suggestions for solving many of the problems of the DiMeMa interface. One, make searching across all collections the default collection setting. Two, for all four of the field selection pull-down menus, make “across all fields” the default field setting. These first two suggestions are based on the observation that the default settings often become the de facto searching methods. As is, the DiMeMa defaults tend to result in the dreaded “No Matches found.” The new proposed defaults would tend to get more hits. Irrelevant search results may crop up but these appear to cause less frustration among users than “No Matches found” plus the student users seemed more adept at handling irrelevant results. DiMeMa’s Browse Term’s Box The “Browse Terms” box was completely unused. Since neither UW or VIUS used controlled vocabulary the result was that the “Browse Terms” contained a long list of seemingly random words (including articles and prepositions) and numbers. This made the feature unusable and most subjects could not even figure out its purpose. This lack of use exacerbated the field selection problem described earlier. At the very least, in the eyes of the users, the presence of this box with an unknown purpose did not detract from their experiences in any way. However, this lack of use is a shame since guidance with selecting the proper terms for specific fields is precisely what is needed when using the DiMeMa interface. As a result, making the purpose and usage of the browse box more clear to the users is extremely important if the DiMeMa interface is to be used. Recommendations specific to this browse box includes removing any field from the pull-down menu that does not use controlled vocabulary since they are utterly unusable. As a corollary to this, be sure to have some fields with controlled vocabulary. A particularly useful field for this would be a “subject area” field with controlled vocabulary. This would allow the users to use something like UW’s subject areas even within the DiMeMa interface. Finally, a better description for the purpose of the browse box than “select field to match terms” is urgently needed. Possible examples include, “select field for common searches” or “select field for particularly useful searches.”


DiMeMa’s Interface Conclusions Taking the recommendations for the Browse Terms box in conjunction with the search related recommendations discussed earlier should result in an out-of-the-box interface that is much more usable than it currently is. With the exception of a controlled vocabulary field these suggestions should not entail too much additional work. Overall Search Page Conclusions for the VIUS, UW, and DiMeMa Interfaces Perhaps it is unfair to compare the DiMeMa search page interface to the two custom interfaces. After all, the two custom interfaces were designed to specifically serve their respective databases while the DiMeMa interface was meant to try to work as best as it could with any database. Maybe the best lesson to take away from this is that putting in the additional programming and metadata work is an investment in the database that, if done correctly, will be appreciated by the database users. In terms of work performed locally, the spectrum that these three interfaces falls across would be using the DiMeMa interface requiring the least amount of effort, the VIUS interface in the middle, and the most local work being put into the UW interface. Clearly the DiMeMa interface fell short of the two custom interfaces—although a couple of changes could improve it somewhat. On the other end of the spectrum, the UW interface, the most intricate, serves the more diverse database. It has been shown through these user studies that using the equivalent of the simpler VIUS interface on this more complex UW collection was not as effective as using the full features of the more multifaceted UW interface. Summarizing the above, using the interface that required the least local effort produced the worst results. The one requiring the most local effort handled the more complex database with aplomb whereas the o thers could not. This should be an important note to potential database administrators that investing effort in developing a sophisticated interface will not only increase user satisfaction but may well be necessary to prevent loss of users due to frustrations brought about by issues discussed in this report. Developing the interface should also be an ongoing concern since as the complexity and diversity of the database increases it will be necessary refine the interface in order to simply avoid “losing ground” in terms of searching effectiveness. Search Results Thumbnails per page The search result page is an area that is minimally customizable. One of the few areas that appears to be changeable is the number of thumbnails that are displayed per page. It turns out that this did vary between the two customized databases. The VIUS database displays 30 thumbnails on each search results page while the UW database displays between 6 to 15 per page depending on the section of the database. A number


of VIUS users commented that the size and number of thumbnail images presented on each page was just right. A few suggested that even more, smaller thumbnails per page would have been appreciated. Users of the UW database generally did not appreciate the pages with only 6 thumbnails and expressed a desire for more. DiMeMa’s interface which only displayed 4 thumbnails at once received similar negative comments. Numerous users indicated they would rather scroll down longer pages of thumbnails and quickly skimming them rather than have to click between more pages of fewer thumbnails since that makes comparing images more difficult. There were no comments good or bad about the UW pages with 15 thumbnails. It is highly recommended that image databases use pages with at least 30 thumbnails since that seems to have generate the most positive response. Top banner The other area of the search results page that is customizable is the top banner area. The VIUS top banner offered the options of: New Search, Browse, My Favorites, Tell Us, and Help. The UW offered users: Home, Explore Collections, Advanced Search, About this site, and UW Libraries. Users particularly liked the “New Search” and “My Favorites” options on the VIUS top banner. The importance of this was uncovered not only by positive comments about these as well as frequent usage of them but also because the UW interface lacked these options. UW users commented that these features would be handy. For some using UW’s interface it was difficult to find “My Favorites” while others commented it would be nice to be able start fresh search with a single click rather than needing to use the browser’s “back” button multiple times to get to a fresh search screen. Conversely, users of the VIUS interface did not experience these problems due to the inclusion of these features on the top banner. General suggestions about Search Results Pages There were some additional improvements to the search results pages that were suggested by the users. These suggestions apply to the CONTENTdm™ system as a whole, rather than one or the other of the customized interfaces, since it currently is not possible to customize the interface to include these features. Many would have liked to see the search terms used listed at the top of the search results page so that any misspellings, typos, omitted words, etc would be apparent should surprises appear in the search results. Additionally, the ability to “search within these results” was a commonly cited missing feature. The users admitted they could go back to the main search screen and redo their searches with additional keywords but that ability to simply add them one by one on the search result page until they reached the desired specificity would be a welcomed convenience. Improved, dynamically changing captioning to match their search terms is a feature requested by multiple users. While searching for a picture of an object they were not familiar with, which was often the case for the user protocol study but also reasonable to expect in a real world situation such as a class assignment, a significant number of


users were confused when their search terms did not match the caption for each thumbnail. They were unsure if the thumbnails were showing the proper object when the caption did not include the search terms. Many web search engines will show sample text from the web pages in the search results with the search terms highlighted. Many users would appreciate something similar where the caption dynamically changes based on the search terms used. When the user searched for an item they were familiar with this was not a problem since they could visually tell from the thumbnail image. Forcing users to wade through large numbers of thumbnail pictures in strictly alphabetical order based on the caption caused much frustration. A major criticism of browsing through the search results is that it was only possible to do so by alphabetical order of the caption. This made the browse collections feature unusable as users are forced to wait through pages and pages of irrelevant results. This problem was exacerbated by the fact that, as discussed above, the caption did not always make sense to the users. In these cases, having all the pictures sorted only by captions that they did not understand only confused and frustrated them further. To alleviate this problem, some users tried to skip ahead using the page navigation at the bottom of the page but since it was only by arbitrary page number (rather than anything meaningful related to the pictures) it was still awkward. Some better form of navigation through search results is urgently needed. Some of this would be alleviated by the “search within these results” feature discussed earlier since it would reduce the length of the search results. Some subjects recommended having the ability to be able to skip through to specific letters rather than meaningless page numbers. This would be particularly helpful in conjunction with the improved captioning discussed earlier as it would then become a more meaningful way to sort the search results. Many subjects would like the ability to sort the images by various other criteria in addition to the caption such as name of object, creator, date, file size, etc. Image Display and Description After skimming through the thumbnails in the search results pages the user will click on one which brings up the Image Display and Description pages. These pages are minimally customizable. The two options are to either have the picture right above the description or to be able to toggle between the picture and description by clicking on a tab. Both custom interfaces used the picture above the description format while DiMeMa’s interface used the toggling option. The two most mentioned favorite image display and description screens are the clickable search terms in the metadata and having the description right beneath the picture. Many subjects started clicking on the search terms just to try it out even when it was not necessary to accomplish their tasks. There were also a lot of comments about how the pictures were nicer than those found on the internet both in terms of artistic and technical quality as well as image resolution.


Unfortunately, there were a number of criticisms related to this display as well. For one thing, the location of the description in relation to the picture varied depending upon which section of the database the user was in. For most areas, the description was beneath the picture and this is where the user got used to seeing it. However, when in “My Favorites” in either the VIUS or UW databases, the description was no longer below the picture and the user had to click on a tab to see the description. It took some users a surprisingly long time to find this tab despite it being right at the top of the picture. Once they found the tab there was no problem but frustration mounted before found they this tab and could not see the picture description. The amount of description was judged to be insufficient on both the VIUS and UW sites. The only way to fix this would be the time consuming addition of metadata. A number of users pointed out that the amount of metadata was higher than the internet but did not quite meet their expectations. It was clear that the users’ expectations were higher because this was a library service. One user in particular was frustrated by the lack of information about the picture and she said that when she saw the links in the description clicked on it thinking it would lead to more complete information about the picture but was dismayed when it led to only more pictures—“pictures that led to more pictures without any information!” While many users appreciated the high quality pictures they found there was agreement among them that the database did not make it easy to view them. Many of the pictures were too large to fit on the screen yet there was no “fit to screen” feature that would allow the user to see the entire picture at once without need to scroll. Users would appreciate the ability to “fit to screen” as well as zooming in and out as desired. Right now the pictures only display at full size which does not work well for larger pictures. My Favorites in the VIUS, UW, and DiMeMa’s interfaces Please note that it is currently not possible to customize the “My Favorites” section. Consequently, this section is identical for the VIUS, UW, and DiMeMa’s interfaces and the user opinions are all discussed in one section rather than a separate section for each interface. In summary, nearly all the subjects were able to successfully accomplish all the tasks which incorporated the features of “My Favorites.” However, most subjects agreed that the procedures for using these features could be a bit more user friendly and accomplishing the tasks entailed more frustration than necessary. A common complaint was that the current “My Favorites” interfaced looked “clunky” and “out of date” but was otherwise serviceable.


Saving images to “My Favorites” Saving images proved to be initially very difficult for a number of subjects to do. Once the proper method was discovered it became easy but many experienced frustration finding the necessary link to save images to “My Favorites.” The link for “Add to my Favorites” only resides on the image display pages and it was very difficult for many users to find. This link only exists at the very bottom of the description and it was not uncommon for users to avoid scrolling all the way to the bottom. In fact, this tendency of not scrolling to the bottom was observed by the research and many users explicitly commented that in general they do not scroll all the way to the bottom of web pages unless there is a strong need to. The result was that many did not see this link until they got desperate enough to do what they normally did not do. Additionally, since the “Add to my Favorites” link is a text based link rather than an icon it tended to blend in with the multitude of text based links within the picture description. Consequently, it did not gain the user’s attention as much as it might have as an icon. This problem was exacerbated in the UW interface since, unlike the VIUS interface, there was no “My Favorites” option in the top banner. This made it difficult for some users to find the objects they had placed in “My Favorites.” Conversely, users of the VIUS interface often clicked on the highly visible “My Favorites” in the top banner BEFORE finding the “Add to my Favorites” link at the bottom. There was an additional confusion related to adding images to “My Favorites.” When asked to save pictures from the image display into “My Favorites” many subjects confused the “My Favorites” section of CONTENTdm™ with the web browser’s “Favorites.” Often subjects would want to add a page to the browser’s “Favorites” instead of CONTENTdm™’s “My Favorites.” This was made worse by the fact that “Favorites” was more visible right at the top and more well known than the “My Favorites” where the links were way at the bottom of the page. One user summed it up as, “I’m more likely to look to the top menu than scroll all the way to the bottom.” There were three major recommendations related to these problems. First, place the “Add to my Favorites” link immediately above and below the picture. Second, make it a colorful, eyegrabbing, icon based link so it stands out from the other text. This added visibility should solve many of the problems. Third, it may also be desirable to change the name of this section so as to avoid the confusion between the various “favorites.” My Favorite’s: Tool Section The “tools” section of “My Favorites” was particularly troublesome and often described as “clunky.” When asked to rearrange the order of the pictures many subjects initially tried “dragging and dropping.” When that did not work the subjects eventually found the pull-down menus to rearrange the order of the pictures. Again, the hesitancy of users to scroll down delayed many from finding this feature but eventually all users found this on their own. Upon finding the appropriate section to rearrange the order of the pictures, a common comment was that this was not the most intuitive way of changing the order.


Most said they wanted drag and drop and one undergraduate multimedia web programmer said he knew such programming was possible. Others indicated that the current interface requires too much scrolling up and down. The users had to scroll down to the “move item” section, saw they needed the image number and scrolled up to obtain that, then scroll down to select the image number from the pull-down, scroll back up for the next image number, and repeat until images were rearranged as desired. The aforementioned drag-and-drop would eliminate this scrolling. Other suggestions included using the image title in the pull-down boxes rather than the picture number. Another subject thought icons underneath the thumbnails to move and delete images would be helpful. Some higher resolution screen settings such as 1024 X 768 also seem to solve this problem by allowing more to be visible on the screen and, hence, reducing the need to scroll whereas 800 X 600 was used for the user testing. A number of users commented that the “compare” feature was not self explanatory from the title alone and they would not have used it if they had not been asked to by the task list. Perhaps a short explanation right by the title “Compare.” While the “My Favorites” tools interface was not optimal, all users agreed that once they figured it out it was fairly easy. Changes in this area would simply reduce the initial looking around and time spent figuring things out. It would also make this area of CONTENTdm™ look more up-to-date with current programming practices and user expectations. My Favorites: Publish or send your favorites Currently naming, creating, and saving a webpage is a three part process. Often the users would enter a name for the web page and hit the “create” button without fully reading the directions. The resulting web page would open up in another window but the users frequently had not read how to save this to the hard drive. There was a small subset of users who thoroughly read the directions and knew what to do when the generated web page popped up but most users were surprised. They expected that the “create” button would save the file. Ideally, the “create” button should both create and save the web page. Or, another user suggested moving the “create” button further down on the page in order to prompt the user to read the directions. In its current placement it was too easy for the users to type in a name and hit the button without reading the directions that are currently below the “create” button. There is a mysterious button on the web page that is created by the above process. This button is labeled “Make Favorites” and comes with the following instructions and warning: “Click "Make Favorites" to store these items as your CONTENTdm™ Favorites. (Warning, this will overwrite any existi ng Favorites you have on this CONTENTdm™ Server.)” This button only confused the users. Many who had not thoroughly read the instructions as described above thought that they need to click this button to save the web page. Since the web page is based on what was in “My


Favorites” it seems redundant to have a button that will make the web page become the contents of “My Favorites.” This button should have a clearer explanation for its purpose. The end result of the confusion between saving the web page and the confusing button was that many subjects thought at one point or another (either when the web page popped up or when they pressed the button) that they had saved the web page to their hard drive when in actuality they had not. It was only the researcher’s comment to them that set them straight. Otherwise many users would have gone on to the next task without actually having saved the web page. My Favorites: Conclusions While a bit clunky and not as user-friendly as could be, the users were able to complete all the tasks without serious obstacles. The main comment about the “My Favorites” section as whole was the many users did not see how it would help them with most real world tasks. A frequent comment was that while “My Favorites” had some nice features, normally they would just download the pictures they needed and then use them using other software such as including them in word processing documents, Power Point presentations, or using imaging software to view or print them out. Another common comment was that in the normal course of events they probably would not have discovered the full features of “My Favorites” and only did so since they were prompted by the task list. It should be noted that this user study did not include the use of DiMeMa’s PowerPoint plugin that could conceivably add value to storing images in My Favorites. Saving images to the user’s hard drive

This problem applies to the throughout the entire CONTENTdm™ database regardless of the interface that is being used. Currently there is no explicitly stated method for saving the images in the database to the user’s hard drive. The only way to save an image is to right click on the image and use the “save image as” feature in the menu. A significant portion of the users were not aware of this right click method and were completely stuck on saving an image to the hard drive. This was the single most common item on the task list that the subjects failed to complete without help. Ideally there should be a “save picture” icon by the thumbnails in “My Favorites” and right in the image display and description page. Given the general user desire to “just download the images” as discussed above, this becomes a particularly crucial fix.


VIUS CONTENTdm™ Protocol Analysis VIUS Custom Interface Checklist

Positive Features Main Search Page: ü Simple main interface ü Boolean radio buttons (helped explain how search worked) ü Fields pull-down menu (mostly didn’t use but many looked at the fields to see

how to search) Search Results: ü Size/number of thumbnails ü Top Banner (particularly “New Search” and “My Favorites”)

Image Display and Description: ü Clickable search terms in metadata (many tried this just for experimentation) ü Description right under picture

My Favorites: ü Ultimately most subjects figured out how to complete all tasks using My Favorites


VIUS CONTENTdm™ Protocol Analysis VIUS Custom Interface

Negative Features Main Search Page: ü Collections as listed under “Browse” do not match the collections on the main

keyword search page. This caused confusion. Search Results: ü Lack of search terms used at top of search results pages to verify spelling and

terms used ü Lack of a “search within these results” feature to more easily narrow the results ü Better captioning (some subjects were confused when searching for an item they

were not familiar with and the key words were not in the caption) ü Browsing strictly by alphabetical order within the collections makes “browse”

unusable Image Display and Description: ü The amount of description was judged to be insufficient by many subjects ü “Add to My Favorites” was only at the very bottom of the description. It took many

subjects a while to find it. This should be duplicated at the top of image. ü Picture should have a “fit to screen” option as well as zoom. Now it only displays

at actual size which can entail lots of scrolling for larger images and does not give a good overview of the entire image.

ü In advanced search, the change from the description below the picture to tabbing between picture/description confused some and many would just assume it was not there and did not find the tab.

My Favorites: ü Many subjects got confused between the CONTENTdm™ “My Favorites” and the

web browser’s “Favorites.” Subjects tried to both save pictures to “Favorites” and retrieve them from there. This is particularly true in light of the poor location of “add to my favorites.” Suggest name change along with previous suggestion of “Add to My Favorites” above picture.

ü Nearly all subjects thought the use of pull down menus in the “Tools” section was not intuitive but most figured it out eventually. Many tried to drag and drop. One subject recommended small icons under thumbnails.

ü Current set up required too much scrolling according to many subjects. Repeatedly look at thumbnails to determine image number and then scrolling down to use pull down menus.


ü Naming, creating, and saving a webpage. This is a three part process of which only the first two parts of naming and creating the web page are clear in “My Favorites.” The third part of saving the resulting web page has caused confusion. Clicking on the “create” button should ideally both create and save the web page all in one step.

ü Lack of “Save this picture to your hard-drive” button. As is, you must right click on image to save it. Many subjects were not familiar with this and had no way to save the image.


VIUS Database (Custom Interface) http://content.libraries.psu.edu/home.html

The VIUS CONTENTdm™ database contains five collections which are heavily tilted towards architectural images. Each collection uses a unique metadata structure. Fields are mapped to Dublic Core for cross-collection searching. The bulk of the images reside in the “VIUS collection” which contained approximately 4100 architectural images from around the world during the user studies. There are also the Davis Art Collection which contains another 300 architectural images, the University Park Campus Buildings Collection with 210 photos and architectural plans of campus buildings, the O’Conner-Yeager Collection with 270 digitized paintings of Pennsylvania, and the Art History Slide Department collection with 47 architectural images. The main search page contains a key word box and two pull-down menus to limit the search by field and/or collection. The defaults are to search across all fields in all collections. The VIUS CONTENTdm™ database currently does not have the ability to search by subject. Since this database is both smaller and primarily focused on the one subject of architectural images this lack is not so critical. Doing a keyword search across all fields in all collections, in general, did not produce irrelevant results. It is essentially an architectural image database at this point and as long as the user is searching for an architectural image that is in the database he will no problem finding them.


VIUS Main Search Page

The default settings for the VIUS main search page are to search across all fields and across all collection with an “and” relationship between keywords.


CONTENTdm™ at the University of Washington http://content.lib.washington.edu/index.html

This database was built under auspices of the University of Washington Libraries' Digital Initiatives Program. The Libraries often partnered with institutions and faculty to build the collections. Projects were selected that emphasized the Libraries' strengths or which provided technological or knowledge management challenges. The Dublin Core Metadata element set is used to create basic object cataloging with CONTENT software providing a flexible template, which is customized for each collection. Collection specific fields are mapped to Dublin Core. The University of Washington’s CONTENTdm™ database contains 45 collections. The subjects and sizes of these collections have a very wide range. Examples of just several of these collections include the American Indians of the Pacific Northwest collection (2300 photos and 1500 pages of reports) the William F. Boyd Photograph Album (43 images), and the Cities and Buildings collection with 7500 images. Many of these collections have names that do not intuitively indicate their contents. For instance, seventeen of the collections (38%) are named after photographers. This would not be helpful unless the user had prior knowledge of the photographers. Further, the subjects covered in each collection often overlap with subjects contained in other collections. While there is a page dedicated to explaining what each collection contains there are so many collections this proves unwieldy. To ease the confusion related to the non-intuitive collection names and to aid in searching subjects which often cross over collections, the University of Washington’s CONTENTdm™ database also contains nine different subject areas that can be used to limit a keyword search. These subject areas are: Pacific Northwest, Seattle, Alaska and the Yukon, Arts and Architecture, Expeditions and Adventures, International, Labor and Industry, Natural World, and Politics. The 45 collections have been mapped into these 9 subject areas. For example, a search in the Arts and Architecture subject area will perform a keyword search across all fields in seven relevant collections that contain a total of approximately 8700 images. On the main search page the top box is to “Do a keyword search across all collections.” The search by subject buttons are prominently displayed on the left margin while a pull-down menu used to select a specific collection is below the previously mentioned keyword search box. Clicking on a subject button or using the collection pull-down menu will take the user to a separate page to search within that subject or collection. In practice, doing a keyword search across all collections frequently produced irrelevant search results from the very diverse collections included in the database whereas limiting the search by subject was very effective. If a collection is chosen the resulting collection specific page will prompt the use to initiate a keyword search or to select a predefined search from a pull-down menu. If a subject area is chosen the subject specific page will indicate which collections are being searched and prompt the user to initiate a keyword search. There does not appear to be predefined searches within subject areas.


University of Washington’s Search Pages

Main Search Page

The Keyword search box visible here is equivalent to the VIUS default search in the sense that it searches across all fields in all collections with an “and” relationship between key words. The subject area icons are along the left margin and user can go to a specific collection by using the pull-down menu. The subject area of “Politics” is not visible on the screen but falls right below “Natural World.”



Subject Area Search Page

This is the search screen for the Arts and Architecture subject area. It is comprised of 7 collections for which there are descriptions and links. The user can either use this page to search across all the collections in this subject area or to follow the link directly to a specific collection.



Collection Search Page

This is the Cities and Buildings collection search page. It contains a more complete description of this collection than on the subject area collection page. Users can either perform a key word search within this collection or use the pull-down menu for a predefined search. In this case, the pull-down menu allows the user to search by cites and countries. An important note, users are currently not able to combine keyword searches with the predefined searches.


DiMeMa’s CONTENTdm™ Out-of-the-Box Search Page DiMeMa, the manufacturer of CONTENTdm™, provides a search interface that can be used right out of the box. In theory, this interface can provide more precise searching of the database than either the VIUS or UW custom interfaces as well as a more precise selection of collections to search across. The more precise searching is made possible by the two by four search matrix that is provided and shown in the screen shot below. Users can select up to four fields and restrict the keyword search by the combination of all those fields and keywords (ie. Nations=Italy, City=Rome, Materials=Rome, Photographer=John Smith). There is also a browse term box that provides a list of searchable terms for a field select by a pull-down menu. This list will either contain all the available terms actually used in any given field or an authority controlled list depending on how each field is set by the database administrator. By clicking on the “Select Collections” icon it is possible to click on check boxes to indicate which collections to search across. The user can search across an individual collection or any combination of collections up to and including searching across all collections. This precise control over the collections to search is not possible using the UW or VIUS custom interfaces. For these other interfaces, it is generally only possible to search across one or all collections. The UW interface adds the possibility of searching across a predefined subset of collections linked by subject area. The more precise control over searching and selecting collections is the reason why UW uses this interfaces as their “Advanced Search” feature.


Content DM™ Pennsylvania State University http://content.libraries.psu.edu You are preparing a short presentation for class comparing the two famous Italian cities Rome and Florence. You will be using the images in the Content DM™ database to create a power point presentation displaying the cities different architectural monuments. You will need to find these images within Content DM™, before you can create the presentation. Follow the instructions below. 1. Use Content DM™ to complete the following steps:

• Find and select one image of each monument listed below. • View their descriptions. • Store each image in “My Favorites.”

a. Florence Cathedral, Florence, Italy b. San Pietro, Rome, Italy (aka St. Peter’s)

2. On your own, find two architectural images within the Content DM™ database using

any means possible. One image should be from the city of Rome and the other should be from Florence, Italy. Note: the images should contain different structures than those listed above.

• After you have chosen your images, store them in “My Favorites.”

3. You are looking for a fountain by Nicola Salvi. Once you find an image, place it in “My

Favorites.” 4. You decide that you would like to include a picture of a building from Japan. You are

not sure what search terms to use and instead would like to browse through the collection to view an image you particularly like. Please browse through the “Davis Art Collection” and select a picture of a Japanese structure to place into “My Favorites.”

5. View “My Favorites” and arrange images by placing all the Rome images before the

Florence images. 6. Compare the image of the Florence Cathedral with San Pietro. Display the images

full screen. 7. Deciding that you would rather use all Florence and Rome images for your

presentation, remove the Asian image from your collection. 8. Please name and publish your collection as a web page. Create the webpage using

the database tools and save it to the desktop - no need to put on the web. 9. Save one full-sized image to the desktop of the computer.

Protocol Analysis #

Date post:	26-Apr-2021
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

VIUS Reports 8The UW interface has the same features as the VIUS --keyword box to search across all...

Documents