Comparative Usability StudyReporting of Results

Robert Murphy

Study Overview• Comparative usability study between SHOP.COM and

Amazon / Shopping.com with a concentration on search UI elements and use of search functionality

Participants

• 18 Shoppers

– 16 women

– 2 men

– 2 – age 20-29

– 4 – age 30-39

– 7 – age 40-49

– 5 – age 50-59

– 6 w/SHOP.COM experienced

– 12 w/out SHOP.COM experience

– 13 shoppers mentioned “Amazon” as a commonly used ecommerce site

Study OverviewMethodology

• Each shopper performed 2 - 4 unaided searches on each website beginning from the home page

• Total of 90 searches performed (search and navigation)

– 25 on SHOP.COM and Shopping.com

– 20 on SHOP.COM and Amazon

– SHOP.COM was always presented first

• Categories of searches included:

– jewelry, shoes, sporting goods, clothing, electronics, auto supplies, books & video

• Questions rating their experience and preference between sites were asked after performing searches and after using both sites

• Various additional pages were visited, involving tasks in both navigation and general site usage

Study OverviewComparison to previous study

• Generally, improved response over similar study conducted April, 2006

– 87% success rate over 60%– Overall satisfaction rating

4.3 over 2.8

• Several favorable responses from shoppers:

– I had a good experience– You guys have a great product– I like the 1-cart concept

Overall Site Satisfaction Rating

2.8

4.3

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

April, 2006 Dec, 2006

Date of Study

Rat

ing

(1-

5)

Overall FindingsSHOP.COM

Overall Ratings for SHOP.COM

87%

70%

60%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Successfully Found Product

Searches Returning Relevant Results

Searches with AcceptableNumber of Results

• Shoppers successfully found the product they were looking for 87% of the time

• Shoppers felt the results were relevant in 70% of the searches

• Shoppers felt an acceptable number of results were returned in 60% of the searches

Use of Navigational ElementsSHOP.COM

• Breakdown of total usage of navigational elements:

– Performing a search again

– Use of Left Navigation links

– Using Search Within function

– Clicking to view the “next” page of results

– Sorting of results

Percentage Usage of Navigational Elements

0%

5%

10%

15%

20%

25%

30%

35%

Search Again9 Shoppers

Left Nav Links4 Shoppers

Search Within7 Shoppers

Additional (Next)Pages

6 Shoppers

Sort Results9 Shoppers

Navigational Elements

Per

cen

tag

e o

f U

se

Use of Navigational ElementsSHOP.COM

• Breakdown of total usage of navigation functions by experience on SHOP.COM

– Inexperienced shoppers relied primarily on searching again to refine their results

– Experienced shoppers utilized searching again, Search Within and (primarily) Left Nav links

– Experienced shoppers did not need to go from page to page of results

Percentage of Use of Navigational Elements by SHOP.COM Experience

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Search Again Left Nav Links Search Within Additional(Next) Pages

Sort Results

Pe

rce

nta

ge

of

Us

e

No SHOP.COM Experience

SHOP.COMExperience

Comparative AnalysisSuccessful Searches

Percentage of Successful Searches

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

SHOP.COM Shopping.com SHOP.COM Amazon.com

• Percentage of searches where shopper successfully found the product they were looking for

• Q: “Do you feel you successfully found the product you were looking for?”

Comparative AnalysisRelevancy of Results

• Percentage of searches in which shopper felt their search returned relevant results

• Areas identified for improvement on SHOP.COM:

– shoes, books, sporting equip, clothing

• Underperforming category on Amazon:

– household

Percentage of Relevant Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

SHOP.COM Shopping.com SHOP.COM Amazon.com

• Q: “When you were searching or navigating through the results, was the relevancy of the products displayed what you expected?”

Comparative AnalysisQuantity of Results

Quantity of Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

SHOP.COM Shopping.com SHOP.COM Amazon.com

Fre

qu

ency

of

Occ

ura

nce

Too Few

Just Right

Too Many

• Shopper’s perception of the amount of results returned for their search

• NOTE : Amazon only returns the first 48 products (3 pages of results) and does not display the number of results for their “all department” search

• Q: “Were the amount of results initially returned: too few, too many or an acceptable amount?”

Comparative AnalysisEase of Navigation

Preference of Ease of Navigation

0%

10%

20%

30%

40%

50%

60%

70%

• Shopper’s impression of which site was easier to navigate around in

• Q: “Which site was easier to navigate around?”

Comparative AnalysisOverall Layout

Preference of Overall Layout

0%

10%

20%

30%

40%

50%

60%

70%

• Shopper’s preference of overall layout of the sites

• Q: “In general, which site’s overall layout do you prefer?”

Comparative AnalysisSite Preference

Preference of Overall Usage

0%

10%

20%

30%

40%

50%

60%

70%

• Shopper’s overall preference of sites

– 6 of 6 shoppers preferring Amazon, listed Amazon as a commonly used site

– 2 of 3 shoppers preferring SHOP.COM, listed Amazon as a commonly used site

• Q: “Overall, which site do you prefer?”

Qualitative Comparative FeedbackShopping.com (vs. SHOP.COM)

Pros• Price links on top of

Left Navigation• Cleaner UI• Easier to read• Less cluttered

Cons• Fixed width• Not enough products above

the fold• Too far to scroll (defaults to

30 products per page)• Some text too small• Sort was difficult to locate

Qualitative Comparative FeedbackAmazon (vs. SHOP.COM)

Pros• Easier to read• Not as busy• Better use of spacing;

more open feeling• Minimum use of boxes, lines• Better color usage – organic,

subtle colors• Price closer to photos• Attributes more visible• Better logic in attribute

(Left Nav) hierarchy• Friendlier; more inviting,

personal, shopping-like

Cons• Only 3 pages of results

(misleading)• Too stretched out• Too open• No descriptions• In-line department links

caused confusion• Too much information in the

lower left navigation bar

Qualitative Comparative FeedbackSHOP.COM

Cons• Fonts too small• Overwhelming amount of info;

too much to read• Too crowded• Too grid-like; too many boxes

and lines• Too top-heavy; red search too

overwhelming• Column of red buttons too much

attention• No color harmony; too much

contrast• Grey too dull• Cold, clinical, techy, utilitarian

Pros• Search within• Colors OK;

easy on the eyes• More products above

the fold• Easy to narrow-by• Clean

NOTE

• Row view was primary format of search results

Use of Navigational ElementsSHOP.COM

• 9 of 18 shoppers missed the “Next Page” links at the top of the page

• Multiple shoppers complained about the size of the text and links in the grey bar

• Shoppers complained The “Sort By” links and text seemed to be a lot of links and information

• Breadcrumb text seemed easy enough for shoppers to indicate quantity of results and location within department hierarchy

Search Results MessagingLive Site

• 6 of 8 shoppers missed the message: “There are 0 matches…”

• All shoppers missed the message:“Or see matches in all department…” (has since changed to non-italized)

• 5 out of 7 shoppers missed the message:“Here are the closest matches…”

– When pointed out, this particular message was understood

Navigation ElementsLive Site

• Very little use of Left Navigation links and Search Within function for first-time shoppers

• 1 shopper using Left Navigation links, assumed they expanded results

• 3 additional shoppers were unsure whether links would expand or narrow

Navigation ElementsMockup

Changes Made Based on Shopper Feedback:

• Search within moved to left nav

• 2 sets of labels added to Left Navigation bar

• Vertical line added to Left Navigation bar

• Sort removed for “All Department” search

• “Showing 1-24…” enlarged and moved out of gray bar

• Box removed from photos

• Vertical line next to price column removed

• Descriptions shortened

Navigation ElementsMockup

Results of Usage:• Significant increase in usage

of Left Nav and Search Within • All but 2 shoppers preferred

the new “Search Within” position

• All shoppers preferred this mockup to original

– Better; Cleaner; Less Cluttered

• All but 1 shopper preferred photos without the outline

Result Listings• 14 out of 18

shoppers prefer to have the price closer to the product name/photo

• 13 out of 18 shoppers indicated they liked having the description

Attribute Quantities

• 12 out of 18 shoppers preferred to have the quantities displayed with the attribute links

• 6 out of 18 shoppers indicated that large quantities of results may discourage them from clicking the links (supports findings in previous study)

Single Product Search Results• No shopper show any

confusion or surprise when 1 product search result or navigation link returned a product page

• Supports current functionality (recent change to live site)

Search-Specific RecommendationsOverall

• Explore options to enhance visibility of Search Messaging

• Increase font size and visibility of “Showing 1-24 of 86”

• Increase font size of “Next Page” links

• Explore dropping “Next Page” links on top of results

Left Nav Bar

• Keep “Search Within” at top of Left Nav bar

• Use 2 Left Nav headers (“Search Within” and “Narrow Results By”)

• Maintain visibility of Left Nav headings/labels (as per peach colored background)

• Increase font size of attribute listing

• Continue to restrict showing quantity of results until shopper narrows their results

– need to establish rule – is currently inconsistent

Left Nav Bar (con’t)

• Show price attributes high in left nav column

• Remove ‘Showing # and counts’ from long list of attribute listings (i.e. brand listing)

Display of Results

• Minimize usage of lines

• Remove boxes around photos

• Shorten descriptions

• Provide detailed/important information (i.e. books: author, publication date, …)

• Explore moving price closer to photo and product information

• Explore dropping or changing color of “view details” buttons

Home PageInitial Reaction

• 14 out of 18 shoppers (78%) had initial positive reactions to this page

• Half the shoppers thought the “Featured Store” logos were link-offs

Home PageVerbal Feedback

Positive Feedback:• Like store listing on right

• Like the colors

• Broad selection of products

• Clean, wide variety of products

• Nice variety of things

• Looks pretty good

Negative Feedback:• Animated graphic distracting

• Lack of focus

• Not enough access points, few top-level departments displayed

• Crowded and busy, a lot of things going on

• A lot of information to comprehend

• Things too close together

Home PageRatings of Visual Measures

Rating of Visual Measures on Home Page

1.00

1.20

1.40

1.60

1.80

2.00

2.20

2.40

2.60

2.80

3.00

Professional Looking Easy to beginShopping

Attractiveness Organization

Ave

rag

e R

atin

g (

ou

t o

f 3)

• As reflected in Shopper Feedback, Organization of the Home Page was lowest scoring measure

• 3 point scale

• Q sample: “Do you consider this page attractive, unattractive, or neither?”

Merchant Home Page• 8 out of 10 shoppers

expected to see more products

• 6 out of 10 shoppers thought only 3 products were available from the merchant displayed

• 4 Shoppers thought the strike-through pricing meant the product was unavailable for sale (missing explanation on search results also)

• 2 of 3 shoppers navigating in the catalog expressed confusion:

– Different UI

– Looking for “narrow by” functionality

Multi-Merchant Product Page• 10 out of 18 shoppers initially

thought the product review stars indicated “5 Star Rating” (not grayed out = active)

• 9 out of 18 shoppers could not find the Zip Code / shipping calculator (some clicked “View shipping info”)

• No shopper saw the need for pricing at the top and at the merchant level

• Confusion with red price (at merchant level) - not indicated as “Sale”

On-Line Shopping

• Concept of on-line shopping as “fun”:– Unexpected finds– Variety of products

• Things that would most influence your decision to return to an ecommerce site:– Ease of use– Shipping charges– Product pricing– Order fulfillment– Variety of products– Visual appeal of site