Click here! Tap here!Using Chalkmark for adaptive design testing and Excel for analysis

UXPA Boston 2014Azilah Baker

What you’ll learn

• A way to test adaptive designs• How to set up a first-tap study using Chalkmark

• A way to tell a more complete story with numbers• How to calculate statistical data in Excel

Task 1

Task 2

Task 3

Task 4

Task 5

Task 6

Task 7

Task 8

Task 9

Task 10

0%

20%

40%

60%

80%

100%

Accuracy

Agenda

• Introductions • Chalkmark• Set up & high-level Analysis• Take the study• Study results• Analysis spreadsheet in detail• Lessons learned

About Me: Azilah Baker

• User Experience Researcher, Fidelity Investments, Boston• Team of 30• Spectrum of Quant & Qual studies• In 2014 we’ve run studies with almost 15,000 participants to date

You’re here because…

• You’re interested in designing and/or testing for mobile and want to compare it to desktop• You use Chalkmark and want to know what else it can do• You’re here to kill time before Beer o’clock

Adaptive vs Responsive

• Responsive design = fluidly changes according to the screen or device size

• Adaptive design = changes to fit a set number of screen or device sizes

Responsive design on Boston Globe

First-click testing

• Quantitative testing method• Goal is to learn: Where do users begin their task?• Tools: • Chalkmark• UserZoom • UsabilityHub • Qualtrics

When to use First-click testing• Early in the design cycle, after IA• You have wireframes or flat images, but not interactive prototypes• Typically, you have multiple designs to compare

Chalkmark yesterday

Chalkmark yesterday

Chalkmark today

Chalkmark today

Setting up a study

To test adaptive designs, you need: 1. Flat images

• Large screen: 1024px width• Small screen: 320px width• Tip: Use the same image format for all (e.g. avoid mixing pngs and jpgs)

2. For each test condition, create its own study in Chalkmark

3. Create a URL with javascript to randomize study conditions

Setting up a study

Tips for the smartphone (320px wide)• Check display on different phones• Add 100px of blank space to the bottom of the image canvas

Image size: 320 x 640

On some phones, the image is cut off at the bottom

Try it: bit.ly/1nFH3Ap

Analysis: Chalkmark gives you…• Accuracy data • Time data (regardless of task success)

Try it: bit.ly/1nFH3Ap

Define area of correct answer: Heatmaps > Selection

What if you need statistical data that looks like this…

Task 1

Task 2

Task 3

Task 4

Task 5

Task 6

Task 7

Task 8

Task 9

Task 10

0%10%20%30%40%50%60%70%80%90%

100%

Accuracy

Why use the Analysis spreadsheet?1. Calculate accuracy, time & efficiency

(combination of accuracy & time) per condition. Efficiency is a better measure of a page’s usability.

2. Get statistics across all conditions

3. Save coordinates of the correct answers because AOIs (areas of interest) in Chalkmark are not “sticky” and cannot be saved – has to be redefined each time

Preview: Analysis Spreadsheet

Use vlookupSample Condition

Preview: Analysis Spreadsheet

Do both X and Y coordinates fall within the correct answer range? 1 if yes; 0 if no

Efficiency = accuracy and time. If P was accurate, how many correct answers per minute would they get?

Agenda

• Introductions• Chalkmark• Set up & high-level Analysis• Take the study • Study results• Analysis spreadsheet in detail• Lessons learned

Pilot Study

• Does the placement of action buttons affect task accuracy and efficiency?• Are there any differences on a large vs small screen?

• Smartphone:• http://WebUsabilityStudy.com/Adaptive1/• Laptop or tablet:• http://WebUsabilityStudy.com/Adaptive2/

Design ALarge screen

Design ASmall screen

Design BSmall screen

Design BLarge screen

LargeA LargeB SmallA SmallB0%

10%20%30%40%50%60%70%80%90%

100%

Task 1 Accuracy

Results: Task 1

• Design A on large screen more accurate than small screen

• Higher efficiency on large screens

LargeA LargeB0

1

2

3

4

5

6

7

8

9

Task 1 Efficiency (# correct per min)

SmallA SmallB

LargeA LargeB SmallA SmallB0

2000

4000

6000

8000

10000

12000

14000

16000

18000

Task 1 Time (ms)

*

LargeA LargeB SmallA SmallB0

2000400060008000

100001200014000160001800020000

Task 2 Time (ms)

Results: Task 2

• Accuracy much lower than Task 1

• Small screen A worst in accuracy and efficiency

LargeA LargeB SmallA SmallB0%

10%20%30%40%50%60%70%80%90%

100%

Task 2 Accuracy

LargeA LargeB SmallA SmallB0123456789

10

Task 2 Efficiency (# correct per min)

LargeA LargeB SmallA SmallB0%

10%20%30%40%50%60%70%80%90%

100%

Overall Accuracy

LargeA LargeB SmallA SmallB0

5000

10000

15000

20000

25000

Overall Time (ms)

LargeA LargeB SmallA SmallB0123456789

10

Overall Efficiency (# correct per min)

Results: All Conditions

• Overall, participants were less efficient on the smartphone

Lessons Learned: Chalkmark on mobile

Mobile experience similar to desktopEasy to compare data across form factors

• Slightly higher potential for false taps• Task bar eats up space •Minimize pre-study questions• Check display on different devices

Wrap up

• This is ONE way to test adaptive (and responsive) design

• Calculating statistical data tells a more complete story. Analysis is manual – but you can do it!

THANK YOU!

Azilah Bakerazilah.baker@fmr.com

@azibakerwww.linkedin.com/in/azilahbaker

Session materials will be posted on LinkedIn