Ted BorenUser Research Lead

The Church of Jesus Christof Latter-day Saints

THE BEST ONLINE BENCHMARK YOU’VE NEVER MEASURED

TRUE INTENT

Is our site successful?

GAPS IN BENCHMARKING

What do visitors want to do?

(And can they do it?)

Where dothey go?

(How longdoes it take?)

What do theythink of theirexperience?

(Who arethey anyway?)

Whatyou want

to know aboutsite visitors’Experience

…

RemoteUsability Testing

WebAnalytics

OnlineSurveys

TrueIntent

Whatyou want

to know aboutsite visitors’Experience

…

“True Intent” in a Nutshell1. Intercept live visitors.2. Ask them why they came to your site.

3. Ask them to go do that.4. Track where they go and how long it

takes.5. Let them tell you when they succeed—

or give up.

6. Ask satisfaction and profile questions.

True Intent Studies are “Time-Aware”“Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest.”

“Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.”

~ Nate Bolt and Tony Tulathimutte, “Remote Research,”

page 10 (italics added).

CASE STUDY: True Intent Study for LDS.org*

* The flagship website for the Church of Jesus Christ of Latter-day Saints, my employer

Give me the details on thisTrue Intent thing!

“WHEN, HOW, WHO?”

WHEN in the development cycle do I do this?During planning…

… or after deployment.

BEWARE: Variation by hour, day, or special event

Personal study

Find something for my family

Prepare a lesson

Prepare a talk

Search Intents on LDS.org

Personal study

Find something for my family

Prepare a lesson

Prepare a talk

BEWARE: Variation by hour, day, or special event

Search Intents on LDS.org

TYPICAL STREAM: A Whole Week? Day? Month?

Search Intents Declining on the Weekend (spiking on Monday)

Personal study

Find something for my family

Search Intents Spiking on the Weekend

Prepare a lesson

Prepare a talk

So if in Doubt—Slow It Down and Spread It Out!• If you invite 100% of your visitors, you may finish too quickly, especially with:• Heavy traffic• Enthusiastic audience

So:• Meter the rate (preferred)*• Or turn your survey off and on

WHO do I invite? (And how?)•Email from a pool?•Link on the page?•Pop-up on the page?

Sample Popup Invitation

Sample Instructions

Sample First Question

True Intent’s Golden Question:

Why did you cometo this website

today?

Please be as specific as you can.

Prepare a lesson.

80% of your responses may be as exciting (and detailed) as this one.

But the other 10%...

My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.

~ LDS.org visitor

My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.

My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.

My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.

its my homepage. helps remind me who i am everytime i get on the computer.

~ LDS.org visitor

So when asking the Golden Question…

Beware of Qualifiers!

And Seize Today!

The One (and only) Task:“Do What You Came To Do.”

Visitors make their way through your site…

?

… but sometimes they may be hard to track.

?

The tool continues tracking time and path…

… until they say they are done.

Primary Post-Test QuestionsGoals

Satisfaction & Relevancy

Problems, Anything Else?

Secondary Post-Test Questions•Key pivots for slicing and dicing•Demographics / profile•System Usability Scale (SUS) or similar instruments…only if external comparisons are important and feasible

•Be thinking ahead to future comparisons…how will you know you’ve improved?•What data will speak to your stakeholders?•But KEEP it SHORT!

Q: How long do you run this thing?A: “It Depends.”•Site volume• Percentage invited•Survey length•Audience engagement… and incentives*

• I’ve finished in a day or two, or in 4-6 weeks.

• Shoot for at least 300-400 responses:+ More variety of audience and intents

+ More subgroups to slice and dice

= More responses needed

A FEW QUESTIONS?

300-400 responses later…

“ANALYZE THIS!”

Basic Quantitative StatsSuccess Rate

Average Satisfaction Ratings

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%Completed task

Completion Status

I found what I wanted.

It was easy to find what I wanted.

Within the next few weeks, I'll probably study, share, or use the information I found today.

1 2 3 4 5 6 7

StronglyAgree

StronglyDisagree

Breakouts: Stated Goals, by Age Range

• Younger LDS.org visitors more likely looking for personal study or inspiration…

• Older visitors come for these reasons too, but also more for utilitarian purposes.

Personal Study

Find something inspirational

Find something for my calling

Use the calendar, directory, maps, or leader tools

55 and older

35 to 54

18 to 34

Benchmarks: LDS.org Search SuccessVersion 2 was better overall:• Especially for

preparing a lesson…

• But not so much for finding something inspirational…

Prep

are

a less

on

Find

som

ething

insp

iratio

nal

Version 1 Version 2

ANALYZING QUALITATIVE DATA:The Affinity Diagram• A thematic, hierarchical view of qualitative data points.• A way to see what issues arise out of the data,instead of imposing a pre-determined structure.

Sample Affinity Diagram Excerpt:1. I visit LDS.org often for inspiration and information

1.1. I want to be inspired and strengthened

1.1.1. I am looking for something to help me feel the Spirit

1.1.2. I want quick inspiration during a break in my day

1.1.3. I start on LDS.org to keep “spiritually minded”

This is the data we start

with...

These are categories we generate.

Intent: “I try to read a quote or watch a Mormon Message every time I get on the internet. It helps me stay spiritually minded.” (Female, 35 yrs)

Intent: “I come every day as I start my browser, to keep me focused.” (Male, 62 yrs)

Intent: “LDS.org is my homepage. It helps me remember who I am and what I stand for before perusing the internet.” (Female, 17 yrs)

Intent: “Its my homepage. helps remind me who i am everytime i get on the computer” (Male, 25 yrs)

Preparing Data for the Affinity Diagram•Export the data to a spreadsheet•Create one row per comment*, including all data needed for context•Use “mail-merge” functionality to pull quotes and key contextual data (age, experience, success ratings, etc.) into a document formatted for printing the notes.•And then…

Meet Your New Best Friend…

•Post-It Notes for Printers•Print a few merged notes onto regular paper to check alignment•Print a few at a time on manual feed to avoid jams

Organizing over800 comments

Naming and grouping169 categories(11 at the top level)

Conduct an Affinity Diagram Workshop

Finished diagram, with three levels of headings.

Record the DiagramHeadings OnlyRecord only the headings in a document or spreadsheet

More CompleteAssociate each data point with its headings in a spreadsheet

Formal ReportRe-merge the headings and data back into a formatted document, with an introduction, findings, etc.

REPORTBackgroundKey Findings

A FEW QUESTIONS?

Analysis Steps LDS.org Example

1. Filter to those who a failed a specific task. *

Looking for a location

2. Review all the data you have for each instance:• Stated intent

• Path taken, search terms, etc.

• Biggest problem• Other survey questions

that might be relevant

Trying to find the DC Temple

Home > Maps < Home > Directory < Home > Temples > Find a Temple > Washington DC Temple

Didn’t see Temples under Families

Identify patterns and themes Temples not seen under “Families”

Suggest solutions, or follow-up questions

Broaden the heading.Conduct some tree testing.

Failure Analysis

Failure Cases for LDS.orgAlmost all problem cases fell into one or more of the following categories:•Difficulty finding things in menus•Problems with lack of quotation marks in Search•Content lacking or spread across multiple sites

Mitigating ChallengesIt can be difficult to get enough context to conduct an effective failure analysis.This works best when you have:• Additional information derived from the URLs captured (e.g. search terms or other parameters)• Live pilot sessions to provide context for common experiences.• “Biggest problem” survey data.

Be ready to accept that some paths will still be inexplicable.

What Next?•Conduct interviews or tests on key areas.•Adjust or create audience segments. Enrich personas.•Prioritize top intents for future development(or conduct a follow up survey to quantify themes from the affinity diagram).•Fix common problems from failure analysis.

•Re-test after you’ve had new changes deployed for a while; compare quantitative results.

Archive It!

But don’t just forget about it.

True Intent data has a good shelf-life…

How Long Does This Take?• For 300-400 intents:• Creating the online study, collecting data, running quantitative

stats:40-60 hours for the facilitator*• Creating the affinity diagram:

20-40 hours for the facilitator*• Failure analysis:

20-60 hours for one analyst**

• Total: 80-160 hours of your time, plus team time• Can be less with shortcuts• Should be less the second time around

But is it worth it?

Some unsolicited quotesfrom team members on

three different projects…

“Thanks for leading this exercise. I will admit I was a bit skeptical going into these meetings (who isn’t skeptical of meetings that take you

away from already-pressing deadlines?). But I’ve been really pleased with the process and

results, and I think they will be of great help in determining our future strategy.”

~ Member of Team 1

“I mentioned this study in two different high-level meetings this week, and

there is a great deal of interest and enthusiasm about the way the study

was done and in seeing the results.”

~Member of Team 2

“Thanks a ton for your work. As I’ve said before, the results of the study

will help inform our policies & site development going forward in big ways.

We’ve learned a lot – I’ll share appropriate info with [other key stakeholders].”

~Member of Team 3

No time for all of this?

Here are some ways to trimtime and budget.

SHORTCUTS

Shortcut #1: TRIM SCOPESkip 1 or 2 of the 3 main activities:•quantitative benchmark•affinity diagram•failure analysis*

Or collect the data for all 3, but wait on the analysis for 1 or more.

Shortcut #2:USE WORD CLOUDS•One cloud per open-ended question(or per set of closely related questions)•Dump all comments into a single text file•Standardize closely related word forms

“Please tell us why you came to this website today.”

“What was the biggest problem you encountered in your session today?”

“Anything else you'd like to tell us about your experience today?”

Conduct a “True Intent” Study1. Intercept live visitors.2. Ask them why they came to your site.

3. Ask them to go do that.4. Track where they go and how long it

takes.5. Let them tell you when they succeed—

or give up.

6. Ask satisfaction and profile questions.

Shortcut #3:Conduct a “True Intent” Study SURVEY1. Intercept live visitors.2. Ask them why they came to your site.

3. Ask them to go do that.4. Track where they go and how long it

takes.5. Let them tell you when they succeed—

or give up.

6. Ask satisfaction and profile questions.

FINAL THOUGHTS

Evaluating success on an actual, self-defined

task

Largenumbers

of paths andtimes on task

Feedback& profile data (qualitative &quantitative)

TrueIntent

We could call this:A single-task

remoteunmoderated

interview-guidedusability test

with integrated analyticsand survey questions.

But it’s really all about:

True TasksTrue Success

TruePaths

True ProfilesTrue Satisfaction

TrueIntent

FINAL QUESTIONS?

References & LinksOfficial Website of the Church of Jesus Christ of Latter-day Saints:• http://lds.org

Nate Bolt and Tony Tulathimutte, “Remote Research”• Book: http://amzn.to/18nEWtb

• Supplementary Site:http://remoteresear.ch

Jared Spool, “The KJ Technique”:• http

://www.uie.com/articles/kj_technique/

Caroline Jarrett, “Surveys That Work”:• http://rosenfeldmedia.com/books/s

urvey-design/blog/do_incentives_help_to_improve/

Creative Commons Licenses:

• Attribution, Non-commercial: http://creativecommons.org/licenses/by-nc/2.0/

• Attribution, Non-commercial,No Derivatives: http://creativecommons.org/licenses/by-nc-nd/2.0/