Seeing the Elephant: Defragmenting User Research

transcript

Seeing the ElephantDefragmenting User Research

Lou Rosenfeld • lou@rosenfeldmedia.com UXPA Boston • May 15, 2014

November 14, 2013: User Researcher-in-Chief Barack Obama

What does victory look like?

User research in today’s organization

Reports from the user research group

Query data gleaned fromsite search team

XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=regional+transportation+governance+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17

Logs from thecall center

Reports fromanalytics applications

Insights fromVoice of the Customer research

Reports from CRM applications

Papers from the research center

One agency’suser mental model

Another agency’sbrand architecture research

Surveys behind Net Promoter Score

So why does so much design still SUCK?

The blind men and the elephant

What Why

Methods employed: quantitative versus qualitative

Goals: help org or users

Organizational goals

Users’ goals

How they use data: measuring world we know versus world we don’t

Measuring the world we know

Exploring the world we don’t

Kind of data they use: statistical vs. descriptive

Descriptive dataStatistical data

Lou’s TABLE OF OVERGENERALIZED

DICHOTOMIESWeb Analytics User Experience

What they analyze

Users' behaviors (what's happening)

Users' intentions and motives (why those things happen)

What methods they employ

Quantitative methods to determine what's happening

Qualitative methods for explaining why things happen

What they're trying to achieve

Helps the organization meet goals (expressed as KPI)

Helps users achieve goals (expressed as tasks or topics of interest)

How they use data

Measure performance (goal-driven analysis)

Uncover patterns and surprises (emergent analysis)

What kind of data they use

Statistical data ("real" data in large volumes, full of errors)

Descriptive data (in small volumes, generated in lab environment, full of errors)

Four themes for getting to synthesis

1. Balance

2. Cadence

3. Conversation

4. Perspective

1. Balance

Rohrer/Mulder/Yaar’s map

Qualitative (direct) Quantitative (indirect)

Attitudinal

Behavioral

Approach

Scripted (often lab-based) use of product

Natural use of product De-contextualized / not using product

Key for Context of Product Use during data collection

Combination / hybrid

Focus Groups

Phone Interviews

Ethnographic Field Studies

Cardsorting

Diary/Camera Study

Intercept Surveys

Usability Lab Studies

Eyetracking

Usability Benchmarking (in lab)

A/B (Live) Testing

Online User Experience Assessments (“Vividence-like” studies)

Desirability studies

Data Mining/Analysis

Email Surveys

Message Board Mining

Participatory Design Customer feedback via email

Landscape of User Research Methods

Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

Web analytics’ question:“Are we converting license plate renewals?”

UX practitioner’s question:“What are people searching the most?”

Balanced analysis

Balance within a method

2. Cadence

A research cadence from Whitney Quesenbery

Cadence

WeeklyWeeklyWeekly

Call center data trend analysis 2 – 4 hours behavioral/quantitative

Task analysis 4 – 6 hours behavioral/quantitative

QuarterlyQuarterlyQuarterly

Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative

User survey 16 – 24 hours attitudinal/quantitative

AnnuallyAnnuallyAnnually

Net Promoter Score study 3 – 4 days attitudinal/quantitative

Field study 4 – 5 days behavioral/qualitative

Cadence

WeeklyWeeklyWeekly

Cadence + Balance

Cadence

WeeklyWeeklyWeekly

Cadence + Balance

3. Conversation

Develop a pidgin

Dave Gray’s boundary matrix: http://bit.ly/gWoZQm

segments

personas

Ban words that impede conversations

• Product names: Omniture,, SharePoint...

• Methods: focus group,, usability test...

• Departments: market research,, analytics...

• Disciplines: business analysis,, information architecture...

• Outcomes: portal, social media layer...

Tell Stories

SKU: #39072-2AH1

Buy Candy for Strangers

4. Perspective

Maps help us make sense by seeing things in new ways

Rohrer/Mulder/Yaar’s map

Qualitative (direct) Quantitative (indirect)

Attitudinal

Behavioral

Approach

Scripted (often lab-based) use of product

Natural use of product De-contextualized / not using product

Key for Context of Product Use during data collection

Combination / hybrid

Focus Groups

Phone Interviews

Ethnographic Field Studies

Cardsorting

Diary/Camera Study

Intercept Surveys

Usability Lab Studies

Eyetracking

Usability Benchmarking (in lab)

A/B (Live) Testing

Online User Experience Assessments (“Vividence-like” studies)

Desirability studies

Data Mining/Analysis

Email Surveys

Message Board Mining

Participatory Design Customer feedback via email

Landscape of User Research Methods

Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right

Avinash Kaushik’s visualization (from Web Analytics 2.0)

“...while I have a bucket for ‘Voice of Customer,’ in hindsight I should have worked harder still to paint the full

qual and quant picture....”

Containers help us make sense by doing things in new ways

MailChimp’s UX team: drowning in data

• Analytics

• Account closing surveys

• Blog comments

• Competitor news

• Delivery stats

• Industry research

• Release notes

• Support data

MailChimp + Evernote

• Shared bucket of buckets (60 notebooks)

• Email is the API

• OCR’d (nice for SurveyMonkey reports)

• Searchable!

• Led to “regular data nerd lunches”

MailChimp is on the threshold of synthesis

Map + Container = Dashboard

Map + Container = Dashboard ?

A very helpful book

Helps decision-makers understand that silos are

your problem—and theirs too

A parting question

If you were going to build your organization’s brain—its decision-making capability—from scratch...

What would it look like?

Thanks!

slides: http://rfld.me/11FrI3o

article:http://rfld.me/145ZccP

Lou Rosenfeld

Rosenfeld Media

www.louisrosenfeld.com • @louisrosenfeldwww.rosenfeldmedia.com • @rosenfeldmedia