Post on 18-Jul-2015
transcript
Qual vs QuantUsing Better Data to Build Better Products
Laura Klein laura@usersknow.com
@lauraklein
Who’s Tired Of Hearing About Data?*
*be honest
A/B Testing
A/B Testing
Funnel Analysis
A/B Testing
Cohorts
Funnel Analysis
A/B Testing
Cohorts
Funnel Analysis
Blah
A/B Testing
Cohorts
Funnel Analysis
Blah
Blah
A/B Testing
Cohorts
Funnel Analysis
Blah
Blah
Blah
Sorry. I’m Going To Talk About Data Some More.
Better Data ==
Better Products
Well…Sometimes
What Data Can’t DoReplace Design
Replace Listening to Users
Tell Us What We Should Be Building
What Data Can DoInform Design
Inform Research
Tell Us What We Should Be Investigating
How About An Example?
A Funnel
A Checkout Funnel
Create Account
Select Plan
Payment Info
Address
ConfirmThanks & Up Sell
A Leaky Checkout Funnel
100%
Create Account
Select Plan
Payment Info
Address
ConfirmThanks & Up Sell
-20%
-10%
-50%
-33%
-25%== ???
What That Means to You
100 Buyers
Create Account
Select Plan
Payment Info
Address
ConfirmThanks & Up Sell
-20 Buyers
-8 Buyers
-36 Buyers
-12 Buyers
-6 Buyers== 18 Buyers
100 To 18????How Do We Fix This?
Identifying a Friction Point
100 Buyers
Create Account
Select Plan
Payment Info
Address
ConfirmThanks & Up Sell
-20 Buyers
-8 Buyers
-36 Buyers
-12 Buyers
-6 Buyers== 18 Buyers
Stop It
Quantitative Data Qualitative Data
Understanding the Friction Point
Quantitative Data !
WHAT
Qualitative Data !
WHY
How Can You Use Them Together?
Step 1: Identify the Biggest Problem
Step 1: Identify the Biggest Problem
Use Funnel Metrics
The Biggest Problem
100 Buyers
Create Account
Select Plan
Payment Info
Address
ConfirmThanks & Up Sell
-20 Buyers
-8 Buyers
-36 Buyers
-12 Buyers
-6 Buyers== 18 Buyers
Step 1: Identify the Biggest Problem
Use Funnel Metrics Quant
Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability Qual
Problem Possibilities
Create Account
Select Plan
Payment Info
Address
ConfirmThanks & Up Sell
-36 People • Didn’t have payment info
handy • Confused by specific
question • Looked for promo code
Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Step 3: Propose Solutions
Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Step 3: Propose Solutions
Create Solution Hypotheses
Solution Hypothesis!
Problem
People looked for a promo code.
Solution Hypothesis!
Possible Solution
Remove promo code entirely.
Solution Hypothesis!
What Do We Think Would Happen?
An increase in conversion.
Solution Hypothesis!
What Might Go Wrong?
A decrease in acquisition or revenue.
Step 4: Learn & Iterate*
*The last step is always learn & iterate!
Quant & Qual
Learn…!
How Would We Know We Were Right?
A/B test of the checkout flow with and without the promo code.
What Really Happened?
Create Account
Select Plan
Payment Info
Address
ConfirmThanks & Up Sell
Why Did It Happen?
Qual!
…And Iterate!
Another Possible Solution
Create custom landing pages rather than promo codes.
Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Step 3: Propose Solutions
Create Solution Hypotheses
Step 4: Learn & Iterate!
Or…
Step 1: Identify the Biggest Problem
Step 1: Identify the Biggest Problem
Interview Current Users Qual
Step 1: Identify the Biggest Problem
Interview Current Users
Step 2: Understand What
Step 1: Identify the Biggest Problem
Interview Current Users
Step 2: Understand What
Study Relevant Metrics Quant
But Why?
Step 1: Identify the Biggest Problem
Interview Current Users
Step 2: Understand What
Study Relevant Metrics
Step 3: Propose Solutions
Create Solution Hypotheses
Step 1: Identify the Biggest Problem
Look at Your Sales Pipeline
Step 2: Understand Why
Interview People Who Said No
Step 3: Propose Solutions
Create Solution Hypotheses
Step 4: Learn & Iterate*
*The last step is always learn & iterate!
Once Again…
Quantitative Data !
WHAT
Qualitative Data !
WHY
Better Data ==
Better Products
The Velocity Conference Santa Clara, May 27-29
Join us for 3 days of keynotes, sessions and training and learn to optimize your business
end-to-end with speakers like Jez Humble, Gary Gruver, Charity Majors & more.
Use discount code LKLEAN20 during
registration for 20% off All-access, 3-day & 2-day passes
http://velocityconf.com/
velocity2015
Bringing a team? Contact velocityreg@oreilly.com for up to 30% off.