Post on 14-Apr-2017
transcript
Lessons from the World’sGreatest Testers
May 2016
@Hazjier
Hazjier PourkhalkhaliLead Strategy Consultant
@Hazjier
Optimizely is the world’s largest digital laboratoryAlexa 10k websites, Q1 2016
We researched the world’s top companies
Interviewed experts
§ Spoke with thought leaders in the industry
§ Met with top customers
Built maturity model
§ Developed comprehensive maturity model
§ Assessed 100 customers
§ Surveyed 50+ customers
Held 100+ Business Reviews
§ Met with executives at over 100 Optimizely clients
Analyzed Top Revenue Experiments
§ Analyzed the 100 best experiments run on Revenue Per Visitor
§ Asked customers to describe test and process
1 Understand yourrevenue levers
3 Solve realproblems
2 Test all changes
4 Test multiplevariations
1 Understand yourrevenue levers
User acquisition:
1.5m
User retention
33%
Add-to-Cart:4.00%
Cart Checkout:
50.0%
Average quantity:
2.00
Average per unit price:
£25.00
Average Order Value:
£50.00
Conversion Rate:2.00%
Revenue Per Visitor:£1.00
Visitors:2.0m
Revenue:£2.0m
Understand the revenue drivers
Goals
Strategies
Tactics
Cart Checkout
Rate
Wording / messaging
Location / size
Design / color /
iconography
Establish a visual
hierarchy
Remove non-
critical content
Limit number of
choices
Select/target the right USP’s
Refine the messaging
Crystallize visually
Clear and consistent
UI
Structure, naming,
and order of sections
Describe visually
Test ideal product visuals
Use the right
review systems
DescriptionsReviewsImages
Emphasize the Primary
Call to ActionMinimize
DistractionsCommunicate
Unique Selling Points
Design Intuitive
Navigation
Market Products
EffectivelyBuild a Sense
of Urgency
Show quantity
left
Push temporary
offers
Promote buying
before set times
2 Test all changes
“Every change we make to ourwebsite has to be approvedby our senior staff”
-Financial Services company
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Our Brand Styleguide is key.We cannot make changes that do not fit our design book”
-Luxury goods company
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“We would even change our logo if it converted better”
-Travel Company
3 Solve realproblems
NO BEARD CAVEMAN
.30% .29% .28% .32% .25% .74%
.11% .28% .24% .32% .09% .59%
.74%
.59%
Minimize Distractions:Remove Non-Critical Content +14%
Revenue PerVisitor
BASED ON KEYWORDCURATED
+24%Click to Sale
+30%Add-to-Cart
+210%Conversions
Problem Solution Result
All great hypotheses begin with the problem, NOT the solution!
4 Test multiplevariations
The problem with “A/B Testing”
2
A B
4
A B C D
Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests 65% 17% 17%
# of Variations
Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests
# of Variations
Test 5 variations and nearly double your win rate!Win Rate
# of Variations
14% 18% 20% 27% 20% 22% 23%0%
10%20%30%40%50%60%70%80%90%
100%
2 3 4 5 6 7 >8
+90%
Running more variations will significantly improve your testing!
1 Understand yourrevenue levers
3 Solve realproblems
2 Test all changes
4 Test multiplevariations
Workshop
§ The <user persona>§ Cannot find…§ Does not know…§ Does not see…§ Wants to…§ Is distracted by…§ Is forced to…§ Needs…
Problem Solution Result
§ By <…> the user will… <…>
§ If <solution> solves <problem>, then <metric> will rise
Build powerful hypotheses
§ Discover 5-10 problems for your users
§ Group problems by themes
§ Prioritize most important problems
Problem Solution Result
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
§ Primary Goals:What do you seek to impact immediately
§ Secondary Goals:What would you like to impact, but do not have to
§ Monitoring Goals:What are metrics you want to track
Exercise: Build Powerful Hypotheses
Optimizely funnel analysisHomepage
FlightResults
BookFlight
PassengerDetails
FlightOptions
Payment Page
Step
45%30%
65% 65%
90%
70%
40%
40%
15% 20%
5%
25%
15%30%
20% 15%5% 5%
Optimizely funnel analysisHomepage
FlightResults
BookFlight
PassengerDetails
FlightOptions
Payment Page
45%30%
65% 65%
90%
70%
40%
40%
15% 20%
5%
25%
15%30%
20% 15%5% 5%
Step
Completion
Optimizely funnel analysisHomepage
FlightResults
BookFlight
PassengerDetails
FlightOptions
Payment Page
45%30%
65% 65%
90%
70%
40%
40%
15% 20%
5%
25%
15%30%
20% 15%5% 5%
Step
Divergence
Optimizely funnel analysisHomepage
FlightResults
BookFlight
PassengerDetails
FlightOptions
Payment Page
45%30%
65% 65%
90%
70%
40%
40%
15% 20%
5%
25%
15%30%
20% 15%5% 5%
Step
Exit Rate
Optimizely funnel analysisHomepage
FlightResults
BookFlight
PassengerDetails
FlightOptions
Payment Page
Step
45%30%
65% 65%
90%
70%
40%
40%
15% 20%
5%
25%
15%30%
20% 15%5% 5%
Exit Rate
Divergence
Completion
Optimizely funnel analysisHomepage
FlightResults
BookFlight
PassengerDetails
FlightOptions
Payment Page
Step
Primary /SecondaryGoals
§ Searches Submitted
§ Category tabs§ Deals§ Frequent Flyer
§ Flight Selection§ Filter Usage§ Refine Search§ Sort Order
§ Continue button
§ Edit search§ Upsells
§ Field completion rate
§ Continue button
§ Continue button
§ Upsells§ Seat
selection
§ Conversion rate
§ Form field completion
§ Payment Type
MonitoringGoals
§ Checkboxes§ Flight tracker
§ Home page clicks
§ Menu interactions
§ Site search
§ Home page clicks
§ Menu interactions
§ Site search
§ Field error rate
§ Home page clicks
§ Menu interactions
§ Site search
§ Shopping cart pageviews
§ Home page clicks
§ Return to previous page
§ Edit address, information
§ Error rate
§ Discover 5-10 problems for your users
§ Group problems by themes
§ Prioritize most important problems
Problem Solution Result
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
§ Primary Goals:What do you seek to impact immediately
§ Secondary Goals:What would you like to impact, but do not have to
§ Monitoring Goals:What are metrics you want to track
Exercise: Build Powerful Hypotheses