Date post: | 15-Jul-2015 |
Category: |
Marketing |
Upload: | marketingexperiments |
View: | 485 times |
Download: | 2 times |
November 3–6, 2014
#CZLCHI | @ClickZLive
The Global Conference Series Designed by Digital Marketers, for Digital Marketers
3 Common Mistakes in Online Testing(And How to Avoid Them)Taylor KennedyMECLABSSenior Manager, Optimization & Strategy
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
30%
16%
Do you understand your results?60
%
12%
43%
8%
62%
6%
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
You’re (probably) testing wrong if you:
• Are stuck optimizing a part of your funnel that lacks significant impact on the bottom line
• Have been blindly following “best practices”
• Are not further establishing customer theory with each test
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Cart Test 1: Results
Results were flat.
Shopping Cart Conversion Rate
Rel. Difference
Level of Confidence
Control 14.3% NA ---
Treatment 13.5% - 4.99% 44%
?
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Cart Test 2: Results
Results were flat, even trending downward.
Shopping Cart Conversion Rate
Rel. Difference
Level of Confidence
Control 13.7% NA ---
Treatment 11.1% -19.26% 88%
?
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Why aren’t tests getting results?
5.7% 41%All visitors Add to cart Payment info Convert40%
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Through proper planning you:Identify the highest impact area of your site to test.
Find potential validity threats linked to normal traffic that may impact results.
Discover visitor behavior that helps you create long-term test plans.
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Product Page Test 1: Results
• “Add to Cart” increased by over 15%• Conversion increased by 40%
Shopping Cart Conversion Rate
Rel. Difference
Level of Confidence
Control 1.07% NA ---
Treatment 1.50% 40.05% 98%
!
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
What motivated the cart test?A. Needed improvements were obvious
B. Our client or boss told us to
C. Testing close to the conversion event has the biggest ability to increase revenue
D. The data told us to
November 3–6, 2014 | #CZLCHI | @ClickZLive
You’ve been following “best practices”Case study examples
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
How are successful and unsuccessful ecommerce companies approaching testing?
MarketingSherpa asked marketers:
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Your peers are following best practices too.
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
How often do you hear the following?
“Keep your call-to-action above the fold”
“Video converts”“Use a compelling image”
“Mobile-friendly is critical”
“Include privacy and security statements to establish trust”
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Security Seal Test: Results
Shopping Cart ClickthroughRate
Rel. Difference
Level of Confidence
Control 66.14% NA ---
Treatment 1 66.95% 1.2% 50%
Treatment 2 65.00% -1.7% 57%
Treatment 3 64.60% -2.3% 54%
Results were flat.?
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Mobile Redesign Test: Results
Results were significantly negative.
Shopping Cart Conversion Rate
Rel. Difference
Level of Confidence
Control 0.5% NA ---
Treatment 0.3% -40.75% 96%
!
November 3–6, 2014 | #CZLCHI | @ClickZLive
Redefine best practice:• There are no shortcuts to customer wisdom.
• Just because a test worked for someone else, doesn’t mean it will work for you.
• Rigorous, methodology-driven, hypothesis testing ensures you are generating real customer insights.
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Framework
©C = 4m + 3v + 2(i-f) - 2a
Probability of conversion
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Framework
©C = 4m + 3v + 2(i-f) - 2a
Motivation of user
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Framework
©C = 4m + 3v + 2(i-f) - 2a
Clarity of value proposition
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Framework
©C = 4m + 3v + 2(i-f) - 2a
Incentive - Friction
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Framework
©C = 4m + 3v + 2(i-f) - 2a
Anxiety
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:“By reducing friction on the page associated
with product selection, we will see an increase in clickthrough.”
• Unclear link to a particular problem• Not related to specific elements on a
particular page or process• Not linked to information or data about the
page
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:“By reducing friction on the page associated
with product selection, we will see an increase in clickthrough.”
• Unclear link to a particular problem• Not related to specific elements on a
particular page or process• Not linked to information or data about the
page
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:“By reducing friction on the page associated
with product selection, we will see an increase in clickthrough.”
• Unclear link to a particular problem• Not related to specific elements on a
particular page or process• Not linked to information or data about the
page
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:“By reducing friction on the page associated
with product selection, we will see an increasein clickthrough.”
• Unclear link to a particular problem• Not related to specific elements on a
particular page or process• Not linked to information or data about the
page
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:
CHANGING __________________ INTO__________________
WILL_______________________
What your analysis indicates is the
problem
What change you think will solve the
problem
The effect it will have on your KPI
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:“Changing the presentation of products from a single column format into a dual column format will increase the number of product
options visible and boost clickthrough to the product details page”
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:“Changing the presentation of products from a single column format into a dual column format will increase the number of product
options visible and boost clickthrough to the product details page”
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:“Changing the presentation of products from a single column format into a dual column format will increase the number of product
options visible and boost clickthrough to the product details page.”
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis is:• A proposed statement
• Made on the basis of limited evidence related to a specific problem
• Can be proved or disproved
• Used as a starting point for further investigation
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Motivation Test: ControlHypothesis:
Changing the right-column form into a call-to-action requiring no personal information will reduce friction and boost clickthrough to the subsequent page.
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Removed form, added starting priceTreatment 2:
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Kept form, added starting priceTreatment 3:
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Homepage Motivation Test: Treatments
Shopping Cart Rel. CTR Level of Confidence
Control -- ---
Treatment 1 202% 99%
Treatment 2 149% 99%
Treatment 3 - 20% 99%
@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Remember:• Proper planning allows you to run tests
with clear outcomes predetermined in advance
• Formulate test sequences that generate big-picture customer analysis
!
November 3–6, 2014 | #CZLCHI | @ClickZLive
Key takeawaysFollow a “testing process.” Don’t just “test.”
1. Always perform at least a basic site data analysis (funnel analysis) before beginning testing.
2. Analyze each page through the mind of your prospect to identify areas of opportunity for testing.
3. Formulate hypotheses that enable you to learn from each and every test. That will lead you to a more complete picture of your ideal prospect.
November 3–6, 2014 | #CZLCHI | @ClickZLive
http://bit.ly/OptimizeTKPresentation Link