< H a v e m o r e f u n A / B - t e s t i n g >
To n W e s s e l i n g – C E O Te s t i n g . A g e n c y # D D T T A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
A / B - t e s t i n g E r r o r s
Email: [email protected] Twitter: @TonW #DDTT
Why me?
ü 10 years+ A/B-test experience
ü I always loved the numbers side
ü I’ve made all these mistakes
Email: [email protected] Twitter: @TonW #DDTT
Email: [email protected] Twitter: @TonW #DDTT
Better Dialogue Optimizat ion
Winner!"
Variation B
Variation A
Default
Email: [email protected] Twitter: @TonW #DDTT
This is why you’re A/B-test ing
Small steps to learn how
to make the next big step
Email: [email protected] Twitter: @TonW #DDTT
Look at your A/B-test ing results!
+24% +52%
+83% +12%
+126% +16%
Email: [email protected] Twitter: @TonW #DDTT
Did you push the winners l ive to your si te?
You must be rich now!
Email: [email protected] Twitter: @TonW #DDTT
Are these real winners?
+24% +52%
+83% +12%
+126% +16%
Email: [email protected] Twitter: @TonW #DDTT
1: low signif icance levels
ü Big chance the winner is not a real winner
Email: [email protected] Twitter: @TonW #DDTT
2: low power levels
ü Big chance a real winner is not recognized
Email: [email protected] Twitter: @TonW #DDTT
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner
ü With an average uplift of 10%
Email: [email protected] Twitter: @TonW #DDTT
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 15%+
Email: [email protected] Twitter: @TonW #DDTT
3: Not calculat ing traff ic up front
ü You thought the page had thousands of
visitors
Email: [email protected] Twitter: @TonW #DDTT
4: Not calculat ing what you can improve
ü You thought the page had lots and lots
conversions
Email: [email protected] Twitter: @TonW #DDTT
5: Not choosing your segments up front
ü You just go and dig unitil you find something
(you will always find something that seems true BUT is not!)
Email: [email protected] Twitter: @TonW #DDTT
7: You just let i t run unt i l i t ’s signif icant
Your test should have a fixed lenght!
Email: [email protected] Twitter: @TonW #DDTT
8: Your test takes too long
ü Cookie deletion (you will loose 10% in two weeks)
They will re-enter the the and pollute your samples!
Email: [email protected] Twitter: @TonW #DDTT
9: You just choose a test length
ü At least 1 full purchase cycle!
Email: [email protected] Twitter: @TonW #DDTT
10: You don’t test ful l weeks
ü At least 1 full week (if your not testing for time of day / week effects)
Email: [email protected] Twitter: @TonW #DDTT
11: You think they always use one decive
ü Only test 1 device at a time!
Email: [email protected] Twitter: @TonW #DDTT
12: You’re test ing broken stuff
ü Always check browser compatibility!
ü Dont’ break dynamic stuff!
Email: [email protected] Twitter: @TonW #DDTT
13: You have mult iple tests that overlap
ü Your variation did not won (it was a combination of variations)
Email: [email protected] Twitter: @TonW #DDTT
14: You thought this page matters
ü But it did not (so you decide to use some other metric to declare the winner)
Email: [email protected] Twitter: @TonW #DDTT
15: You forgot to leave out some visi tors
ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)
Email: [email protected] Twitter: @TonW #DDTT
16: You’re not slowing the control down
ü Add the same code to the control
Email: [email protected] Twitter: @TonW #DDTT
17: You’re not test ing on fresh people
ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)
Email: [email protected] Twitter: @TonW #DDTT
18: You forgot to give them t ime to buy
ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)
Email: [email protected] Twitter: @TonW #DDTT
19: You’re not diving into your analyt ics
ü Always measure with analytics software (You will find conversions that are not supposed to be there!)
Email: [email protected] Twitter: @TonW #DDTT
20: You’re not measuring the r ight stuff
ü Use unique visitors
ü Against lifetime value predictors!
Email: [email protected] Twitter: @TonW #DDTT
I f you keep on making mistakes:
Stuff like this happens
Email: [email protected] Twitter: @TonW #DDTT
Bonus: You did not re-test your winners
ü Always re-test to be sure it was not a glitch
< H a v e m o r e f u n A / B - t e s t i n g >
To n W e s s e l i n g – Te s t W i n n e r # D D T T A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
A / B - t e s t i n g E r r o r s