Date post: | 08-Apr-2017 |
Category: |
Marketing |
Upload: | affiliate-summit |
View: | 23 times |
Download: | 0 times |
A/B PUZZLE How A/B tests lie to us, and
how to drive genuine improvement
Jason Cohen, founder & CTO @asmartbear http://wpengine.com
The puzzle
758 out of 24,483 visits 85% confidence
What do stats really tell us?
758 out of 24,483 visits 85% confidence
Finding the Cheaters
105 athletes. 100 are clean. 5 are juicing. Test is 95% accurate.
Finding the CheatersIN REALITY
Clean (100) Juicing (5)
TEST RESULT (95%accuracy)
Clean
Juicing 5
095
5
Insight: Rare events
are indistinguishable from random noise
758 out of 24,483 visits 85% confidence
Easy significance testing with hamsters
Trust in the Hamster.
bit.ly/abhamster
Easy significance testing
N = A+B = total # of conversions.
D = (A–B)/2 = half the difference.
Significant if D2 > N.
bit.ly/abhamster
N = 758 + 685 = 1443
D = ( 758 — 685 ) / 2 = 37
1369 > 1443?
No! Not significant.
Final Result: As predicted
Wasted another month
30
40
50
60
70
12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11
OtherInbox Splash Page
Trust the Hamster
Beware the Rare
Defeat rare by seeking only huge results
Can you A/B test your way to huge results?
1,116
250
4,100
1,151Individuals: 67% off
Group: 3% off
Wisdom of the Crowd
“I’m as clueless as you, but in the equal and opposite direction.”
What’s the funniest joke?
And the winner is…
Jokes, Pizza, Design, Features, Voice,
Values, Culture
Crowds find objective truth but kill creativity
Beware the rare. Seek huge results.
Don’t let data kill creativity.
Jason Cohen, founder & CTO @asmartbear http://wpengine.com