38
Mumbling Crap about A/B Testing
Mumbling Crap
about A/B
Testing
Let’s go back in time…
UX Camp Brighton 2012
Two things happened at UX Camp Brighton 2012:
• An A/B testing talk from Gareth Llewellyn
And
• I presented the following…
My First A/B Test…
A/B Testing is Great!
Back to Today…
Why A/B Testing Might
not be Great…
Some (Obvious) Things
that I Learnt from a
year of A/B Testing
The Positives
Easy to Setup
Clients Love Them
Quantitative Results
Small Changes - Big Difference
A/B Testing is Great!
The Negatives
It can Take a Long Time
Book a Demo
Need Lots of Traffic
Quick results often wrong
Often Inconclusive
Minor Changes = Minor Changes
Dehumanises Users
Use Other Methods
Conversion First Mind-set
Dark Patterns
darkpatterns.org
Problems with the Testing Tool
Limited HTML/CSS
Cloaking?
Google Guidelines
Can’t Polish a Turd
Ethicare
Misses the Bigger Picture
Why is Nobody Clicking that Button?
It’s the wrong colourIt’s not big enoughIt says “Buy Now” not “Add to Cart”It should be roundIt should be flat!
Your product is shitNobody know what your product doesYour product is overpricedYou competitor has a sale onYour website doesn’t look professionalYour user journey is confusingYou charge too much for postageYour website is slowYour product is out of seasonYour website doesn’t work on mobile
No Room for User Learning
Change Aversion
1. Warn users about major changes
2. Clearly communicate the nature and value of the change
3. Let users toggle between old and new versions
4. Provide transition instructions and support
5. Offer users a dedicated feedback channel
6. Tell users how you’re addressing key issues they’ve raised
In Summary• A/B testing should be part of a bigger user testing plan• Don’t rely on tests to be conclusive• Explain their limitations to clients/stakeholders• Most problems cannot be solved by A/B testing• Set aside a lot of time for tests to find a conclusion• Make sure tests are fair 50/50 split, on all browsers and devices• Test your tests thoroughly! • Treat ‘failed’ tests as a learning process
A/B testing can still be useful if run in the right way and for the right reasons. Be sure to include other types of testing, both qualitative and quantitative, to get the full picture.
Remember…
“Nobody has ever A/B tested
their way to innovation!”
Luke Hay
Research & UX Director
@hayluke
Questions?
@hayluke
