15 A/B testing mistakes I see all the damn time (and what to do about it)

by Peep Laja, ConversionXL Call to Action Conference 2014

“You should test that!”

“Dude wtf?”

@peeplaja

Typical scenario: A company runs 100 tests /

year. After 12 months their

conversion rate is still the same.

@peeplaja

Testing mistake #1:Precious time

wasted on stupid tests

@peeplaja

Testing mistake #2:

You don’t have a data-driven,

learning-oriented hypothesis

@peeplaja

Testing mistake #3:You think you

know what will work

@peeplaja

Testing mistake #4:You run tests

when you have no traffic

@peeplaja

Testing mistake #5:

Your absolute conversions are

way too low

@peeplaja

You need *at least* 250-400 conversions PER

variation.

Way more if you want to segment the data

@peeplaja

Our "thumb rule" is: 3000-4000 conversions per variation and 3-4 week test duration. That is enough traffic so we can even talk about valid data if we drill down into segments.

Testing "sin" nr.1: search in segments for uplifts although you have no statistical validity, e.g. 85 versus 97 conversions. That’s bullshit.

@peeplaja

Testing mistake #6:

Your tests don’t run long enough

@peeplaja

Testing mistake #7:

You don’t test full weeks at a time

@peeplaja

Testing mistake #8:

Test data is not sent to third-party

analytics

@peeplaja

Pro tip: If you have over 500,000

pageviews / mo, you don’t (necessarily) need to spend

$100,000 on GA Premium. Get unsampled data with Analytics

Canvas

@peeplaja

Testing mistake #9:

You give up after your first test for

a hypothesis fails.

Full case study: http://conversionxl.com/case-study-how-we-improved-landing-page-conversion/

79.3% uplift in conversions

@peeplaja

Testing mistake #11:

You’re not aware of validity threats

@peeplaja

History effectInstrumentation effect

Selection effect

@peeplaja

Testing mistake #12:

You’re ignoring small gains

@peeplaja

5% monthly increase in your conversion rate will result in

80% uplift for the year. That’s how math works.

@peeplaja

Testing mistake #13:

You don’t know html/css/jquery -

and thus only test the simple stuff

@peeplaja

Testing mistake #14:

You’re not running tests at all times

@peeplaja

Testing mistake #15:

You’re late for your lunch break!

Follow me @peeplaja

@peeplaja

Keep reading ConversionXL!

Thank you