Lars LofgrenMarketing Analyst - August 2012 info@kissmetrics.com - Confidential - Do not distribute

SUMMER 2012 STATE OF A/B TESTING

@larslofgren #KISSwebinar

We’ll cover...

State of A/B Testing1

Q&A2

Demo3

Let’s start with the data!

WHO AREWe?

Who are we? 176 in total

0

15

30

45

60

Ecommerce SaaS Agency/Cons Publishing Mobile Other

What’s the size of our teams?

0

22.5

45

67.5

90

1-5 6-10 11-25 26-60 51-200 200+

What testing tools does everyone use?

0

22.5

45

67.5

90

Visual WebsiteOptimizer

GA ContentExperiments

Optimizely None Other

How many A/B tests did we run last month?

0

37.5

75

112.5

150

0 1-4 5-9 10-19 20+

It’s easy to get ahead of our competitors

How do we generate ideas for testing?

0

15

30

45

60

TestRandomly Experience Best Practices Research Needs

Step 1: Get Qualitative

Step 2: Make a prediction

Step 3: Confirm with an A/B test

How KISSmetrics generates testing ideas

Get qualitative insights1

Predict how to improve2

Confirm prediction with an A/B test3

OURResults

Do we regularly increase conversions?

0

20

40

60

80

Always Sometimes Rarely Never

The revenue we gain from A/B testing

0

17.5

35

52.5

70

None $1-1,000$1001-10,000

$10,001-100,000

No IdeaOver

$100,000

2 ways to make money with A/B testing

Option 1: Understand your customers first

Get qualitative insights1

Predict how to improve2

Confirm prediction with an A/B test3

You’ll find the BIG wins much faster

Option 2: Get relentless

How pros like Facebook and Twitter do it

Build an entire growth team1

Back-to-back testing2

Look for 100s of small improvements3

How do we A/B test with small amounts of data?

Do tests with customer interviews

Talk to people, give different options1

Look for trends2

Build and see if actions match3

Do we track results over the long term?

0

32.5

65

97.5

130

Yes No

The pitfalls of not tracking long term results

Variation B performs the best in this funnel

Variation A gives the most revenue

LET’S TRACK THE RETURN ONA/B Testing

You’ll need customer analytics that can track multiple purchases from the

same person.

First, make sure you can track lifetime value

Where do we find customer analytics?

OR

Second, connect A/B tools to customer analytics

What about Google Analytics?

Connects tests to goals, not ecommerce transactions.

Q&A Time!Lars Lofgren

llofgren@kissmetrics.com@larslofgren