Beyond A/B Testing: How to Take Your Split Tests to the Next Level

transcript

800-920-7227 | www.pinpointe.com

Presented by:Michael Pelosi

ALeaderinBusiness-to-BusinessEmailMarketing

Brought to you byPinpointe

DigitalMediaStrategistatPinpointe On-Demandwww.pinpointe.com

Phone:617-461-1209|mpelosi@pinpointe.comTwitter/Facebook:@Pinpointe

Featured SpeakerMichael Pelosi

ErynBranhamPinpointeOn-Demand,Inc.ebranham@pinpointe.com

800-920-7227

Moderator

BEYOND A | BTESTING

A S K A Q U E S T I O N , F I N D A N A N S W E R , A N D P L A N F O R T H E F U T U R E

WHAT IS A | B TESTING?

A/B Testing, or split testing, tests one component of your marketing campaign (subject line, call-to-action, etc.) against another within a

certain time frame and across a fixed amount of users.

TEST EVERYTHING (ALMOST)

• The preferences and tastes of your customers change as you offer more products and services.

• Don’t just throw darts at the dartboard and hope they stick. A/B testing gives a framework for picking winners.

• Consistent A/B testing builds a more data driven email and digital marketing culture.

REASONS NOT TO A/B TEST

• Your campaign is performing well. If it’s not broken, don’t fix it. Or you just may break it.

• More data is does not lead to better decisions. It only leads to more analysis.

• If something just isn’t working, don’t spend the time testing. Spend the time fixing. A/B testing is all about providing a lift for a metric of interest.

A TOUCH OF RANDOM IN THE AIR

• A/B testing is powerful, but it’s not a crystal ball.

• Customer behavior can be unpredictable and sometimes random.

• There’s always an unseen error in A/B test results.

• The results from an A/B test today may not be the same as the results from the same test tomorrow.

HOW WE THINK PEOPLE READ THEIR EMAILS

Can you go to the grocery store on your

way home?

4 SIMPLE STEPS TO TEST

Select A and B

Select how many subscribers will receive A and how many will receive B. Then select for how long the test will run for.

Conduct an A/A test to establish a baseline.

Develop the hypothesis, wait for the result, and analyze the data.

#1. SELECT A & SELECT B

•Test small differences to start.

•Have a good business reason for selecting B.

EXAMPLE ONE: EMAIL SUBJECT LINECONTROL SUBJECT LINE (A)

VARIATION SUBJECT LINE (B)

We Could Really Use Your Input on This Problem

The 7 Strategies We Used to Solve Our Problem

MORE SUBJECT LINESCONTROL SUBJECT LINE (A)

VARIATION SUBJECT LINE (B)

“This is just a service I can’t live without.”

“7 services curated just for you”

MAKING ONLY ONE CHANGE AT A TIME

If you’re testing a subject line, everything after the subject line has to stay the same. The same goes for email layout, landing

pages, etc.

SUBJECT LINE A

“Our biggest summer sale: 25% off select brushes”

SUBJECT LINE B

“Brush Bonanza is Here! 25% Off Premium Brushes”

EMAIL A

SUBJECT LINE TESTING IDEAS

1. Subject line length (character differences, short versus long subject lines).

2. Using numbers in the subject line vs. no numbers in the subject line.

3. Using “testimonials” or “quotes.”

4. Using emojis.

5. Using urgent words versus a relaxed words.

6. Using personalized lines versus generic lines.

MORE EXAMPLES

EXAMPLE 2: PLAIN TEXT VERSUS IMAGE BASED EMAILS

EXAMPLE 3: CALL TO ACTION BUTTONS

THE CTA RESEARCH

• Best Color: Blue

• Best Size for Mobile: 44 x 44 pixels

• Not Much Other Research: Perfect Place for Your Case Studies

EMAIL CTA: BUTTON PANEL

MORE EXAMPLES

CTA PLAIN TEXT

•Because plain text emails offer larger lifts, A/B test image CTA versus link CTA.

•Use actionable words that are clearly links.

CTA LINK EXAMPLES PLAIN TEXT• Buy now• Buy today• Claim your spot• Claim your spot today• 50% off your purchase• Happy Birthday! Enjoy our gift to you.• Download our whitepaper • Tell us you’ll be there!• Give us a follow on Facebook to stay in touch

PLAIN TEXT CTA IN EMAILS

Multiple CTAs, One Email

MORE PLAIN TEXT EXAMPLES

Multiple CTAs, One Email

#2. SELECT A SAMPLE SIZE AND TIME FRAME

• Your split testing experiment has to have enough people.

• The experiment needs to go on long enough to get valid results.

HOW MANY PEOPLE WILL I NEED?

• For email testing, your sample size needs to be your open rate.

• Landing pages, CTAs on your website, etc., it’s based on traffic.

• In a typical A/B test for email, 50% of subscribers get A, 50% get B.

GENERAL SAMPLE SIZE TABLE

HOW TO CONDUCT AN EMAIL A/B TEST

1. Select variation A and variation B.

2. Select how many subscribers you will need.

3. Send out the emails randomly to the subscribers in a 50/50 split.

4. Wait some time period until everyone has had a chance to open the email and select a winner.

FATAL A/B TESTING FLAW

Never stop an A/B test early. For landing pages and website based tests, 2 to 4 weeks is ideal.

Once you develop a time frame, don’t look until the test is over.

DAYS 1-9. STOP HERE AND A WINS

AT DAY 14, B WINS

AT DAY 30, NO REAL RECENT WINNER

#3. BEFORE A/B, A/A TEST THE ASSET

• What if it wasn’t a change in subject line or layout that resulted in a change?

• What if it was the behavior of your customers instead?

• In an A/A test, you perform a 50% / 50% split within your subscribers, but they both get the A variation.

A/B testers started to become suspicious of high conversion rates given by split testing software (300%

conversion just by changing the color of a button!).

When an A/A test was performed, the detected lift was still pretty high.

EMAIL CTA BUTTON TEST

EXAMPLE SPLIT TEST RESULTS

Blue Conversion Rate = 60%Red Conversion Rate = 50%

LIFT FROM BLUE = 10%

EMAIL CTA BUTTON TEST: A/A

A/A TEST RESULTS

Red Button Conversion: 47.75%Red Button Conversion: 63.03%

Lift: 16%

BLUE IS NOT THE WINNER

THE RESULTS FROM AN A/A TEST SHOULD SHOW NO DIFFERENCE BETWEEN A AND A.

#4. DEVELOP A HYPOTHESIS, ANALYZE THE DATA

• Your hypothesis is your assumption as to how the test will go. Typically, you’ll state: “There is no difference between A and B.”

• Set a desired lift in the desired metric for analyzing the data. “I want to see a 20% lift, + or – 5% for this test to be successful.”

THE UNCOMFORTABLE TRUTH

There are literally millions of variations you could make that would

improve or ruin your desired metric. You’re tests are only as good as

your assumptions as to what should be tested. Don’t get caught up in

testing. Get caught up in gathering data and intelligence to make your

email / website / landing page / subject lines better for your audience.

SPLIT TEST TO CREATE A BETTER EXPERIENCE

The ultimate goal of split testing is to not have to split test anymore until there is a

valid reason for doing so.

PRACTICAL SPLIT TESTING CASES

NETFLIX LANDING PAGE A

Chrome Browser landing page. Netflix giving me, based on past data, what I want.

NETFLIX LANDING PAGE B

Firefox Browser Landing Page Result. Netflix B variation is a scroll through of their original programming. Variation A indulged my current interests. Variation B is about future value.

MORE SENSIBLE SPLIT TESTING

CONTROL EMAIL

Same offer, two different email addresses, eight hours apart. From Macy’s.

BUTTON TESTING

WHAT LARGER TESTS LOOK LIKE

BIGGER CHANGES IN EMAIL

12: 52 PMGmail

1:56 PMBusiness Email

LANDING PAGE TEST SEQUENCE

PROBLEM: After clicking a button in our email, the subscriber arrives at a landing page. We

want to increase conversions on that landing page by 10%.

VARIATION A

THE METHOD

• Sampling Method: With 5,000 subscribers, we must randomly sample 726 customers in our database. 363 receive A, and 363 receive B in a 50/50 split.

• Length of Time: Check results five days from the time the email is sent out.

VARIATION B

THE RESULTS ARE IN

WINNER

BUT WAIT….WE NEED AN EXTRA STEP

Always review the quality of the data to make sure everything was fair

in the experiment. If not, you may be using bad data.

THE IDEAL SCENARIO

BUT DATA CAN BE MESSY

PROBLEMS AND FIXES• Different open rates lower the validity of your data.

• You can’t use the results even though there was a clear winner.

• You can re-run the test with a new sample 50/50 split.

EASY SPLIT TESTING IN PINPOINTE

1. 4 Simple Steps to Split Testing.

2. Have a business justification for the best testing results (you’re only as good as your assumptions).

3. Always establish a baseline metric with an A/A test.

4. Don’t create data for the sake of creating data.

5. Don’t spend all your time testing. Spend all your time improving. Split testing is only a tool.

QuestionsAskMichael…DigitalMediaStrategistPinpointe On-Demandmpelosi@pinpointe.com

Twitter/Facebook:@Pinpointewww.pinpointe.com

We Thank YouWeappreciateyoutakingthetimetowatchthistoday!

Check Out Our Blogwww.pinpointe.com/blog

Getmoregreatmarketingcontent.

Easilycreateautomatedcampaignsequences.www.pinpointe.com/get-started - Selectprogram&entercodePPTWEBNR

FREEMONTHofPinpointeService(limit100Kemails)

TryPinpointe’sEmailMarketingSolutionforFREE

Follow

formarketingtips&info@pinpointe

www.facebook.com/Pinpointewww.linkedin.com/company/pinpointe

Pinpointe