800-920-7227 | www.pinpointe.com
Presented by:Michael Pelosi
800-920-7227 | www.pinpointe.com
ALeaderinBusiness-to-BusinessEmailMarketing
Brought to you byPinpointe
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
DigitalMediaStrategistatPinpointe On-Demandwww.pinpointe.com
Phone:617-461-1209|[email protected]/Facebook:@Pinpointe
Featured SpeakerMichael Pelosi
ErynBranhamPinpointeOn-Demand,[email protected]
800-920-7227
Moderator
800-920-7227 | www.pinpointe.com
BEYOND A | BTESTING
A S K A Q U E S T I O N , F I N D A N A N S W E R , A N D P L A N F O R T H E F U T U R E
800-920-7227 | www.pinpointe.com
WHAT IS A | B TESTING?
A/B Testing, or split testing, tests one component of your marketing campaign (subject line, call-to-action, etc.) against another within a
certain time frame and across a fixed amount of users.
800-920-7227 | www.pinpointe.com
TEST EVERYTHING (ALMOST)
• The preferences and tastes of your customers change as you offer more products and services.
• Don’t just throw darts at the dartboard and hope they stick. A/B testing gives a framework for picking winners.
• Consistent A/B testing builds a more data driven email and digital marketing culture.
800-920-7227 | www.pinpointe.com
REASONS NOT TO A/B TEST
• Your campaign is performing well. If it’s not broken, don’t fix it. Or you just may break it.
• More data is does not lead to better decisions. It only leads to more analysis.
• If something just isn’t working, don’t spend the time testing. Spend the time fixing. A/B testing is all about providing a lift for a metric of interest.
800-920-7227 | www.pinpointe.com
A TOUCH OF RANDOM IN THE AIR
• A/B testing is powerful, but it’s not a crystal ball.
• Customer behavior can be unpredictable and sometimes random.
• There’s always an unseen error in A/B test results.
• The results from an A/B test today may not be the same as the results from the same test tomorrow.
800-920-7227 | www.pinpointe.com
HOW WE THINK PEOPLE READ THEIR EMAILS
800-920-7227 | www.pinpointe.com
Can you go to the grocery store on your
way home?
800-920-7227 | www.pinpointe.com
4 SIMPLE STEPS TO TEST
Select A and B
800-920-7227 | www.pinpointe.com
4 SIMPLE STEPS TO TEST
Select how many subscribers will receive A and how many will receive B. Then select for how long the test will run for.
800-920-7227 | www.pinpointe.com
4 SIMPLE STEPS TO TEST
Conduct an A/A test to establish a baseline.
800-920-7227 | www.pinpointe.com
4 SIMPLE STEPS TO TEST
Develop the hypothesis, wait for the result, and analyze the data.
800-920-7227 | www.pinpointe.com
#1. SELECT A & SELECT B
•Test small differences to start.
•Have a good business reason for selecting B.
800-920-7227 | www.pinpointe.com
EXAMPLE ONE: EMAIL SUBJECT LINECONTROL SUBJECT LINE (A)
VARIATION SUBJECT LINE (B)
We Could Really Use Your Input on This Problem
The 7 Strategies We Used to Solve Our Problem
800-920-7227 | www.pinpointe.com
MORE SUBJECT LINESCONTROL SUBJECT LINE (A)
VARIATION SUBJECT LINE (B)
“This is just a service I can’t live without.”
“7 services curated just for you”
800-920-7227 | www.pinpointe.com
MAKING ONLY ONE CHANGE AT A TIME
If you’re testing a subject line, everything after the subject line has to stay the same. The same goes for email layout, landing
pages, etc.
800-920-7227 | www.pinpointe.com
SUBJECT LINE A
“Our biggest summer sale: 25% off select brushes”
800-920-7227 | www.pinpointe.com
SUBJECT LINE B
“Brush Bonanza is Here! 25% Off Premium Brushes”
800-920-7227 | www.pinpointe.com
EMAIL A
800-920-7227 | www.pinpointe.com
EMAIL A
800-920-7227 | www.pinpointe.com
SUBJECT LINE TESTING IDEAS
1. Subject line length (character differences, short versus long subject lines).
2. Using numbers in the subject line vs. no numbers in the subject line.
3. Using “testimonials” or “quotes.”
4. Using emojis.
5. Using urgent words versus a relaxed words.
6. Using personalized lines versus generic lines.
800-920-7227 | www.pinpointe.com
MORE EXAMPLES
800-920-7227 | www.pinpointe.com
EXAMPLE 2: PLAIN TEXT VERSUS IMAGE BASED EMAILS
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
EXAMPLE 3: CALL TO ACTION BUTTONS
800-920-7227 | www.pinpointe.com
THE CTA RESEARCH
• Best Color: Blue
• Best Size for Mobile: 44 x 44 pixels
• Not Much Other Research: Perfect Place for Your Case Studies
800-920-7227 | www.pinpointe.com
EMAIL CTA: BUTTON PANEL
800-920-7227 | www.pinpointe.com
MORE EXAMPLES
800-920-7227 | www.pinpointe.com
MORE EXAMPLES
800-920-7227 | www.pinpointe.com
MORE EXAMPLES
800-920-7227 | www.pinpointe.com
CTA PLAIN TEXT
•Because plain text emails offer larger lifts, A/B test image CTA versus link CTA.
•Use actionable words that are clearly links.
800-920-7227 | www.pinpointe.com
CTA LINK EXAMPLES PLAIN TEXT• Buy now• Buy today• Claim your spot• Claim your spot today• 50% off your purchase• Happy Birthday! Enjoy our gift to you.• Download our whitepaper • Tell us you’ll be there!• Give us a follow on Facebook to stay in touch
800-920-7227 | www.pinpointe.com
PLAIN TEXT CTA IN EMAILS
Multiple CTAs, One Email
800-920-7227 | www.pinpointe.com
MORE PLAIN TEXT EXAMPLES
Multiple CTAs, One Email
800-920-7227 | www.pinpointe.com
#2. SELECT A SAMPLE SIZE AND TIME FRAME
• Your split testing experiment has to have enough people.
• The experiment needs to go on long enough to get valid results.
800-920-7227 | www.pinpointe.com
HOW MANY PEOPLE WILL I NEED?
• For email testing, your sample size needs to be your open rate.
• Landing pages, CTAs on your website, etc., it’s based on traffic.
• In a typical A/B test for email, 50% of subscribers get A, 50% get B.
800-920-7227 | www.pinpointe.com
GENERAL SAMPLE SIZE TABLE
800-920-7227 | www.pinpointe.com
HOW TO CONDUCT AN EMAIL A/B TEST
1. Select variation A and variation B.
2. Select how many subscribers you will need.
3. Send out the emails randomly to the subscribers in a 50/50 split.
4. Wait some time period until everyone has had a chance to open the email and select a winner.
800-920-7227 | www.pinpointe.com
FATAL A/B TESTING FLAW
Never stop an A/B test early. For landing pages and website based tests, 2 to 4 weeks is ideal.
Once you develop a time frame, don’t look until the test is over.
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
DAYS 1-9. STOP HERE AND A WINS
800-920-7227 | www.pinpointe.com
AT DAY 14, B WINS
800-920-7227 | www.pinpointe.com
AT DAY 30, NO REAL RECENT WINNER
800-920-7227 | www.pinpointe.com
#3. BEFORE A/B, A/A TEST THE ASSET
• What if it wasn’t a change in subject line or layout that resulted in a change?
• What if it was the behavior of your customers instead?
• In an A/A test, you perform a 50% / 50% split within your subscribers, but they both get the A variation.
800-920-7227 | www.pinpointe.com
WHY?
A/B testers started to become suspicious of high conversion rates given by split testing software (300%
conversion just by changing the color of a button!).
When an A/A test was performed, the detected lift was still pretty high.
800-920-7227 | www.pinpointe.com
EMAIL CTA BUTTON TEST
800-920-7227 | www.pinpointe.com
EXAMPLE SPLIT TEST RESULTS
Blue Conversion Rate = 60%Red Conversion Rate = 50%
LIFT FROM BLUE = 10%
800-920-7227 | www.pinpointe.com
EMAIL CTA BUTTON TEST: A/A
800-920-7227 | www.pinpointe.com
A/A TEST RESULTS
Red Button Conversion: 47.75%Red Button Conversion: 63.03%
Lift: 16%
800-920-7227 | www.pinpointe.com
BLUE IS NOT THE WINNER
800-920-7227 | www.pinpointe.com
THE RESULTS FROM AN A/A TEST SHOULD SHOW NO DIFFERENCE BETWEEN A AND A.
800-920-7227 | www.pinpointe.com
#4. DEVELOP A HYPOTHESIS, ANALYZE THE DATA
• Your hypothesis is your assumption as to how the test will go. Typically, you’ll state: “There is no difference between A and B.”
• Set a desired lift in the desired metric for analyzing the data. “I want to see a 20% lift, + or – 5% for this test to be successful.”
800-920-7227 | www.pinpointe.com
THE UNCOMFORTABLE TRUTH
There are literally millions of variations you could make that would
improve or ruin your desired metric. You’re tests are only as good as
your assumptions as to what should be tested. Don’t get caught up in
testing. Get caught up in gathering data and intelligence to make your
email / website / landing page / subject lines better for your audience.
800-920-7227 | www.pinpointe.com
SPLIT TEST TO CREATE A BETTER EXPERIENCE
The ultimate goal of split testing is to not have to split test anymore until there is a
valid reason for doing so.
800-920-7227 | www.pinpointe.com
PRACTICAL SPLIT TESTING CASES
800-920-7227 | www.pinpointe.com
NETFLIX LANDING PAGE A
Chrome Browser landing page. Netflix giving me, based on past data, what I want.
800-920-7227 | www.pinpointe.com
NETFLIX LANDING PAGE B
Firefox Browser Landing Page Result. Netflix B variation is a scroll through of their original programming. Variation A indulged my current interests. Variation B is about future value.
800-920-7227 | www.pinpointe.com
MORE SENSIBLE SPLIT TESTING
CONTROL EMAIL
Same offer, two different email addresses, eight hours apart. From Macy’s.
800-920-7227 | www.pinpointe.com
BUTTON TESTING
800-920-7227 | www.pinpointe.com
WHAT LARGER TESTS LOOK LIKE
800-920-7227 | www.pinpointe.com
BIGGER CHANGES IN EMAIL
12: 52 PMGmail
800-920-7227 | www.pinpointe.com
1:56 PMBusiness Email
800-920-7227 | www.pinpointe.com
LANDING PAGE TEST SEQUENCE
PROBLEM: After clicking a button in our email, the subscriber arrives at a landing page. We
want to increase conversions on that landing page by 10%.
800-920-7227 | www.pinpointe.com
VARIATION A
800-920-7227 | www.pinpointe.com
THE METHOD
• Sampling Method: With 5,000 subscribers, we must randomly sample 726 customers in our database. 363 receive A, and 363 receive B in a 50/50 split.
• Length of Time: Check results five days from the time the email is sent out.
800-920-7227 | www.pinpointe.com
VARIATION B
800-920-7227 | www.pinpointe.com
THE RESULTS ARE IN
800-920-7227 | www.pinpointe.com
WINNER
800-920-7227 | www.pinpointe.com
BUT WAIT….WE NEED AN EXTRA STEP
Always review the quality of the data to make sure everything was fair
in the experiment. If not, you may be using bad data.
800-920-7227 | www.pinpointe.com
THE IDEAL SCENARIO
800-920-7227 | www.pinpointe.com
BUT DATA CAN BE MESSY
800-920-7227 | www.pinpointe.com
PROBLEMS AND FIXES• Different open rates lower the validity of your data.
• You can’t use the results even though there was a clear winner.
• You can re-run the test with a new sample 50/50 split.
800-920-7227 | www.pinpointe.com
EASY SPLIT TESTING IN PINPOINTE
800-920-7227 | www.pinpointe.com
EASY SPLIT TESTING IN PINPOINTE
800-920-7227 | www.pinpointe.com
EASY SPLIT TESTING IN PINPOINTE
800-920-7227 | www.pinpointe.com
RECAP
1. 4 Simple Steps to Split Testing.
2. Have a business justification for the best testing results (you’re only as good as your assumptions).
3. Always establish a baseline metric with an A/A test.
4. Don’t create data for the sake of creating data.
5. Don’t spend all your time testing. Spend all your time improving. Split testing is only a tool.
800-920-7227 | www.pinpointe.com
QuestionsAskMichael…DigitalMediaStrategistPinpointe [email protected]
Twitter/Facebook:@Pinpointewww.pinpointe.com
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
We Thank YouWeappreciateyoutakingthetimetowatchthistoday!
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
Check Out Our Blogwww.pinpointe.com/blog
Getmoregreatmarketingcontent.
Easilycreateautomatedcampaignsequences.www.pinpointe.com/get-started - Selectprogram&entercodePPTWEBNR
FREEMONTHofPinpointeService(limit100Kemails)
TryPinpointe’sEmailMarketingSolutionforFREE
800-920-7227 | www.pinpointe.com
800-920-7227 | www.pinpointe.com
Follow
formarketingtips&info@pinpointe
www.facebook.com/Pinpointewww.linkedin.com/company/pinpointe
Pinpointe
800-920-7227 | www.pinpointe.com