Date post: | 30-Mar-2015 |
Category: |
Documents |
Upload: | donte-emmitt |
View: | 217 times |
Download: | 0 times |
The Art of Testing
August 2012
Agenda This is a Test. This is only a
Test. Testing in Direct Mail Show Me the Numbers Oooops! Pop… TEST…. Valid or
Invalid – you be the Judge!
What Is a test?
A procedure intended to establish the quality, performance, or reliability of something, esp. before it is taken into widespread use.
A/B testing, split testing or bucket testing compares a control sample to other samples in order to discover how to improve response or conversion rates
What Is NOT a Test?
NO, REALLY - A TEST Is NOT…
…..Comparing results of two different campaigns mailed during two different times of the year.
A TEST Is NOT…
… Comparing results of two mailings at two different organizations.
A TEST Is NOT…
… Mailing two completely different packages, with different offers, and different asks , with different inserts.
Testing in Direct Mail
Offer 40%
List40%
Creative20%
Direct Mail Success
Testing in Direct Mail
Component Test
• Postage• Ask• List• Offer• Who signs the
letter?• Color
Creative Test
• Package• 2 page vs. 4 page
letter.
Low Risk/ Lower Reward
Higher Risk/Higher Reward
Testing in Direct Mail: Knowing When & What to Test
When – Quantity is RIGHT Timing is RIGHT There is Rollout potential
What – Tests should be designed to meet both
station goals and budget. All tests should start with your
hypothesis!
Splitting the File
RANDOM- Tips on pulling a random file:
Team Approach has a “Random”” field! EQUAL:
The Test Bucket and the Control bucket need an equal # of records (which may mean a third bucket (“fall out”) if it is not a 50/50 split!
CODED: Each segment needs a unique code for
tracking purposes.
Don’t forget those pesky follow-ups!
Set yourself up for Statistical Success…
Final Objective: Statistic validity
What makes a test statistically valid? Significance Sample Size
What does it mean to be statistically significant? Results are repeatable, not by random chance
Significance
Significance measures the likelihood of
repeatable results. Determined by mathematical formulas and expressed as a
percentage (the confidence level) 95% Confidence Level is best practice Significance tests differ for Response Rates & Average Gifts A winning test does not necessarily mean you have a
winning package Free significance calculators are available online
Significance Calculators
Sample Size
Why is sample size important? Measuring significance Minimizing risk & controlling costs
How do we determine sample size? In general: There is no “right” quantity to test
The larger the sample size, the better Rule of 100 Statistical Calculations
Sample Size: Rule of 100
“100 responses will make this test statistically significant.”
Some say that this rule-of-thumb has never proved them wrong.
Some believe this statement underestimates the actual sample size that is needed.
Sample Size Calculator:Response Rate
Sample Size: Statistical Calculations
Response Rate:
N = 31.38 * p * (1-p)/(significant effect)²
Average Gift:
N = 31.38 * σ² /(significant effect)²
Have no fear! Many free calculators are available online
Rule of 100 vs. CalculationsHead to head test… Scenario 1 Control RR: 2% Expected Lift: 25%
Calculations show that… Responses needed: 308 Control/Test Quantity:
12,302
Scenario 2 Control RR: 2% Expected Lift: 47%
Calculations show that… Responses needed: 102 Control/Test Quantity:
3,481
Results: In scenario 1, the Rule of 100 underestimates the quantity needed.In scenario 2, the Rule of 100 holds true.
Quick Anatomy Takeaways
The goal is to set yourself up for statistically valid results from the start
Make sure that you have an adequate sample size To determine the right size you need to know the expected
response rate, effect, and confidence level.
Run the test… One variable at a time; multivariate testing increases sample
size.
Results are In! Check for significance. Use the calculators to see if your test was significant at the 95%
confidence level.
OOOPS!
Mistakes that
Muddle
Mistakes that Muddle
1. Knowing when something “just isn’t right.”
2. Uncovering what that something is.
YOU BE THE JUDGE!
Questions?
Thank You!!!