1

A Crazy Culture of Testing

2

Plethora of Piñatas

1. Webinars: Marketing Experiments & Site Tuners

2. Landing Page Optimization: Optimizely

3. User Testing: Usertesting.com

4. Browser Extensions: Cookies tab in Firebug & Responsive design view in Web developers

5. Tag management – Brighttag.com, Google Tag Manager

Useful Tools

Landing Page vs. Conversion optimization

Shiny Objects

3

Getting in the game: some webinars, look into crazyegg and Google Experiments

Get me some tools: Usertesting, Optimizely, more webinars, analytics based projects

Fuel on the Fire: statistical significance, full A/B, adopted methodology

Phase I

Phase II

Phase III

4

Phase I & II

Landing Page optimizationOptimizely/Google ExperimentsVersion testing – compare time periodsGut based w/ mix of some research (webinars or competition

• Seasonality, external factors (economy, news, weather), time of day, marketing & advertising efforts

• STATISTICAL SIGNIFICANCE• MATHMATICAL METHODOLOGY

5

The “Next Level”

Phase III

MethodologyRealistic mathematical understanding of site funnel metricsFull A/B testing platformMix of large scale A/B & LPO – dual trackFull tracking system & statistical baselines

6

Methodology - Conversion Heuristic

C = probability of conversion

m = motivation of user (when)

v = clarity of the value proposition (why)

i = incentive to take action

f = friction elements of process

= anxiety about entering information/taking action

7

427% Lift

8

Math & the Funnel

1,000 * 40% bounce = 600

1,000 * 30% bounce = 700

3.5% conversion = 21 vs. 25

WIN?

1,000 * 40% bounce * 45% funnel entries@ 4% & 55%@1% = 13

1,000 * 30% bounce * 45% funnel entries@ 4% & 55%@1% = 15

9

The Basic Funnel

Landing Pages

Search Results

Property

Cart

Billing

Purchase

3.5%

1.7%

40%

“rising tide” A/B

Single page LPO

10

A/B & LPO – The Dual Track

A/B Tests

Internal system with server-side splitting,

custom vars

Full manual reporting toolset

LPO

Optimizley

Software determines

Headlines/CTA

PPC/SEO

Lead Forms

Images

Search Results

Cart

Product

11

Determining the Priority & Process

Landing Pages

Search Results

Property

Cart

Billing

Purchase

“rising tide” A/B

Single page LPO

Book Now or Web Lead

Checkout

Book

10% - 30%

Optimizely

Cart 1 Cart 2

Results• 32% Conversion rate increase

• 96% Level of confidence

• $800,000 Additional annual online sales

Rigorous Testing

29.8% Lift

18.7% Lift

54% Total Lift

Rigorous Testing – Sample of current tests

Site / Page Lift

Beaver Creek Lodging 29.8%

Breckenridge Lodging 427.4%

Park City Resort 128.9%

Vail Destination 69.1%

Mexico Destination 8.7%

Mexico Lodging Ideas 5.8%

Mexico Travel Ideas 5.6%

And many more…

17

Tidbits from Testing

• Orange buttons for main CTA

• One main CTA button vs. multiple options

• People on pages

• Security/value icons work every time

• Consistent value propositions

• Fold doesn’t matter much

• Typically the vertical layout works

• Smaller “hero” images

18

PowerPack Punch of Death

1. Marketing Experiments

• Webinars, research library, methodology

2. A/B Split Testing Capabilities

• Split testing across the whole site, server side traffic splitting, custom vars

3. Full Tracking based on vars

• Online & Offline conversions based on split

4. Optimizely

• Ability to deploy on the fly, integrate w/GA & crazyegg

5. GA Reporting

• Custom reports & custom segments

6. Statistical Tracking & “Funnel Math”

• Excel report and/or online tool (visualwebsiteoptimizer.com/ab-split-significance-calculator/ – based on 5% p-value, or 95% confidence level)