© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.© 2009 WEBTRENDS INC. ALL RIGHTS RESERVED.
Webinar #3: March 18, 2010
Step-by-step Overview of Multivariate Testing
Frans KeylardDirector, Optimization Services
Bob Garcia Director, Product Marketing
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Agenda
Introduction
Multivariate Testing Process
1. Strategy
2. Test Selection
3. Test Planning
4. Test Setup
5. Test Activation
6. Reporting & Analysis
Wrap-up
Sample Site
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Rationale - Optimization as #1 Priority
Optimization is a quadruple win:
1. FAST TIME TO VALUE - produces results quickly
2. SELF FUNDING - pays for itself many times over
3. RELEVANCE - improves user experience and site “stickiness”
4. BROADER ROMI - maximizes conversions (success events)
Your audiences will reward you:
1. Loyalty
2. Customer satisfaction
3. Share of wallet
4. All the big metrics – revenue, conversions, leads, etc.
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
What Can be Optimized?pa
ges
Landing pages
Microsites
Sales funnels
Shopping cart pages
Credit card pages
E-tail templates
Forms
Interactive pages & templates
Web-based UIs
cont
ent
Copy
Images
Offers
Pricing
Promotions
Rich media
Headlines
Quotes
Graphics
Colors
Rich media
traf
fic Customer segments
Search traffic
Display traffic
Affiliate traffic
New & repeat visitors
Traffic by geography
Weekend vs. weekday
Demographic
Behavioral
Site types that can benefit:1. LEAD GENERATION Boost campaign performance
2. SELF-SERVICE Improve performance of processes
3. ECOMMERCE Maximize revenue & optimize funnels
4. CONTENT Improve relevance to drive advertising revenue
Examples of opportunities:
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Site Optimization Continuum
Segment Testing General Testing Targeting Personalization
ROMI ROMI ROMI ROMI
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Requirements for Success
Test PlanningTest PlanningStrategy Test
SelectionTest
PlanningTest Setup
Test Activation
Reporting & Analysis
PROCESS
Project Management Strategy/Planning Analysis & Reporting
Test development Implementation QA
Creative strategy Asset generation Usability
Optimization Analyst Creative Specialist Web Developer
PEOPLE
TECHNOLOGY
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
THE MULTIVARIATE TEST PROCESSUsing Azul Travel as a Sample Site
Sample Site
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Agenda
Introduction
Multivariate Testing Process
1. Strategy
2. Test Selection
3. Test Planning
4. Test Setup
5. Test Activation
6. Reporting & Analysis
Wrap-up
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Azul Travel Business Objective
Sample Site
Drive registrations for MyAzul Membership Program that will provide users with
travel incentives and help marketers build customer loyalty
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Strategy
Evaluate - General opportunity assessment
• Use web analytics to ID areas of opportunity
• Understand your marketing investments
• Consider the business value of opportunities
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
PPC, Social, Email, Display, etc.
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Selection
Picking a page to optimize. Things to consider:
• Simple and easily measurable conversion objectives
• Clear marketing goals and shared understanding of importance
• Look to analytics to validate selection
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Planning
How and what to test
• Test design options
• Evaluate test ideas & define page elements to test
• Create test recipe matrix & creative assets
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
A/B/n Multivariate Testing
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Planning | Page Elements to Test
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Planning | Recipe Matrix
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Setup
Execute your test recipe matrix
• Set-up test in platform
• Implement tags
• 1 page/table set-up from RC
• 1-time
• QA
• Preview content
• Ensure conversion points are being recorded
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Setup | Webtrends Optimize
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
Site Optimization Platform
Test Campaign Setup
Multivariate Test Setup
Segment Definition & Use Content Management
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Setup | Preview Experiments
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
Preview Experiments 1 of 16
2 of 16
3 of 16
8 of 16
16 experiments using Webtrends Optimize Fractional Factorial approach versus 256 experiments using a Full Factorial Approach
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Test Activation
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
Go!
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Reporting & Analysis
Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Site Optimization Continuum
Segment Testing General Testing Targeting Personalization
ROMI ROMI ROMI ROMI
© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.
Bob Garcia(e) [email protected](t) @bob_garcia
Frans Keylard(e) [email protected]
blog: For more on site optimization check out Billy’s Bloghttp://blogs.webtrends.com/optimization/
FOR MORE INFO: