Web Analytics: A Measurement System of Website Performance for Optimization
Dave Rogers
Former Senior Web Analytics Researcher @ PayPal
Who is Dave Rogers?
Almost a decade in research and internet 1/3 Researcher, 1/3 Analyst, 1/3
Psychologist Education: MA Stats/Research; MBA Industry experience in academia, agency
work, e-commerce, finance and social networking
Was @ PayPal for over a year
Source: ‘mutt’ idea stolen from Angel Morales; LightsOut Marketing
answer: a mutt
Discussion
Balkans & the Internet
Web Optimization Ecosystem
Website Optimization
Web Analytics Resources
Q&A
Balkans & the Internet
Great time to get started! Tools are all figured out (consolidations) Many agencies now (available experts!)
Compete on analytics (hunt vs. get hunted)
-4-2
0
-3
??
The Balkans
Web Optimization Ecosystem
Web Site Optimization Ecosystem
Source: Web Analytics Demystified (www.webanalyticsdemystified.com)
WA
Web Analytics
VOC
Voice of the Consumer
TESTINGConversion Optimization
CEM
Customer Experience
Management
Web Analytics WA VOC
TESTING
CEM
The WHAT: Visits / Visitors Bounce/ Exit rates
Marketing tasks Acquisition
SEO and SEM Affiliate Networks Comparison Shopping
Retention Email Affiliate Networks Comparison Shopping
SEMSEO
Voice of the ConsumerThe WHY: Measure customer
satisfaction continuously
Identify what’s driving customer satisfaction and what will have the greatest impact on behavior and financial performance
WA VOC
TESTING
CEM
Customer Experience Management WA VOC
TESTING
CEM
REPLAYS: Tealeaf enables you to see
your customers' actual online experiences, analyze their motivations and ultimately, gain insight as to why abandonment or other site actions occur
AcquisitionMarketing
RetentionMarketing
Testing WA VOC
TESTING
CEM
OPTIMIZING:
AcquisitionMarketing
ConversionOptimization
RetentionMarketing
Typical consumer engagement model from Marketing team perspective:
Now, who owns the conversion task?
Website Optimization
What is MV Testing?
Framework in 8 Steps
Case Study
Advanced Testing Techniques
Ok… now how do I start?
Bonus: Elements to Start Testing Today!
Optimization Discussion
What is MV Testing? … Why?
What is average conversion rate?
What is MV Testing?
Framework in 8 Steps
Case Study
Advanced Testing Techniques
Ok… now how do I start? Bonus: Elements to Start Testing Today!
Website Testing Framework
1. Split traffic
All Traffic
Version A
Version B
Technology
www.books.com
Website Testing Framework
1. Split traffic
2. Display content
www.books.com
Version A (Original, Control) Version B (Test)
Website Testing Framework
1. Split traffic
2. Display content
3. Log which version
Website Testing Framework
1. Split traffic
2. Display content
3. Log which version
4. Log success event
Website Testing Framework
1. Split traffic
2. Display content
3. Log which version
4. Log success event
5. Measure results
Conversion Rate =
# of success events# of ‘starts’ / version
Website Testing Framework
1. Split traffic
2. Display content
3. Log which version
4. Log success event
5. Measure results
6. Use stats!
Website Testing Framework
1. Split traffic
2. Display content
3. Log which version
4. Log success event
5. Measure results
6. Use stats!
7. Decide winner
Website Testing Framework
1. Split traffic
2. Display content
3. Log which version
4. Log success event
5. Measure results
6. Use stats!
7. Decide winner
8. Push winner live
What is MV Testing?
Framework in 8 Steps
Case Study
Advanced Testing Techniques
Ok… now how do I start?
Bonus: Elements to Start Testing Today!
Test Plan: Improve the efficiency of the product detail page
Success Metric: Percent of product page views that result in shopping cart “puts”
The test consisted of 8 elements, or variables, that can take on 18 different values. The resulting search space contains 576 unique combinations of variable values (‘recipes’).
‘Small changes often have huge impact’
8 Elements
1. Page Order
Fold Line
8 Elements
1. Page Order2. Title
8 Elements
1. Page Order2. Title 3. Price Presentation
8 Elements
1. Page Order2. Title 3. Price Presentation4. GiftBox Presentation
8 Elements
1. Page Order2. Title 3. Price Presentation4. GiftBox Presentation 5. Continue Shopping
8 Elements
1. Page Order2. Title 3. Price Presentation4. GiftBox Presentation 5. Continue Shopping 6. Button Text
8 Elements
1. Page Order2. Title 3. Price Presentation4. GiftBox Presentation 5. Continue Shopping 6. Button Text 7. Button Shape
8 Elements
1. Page Order2. Title 3. Price Presentation4. GiftBox Presentation 5. Continue Shopping 6. Button Text 7. Button Shape 8. Button Color
Results of the Multivariate Test
The purpose of the test was to identify which of the 576 unique versions (‘recipes’) of the product-detail page was the best (MV), and to determine how much better it was than the original.
After the data was collected and analyzed, a new and improved version of the product-detail page was identified and compared head-to-head against the original (essentially an A/B test).
$5.25 M Incremental
Annual Revenue
$5.25 M Incremental
Annual Revenue
The new and improved page performed 4.4% better
than the original.(with 99% confidence)
The new and improved page performed 4.4% better
than the original.(with 99% confidence)
What is MV Testing?
Framework in 8 Steps
Case Study
Advanced Testing Techniques
Ok… now how do I start?
Bonus: Elements to Start Testing Today!
Testing outside of Web Pages
Landing Pages Banners
Person v. product Different value propositions
Emails Product v. brand Category sale v. sale across categories Discount v. free shipping Time-of-day / day-of-week
Targeted! Triggered!Targeted! Triggered!
Behavior Targeting & ‘Personas’
If like this, then show ___. Database construction Test & develop business rules Segmentation & predictive analytics
Personas Represent different user types and their behavior,
attitude, ‘world view’ User-centered design “Conversion: What About the Other 98 Percent?”
Predictive Modeling!Predictive Modeling!
‘Manufacturing’ Testing Structure
Build a reporting face Alert when sample size reached Automated decision making Output to Dev Team
Develop the long-term ‘Program’ Share learnings throughout org Business unit ‘on the hook’ as driver of continually
increased conversions
Testing Opportunities
DemographicsPsych/Social
Web Analytics/ Website Behavior
Attitudinal Data
?
Customer Support Calls & Outcome
TransactionData (R,F,M)
Databases Speaking to One Another
What is MV Testing?
Framework in 8 Steps
Case Study
Advanced Testing Techniques
Ok… now how do I start?
Bonus: Elements to Start Testing Today!
Build Yourself or Find a Partner?
BUILD Long-term cost
savings Back-end advanced
testing structure Can build custom
monitoring and notification system
Data easily integrated into web analytics
PARTNER Realize continuous
optimization earlier Some tools include
Professional Services
Can ‘learn’ from the experts
“Which pages do I test first?”
Use your web analytics tool - think ‘funnel’ All pages in the check-out process are important Where do visitors leave/bounce? What are your best selling products/categories?
Know your audience understand needs and create experience for user guest or member/account? first-time buyer or frequent buyer?
Landing Pages?
Landing Pages?
Product Pages?
Product Pages?
HomePage?
HomePage?
MV Testing is ‘doable’ See above- up on the right Think: incremental and continuous Many times, a small win builds
psychological momentum at the organization for continual testing
Wrap-Up: Key Considerations
‘Small changes often have huge impact’
‘Small changes often have huge impact’
Asked to many web analytics professionals at the 2008 e-Metrics summit:
What is MV Testing?
Framework in 8 Steps
Case Study
Advanced Testing Techniques
Ok… now how do I start?
Bonus: Elements to Start Testing Today!
What are different versions of the value proposition? Have they been tested before? Can they be seasonal?
Try various formatting techniques Font, u, size, bold, color, centered
Use of exclamation point! or? # of words: Many v. less Consider language
colloquial; education level
‘Small changes often have huge impact’Testing Element: Headlines
Sale % off $ off Buy1Get1 Discount for future purchase
Coupons Can pilot test campaigns!
Testing Element: Offers
Free shipping is king!!!
Free shipping is king!!!
… but what kind of shipping?
‘Small changes often have huge impact’
Value proposition emotional v. logical
Word choice Add to Bag v. Add to Cart; Shop Now v. Order Now
Color Shape Size Placement
Testing Element: Call-to-Action
‘Small changes often have huge impact’
% image to text to white space Use very similar images in the flow Size Placement Contrast to site look & feel Personalize! Images of people are different
Which direction are they facing? Where are the eyes focusing?
Testing Element: Image
‘Small changes often have huge impact’
Common Mistake: Clutter
Too much text Competing for attention Overstimulation Rest Hype-free zone ‘Less is more’
‘Small changes often have huge impact’
Web Analytics Resources
BLOG ROLL
• kaushik.net/avinash/• blog.webanalyticsdemystified.com/weblog/• semphonic.blogs.com/• grokdotcom.com/• blogs.omniture.com/• juiceanalytics.com/weblog/• webanalysis.blogspot.com/
BLOG ROLL
• kaushik.net/avinash/• blog.webanalyticsdemystified.com/weblog/• semphonic.blogs.com/• grokdotcom.com/• blogs.omniture.com/• juiceanalytics.com/weblog/• webanalysis.blogspot.com/
EVENTS & TRAINING
• webanalyticsassociation.org• eMetrics Summit• Web Analytics Wednesdays• Web Analytics Forum Yahoo Group • UBC Web Analytics • X Change Conference: Gary Angel• Predictive Analytics: Eric Siegel
EVENTS & TRAINING
• webanalyticsassociation.org• eMetrics Summit• Web Analytics Wednesdays• Web Analytics Forum Yahoo Group • UBC Web Analytics • X Change Conference: Gary Angel• Predictive Analytics: Eric Siegel
WEB ANALYTICS BOOKS OPTIMIZATION BOOKS
Web Analytics Resources
