Three-Step Framework for Overcoming Challenges of Site Optimization

transcript

June 19th, 2012

It’s all about relevance

Relevance means understanding

a consumer’s digital behavior

across channels and then being

able to use that intelligence to

optimize digital marketing.

Webtrends Focus

Enabling digital marketers to deliver real-time relevance to their customers

what?who?

Where Does Optimization Fit?

Analytics

Identify and quantify opportunities thru insight

Optimization

Test, segment and target to improve performance

Site Optimization Includes …

Optimization Team

Optimization Consultant

• Strategy and roadmap • Analytics evaluation• Optimization project lead• Campaign performance / KPIs

Web Developer

• Build out/development• Implementation• QA

Designer

• Creative strategy• Asset generation• Usability

• Verify against plan• Multiple Browsers• Conversion Checks

Joe Stanhope (Guest Speaker)+1 617.613.8929

jstanhope@forrester.com

@joestanhope

www.forrester.com

blogs.forrester.com/joseph_stanhope

PresentersForrester + Webtrends

Bob Garcia+ 1 877.932.8736

Bob.garcia@webtrends.com

@bob_garcia

www.webtrends.com

blogs.webtrends.com

A Complete Approach To Site Optimization: Anticipating Challenges and Harnessing Opportunities To Build An Online Testing Program

Joe Stanhope, Senior Analyst, Forrester Research

June 19, 2012

Website traffic is a valuable commodity. To make the most of this

asset, ensure that your marketing and customer experience efforts will

resonate with visitors and drive measurable benefits.

#optimizeplan

@bob_garcia

@joestanhope

What is Online Testing & Why Do You Need It?

Online Testing Maturity: The Scale Conundrum

Success Factor 1: Process

Success Factor 2: Organization

Agenda

#optimizeplan

@bob_garcia

@joestanhope

What is online testing?

Online testing defined: A statistical methodology that allows site owners to

test the impact of specific Web site components on target metrics

The primary experimental techniques of online testing are:

– A/B testing - Test multiple variations of a site

element against one another

– Multivariate testing - Test multiple variations

of multiple site elements against one

another simultaneously

#optimizeplan

@bob_garcia

@joestanhope

How is online testing useful?

Meet marketing or campaign goals (e.g. conversion, sales,

click-throughs, etc.)

#optimizeplan

@bob_garcia

@joestanhope

Improve customer experience

#optimizeplan

@bob_garcia

@joestanhope

Validate segmentation strategies

#optimizeplan

@bob_garcia

@joestanhope

Validate segmentation strategies

Statistically based insight into what works and what doesn’t

– Understanding of:

– Factor importance

– Factor interaction

#optimizeplan

@bob_garcia

@joestanhope

Testing addresses a variety of challenges

But users test primarily to enhance revenue and usability

Other, please specify

Marketing integration with other online channels (suchas search, email, and offsite display advertising)

Content relevance (such ascontent presentedonsite, searchresults, or on landing pages)

Visitor segmentation and targeting

Overall customer engagement and satisfaction

Siterevenue generation (such as via offers,promotions and cross sell/up sell)

Site usability (such as site design, navigationand multistep processes)

“Whichtopicsrepresent the three greatest challenges thatyou seektoresolve withyour onlinetesting program?”

Base: 71respondents who use an online testing platform(multiple responses accepted)

Source: Q4 2011 Global online Testing User Online Survey

#optimizeplan

@bob_garcia

@joestanhope

Testing works for sites of all sizes

Upto 50,000 unique visitors

Upto 250,000 unique visitors

Upto 1 million unique visitors

Upto 2.5 million unique visitors

Over 50 million unique visitors

Base: 71respondents who use an online testing platform(percentages may not total 100 because of rounding)

“Onaverage, what is the monthlyvolume of unique visitors on theweb propertieson whichtesting is performed?”

#optimizeplan

@bob_garcia

@joestanhope

Ultimately, online testing is effective

No benefits

Increased customer satisfaction (surveycontrolledgroup of customers)

Increased formcompletion (form submittals/visitsthat start form)

Increased site based searchconversion (ordersattributedto searchers/visitsto searchresults page)

Increasedregistration (registrations/visits)

Improved customer segmentation and targeting

Increasedaverage ordervalue (online revenue/orders)

Increased landing page conversion(orders/visitsto landing page)

Increasedconversion (orders/visits)

“What are thetop three benefitsyourcompany hasrealized through A/B and/or multivariatetesting?”

Increased off-site display ad clickthough(clicks/ad impressions)

#optimizeplan

@bob_garcia

@joestanhope

Ultimately, online testing is effective

No benefits

Increased customer satisfaction (surveycontrolledgroup of customers)

Increased formcompletion (form submittals/visitsthat start form)

Increased site based searchconversion (ordersattributedto searchers/visitsto searchresults page)

Increasedregistration (registrations/visits)

Improved customer segmentation and targeting

Increasedaverage ordervalue (online revenue/orders)

Increased landing page conversion(orders/visitsto landing page)

Increasedconversion (orders/visits)

“What are thetop three benefitsyourcompany hasrealized through A/B and/or multivariatetesting?”

Increased off-site display ad clickthough(clicks/ad impressions)

#optimizeplan

@bob_garcia

@joestanhope

Source: Insurance Institute for Highway Safety

Online Testing Maturity: The Scale

Conundrum

Online Testing Program Longevity Is Diverse But Skews YoungFebruary 2012 “The State Of Online Testing 2012”

#optimizeplan

@bob_garcia

@joestanhope

Few Online Testing Users Run Large Numbers Of ExperimentsFebruary 2012 “The State Of Online Testing 2012”

#optimizeplan

@bob_garcia

@joestanhope

“Onaverage, approximately how many A/B and/or multivariatetestsdoyou runconcurrently onyourweb properties?

20 or more

Concurrency is low

#optimizeplan

@bob_garcia

@joestanhope

Breadth of available online testing techniques

Strength of vendor’sreference customers

Availability of advanced or proprietaryfeatures andfunctionality

Vendor’sreputation (e.g.known as a leader in this field)

The online testing application is a module within a largersuite of analytics and/or marketing tools

Test deployment method (i.e.type of site tags, etc.)

Vendor’s abilityto servicemy business needs

Total cost of ownership

Ease of use and automation

Vendor’s thought leadership

User interface

Professional services, managed services,and training capabilities

Standard features and functionality (such asreporting,segmentation and targeting)

Ease of implementation and ongoing management

“Which of thefollowing are the five most important factors whenselecting an onlinetesting solution?”

Experience in our industry

Product support capabilities

Selection criteria reflects maturity and support needs

Steady

#optimizeplan

@bob_garcia

@joestanhope

Breadth of available online testing techniques

Strength of vendor’sreference customers

Availability of advanced or proprietaryfeatures andfunctionality

Vendor’sreputation (e.g.known as a leader in this field)

The online testing application is a module within a largersuite of analytics and/or marketing tools

Test deployment method (i.e.type of site tags, etc.)

Vendor’s abilityto servicemy business needs

Total cost of ownership

Ease of use and automation

Vendor’s thought leadership

User interface

Professional services, managed services,and training capabilities

Standard features and functionality (such asreporting,segmentation and targeting)

Ease of implementation and ongoing management

“Which of thefollowing are the five most important factors whenselecting an onlinetesting solution?”

Experience in our industry

Product support capabilities

Selection criteria reflects maturity and support needs

Steady

On the Rise

#optimizeplan

@bob_garcia

@joestanhope

Components of an online testing program

Each element is crucial

#optimizeplan

@bob_garcia

@joestanhope

Culture of measurement and innovation

– Online testing is an iterative program, not a discrete test

– Online testing is an experiment, not a pass/fail exercise

#optimizeplan

@bob_garcia

@joestanhope

Process

– Plan, design, execute, analyze, validate, repeat

– Marketing input

– Program goals

#optimizeplan

@bob_garcia

@joestanhope

Process

– Marketing input

– Program goals

Organization

– Defined ownership for the program is critical

– Skills to support the process

#optimizeplan

@bob_garcia

@joestanhope

Process

– Marketing input

– Program goals

Organization

– Defined ownership for the program is critical

– Skills to support the process

Technology

– Site management

– Analytics

#optimizeplan

@bob_garcia

@joestanhope

Success Factor 1: Process

Conducting tests is predominantly an ad hoc process

Testing followsformalized processesfor some tasksbut is mostly ad hoc

Testing follows a highly formalized process throughoutthe development, execution, and analysis of experiments

Testing followsformalized processesfor most tasks,but some tasks are ad hoc

“To what degree doesyour firm adhereto a structured processtoconduct onlinetesting?”

The majority of firms struggle with processes

#optimizeplan

@bob_garcia

@joestanhope

The elements of a testing plan

#optimizeplan

@bob_garcia

@joestanhope

The process is critical

#optimizeplan

@bob_garcia

@joestanhope

Online Testing Processes

Who are the stakeholders?

How do we ideate experiments?

What is the criteria for prioritizing experiments?

Can we get support from existing process management expertise such as

our PMO?

How does the team stay informed?

What information populates our online testing knowledgebase?

It’s important to consider:

#optimizeplan

@bob_garcia

@joestanhope

Success Factor 2: Organization

Image source: Hostile Work Environment (http://hostilework.blogspot.com/)

Usability of thetesting interface

Abilityto segment/target test participants

Lack of corporate buy-in

Modifyingweb pagesfortesting

Establishing testing scenarios (i.e.concept, areasto test)

Acting upon testing results

Securing ITresourcesto deploy and manage testcodeon thewebsite

Demonstrating ROI

Interpreting test results

Developing testcontent (such as creative elementsand/or promotionsto serve astestvariations)

Ability of testing application to accommodate desiredtesting techniques and designs

Attaining enough site trafficfor statistically significant tests

Lack of staffto manage the testing program and utilizetesting applications

Prioritizing testing initiatives

“What challenges doyouexperience in operatingyourcompany'sA/B and/or multivariatetesting application?

No challenges

Securing funding for testing initiatives

#optimizeplan

@bob_garcia

@joestanhope

Dedicated onlinetestingor site optimizationgroup

Business intelligence orcorporate analyticsgroup

Web analyticsgroup

eCommerce

Marketing

Base: 71respondents who use an online testing platform

“What department inyour firm isresponsiblefor onlinetesting?”

Online testing is part of the business

#optimizeplan

@bob_garcia

@joestanhope

The Ownership Model SpectrumAugust 2010 “Organizing For Site Optimization”

#optimizeplan

@bob_garcia

@joestanhope

Other, please describe

A single full time resource

Acombination of part and full time resources

A single part timeresource

Multiple full time resources

Multiple part timeresources

“How is the onlinetesting program staffedatyour firm?”

Online testing is a team effort

#optimizeplan

@bob_garcia

@joestanhope

Allow external agencyto manage the entire process

Allowvendorto manage the entire process

Work with an external agency orconsultancy

Manage entirely in house

Work with our vendor’s professional servicesgroup

“What isyour preferred method of managing A/B and/or multivariatetesting onyourwebsite(s)?”

Most firms leverage third party resources

#optimizeplan

@bob_garcia

@joestanhope

How can a third party service provider help?

Services to support all stages of online testing

#optimizeplan

@bob_garcia

@joestanhope

Technical services

– Implementation

– Training

– Integrations

#optimizeplan

@bob_garcia

@joestanhope

Technical services

– Implementation

– Training

– Integrations

Program support

– Test ideation and prioritization

– Test design

– Creative services

– Experiment analysis

– Strategic advisory

#optimizeplan

@bob_garcia

@joestanhope

Technical services

– Implementation

– Training

– Integrations

Program support

– Test ideation and prioritization

– Test design

– Creative services

– Experiment analysis

– Strategic advisory

Managed services

#optimizeplan

@bob_garcia

@joestanhope

Third Party Services

Which components of the testing program would benefit from external

support?

How much help do you need to support a robust online testing program?

How much control do you want?

Do we want to become self-sufficient (eventually)?

What is the opportunity cost of delayed testing versus the expense of

services?

It’s important to consider:

#optimizeplan

@bob_garcia

@joestanhope

Make the business case for testingProve the concept with a modest, well chosen pilotJustify investment to grow and overcome challenges

Start with the basicsWork the maturity curve from A/B to Multivariate testingBuild up cadence, complexity, and concurrency over time

Online testing programs require ownershipA named resource must own the online testing program, even if in a

part-time capacity

Online testing is a people businessRealistically assess your current organization and skills relative to

managing an effective online testing program

part-time capacity

Online testing programs require ownershipA named resource must own the online testing program, even if in a part-time capacity

Online testing is a people businessRealistically assess your current organization and skills relative to managing an effective online testing program

Summary + Close

Summary - Why It Matters?

1. Make More Money – You are likely leaving money on the table

We often deliver 20-50% lift translating to thousands and millions of new found revenue.

2. Customer Experience – Customers own your future

A business that listens and relates will win. What are your competitors doing?

3. Eliminate Risk – You’re taking big and unnecessary risks

Not only does testing make money, but it helps you avoid making costly mistakes.

British fashion drives

e-commerce relevance

Business Challenge

Reduce cart abandonment and encourage repeat

website visitors for new merchandise launches,

both on the UK and USA sites.

Solution

Webtrends Optimize for testing and targeting

during both sales events and non-sales time

periods, and for testing website components for

best responses.

Gearing Emails for

One-to-One Merchandising

Business Challenge

Find more effective ways to act on rich customer

data and provide dynamic, one-to-one offers based

on individual views, searches and cart activity

Solution

Webtrends Analytics identifies promotional

opportunities for products that generate a high

number of views, searches or purchases.

Great Forrester Quotes

Realities …

“Testing Programs Require Investment to Achieve Scale.”

“… experience, process, and resource constraints throttle their efforts to scale up online testing program.”

“… firms must commit to investing in online testing programs to achieve scale and long-term success.”

“… testing delivers measureable benefits across a range of success metrics”

Forrester Research, Inc., The State of Online Testing 2011, February 2012

#OptimizePlan | @joestanhope | @bob_garcia

Joe Stanhope (Guest Speaker)+1 617.613.8929

jstanhope@forrester.com

@joestanhope

www.forrester.com

blogs.forrester.com/joseph_stanhope

Thank YouForrester + Webtrends

Bob Garcia+ 1 877.932.8736

Bob.garcia@webtrends.com

@bob_garcia

www.webtrends.com

blogs.webtrends.com