Date post: | 06-Aug-2015 |
Category: |
Business |
Upload: | optimizely |
View: | 103 times |
Download: | 2 times |
Benchmark Your Optimization Program Shana RusonisContent Marketing Manager, Optimizely
#opticon2015
#opticon2015@srusonis
optimize.ly/optimizationbmrkSlides will be posted, and this session is also being recorded.
“What gets measured, gets managed.”
—Pete Drucker
Stories
Data
#opticon2015@srusonis
Overview of today’s session
• Debuting industry research from Optimizely
• Key findings: Benefits & budget
• Key findings: People behind optimization
• Key findings: Processes for optimization
• Implications & takeaways
#opticon2015@srusonis
Let’s collect some data.
How does optimization impact businesses?
#opticon2015@srusonis
Benchmark Methodologyn = 417
1/4 Large Enterprise
1/2 Mid-Size to Small Enterprise
1/4 Small Businesses
Company Size
1/4 Executives
1/2 Managers
1/4 Associate, Consultants
Level
Marketers
Product Managers
Engineers
Titles
#opticon2015@srusonis
94%Improves my understanding of my customer’s needs and desires
Helps me to deliver relevant content to my customers6%
Do you agree or disagree with the following statements about optimization?
93%Helps me to deliver relevant content to my customers
Increases my conversion and engagement rates7%
Do you agree or disagree with the following statements about optimization?
87%Allows me to take action on data quickly
Has created a culture of learning and experimentation at my company13%
Do you agree or disagree with the following statements about optimization?
87%Has created a culture of learning and experimentation at my company
Improves the creative output of my team13%
Do you agree or disagree with the following statements about optimization?
91%Increases my conversion and engagement rates
9%
Do you agree or disagree with the following statements about optimization?
88%Improves the ROI of my marketing campaigns
Allows me to take action on data quickly12%
Do you agree or disagree with the following statements about optimization?
80%Improves the creative output of my team
Improves my team’s speed to execute on ideas20%
Do you agree or disagree with the following statements about optimization?
98%
94%
93%
91%
88%
87%
87%
80%
76%
Is valuable to the success of my marketing campaigns and/or product strategy
Improves my understanding of my customer’s needs and desires
Helps me to deliver relevant content to my customers
Increases my conversion and engagement rates
Improves the ROI of my marketing campaigns
Allows me to take action on data quickly
Has created a culture of learning and experimentation at my company
Improves the creative output of my team
Improves my team’s speed to execute on ideas
6%
7%
9%
12%
13%
13%
20%
24%
Agree
Disagree
Do you agree or disagree with the following statements about optimization?
98% of optimizers agree that it’s valuable to their success
98%Is valuable to the success of my marketing campaigns and/or product strategy
Do you agree or disagree with the following statements about optimization?
#opticon2015@srusonis
… Along with many other qualitative and quantitative benefits
94%Improves my understanding of my customer’s needs and desires
Helps me to deliver relevant content to my customers6%
Do you agree or disagree with the following statements about optimization?
87%Has created a culture of learning and experimentation at my company
Improves the creative output of my team13%
Do you agree or disagree with the following statements about optimization?
91%Increases my conversion and engagement rates
9%
Do you agree or disagree with the following statements about optimization?
88%Improves the ROI of my marketing campaigns
Allows me to take action on data quickly12%
Do you agree or disagree with the following statements about optimization?
80%Improves the creative output of my team
Improves my team’s speed to execute on ideas20%
Do you agree or disagree with the following statements about optimization?
Quantitative Qualitative
#opticon2015@srusonisIncrease
57%61%
43%
How do you plan to change the amount of budget allocated to optimization in the next 12 months?
Optimization budgets are set to increase
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
55% 53%
4%
43%
Increase
Decrease
No change
How do you plan to change the amount of budget allocated to optimization in the next 12 months?
#opticon2015@srusonisIncrease
57%61%
43%
How do you plan to change the amount of budget allocated to optimization in the next 12 months?Executives & Managers are more certain of budget increases
• 57% of Executives reported a planned increase
• 61% of Managers reported a planned increase
Executive
Manager
Individual
How do you plan to change the amount of budget allocated to optimization in the next 12 months?
“Optimization has become even more crucial in today's market,
so more budget is required to make a position in the market.”
– Anonymous Survey Respondent
#opticon2015@srusonis
The human capital behind optimization.
#opticon2015@srusonis
#opticon2015@srusonis
Who optimizers are planning to hire
Web Analyst UX DesignerMarketerFront-end Developer
Program Manager
#opticon2015@srusonis
0%
5%
10%
15%
20%
25%
30%
One, full-timeMultiple, full-time
One, part-timeMultiple, part-time
An agencyNone
23%
29%
17%
29%
2% 2%
How many members of your team are focused on optimization?
Multiple,full-time
Multiple,part-time
52% of optimization teams have 1 or more full-time hires
“I think the single best thing to do is to allocate headcount towards optimization.
If you can swing it, hire a dedicated person. If you can’t, get everyone on the team
(designers, engineers, product managers) evangelized. Let everyone run an experiment
so they can see how mission critical it is.”– Kyle Rush, Director of Frontend
Engineering and Optimization, Hillary for America
#opticon2015@srusonis
Optimization draws from a wide range of teams and skills
Marketing62%
Design53%
UI or Front-End Development40%
Product36%
Engineering35%
E-commerce31%
Sales30%
Customer Service or Support23%
Executive Staff20%
Which teams are involved with optimization at your company?
Marketing32%
E-commerce14%
UI or Front-End Development12%
Product12%
Engineering9%
Design8%
Executive Staff5%
Other3%
Customer Service or Support3%
Sales
Which team leads optimization at your company?
#opticon2015@srusonis
Move, Inc. Team Structure
#opticon2015@srusonis
The processes needed for optimization.
How do we scale testing to maximize our impact?
#opticon2015@srusonis
OPTIMIZATION LEAD
Build, QA, and launch
experiments
Conclude and analyze
experiments, share results
Prioritize backlog of
experiment ideas
Research to create solid experiment
hypotheses
Processes help scale optimization efficiency
#opticon2015@srusonis
Research to create solid experiment
hypotheses
Processes help scale optimization efficiency
OPTIMIZATION LEAD
“If you can’t state your reason for running a test simply, then you probably need to examine
why and what you are testing.”
.”
– Brian Schmitt, CROmetrics
#opticon2015@srusonis
Analytics are the top input for optimization
Analytics72%
Results from Previous Experiments57%
Customer Surveys51%
User Research Studies50%
A/B Testing Case Studies44%
Support Tickets or Live Chat Feedback38%
Heat Maps28%
Which of the following inform your optimization strategy?
Analytics72%
Which of the following inform your optimization strategy?
Results from Previous Experiments57%
Which of the following inform your optimization strategy?
Customer Surveys51%
Which of the following inform your optimization strategy?
User Research Studies50%
Which of the following inform your optimization strategy?
A/B Testing Case Studies44%
Which of the following inform your optimization strategy?
A/B Testing Case Studies44%
Which of the following inform your optimization strategy?
Support Tickets or Live Chat Feedback38%
Which of the following inform your optimization strategy?
Heat Maps28%
Which of the following inform your optimization strategy?
#opticon2015@srusonis
… Along with other qualitative inputs and sources of inspiration
Customer Surveys51%
Which of the following inform your optimization strategy?
User Research Studies50%
Which of the following inform your optimization strategy?
Support Tickets or Live Chat Feedback38%
Which of the following inform your optimization strategy?
Heat Maps28%
Which of the following inform your optimization strategy?
Results from Previous Experiments57%
Which of the following inform your optimization strategy?
A/B Testing Case Studies44%
Which of the following inform your optimization strategy?
Qualitative InputsInstitutional &
Industry Knowledge
#opticon2015@srusonis
Research to create solid experiment
hypotheses
Processes help scale optimization efficiency
Prioritize backlog of
experiment ideas
OPTIMIZATION LEAD
#opticon2015@srusonis
Optimizers prioritize for revenue, customer satisfaction
Potential to increase the value of each customer interaction, or increase overall revenue
65%
The impact on ease of use and satisfaction for my customer
58%
Improvement to website or app stability or performance
48%
The inefficiencies that currently exist in my marketing and product funnels
33%
The cost to acquire customers on various platforms
31%
We don’t do this kind of prioritization
2%
How do you prioritize which digital interactions with your audience require optimization?
#opticon2015@srusonis
Optimizers care deeply about the customer experience, in addition to positive ROI.
The values we use to prioritize our tests are a
snapshot of why we choose to optimize in the first place.
“The way your customers use your product and the way they experience it … is everything.”
”
.”
—Lev Tatarov, Marktplaats, eBay
#opticon2015@srusonis
Research to create solid experiment
hypotheses
Processes help scale optimization efficiency
Prioritize backlog of
experiment ideas
Build, QA, and launch
experimentsOPTIMIZATION LEAD
#opticon2015@srusonis
Optimizers are testing a variety of elements, each with different degrees of complexity
User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)63%
Landing Pages61%
Messaging and Copy57%
Forms54%
Offers and Promotions42%
Recommended Content Functionality38%
Social Sharing Functionality37%
Payment Functionality37%
Other
What do you test when you are optimizing?
User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)63%
Landing Pages61%
Messaging and Copy57%
Forms
What do you test when you are optimizing?
#opticon2015@srusonis
What you’re likely to test depends on which industry you’re in
• B2B more likely to test landing pages
E-Commerce B2B Media
What do you test when you are optimizing?
65%74%
67%
65%
Landing Pages
Messaging and Copy
What do you test when you are optimizing?
51%46%
53%
44%
Offers and Promotions
What do you test when you are optimizing?
What do you test when you are optimizing?
32%36%
42%
50%
Social Sharing Functionality
Payment Functionality
What do you test when you are optimizing?
• E-commerce more likely to test offers, payments
• Media more likely to test social sharing
#opticon2015@srusonisMultiple Times per Week
WeeklyMultiple Times per Month
MonthlyMultiple Times per Year
37%
21%
15%
12%
9%
Approximately how often do you run A/B or multivariate tests to opti-mize your website, mobile app, or other digital channels?
Multiple Timesper Week
Multiple Timesper Month
58% of optimizers are testing at least weekly
#opticon2015@srusonis
#opticon2015@srusonis
Research to create solid experiment
hypotheses
Processes help scale optimization efficiency
Prioritize backlog of
experiment ideas
Build, QA, and launch
experiments
Conclude and analyze experiments,
share results OPTIMIZATION LEAD
#opticon2015@srusonis
Measuring optimization success is multi-faceted
Conversion Improvements from Experiments
58%
Revenue Generated from Experiments
52%
ROI (Combination of Conversion Improvement and Revenue)
49%
Output or Velocity of Experiments
37%
Number of Team Members or Departments Involved
25%
Other
1%
0 5 10 15 20 25 30 35 40 45 50 55
How do you measure the success of your optimization program?
#opticon2015@srusonis
What does this mean for optimization as
an industry?
#opticon2015@srusonis
An emerging skills market for optimization
#opticon2015@srusonis
Widening gap between optimizing companies and those who have yet to start
#opticon2015@srusonis
How to leverage these findings to improve your optimization.
What can I do today?
#opticon2015@srusonis
• Evaluate the makeup of your team; focus on communication and removing bottlenecks
• Diversify your inputs for optimization beyond analytics
• Maintain a relentless focus on process to improve experiment quality and quantity
Conversation starters and next steps
#opticon2015@srusonis
Focus on building cross-functional teams
#opticon2015@srusonis
optimize.ly/optimizationbmrkDownload the full Benchmark Report
#opticon2015
Thank you!Shana RusonisContent Marketing Manager Optimizely
#opticon2015@srusonis
Template Slides
Appendix
“The goal is to turn data into information, and
information into insight.” – Carly Fiorina, former CEO of HP
“As revenue and budget allowance increases, we plan
to put this money back into better optimization.”
– Anonymous Survey Respondent
#opticon2015@srusonis
Optimization is a cycle of continuous steps:
• Conduct research, uncover conversion points to optimize
• Create hypotheses, suggest variations and improvements
• Build, QA, and launch experiments
• Analyze experiment results
• Share findings and determine next steps
#opticon2015@srusonis
#opticon2015@srusonis
We all have challenges …
• Maximizing ROI
• Delivering custom content
• Making quick decisions
• Understanding, taking action on data
0 10 20 30 40 50 60 70 80 90 100
20%
24%
24%
27%
30%
31%
31%
Maximize the ROI of my resources
Deliver customized experiences to customer segments
Make quick decisions
Take action on company data
Embrace the creative talents of my team
Make data-driven decisions
Understand company data
37%
26%
35%
34%
35%
39%
35%
31%
26%
23%
31%
24%
21%
27%
12%
15%
15%
6%
10%
8%
6%
9%
Very Good Good OK Not Good Very Bad
#opticon2015@srusonis
Color palette
Primary
Secondary
AaAdelle Sans
AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz
0123456789