Date post: | 20-Aug-2015 |
Category: |
Marketing |
Upload: | optimizely |
View: | 413 times |
Download: | 3 times |
Conversion Op,miza,on: The World Beyond
Headlines and Bu:on Color Patrick McKenzie Founder
Kalzumeus So>ware @pa,o11
What Did We Learn From That Test?
• ~60% increase in sales going from 15 -‐> 8 cards • Synergis,c effects can occasionally actually happen. – More on that later.
• You don’t know what you don’t know un,l you test for it. This post on HN was me losing $100,000:
Key Points With TesPng Quotas
• Grandfather in exis,ng users. • Set quota levels intelligently. – Quotas should segment people by willingness to pay – Break out a histogram of exis,ng accounts
• Be ready to tolerate small up,ck in support load. • Your product is s%ll too cheap.
What We Learned From This Test
• A single A/B test can help your boZom line more than a product launch. • Dealing with synergis,c effects in tes,ng can occasionally be maddening. – First a:empts at implemen,ng on-‐site checkout had null result a>er null result
– Combina,on of on-‐site checkout plus redesign plus aforemen,oned card limit test plus hillclimbing won big
• The data is never wrong, but some,mes the data isn’t telling you what you think it is.
Key Points With TesPng Checkout Flow
• Make absolutely sure you don’t break this! • Have UX-‐focused team (product / marke,ng / etc) go over every step of checkout funnel with zero tolerance for bad UX. • Checkout flows go “stale” at many successful companies. Schedule a deep dive on them twice a year, at least. • Upsells and cross selling are classics for a reason!
What We Learned From This Test
• Important to communicate with sales/CS. – Allay fears about being replaced, leads creamed, etc. – Demonstrate it directly makes them money.
• Lead to resource-‐intensive but hugely successful lead-‐scoring efforts. • Jus,fied very expensive interven,ons like e.g. onboarding help for everybody.
Key Points With TesPng Offline Processes
• Your so>ware/workflow has to close the loop back to the tes,ng so>ware. This can be tricky. • Virtually no CRM or A/B tes,ng so>ware accurately models enterprise decisionmaking process. You might be forced to op,mize for proxy conversions. • Volumes required for sta,s,cal significance are really punishing, so some,mes we “proxy and pray.”
Key Goals of Onboarding Process
• Get user to a concrete success as soon as possible. • For more complex sales, lay groundwork for future success by e.g. invi,ng in team or gepng permission to follow-‐up. • For B2C, especially in social, you live and die on virality during the onboarding experience.
What We Learned From This Test
• Users who start onboarding tour 25% to 100%+ more likely to purchase so>ware • Users some,mes have visceral reac,on to elements of the onboarding experience. – The wise-‐alecky Kiwi – “Beau,ful So>ware Lady Susan”
• Engineers prefer in-‐app onboarding but it is expensive as heck to build and does not always jus,fy itself versus e.g. a Youtube video.
Key Points With TesPng Offline Processes
• Your so>ware/workflow has to close the loop back to the tes,ng so>ware. This can be tricky. • Virtually no CRM or A/B tes,ng so>ware accurately models enterprise decisionmaking process. You might be forced to op,mize for proxy conversions. • Volumes required for sta,s,cal significance are really punishing, so some,mes we “proxy and pray.”
What We Learned From This Test
• Just hiding the advanced features increased task success by 10%, which directly improved sales • Further jump of 15% when we did the redesign – Absolutely no increase in sales… – … but we saw in the funnel that this was solely due to less people hipng the trial limits …
– … which we were able to profitably exploit with the quota limit test described earlier
Key Points With TesPng Core InteracPons
• Test the en,re funnel. Some,mes task success doesn’t (unfortunately) help bo:om line metrics. • A good understanding of where people spend their ,me/energy is vital – test the 90% interac,on, not the 2%.
ROI on ~$500 A Month TesPng Budget
0.00%
0.50%
1.00%
1.50%
2.00%
2.50%
3.00%
3.50% February, 2010
March, 2010
April, 2010
May, 2010
June
, 2010
July, 2010
August, 2010
Septem
ber, 2010
Octob
er, 2010
Novem
ber, 2010
Decembe
r, 20
10
Janu
ary, 2011
February, 2011
March, 2011
April, 2011
May, 2011
June
, 2011
July, 2011
August, 2011
Septem
ber, 2011
Octob
er, 2011
Novem
ber, 2011
Decembe
r, 20
11
Janu
ary, 2012
February, 2012
March, 2012
April, 2012
May, 2012
June
, 2012
July, 2012
August, 2012
Septem
ber, 2012
Octob
er, 2012
Novem
ber, 2012
Decembe
r, 20
12
Janu
ary, 2013
February, 2013
March, 2013
April, 2013
May, 2013
June
, 2013
July, 2013
August, 2013
Septem
ber, 2013
Octob
er, 2013
Novem
ber, 2013
Decembe
r, 20
13
Janu
ary, 2014
February, 2014
Total Freaking Madness
Our Marke,ng Site
A/B
The Web Applica,on
iPhone app
Offline processes Android app
Partner site
Content minisite
Offsite checkout Japanese subsidiary
Lessons Learned With This
• Think about strategy for transi,oning people between systems – easier to build into design earlier than later • Cookies with unique iden,,es are a stopgap, not a solu,on. You need to understand the sales process. • This is a good opportunity to standardize on an A/B tes,ng tool and workflow… or you’ll hate your life • This is a “high class problem” which can be deferred un,l $50M+ in revenue, but when you solve it, it is going to cost serious ,me/money.
Single Most Important Lesson: One Line of Code
@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
Other Lessons Learned
• Organiza,on should learn rather than guess be:er. – Best investment ever: organiza,on-‐wide wiki. – Next best: weekly “What we tried this week” email.
• Engineers love A/B tes,ng when you describe it like a puzzle or a video game, with bragging rights. • Marketers love A/B tes,ng when it’s posi,oned as a way to increase their status in the organiza,on rather than a replacement for their talents.
Things To Never Be Without
• CS should be able to force-‐override a user’s par,cipa,on in an A/B test (and, relatedly, “see what users is seeing” is great if you can get it). • A/B tests deeper than surface level need unit/etc tests. Make sure your framework supports it. • Test switcher UI in development/staging is a huge win. • URL query parameter to flip tests a huge win.
Thanks for Listening!
• I love talking about this and you cannot possibly waste my ,me: @pa,o11 or [email protected] • Free email mini-‐course on CRO in so>ware companies: h:p://bit.ly/patrick-‐op,mizely • Liked this presenta,on? I deliver something like this twice a month, free, to my email list. h:p://training.kalzumeus.com to sign up