Post on 11-Aug-2014
description
transcript
Lean Analytics
Lean UX NYCApril, 2014
@acroll
Some housekeeping.
Don’t sell what you can make. Make what you can sell.
Kevin Costner is a lousy entrepreneur.
The core of Lean is iteration.
Most startups don’t know what they’ll be when they grow up.
Hotmailwas a database company
Flickrwas going to be an MMO
Twitterwas a podcasting company
Autodeskmade desktop automation
Paypalfirst built for Palmpilots
Freshbookswas invoicing for a web design firm
Wikipediawas to be written by experts only
Mitelwas a lawnmower company
Unfortunately,we’re all liars.
Everyone’s idea is the best right?
People love this part!
(but that’s not always a good thing)
This is where things fall apart.
No data, no learning.
Analytics can help.
Analytics is the measurement of movement towards your business
goals.
In a startup, the purpose of analytics is to iterate to product/market fit
before the money runs out.
I have two kids.At least one of them is a girl.
What are the chancesthe other is a boy?
BB BG
GB GG
2 of 3 (66%) are boys.
GB GG BG
Some fundamentals.
A good metric is:
Understandable
If you’re busy explaining the data, you won’t be busy acting on it.
Comparative
Comparison is context.
A ratio or rate
The only way to measure change and roll up the tension between two metrics (MPH)
Behaviorchanging
If you’re busy explaining the data, you won’t be busy acting on it.
Thesimplestrule
badmetric.
If a metric won’t change how you behave, it’s a
h"p://www.flickr.com/photos/circasassy/7858155676/
Metrics help you know yourself.
Acquisition
Hybrid
Loyalty
70%of retailers
20%of retailers
10%of retailers
You are just like
Customers that buy >1x in 90d
Once
2-2.5per year
>2.5per year
Your customers will buy from you
Then you are in this mode
1-15%
15-30%
>30%
Low acquisition cost, high checkout
Increasing return rates, market share
Loyalty, selection, inventory size
Focus on
(Thanks to Kevin Hillstrom for this.)
Qualitative
Unstructured, anecdotal, revealing, hard to aggregate, often too positive & reassuring.
Warm and fuzzy.
Quantitative
Numbers and stats. Hard facts, less insight, easier to analyze; often sour and disappointing.
Cold and hard.
Exploratory
Speculative. Tries to find unexpected or interesting insights. Source of unfair advantages.
Cool.
Reporting
Predictable. Keeps you abreast of the normal, day-to-day operations. Can be managed by exception.
Necessary.
Rumsfeld on Analytics
(Or rather, Avinash Kaushik channeling Rumsfeld)
Things we
know
don’tknow
we know Are facts which may be wrong and should be checked against data.
we don’tknow
Are questions we can answer by reporting, which we should baseline & automate.
we knowAre intuition which we should quantify and teach to improve effectiveness, efficiency.
we don’tknow
Are exploration which is where unfair advantage and interesting epiphanies live.
MayAprMarFeb
Slicing and dicing data
Jan
0
5,000
Activ
e use
rs
Cohort:Comparison of similar groups along a timeline.(this is the April cohort)
A/B test:Changing one thing (i.e. color) and measuring the result (i.e. revenue.)
MultivariateanalysisChanging several things at once to see which correlates with a result.
☀☁☀☁
Segment:Cross-sectional
comparison of all people divided by
some attribute (age, gender, etc.)
☀
☁
Which of these two companiesis doing better?
January February March April May
Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50Is this company growing or stagnating?
Cohort 1 2 3 4 5
January
February
March
April
May
$5 $3 $2 $1 $0.5
$6 $4 $2 $1
$7 $6 $5
$8 $7
$9
How about this one?
Cohort 1 2 3 4 5
January
February
March
April
May
Averages
$5 $3 $2 $1 $0.5
$6 $4 $2 $1
$7 $6 $5
$8 $7
$9
$7 $5 $3 $1 $0.5
Look at the same data in cohorts
Lagging
Historical. Shows you how you’re doing; reports the news. Example: sales.
Explaining the past.
Leading
Forward-looking. Number today that predicts tomorrow; reports the news. Example: pipeline.
Predicting the future.
A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)
If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)
A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)
Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)
A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
Some examples
(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)
Which means it’s time to talk about correlation.
1
10
100
1000
10000
Ice cream consumption DrowningsJan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Correlated
Two variables that are related (but may be dependent on something else.)
Ice cream & drowning.
Causal
An independent variable that directly impacts a dependent one.
Summertime & drowning.
A leading, causal metricis a superpower.
h"p://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/
Growth hacking, demystified.
Find correlation
Test causality
Optimize the causal factor
Pick a metric to change
Is social action a leading indicator of donation?
http
://bl
og.ju
stgi
ving.
com
/nine
-reas
ons-
why
-soc
ial-a
nd-m
obile
-are
-the-
futu
re-o
f-fun
drais
ing/
Is mobile use?ht
tp://
blog
.just
givin
g.co
m/n
ine-re
ason
s-w
hy-s
ocial
-and
-mob
ile-a
re-th
e-fu
ture
-of-f
undr
aising
/
Why is Nigerian spam so badly written?
Aunshul Rege of Rutgers University, USA in 2009
Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages emailed; they expect to land 2 or 3 “Mugu” (fools) each week.One scammer boasted “When you get a reply it’s 70% sure you’ll get the money”“By sending an email that repels all but the most gullible,” says [Microsoft Researcher Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts the true to false positive ratio in his favor.”
1000 emails
1-2 responses
1 fool and their money, parted.
Bad language (0.1% conversion)
Gullible (70% conversion)
1000 emails
100 responses
1 fool and their money, parted.
Good language (10% conversion)
Not-gullible (.07% conversion)
This would be horribly inefficient since
humans are involved.
Turns out the word “Nigeria” is the best way to identify promising prospects.
Nigerian spammersreally understand their target market.
They see past vanity metrics.
The Lean Analytics framework.
Eric’s three engines of growth
Virality
Make people invite friends.
How many they tell, how fast they
tell them.
Price
Spend money to get customers.
Customers are worth more than
they cost.
Stickiness
Keep people coming back.
Approach
Get customers faster than you
lose them.
Math that matters
Dave’s Pirate MetricsAARRR
AcquisitionHow do your users become aware of you?
SEO, SEM, widgets, email, PR, campaigns, blogs ...
ActivationDo drive-by visitors subscribe, use, etc?
Features, design, tone, compensation, affirmation ...
RetentionDoes a one-time user become engaged?
Notifications, alerts, reminders, emails, updates...
RevenueDo you make money from user activity?
Transactions, clicks, subscriptions, DLC, analytics...
ReferralDo users promote your product?
Email, widgets, campaigns, likes, RTs, affiliates...
Stage
EMPATHY I’ve found a real, poorly-met need that a reachable market faces.
STICKINESS I’ve figured out how to solve the problem in a way they will keep using and pay for.
VIRALITY I’ve found ways to get them to tell their friends, either intrinsically or through incentives.
REVENUE The users and features fuel growth organically and artificially.
SCALE I’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.
GateTh
e fiv
e st
ages
Empathy stage:Localmind hacks Twitter
Needed to find out if a core assumption—strangers answering questions—was valid.Ran Twitter experiment instead of writing codeAsked senders of geolocated Tweets from Times Square random questions; counted response rateConclusion: high enough to proceed
Stickiness stage:qidiq streamlines invites
Survey owner adds recipient to groupSurvey owner asks question
Recipient reads survey questionRecipient responds to questionRecipient sees survey results
(Later, if needed…)Recipient visits site; no password!Recipient does password recovery
One-time link sent to emailRecipient creates password
Recipient can edit profile, etc.
Survey owner adds recipient to group
Survey owner asks question
Recipient gets invite
Recipient reads survey question
Recipient responds to question
Recipient installs mobile app
Recipient creates account, profile
Recipient sees survey results
Recipient can edit profile, etc.
10-2
5% R
ESPO
NSE R
ATE
70-9
0% R
ESPO
NSE R
ATE
Six business model archetypes(Yours is probably a blend of these.)
E-commerce SaaS (freemium?) Mobile app (gaming) Two sided marketplace Media User generated content
(Which means eye charts like these.)
Customer Acquisition Cost
paid direct search wom inherent virality
VISITOR
Freemium/trial offer
Enrollment
User
Disengaged User
Cancel
Freemium churn
Engaged User
Free user disengagement
Reactivate
Cancel
Trial abandonment rate
Invite Others
Paying Customer
Reactivationrate
Paid conversion
FORMER USERS
User Lifetime Value
Reactivate
FORMER CUSTOMERS
Customer Lifetime Value
Viral coefficientViral rate
Resolution
Support data
Account Cancelled Billing Info Exp.
Paid Churn Rate
Tiering
Capacity Limit
Upselling rate Upselling
Disengaged DissatisfiedTrial Over
Model + Stage = One Metric That Matters.
One MetricThat Matters.
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCGEmpathy
Stickiness
Virality
Revenue
ScaleThe
stag
e yo
u’re
at
Really? Just one?
Yes, one.
In a startup, focus is hard to achieve.
Having only one metricaddresses this problem.
www.theeastsiderla.com
Moz cuts down on metricsSaaS-based SEO toolkit in the scale stage. Focused on net adds.
Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable?
Net adds up:
Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support?
Net adds flat:
Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart?
Net adds down:
Metrics are like squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/
Empathy
Stickiness
Virality
Revenue
Scale
E-commerce SaaS MediaMobile
appUser-gencontent
2-sidedmarket
Interviews; qualitative results; quantitative scoring; surveys
Loyalty, conversion
CAC, shares, reactivation
Transaction, CLV
Affiliates, white-label
Engagement, churn
Inherent virality, CAC
Upselling, CAC, CLV
API, magic #, mktplace
Content, spam
Invites, sharing
Ads, donations
Analytics, user data
Inventory, listings
SEM, sharing
Transactions, commission
Other verticals
(Money from transactions)
Downloads, churn, virality
WoM, app ratings, CAC
CLV, ARPDAU
Spinoffs, publishers
(Money from active users)
Traffic, visits, returns
Content virality, SEM
CPE, affiliate %, eyeballs
Syndication, licenses
(Money from ad clicks)
Better: bit.ly/BigLeanTable
What other metricsdo you want to know about?
Drawing some lines in the sand.
A company loses a quarter of its customers every year.
Is this good or bad?
Not knowing what normal ismakes you do stupid things.
Baseline:5-7% growth a week
“A good growth rate during YC is 5-7% a week,” he says. “If you can hit 10% a week you're doing exceptionally well. If you can only manage 1%, it's a sign you haven't yet figured out what you're doing.” At revenue stage, measure growth in revenue. Before that, measure growth in active users.
Paul Graham, Y Combinator
• Are there enough people who really care enough to sustain a 5% growth rate?
• Don’t strive for a 5% growth at the expense of really understanding your customers and building a meaningful solution
• Once you’re a pre-revenue startup at or near product/market fit, you should have 5% growth of active users each week
• Once you’re generating revenues, they should grow at 5% a week
Baseline:10% visitor engagement/day
Fred Wilson’s social ratios
30% of users/month use web or mobile app
10% of users/day use web or mobile app
1% of users/day use it concurrently
Baseline:2-5% monthly churn• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s
on the job.• Get below a 5% monthly churn rate before you know you’ve got a
business that’s ready to grow (Mark MacLeod) and around 2% before you really step on the gas (David Skok)
• Last-ditch appeals and reactivation can have a big impact. Facebook’s “don’t leave” reduces attrition by 7%.
Who is worth more?
Today
A Lifetime:$200
Roberto Medri, Etsy
B Lifetime:$200
Visits
Baseline:Calculating customer lifetime
25%monthly churn
100/25=4The average
customer lasts 4 months
5%monthly churn
100/5=20The average
customer lasts 20 months
2%monthly churn
100/2=50The average
customer lasts 50 months
Baseline:CAC under 1/3 of CLV• CLV is wrong. CAC Is probably wrong, too.• Time kills all plans: It’ll take a long time to find
out whether your churn and revenue projections are right
• Cashflow: You’re basically “loaning” the customer money between acquisition and CLV.
• It keeps you honest: Limiting yourself to a CAC of only a third of your CLV will forces you to verify costs sooner.
Lifetime of 20 mo.$30/mo. per
customer$600 CLV
$200 CACNow segment those users!
1/3 spend
The Lean Analytics cycle
Draw a new linePivot orgive up
Try again
Success!
Did we move the needle?
Measure the results
Make changes in production
Design a test
Hypothesis
With data:find a
commonality
Without data: make a good
guess
Find a potential improvement
Draw a linePick a KPI
Do AirBnB hosts get more business if their property is professionally photographed?
Gut instinct (hypothesis)Professional photography helps AirBnB’s business
Candidate solution (MVP)20 field photographers posing as employees
Measure the resultsCompare photographed listings to a control group
Make a decision Launch photography as a new feature for all hosts
5,000 shoots per month by February 2012
Hang on a second.
Gut instinct (hypothesis)Professional photography helps AirBnB’s business
SRSLY?
Draw a new linePivot orgive up
Try again
Success!
Did we move the needle?
Measure the results
Make changes in production
Design a test
Hypothesis
With data:find a
commonality
Without data: make a good
guess
Find a potential improvement
Draw a linePick a KPI
“Gee, those houses that do well look really
nice.”
Maybe it’s the camera.
“Computer: What do all the
highly rented houses have in
common?”
Camera model.
With data:find a commonality
Without data: make a good guess
Landing page design A/B testing
Cohort analysis General analytics
URL shortening
Funnel analytics
Influencer Marketing
Publisher analytics
SaaS analytics
Gaming analytics
User interaction Customer satisfaction KPI dashboardsUser segmentation
User analytics Spying on users
Some non-tech examples.
I lied. Everyone is a tech company.
http://www.flickr.com/photos/puuikibeach/4789015423 http://www.flickr.com/photos/elcapitanbsc/3936927326
Cost of experiments: down. Cost of attention: way up.
Let’s pick on restaurants for a while.
A line in the sand
Labor costs
Gross revenue
30%
24%
20%
Too costly?
Just right
Understaffed?
=
A leading indicator
http://www.flickr.com/photos/avlxyz/4889656453http://www.flickr.com/photos/mysticcountry/3567440970
50 reservationsat 5PM
250 coversthat night
(Varies by restaurant. McDonalds ≠ Fat Duck.)
http://www.flickr.com/photos/southbeachcars/6892880699
Restaurant MVP
Is tip amount a leading indicator of long-term revenue?
Why does every table get the same menu?
Is purple ink better?http://tippingresearch.com/uploads/managing_tips.pdf
Growth hacking
(is a word you should hate but will hear a lot about.)
Growth hacking, demystified.
Find correlation
Test causality
Optimize the causal factor
Pick a metric to change
Guerrillamarketing
Data-drivenlearning
Subversiveness
GROWTHHACKING
A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)
If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)
A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)
Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)
A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
(from the 2012 Growth Hacking conference) (These are also great segments to analyze.)
The leading indicator
• Growth hacking is simply what marketing should have been doing, but it fell in love with Don Draper and opinions along the way
• Optimize a factor you think is correlated with growth
The growth hack
AirBnB and Craigslist
What if you’re ina big organization?
If a startup is an organization designed to search for a sustainable, repeatable business model, then an established company is an organization designed
to perpetuate one.
Intrapreneurs often have to use proxiesStage Startup metrics Intrapreneur metrics
EmpathyCustomers interviewed (needs &
solutions), assumptions quantified, TAM, monetization possibility
Non-customers interviewed; assumptions quantified, constraints identified, TAM, disruption potential
Stickiness Churn, engagement Support tickets, integration time, call center data, delays
Virality Viral coefficient, viral cycle time Net Promoter Score, referrals, case study willingness
Revenue Attention, engagementBillable activity; signed LOIs; pilot
programs; after-development profitability
Scale Automation Contribution, training costs, licensing
The Lean Analytics lifecycle of an Intrapreneur
Empathy Find problems; don’t test demand. Skip the business case, do analytics
Entitled, aggrieved customers
Stickiness Know your real minimum based on expectations, regulations
Hidden “must haves”, feature creep
Virality Build inherent virality in from the start; attention is the new currency
Luddites who don’t understand sharing
Revenue Consider the ecosystem, channels, and established agreements
Channel conflict, resistance, contracts
Scale Hand the baton to others gracefully Hating what happens to your baby
Beforehand Get buy-in Political fallout
Some things that work.
Frame it like a studyProduct creation is almost accidental.Unlike a VC or startup, when the initiative fails the organization still learns.
http://www.flickr.com/photos/creative_tools/8544475139
Transformative isolation:Skunkworks
Use outliers and missed searches to hunt for good ideas & adjacencies
(Multi-billion-dollar hygiene product company)
1/8 men have an incontinence issue. 1/3 women do.When search results show a significant number of men searching, this suggests the adjacent (male) market is underserved.
Use data to create a taste for data
Sitting on Billions of rows of transactional dataDavid Boyle ran 1M online surveysOnce the value was obvious to management, got license to dig.
Focus on the desired behavior, not just the information.
http://www.psychologytoday.com/blog/yes/200808/changing-minds-and-changing-towels
26% increase in towel re-use with an appeal to social norms; 33% increase when tied to
the specific room.
Energy Conservation “Nudges” and Environmentalist Ideology: Evidence from a Randomized Residential Electricity
Field Experiment - Costa & Kahn 2011
The effectiveness of energy conservation “nudges” depends on an individual’s political ideology ... Conservatives who learn that their
consumption is less than their neighbors’ “boomerang” whereas liberals reduce their consumption.
Understand hidden constraints
That pencil story is a myth. Graphite is conductive and explosive. The Minimum Viable Product is Viable for a reason.
Know what has tobe built in-house
SAP integrationEmployment law
Run it as a consulting business first.
(Just don’t get addicted to it. Your goal is to learn and overcome integration challenges and find the 20% of features that 80% of the market
will pay for.)
When in doubt, collect dataFrom tackling the FTA rate to visualizing the criminal justice supply chain.
Everything’s an excuse to experiment
Find other ways to collect data; everything is an experiment.
Don’t just collect data, chase it.
Some tools and traps
Traction graphs
Your business model
The stage you’re at
Your one metric
... change often if you’re doing it right.
So how do you track that over time?
Traction graphs
Jan Feb Mar Apr May Jun
Signupsper day
Conversionrate
Churnrate
Viralcoefficient
This axis changes for each metric
Traction graphs
Jan Feb Mar Apr May Jun
Signupsper day
Conversionrate
Churnrate
Viralcoefficient
0%
Use vanity to get to meaningful metrics
Your goal is to produce outcomes
If the outcomes require action, and vanity motivates actors, use it
But show how the vanity metric is a leading indicator of the real one
x
Web traffic
Revenue
Activation
CartSize
Conversion rate
The three threesThreeassumptions
What big bets are you making?•“People will answer questions”•“Organizers are frustrated with how to run conferences”•“We'll make money from parents”•“Amazon is reliable enough for our users.”
Three actionsto take
What are you doing to make these assumptions happen (or identify they’re wrong and change course?)•Product enhancements•Marketing strategies
Three experimentsto run
•Feature tests•Continuous deployment•A/B testing•Customer survey
The three threes
Threeassumptions
Three actionsto take
Three experimentsto run
Monthly
Weekly
Daily
Board, investors, founders
Executive team
Employees
Strategy
Tactics
Execution
The three threesThreeassumptions
Three actionsto take
Three experimentsto run
Get more people
Increase answer %
Test betterquestions
Change the UI
Test timings
Questions from peers
Many people will answer questions
The problem-solution canvasCURRENT STATUS
• List key metrics you’re tracking, where they’re at, and compare with last few weeks• How are things trending?
LAST WEEK’S LESSONS LEARNED AND ACCOMPLISHMENTS)
• What did you learn last week?• What was accomplished?• On track: YES / NO?
The Goal is to Learn
The problem-solution canvasHYPOTHESIZED SOLUTIONS
• List possible solutions that you’ll start working on next week. Rank them.• Why do you believe each solution will help you solve or complete solve the problem?
METRICS / PROOF + GOALSProblem #1 (put name here)
• Metrics you’ll use to measure whether or not the solutions are doing what you hoped (solving the problem)• List proof (qualitative) you’ll use as well• Define goals for the metric
HYPOTHESIZED SOLUTIONS
• List possible solutions that you’ll start working on next week. Rank them.• Why do you believe each solution will help you solve or complete solve the problem?
METRICS / PROOF + GOALS
• Metrics you’ll use to measure whether or not the solutions are doing what you hoped (solving the problem)• List proof (qualitative) you’ll use as well• Define goals for the metric
Problem #2 (put name here)
“The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”
Lloyd S. Nelson
Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844
ARCHIMEDES HAD TAKEN
BATHS BEFORE.
Once, a leader convinced othersin the absence of data.
Now, a leader knowswhat questions to ask.
Alistair Crollacroll@gmail.com@acroll
Ben Yoskovitzbyosko@gmail.com@byosko