Measuring for Growth:Understanding Your Data & Next Steps

WMD 2015

Jonathan Hsu

Head of Data Science

The Social+Capital Partnership

whoamiPast Current

Analytics @ FB

• Pre-2009 : Growth and Data• IPO FB (mid-2012) : ~50 distributed data scientists/analysts• Sometimes centralized, other times distributed• Current FB: ~250 analysts

Unit of Product Development1x PM: Mini-CEO1x Data: Evidence

1x Design: Aesthetics/UEX3-10x Eng: Feasibility

Pro Con

CentralizedUniform

expectations/culture

Product mis-alignment

DistributedProduct

Alignment

Unevenexpectations/c

ulture

No “right” answer.

Accounting for MAU Growth• MAU growth alone can be misleading.

• Account for the deltas:MAU(t+∆t) = MAU(t) + new(t) + resurrected(t) - churned(t)

MAU(t) = retained(t+∆t) + churned(t+∆t)

-

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,000

200,000

1 2 3 4 5 6 7 8 9 10 11 12

MAU growing at 5% per month

Accounting for MAU Growth

• Very different companies underneath!• Both high retention (90%+) but very different growth

dynamics and implied courses of action.• Typical consumer MAU added/MAU lost ≳ 1

-20,000

-10,000

-

10,000

20,000

30,000

1 2 3 4 5 6 7 8 9 10 11 12

Low churn, low acq Ramp up new

churned

resurrected

new

-20,000

-10,000

-

10,000

20,000

30,000

1 2 3 4 5 6 7 8 9 10 11 12

High churn, high user acq Fix churn

churned

resurrected

new

Accounting for MRR Growth

• Also works for subscription/saas revenue.

MRR(t+∆t) = MRR(t) + new(t) + expansion(t) - churned(t) - contraction(t)

MRR(t) = retained(t+∆t) + churned(t+∆t) + contraction(t+∆t)

“Quick Ratio” = (Expansion + New)/(Cancelled + Contraction)

= 1.6x = 4.5x *See Mamoon Hamid’s deck

Understanding it at the cohort level

• Phenomena at fixed calendar date vs. fixed age.

• Can be used for things other than retention. Cumulative revenue for instance..

What are the drivers?

Be Bayesian

P(X | E) = P(E | X) P(X)

P(E)

• Tests are great for micro-optimization, but often not appropriate for macro questions.

• Correlation is not the best, but it’s also not valueless.

• Example:– At FB, “10 friends in

14 days”

– Notion of “time-to-value” in enterprise is similar.

How to hire this function

• Hire to achieve objectives, which may or may not include building features.

• Three areas, above the line on all, great in at least one.– Product/Business sense

– Engineering/Hacking

– Math/Statistics

• Interviews should cross boundaries as much as possible.

• Reporting chain based on objectives.

• Very hard to hire experienced folks, consider the Insight Data Science Fellowship.

Questions?