StartUp Metrics Lesson Learned - LeanCamp Barcelona

transcript

METRICSLESSONS

LEARNED

@andreasklinger

#LEANCAMP | @andreasklinger

Andreas Klinger

CWTFO of LOOKK @andreasklinger

Slides: http://www.slideshare.net/andreasklinger

LOOKK | CONDÉ NAST | 10-14-2011 | 3

STARTuP

BEFORE PRODuCT MARKET FIT

STARTuP

“BLuE OCEAN”

STARTuP

“BLuE OCEAN”

MADNESS

www.LOOKK.com

4 Biggest (development) Problemswe had/have:

OvER ENgINEERED

AND OvER CONCEPTED

FEATuRES

SOLuTION?FEATuRE

ROLLOuTS IN A/B

NO DATA

DRIvEN STARTuP CuLTuRE

SOLuTION?gOINg MENTAL

ON METRICS.

BLOCKED PROCESSES

SOLuTION? KANBAN

uNCLEARuSER vALuE

PROP(PRE-PMF)

SOLuTION?CuSTDEv

4 Biggest (development) Problemswe had/have:

OvER ENgINEERED

AND OvER CONCEPTED

FEATuRES

SOLuTION?FEATuRE

ROLLOuTS IN A/B

NO DATA

DRIvEN STARTuP CuLTuRE

SOLuTION?gOINg MENTAL

ON METRICS.

BLOCKED PROCESSES

SOLuTION? KANBAN

uNCLEARuSER vALuE

PROP(PRE-PMF)

SOLuTION?CuSTDEv

+uSERgROuP

LOOKK | CONDÉ NAST | 10-14-2011 | 10

BuZZ IS OK

NuMBERS ARE OK

FEEDBACK IS OK

MEDIA IS OK

LOOKK | CONDÉ NAST | 10-14-2011 | 11

BuZZ IS OK

NuMBERS ARE OK

FEEDBACK IS OK

MEDIA IS OK

SOMETHINgIS WRONg

TRAFFIC SPIKES

NO STICKYNESS

PuSHES DON’T LIFT TRAFFIC

LOOKK | CONDÉ NAST | 10-14-2011 | 12

BuZZ IS OK

NuMBERS ARE OK

FEEDBACK IS OK

MEDIA IS OK

SOMETHINgIS WRONg

TRAFFIC SPIKES

NO STICKYNESS

PuSHES DON’T LIFT TRAFFIC

vANITY WHAT ACTIONS WORK?

vANITY METRICS

ESPECIALLY:- Registered users- Facebook Fans

Always goes up.

vANITY METRICS

ESPECIALLY:- Registered users- Facebook Fans

Always goes up.

Not actionable…

vANITY METRICS v2

- visitors / Month- users / Month

Tell you nothing

Usually more related to external factors (PR, Search, etc)

OF THIS MONTH

vANITY METRICS v2

- visitors / Month- users / Month

Tell you nothing

Usually more related to external factors (PR, Search, etc)

OF THIS MONTH

* Tell you nothing about the health of your business or product !�K\cc�pfl�efk_`e^�XYflk�k_\�\÷\Zkj�f]�pfli�nfib%� �\oZ\gk`fe1�`k�d`^_k�j_fn�k_Xk�m`iXc�fi�gX`[�kiXùZ�nfibj

It’s a like a shop saying “we do good… there are now more people on the street”… and not actionable

FOCuS ONRETENTION

BEFORE PMFFOCuS ON RETENTION

http://www.ashmaurya.com/2010/07/3-rules-to-actionable-metrics/

Summary:In Discovery/validation

Focus on Retention & Activation

In validation/growth Focus on Revenue and Referral

THE REASON:

RETENTION=

f(uSERHAPPINESS)

RETENTION = f(uSERHAPPINESS)

Actually you don’t want to measure retention. You want to measure user Happiness.

Especially before PMF.

Rentention is the best signal for userhappiness.

But if you can - narrow down your KPIs to show user happiness and the health of your product.

If you are multi sided (eg. buyers/sellers) segment each group seperately

IF YOu WILL DO ONE THINg ONLY:FOCuS ON uSER HAPPINESS

KPI’S NEED TO BE

YOuR HEALTH MONITOR

KPI’S NEED TO BE

YOuR HEALTH MONITOR

What numbers do actually show you the (real) health of your product?

What numbers could show user happiness?

E.g. Crashpadder.com (privat room rentals) calculates an activity/happiness score for each Host. And then creates groups (cohorts) for each city and

compares over time.

IN PRODuCT DISCOvERY

MARKETINgCAN HuRT

YOuR NuMBERS

#HNLONDON | @andreasklinger

(WRONg) MARKETINg STuNTS (CAN) HuRT YOuR (ACTIONABLE) NuMBERS

E.g. PR BuRSTS OR COMPETITIONS- Created visits in google Analytics- Created userentries in Database- Disappeared.- Lower your retention and activation (%) rates - Defocus you from understanding our product.

Example: LOOKKWe asked users to vote designers in a competition. <÷\Zk1�G\fgc\�mfk\[�k_\`i�]i`\e[j�$�[`[eÌk�^Xm\�X�iXkÌj�ass about LOOKK.

?l^\�kiXùZ�jg`b\j%�?l^\�É]\\c^ff[Ê%�C`kkc\�c\Xie`e^j�Xe[�cfe^k\id�\÷\Zk%

They just created “Dataschmutz”. useless data that obfuscates your real user behaviour

DRILL DOWNTO REMOvE

DATASCHMuTZ

#HNLONDON | @andreasklinger

DRILL DOWN TO REMOvEDATASCHMuTZ*

Q1: What numbers really show the health of your Ylj`e\jj�Xe[�Xi\�X÷\Zk\[�Yp�pfli�gif[lZk�Z_Xe^\j

Q2: How can you make them more “stable” against flkj`[\�\÷\Zkj%�

Example:Base your activation and retention KPIs on visits of

i\^`jk\i\[�lj\ij�efk�Xcc�m`j`kfij�kf�i\dfm\�kiXùZ�jg`b\j%

If you are focusing on your onboarding experience (eg. services not communities) focus base them on the user

registration date.

* Schmutz (noun, german/jiddish) means “dirt”

THE METRICS FAIRY SAYS…

METRICSNEED

TOHuRT

METRICS NEED TO HuRT

If the metrics you focus on do not hurt you everytime you look at them and you are too

embarassed to show them around.

They are eithera) not narrowed down enough

orb) nothing you should focus on.

START WITHAARRR

EXAMPLE - LOOKK

ACQuISITION - user registeredACTIvATION - user voted (eg 90%)RETENTION - user visited AgainREFERRAL - …REvENuE - …

Nice to look at. useless to work with.

Eg. We changed activiation from “Registered users that voted” to “Registered users that voted for more than two designers”

Everybody else is just voting for a friend.

Now: AARRRR = THE PAIN THE PAIN = ACTIONABLE

“AARRR” IS THESOuND OF PAIN

ACQuISITIONYou need a way to reach the user.

(Dave is wrong. A visit is just fog.s)�\^%�i\^`jk\i\[�"�ZfeÕid\[�\dX`c

ACTIvATIONThe user has seen the real core of your

product and used it. (segment in user groups if multifaced)

RETENTION The user did an action that leads to value. (if

possible close to money or coreaction)

REFERRALNot only shared but invited users came.

REvENuEMoney.

TOO MANYMETRICS

AREBAD FOR

YOuR EYES

TOO MANYMETRICS

AREBAD FOR

YOuR EYES

On your dashboards only watch those metrics you are currently iterating with

your product on.

There are always numbers going up and [fne�lek`c�pfl�_Xm\�X�jkXYc\�lj\iÖfn%�

Don’t go crazy because of that.

Focus your product discovery around assumptions. Align your metrics to that.

Example: At LOOKK we removed revenue from our dashboards while focusing on

newsfeed and community features.

LESSONSLEARNED

TL;DR: All Suck.

…but are useful for certain parts:

TL;DR: All Suck.

>ff^c\�8eXcpk`Zj�4�KiXùZ�8eXcpj`jKissmetrics, Mixpanel = People AnalysisYour own Database = drilldown

Be consistent throughout all tools:�$�N_\e&N_\i\�kf�d\Xjli\�X�ÕeXc�^fXc - Naming of goals - Logging and Naming of user types (eg. user, Designer, Admin, visitor)

TL;DR: All Suck.

>ff^c\�8eXcpk`Zj�4�KiXùZ�8eXcpj`jKissmetrics, Mixpanel = People AnalysisYour own Database = drilldown

There are three things you want to do:

* Insights (how many…)* Accounting (compare birdeye weeks/months)* Comparing (eg. A/B or focused cohorts)

google Analytics

Dashboards rock.

Unfortunately single

player only (yet)

- Referrals - Optimize Online Marketing, - Bird eye view Website - Suck for funnel analysis (no ad-hoc eg) - Suck for consistancy (goals are easy to break)

Don’t do funnel testing in gA - only track the last step

Don’t go too crazy with customization - it will break.

READ: Cohorts in google Analyticsby http://danhilltech.tumblr.com/

KissMetrics

“View this people” feature is genius

- Insights on Conversions- Quick reports on features/pages- Suck (yet) at dashboards

use it for funnel optimization.Lj\�`k�]fi�hl`Zb�Ö\o`Yc\�`ej`^_kj%Lj\�`k�kf�Õe[�lj\ij�^iflgj

If you are a SaaS use it for everything ;)

Don’t expect too much and also test mixpanel.

LOOKKButler

- SQL = quick, Tables = cheap- You can drilldown to your actionable KPIs- Dashboards for team members- Suck at bird-eye view, not ad-_fZ�Ö\o`Yc\

Questions:Who are the most popular designers if you only count people who vote for more than one designer?

N_f�Xi\�fli�dfjk�`eÖl\ek`Xc�users?

LOOKKButler

Team dashboards are easy to do

BuT…ALWAYS REMEMBER

IN PRE-PMF

CuSTDEv > METRICS

THANKS.

PLEASESHAREYOuR

LESSONS

@ANDREASKLINgER

Presentation slides: http://slideshare.net/andreasklinger

StartUp Metrics Lesson Learned - LeanCamp Barcelona

Technology