Metrics: Lessons Learned - Killing your pre market product illusions

transcript

Metrics LessONs LeArNeD

KiLLiNG Pre-MArKetFit PrODUct iLLUsiONs

@andreasklinger

LOOKK | CONDÉ NAST | 10-14-2011 | 1

ANDreAs KLiNGer

cO-FOUNDer & cWtFOLOOKK.cOM

BAcKGrOUND iN sOciAL/ONLiNe

VP PrODUctsiNce

2,5 MONths

MFP - Maria Francesca Pepe | Veleno | www.lookk.com/mfp-maria-francesca-pepe/veleno

4 Biggest (development) Problemswe had/have:

OVer eNGiNeereD

AND OVer cONcePteD

FeAtUres

sOLUtiON?FeAtUre

rOLLOUts iN A/B

NO DAtA

DriVeN stArtUP cULtUre

sOLUtiON?GOiNG MeNtAL

ON Metrics.

BLOcKeD PrOcesses

sOLUtiON? KANBAN

UNcLeArUser VALUe

PrOP(Pre-PMF)

sOLUtiON?cUstDeV

+UserGrOUP

4 Biggest (development) Problemswe had/have:

OVer eNGiNeereD

AND OVer cONcePteD

FeAtUres

sOLUtiON?FeAtUre

rOLLOUts iN A/B

NO DAtA

DriVeN stArtUP cULtUre

sOLUtiON?GOiNG MeNtAL

ON Metrics.

BLOcKeD PrOcesses

sOLUtiON? KANBAN

UNcLeArUser VALUe

PrOP(Pre-PMF)

sOLUtiON?cUstDeV

+UserGrOUP

Metrics

Mark Goldenberg | Woven Bird | www.lookk.com/mark-goldenberg/woven-bird

stArtUP

BeFOre PrODUct MArKet Fit

stArtUP

“BLUe OceAN”

LOOKK | CONDÉ NAST | 10-14-2011 | 10

stArtUP

“BLUe OceAN”

MADNess

LOOKK | CONDÉ NAST | 10-14-2011 | 11

BUZZ is OK

NUMBers Are OK

FeeDBAcK is OK

MeDiA LiKes YOU

LOOKK | CONDÉ NAST | 10-14-2011 | 12

BUZZ is OK

NUMBers Are OK

FeeDBAcK is OK

MeDiA LiKes YOU

sOMethiNGis WrONG

trAFFic sPiKes

NO sticKYNess

PUshes DON’t LiFt trAFFic

LOOKK | CONDÉ NAST | 10-14-2011 | 13

BUZZ is OK

NUMBers Are OK

FeeDBAcK is OK

MeDiA LiKes YOU

sOMethiNGis WrONG

trAFFic sPiKes

NO sticKYNess

PUshes DON’t LiFt trAFFic

VANitY WhAt ActiONs WOrK?

VANitY Metrics

esPeciALLY:- registered Users- Facebook Fans

But also:- Visitors / Month- Users / Month

Always goes up.

@j�dfi\�`eÖl\eZ\[�by external factors than by internal.

VANitY Metrics

esPeciALLY:- registered Users- Facebook Fans

But also:- Visitors / Month- Users / Month

Are usually controlled by external ]XZkfij��GI�\÷\Zkj�fi�\m\e�J<F %

tell you nothing about the health of your Ylj`e\jj�Xe[�k_\�\÷\Zkj�f]�pfli�nfib%

(exception: it might show that viral or gX`[�kiXùZ�nfibj�$�jk`cc�[f\jek�_\cg�pfli�product)

eNGiNes OF GrOWth

sticKY (retention)

VirAL (referral)

PAiD (works technically like viral)

eric ries

eNGiNes OF GrOWth

sticKY (retention)

VirAL (referral)

PAiD (works technically like viral)

eric ries

in Discover/Validation Focus on retention and Activation

(in Validation/Growth Focus on revenue and referral)

eNGiNes OF GrOWth

VANitY Metrics

Pr resULts- created visits in Google Analytics- created userentries in Database- Disappeared.

cOMPetitiONs- created visits in Google Analytics- created userentries in Database- Disappeared.

VANitY Metrics

Pr resULts- created visits in Google Analytics- created userentries in Database- Disappeared.

cOMPetitiONs- created visits in Google Analytics- created userentries in Database- Disappeared.

DriLL DOWNreMOVe the DAtA schMUtZ

AND Get tO ActiONABLe NUMBers

DriLLDOWN

Metrics

ActiONABLe Metrics

Where are your weaknesses?

if it’s not a weakness you are working on the wrong thing.

ActiONABLeMetrics

Where are your weaknesses?

if it’s not a weakness you are working on the wrong thing.

start with the AArrr Funnel

eXAMPLe - LOOKK

AcQUisitiON - User registeredActiVAtiON - User VotedreteNtiON - User Visited AgainreFerrAL - …reVeNUe - …

eXAMPLe - LOOKK

AcQUisitiON - User registeredActiVAtiON - User Voted (eg 90%)reteNtiON - User Visited AgainreFerrAL - …reVeNUe - …

ALL NUMBers LOOKeD OK.

GOOGLe ANALYtics WAsALsO Nice tO LOOK At.

MetricsNeeD

tOhUrt

Lilach Eliyahu | Peek a Boo | www.lookk.com/lilach-eliyahu/peek-a-boo

eXAMPLe - LOOKK

AcQUisitiON - User registeredActiVAtiON - User Voted (eg 90%)reteNtiON - User Visited AgainreFerrAL - …reVeNUe - …

<- DriLLDOWN!

Analyse what it truely means to be retentive or activated on your website.

eg. We changed activiation to “Users that voted for more than two designers”

everybody else is just voting for a friend.

Now: AArrrr = the PAiN

the PAiN = ActiONABLe

AArrr NeeDs tO hUrt

AcQUisitiONYou need a way to reach the user.

(Dave is wrong )�\^%�i\^`jk\i\[�"�ZfeÕid\[�\dX`c

ActiVAtiONthe User has seen the real core of your

product and used it. (segment in user groups if multifaced)

reteNtiON the user did an action that leads to value. (if possible close to money or

coreaction)

reFerrALNot only shared but invited users

ONLY LOOKAt%

VALUes

Lilach Eliyahu | Peek a Boo | www.lookk.com/lilach-eliyahu/peek-a-boo

LessONs LeArNeD

ignore everything you currently do not work on within your product. (too much data is bad for your eyes)

eg. We ignore revenue atm.

Look for the 4 values that can tell you about your state of your product.

Only look at relative values displayed as % or .

Do not lie to yourself. if you are Pre-PMF you suck.

it’s all about iterative sucking less.

tL;Dr: All suck.

…but are useful for certain parts:

tL;Dr: All suck.

…but are useful for certain parts:

>ff^c\�8eXcpk`Zj�4�KiXùZ�8eXcpj`jKissmetrics, Mixpanel = People AnalysisYour own Database = drilldown

Be consistent throughout all tools:�$�N_\e&N_\i\�kf�d\Xjli\�X�ÕeXc�^fXc - Naming of goals - Logging and Naming of user types (eg. User, Designer, Admin, Visitor)

Google Analytics

Dashboards rock.

Unfortunately single

player only (yet)

- referrals - Optimize Online Marketing, - Bird eye view Website - suck for funnel analysis (no ad-hoc eg) - suck for consistancy (goals are easy to break)

Don’t do funnel testing in GA - only track the last step

Don’t go too crazy with customization

KissMetrics

“View this people” feature is genius

- insights on conversions- Quick reports on features/pages- suck (yet) at dashboards

Use it for funnel optimization.Lj\�`k�]fi�hl`Zb�Ö\o`Yc\�`ej`^_kj%Lj\�`k�kf�Õe[�lj\ij�^iflgj

if you are a saas use it for everything ;)

Don’t expect too much and also test mixpanel.

LOOKKButler

- sQL = quick, tables = cheap- You can drilldown to your actionable KPis- Dashboards for team members- suck at bird-eye view, not ad-_fZ�Ö\o`Yc\

Questions:Who are the most popular designers if you only count people who vote for more than one designer?

N_f�Xi\�fli�dfjk�`eÖl\ek`Xc�users?

cOMPAre reteNtiON

BehAViOr

Weekly cohorts

if you group them good for comparing behavioural patterns over time.eg. Visited Again

cOMPAre reteNtiON

BehAViOr

AArrr+

WeeKLY cOhOrts

iF BAseD ON reGistrAtiON WeeK

hArD tO ActiON ON

- Better: ActiON WeeK

tAKe AWAYs:

Use YOUr DB tO DriLLDOWNBe cONsisteNt BeYOND

tOOLsFOcUs ON reteNtiON AND

ActiVAtiON

shAreYOUr

LessONs LeArNeD.

@andreasklinger

MFP - Maria Francesca Pepe | Veleno | www.lookk.com/mfp-maria-francesca-pepe/veleno

Metrics: Lessons Learned - Killing your pre market product illusions

Technology