Data  Driven  Done  RightDan  McKinley @mcfunley mcfunley.com

Lean  Startup  Conference  -­‐  San  Francisco,  CA  December  10th,  2014

IntroducHons

20072003

20141998

20072003

20141998

Handmade  &  vintage  marketplace.  

$1.3  Billion  in  Gross  Sales  in  2013.  

600ish  Employees.

Less  than  20  people  in  2007.

photo:  Ian  Norman  on  Flickr.

“What  do  we  build?”

photo:  Wilhelm  Joys  Andersen  on  Flickr.

“We  should  be  data-­‐driven.”

We  should  be  data  driven.  

This  is  data.  

∴  we  should  be  driven  by  this.

Data-­‐Driven  PrioriHesThis  is  a  thing  that’s  possible.  !

I  fell  ass-­‐backwards  into  learning  a  thing  or  two  about  it.

War  Stories

In  the  Beginning

Skateboard  ramp  demoli;on,  2007.

Ideas  that  sound  awesome  are  awesome.

-­‐  Etsy  product  philosophy  in  the  early  days.

photo:  srslyguys  on  Flickr.

photo:  JusDn  Brockie  on  Flickr.

Products  in  the  Free-­‐Growth  Universe

(They  RULE)

Released  a  thing

Released  a  thing

Released  a  thing

Oh  man  that  thing  we  released,  I  tell  you  what

Pimply,  Awkward  Company

Product  Release  Plan1. Build  feature.  

2. Rent  warehouse  for  launch  party.  

3. Release  feature.  

4. Launch  party.  

5.Wait  20  months.  

6. Delete  feature  for  lack  of  usage.photo:  James  Willamor  on  flickr.

2008

2011

A/B  tesHng:  a  soluHon?

Product  selecHon:  sHll  “coolest  idea.”

photo:  nanpalmero  on  Flickr.

photo:  nanpalmero  on  Flickr.

Product  selecHon:  sHll  “coolest  idea.”

But  the  measurement  had  changed!

photo:  masterpunk.etsy.com

photo:  ketrin1407  on  Flickr.

“Only  Hope  stayed    under  the  rim  of  the  jar.”

Something  that  Actually  Started  to  Work

TimeIdea

Code

Release

2005

Idea

Code

Release

Idea

Code

A/B  testRelease?

2005 2010

“Oh,  crap”

Idea

Code

Release

Idea

Code

A/B  test

Release?

2005 2010

Wasted  effort  (maybe).

THE  HELL  WITH  IT!

Idea

Code

Release

Idea

Code

A/B  test

Release?

2005 2010

Idea

Code

Release

Idea

Code

A/B  test

Release?

IdeaValidatePrototype

A/B  Test

Refinement

A/B  test

Release

2005 2010 2012

Use  data  here.Idea

Code

Release

Idea

Code

A/B  test

Release?

IdeaValidatePrototype

A/B  Test

Refinement

A/B  test

Release

2005 2010 2012

Idea

Code

Release

Idea

Code

A/B  test

Release?

IdeaValidatePrototype

A/B  Test

Refinement

A/B  test

Release

2005 2010 2012

Possibly  quite  crappy.

Made  less  crappy  here.

Idea

Code

Release

Idea

Code

A/B  test

Release?

IdeaValidatePrototype

A/B  Test

Refinement

A/B  test

Release

2005 2010 2012

Possibly  quite  crappy.

Idea

Code

Release

Idea

Code

A/B  test

Release?

IdeaValidatePrototype

A/B  Test

Refinement

A/B  test

Release

2005 2010 2012

Maybe  give  up.

Maybe  give  up  (hopefully  not).

Maybe  give  up.

Examples

Here  are  two  real  ideas.Neither  should  sound  completely  nuts.

photo:  machineisorganic  on  Flickr.

Idea  #1  Let’s  build  a  landing  page  for  local  furniture.

Step  1:    Who  would  see  it?

Lis$ng

Search

Shop

Home

Other

“A  significant  fracDon”

Traffic

Percen

t  (%)

Browse

Lis$ng

Search

Shop

Home

Other

Traffic

Furniture

Percen

t  (%)

Browse

Lis?ng

Search

Shop

Home

Other

Not  a  significant  fracDon.

TrafficItems  soldPe

rcen

t  (%)

Browse

Step  2:    What  would  change  imply?

visits  ×  conversion  rate  ×  order  value  ×  liH  ≈  ?  

visits  ×  conversion  rate  ×  order  value  ×  liH  ≈  ?  Es#mated Modeled

50K  ×  0.25%  ×  $400  ×  lig  ≈  ?  

(fake  numbers,  by  the  way)

$0

$125,000

$250,000

$375,000

$500,000

+0.5%?

+1%?

+2%?

One  Year

50K  ×  0.25%  ×  $400  ×  lig  ≈  ?  

$0

$125,000

$250,000

$375,000

$500,000

+0.5%?

+1%?

+2%?

One  Year(fake  numbers,  by  the  way)

50K  ×  0.25%  ×  $400  ×  lig  ≈  ?  

$0

$125,000

$250,000

$375,000

$500,000

+0.5%?

+1%?

+2%  ≈  $360,000

(fake  numbers,  by  the  way)

One  Year

$0

$125,000

$250,000

$375,000

$500,000

$360,000

$13K  (our  cut)

50K  ×  0.25%  ×  $400  ×  lig  ≈  ?  

(fake  numbers,  by  the  way)

One  Year

Verdict:    Let’s  do  something  else.

Idea  #2  Let’s  email  people  that  give  up  in  the    

middle  of  purchases.

users  ×  order  value  ×  resurrec?on  rate  ≈  ?  

20K  ×  $40  ×  resurrecHon  rate  ≈  ?  

$0

$5,000,000

$10,000,000

$15,000,000

$20,000,000

+1%?

+2%?

+5%?

One  Year(fake  numbers,  again)

20K  ×  $40  ×  resurrecHon  rate  ≈  ?  

$0

$5,000,000

$10,000,000

$15,000,000

$20,000,000

+1%?

+2%?

+5%?

One  Year(fake  numbers,  again)

20K  ×  $40  ×  resurrecHon  rate  ≈  ?  

$0

$5,000,000

$10,000,000

$15,000,000

$20,000,000

$3,000,000!

One  Year(fake  numbers,  again)

$0

$5,000,000

$10,000,000

$15,000,000

$20,000,000

$3,000,000

20K  ×  $40  ×  resurrec?on  rate  ×  3.5%  ≈  ?  

$100K!  (our  cut)(fake  numbers,  again)

One  Year

Verdict:    Let’s  build  it!

Other

Abributed  Source  of  Etsy’s  Sales

My  email  (epic)

Systemic  improvement  through  basic  arithmeHc!

GeneralizaHons

Who?  !

How  many?

Close  is    good  enough.

YMMV

Startup Mature  Company

Moonshots Strategic  Projects Vebed  OpportuniDes

h/t  Marc  Hedlund

Conclusions

“What  do  we  build?”

photo:  Wilhelm  Joys  Andersen  on  Flickr.

Many  product  ideas    sound  awesome.

photo:  mindonfire  on  Flickr.

“Upon  what  meat  doth  this  our  Caesar  feed,  that  he  is  

grown  so  great?”

“He  who  refuses  to  do    arithme;c  is  doomed    to  talk  nonsense.”

-­‐  John  McCarthy

mcfunley@gmail.com  mcfunley.com  @mcfunley