So hey

My name’s Dan McKinley. I’m a general purpose web …ish programmer guy.

I got my start in the financial industry back in the early 2000’s. I did that for a few years andthen I freaked out and wound up at a startup called Etsy, in Brooklyn. That was back in 2007.

Pretty much the whole talk will be about my time at Etsy. I left earlier this year to start a newcompany with some other folks from Etsy. But I still think Etsy is awesome. I had a great timethere and learned an awful lot. So I have a few talks left to do about this I guess.

I realize we’re on the west coast and people may not be familiar with Etsy, so I’ll give you alittle background. Etsy’s a marketplace for handmade and vintage goods. It gets about 4million uniques a day, and it sold well over a billion dollars in goods in 2013.

What I’m trying to do with this slide is convince you that Etsy is pretty freaking big, even if youhaven’t heard of it. It’s certainly within the top 100 websites in the US.

But back when I started in 2007, it was tiny. Less than 20 people.

That journey was a wild ride. It was tumultuous. There were three CEO changes while I wasthere. There was plenty of arguing about plenty of things.

And the most contentious issue of all had to be this. What should we spend our time doing?I’m sure this is a hot topic at most companies. Well it was at Etsy too.

I’d say at all points, this was more or less the company line. We should be data driven.Someone heard that Google was data driven, and we wanted to be like Google. So we shouldbe data driven. There was one problem with this.

And that was that nobody knew what that meant, or what it would imply. You’d get this adviceto “be data driven” all the time, with no additional context or advice.

Or even worse, you’d get this. If you wanted to get a project approved, you’d dress it up indata. You’d sprinkle some numbers on it and wave a dead chicken at it to show you werebeing data driven. It was display behavior.

But eventually I think I started to figure some of this out. I started to figure out what it wouldmean to be data driven while you’re picking priorities.

And I also realized that it’s not that complicated.

That’s what this talk is about.

I want to go through the story of how I figured some of this out in more detail.

And that story starts back in 2007.

Etsy in its early days was a magical place. I would not want anyone to get the wrongimpression here and think I’m talking smack about Etsy in the early days. It was awesome. Imiss 2007 all the time.

But I think it’s also important to be honest. Etsy grew geometrically because it was a thing thatwas out there waiting to be discovered. It didn’t grow that way because of carefully plannedproduct and marketing efforts.

Sure, there was agency in the creation of the site. And people worked hard. But the massivegrowth that happened wasn’t strongly connected to the actions of employees. This is utterlyobvious to those of us that were there.

But few of us, as human beings, are hardwired to see things that way. Instead we all tend tothink that if your work takes off, it means you’re smart and your actions are good.

We assume that the methods that precede success caused it.

We assumed that the success was our doing, and not happenstance. And I dunno, maybe wedeveloped outsized egos.

We didn’t really question ourselves. We created a thing and people loved it. And we believedthat to be a repeatable process.

If you build it, they will come, in other words.

And you know what, if the site’s growth is really insane, it looks like it’s working. You canrelease things and as long as they don’t completely destroy everything it will look like you’re agenius. All the graphs will go up and to the right.

And that’s awesome for as long as you don’t think about it too hard.

But of course we went and spoiled all of that by thinking about it too hard.

A thing I noticed (and I’m sure I wasn’t the only person to notice this) was that we had beenconsistently deleting features after a while because nobody wound up using them.

We’d release a feature, and a year or two later it would need maintenance or become asupport headache. At that point we’d look to see if anyone was using it, and pretty often theanswer would be “no.” So we’d just kill the thing.

A good example of that is a feature called Alchemy. This was a feature that let you describean item you wanted, and then have Etsy shop owners bid on making it for you.

That’s a neat idea, and it was in the New York times a few times and was generallyconsidered awesome when it came out back in 2008.

But then we took it down for good in 2011. I looked into it at the time, and I think the statisticwas that in three years Alchemy sold about as many items as the rest of the site sells in lessthan a day. It was a giant bust.

Around that time I started selfishly wondering if there was some way I could avoidparticipating in projects like alchemy. Because there’s this weird thing about me: I prefer not towork on things that aren’t going to be used and are just destined to be taken down after a yearor two.

Around the same time we started A/B testing things in earnest. And I latched onto that as apossible solution. Instead of pushing things out to everyone all at once we’d do a split test andtry to get a quantitative measure of how we were doing.

The thing that A/B testing revealed right away was: holy crap! We’ve been delusional aboutour abilities up to this point.

A huge percentage of products that we tested either had no effect, or made things worse.Usually only slightly worse, but still. And we certainly weren’t having the positive impact weimagined.

For example in early 2011 I was involved in redesigning our homepage. Five of us put aboutfour months of engineering effort into it. The CEO worked on it with us. We imagined that thiswould be really important, being the homepage and all.

But we released it and its primary effect was: zilch. And it slightly reduced the number ofpeople signing up. We wound up just throwing all of this work away.

And that was a really crappy experience that I didn’t want to repeat.

At this point our problem seemed to be that we were still picking projects based on which onesounded the best.

But the way we decided whether or not things were working after the fact had changed. Wewere evaluating our results with A/B testing. And that turned out to be really important.

Once we were looking at how we were really performing when we were releasing products,we’d opened Pandora’s box.

It would be pretty disingenuous for us to just ignore this problem and go back to not A/Btesting.

But we weren’t happy releasing products that had a minor, neutral, or even negative impact.

Our batting average with products was terrible. But over time I did figure out a way to getslightly better at this.

Back at the beginning we started with this. We’d have an idea, we’d code it, and then we’dpush it out.

Then in 2010 we tacked this A/B testing step on to the end.

But that was blowing up in our faces and pretty often it really looked more like this. We’d haveto spend a lot more time in testing than we’d planned on, because our metrics got worse.

And occasionally we couldn’t resolve that, and we might not even release stuff.

But around 2012 I settled into a process that looked more like this. It’s a little morecomplicated, but it worked a lot better.

The premise of this is basically, “hey, maybe we should incorporate data earlier on in theprocess.”

The notable feature here is an explicit step at the beginning where I try to validate ideas usingdata before doing anything else. I’m going to spend most of the rest of the talk discussing this.

The other thing I started trying to do in earnest was to avoid over-committing. We’d explicitlytry to build minimal versions of things and A/B test with those.

Doing that took a lot of discipline sometimes. For example this is a screenshot of one of myprototypes that was notoriously ugly, even when it wasn’t trying to sell you rhinoceros beetletaxidermy.

I would come into work every day while this experiment was running and have a discussionwith a designer or a support person about how crappy it looked in some edge cases. I’d smileand nod and say we’d fix the rough edges after we were sure it was going to last. Andeventually we did that.

If a project survived the first round of A/B tests, you would go back and apply some polish toit.

The nice thing about this process is that it gives me at least two relatively inexpensive placeswhere I can decide to give up on a project that isn’t working. Of course I still have the option ofscrapping it at the very end, but I’m less likely have to exercise that.

One thing about following this process is that you tend to have a lot of downtime while you’rewaiting for A/B tests to finish running.

So I started pipelining other projects. I’d get the process going on something else while I waswaiting for experimental results. It wasn’t uncommon for me to have two or three or fiverunning experiments at once.

Obviously that’s a bookkeeping challenge. You have many oars in the water at once. Youhave to keep meticulous notes just to remember what you were doing.

It’s also a challenge to get over the feeling that analysis isn’t work, at least not in the sameway that coding features is work.

Time spent doing analysis can be a lot more valuable than time spent coding. But that’scontrary to all of our instincts as engineers.

Instead of just leaving you with that abstract stuff, I’ll go through a couple of examples ofdoing the validation on a potential project.

I’ll go through two ideas, they’re both real things that came up at some point. I think they bothsound like pretty good ideas when you first hear them. And early on that would have beenenough for someone to work on them.

The first idea I’ll go through is building a landing page for furniture that’s local to the visitor.

The general idea would be: Etsy already has these landing pages for all of its categories. Andfurniture is pretty hard to buy on the internet, because shipping it is difficult.

Maybe we could improve the experience by making the pages show items that aregeographically close to the visitor.

To get stated, we assess is how large the audience for this is, and what their current behavioris like.

I’d start by looking at the overall fraction of page views that are on the page we’d be talkingabout changing. And in this case, these pages have quite a bit of traffic. It’s not the mostimportant page, but it’s something.

And of course not all of those pageviews are for furniture, but a decent amount of them are.So far so good.

Going one step further, I'd see how many people buy things that they first encountered onthese pages. And If you do that the project looks a little less attractive, because these pagesreally don’t sell many items.

That doesn’t necessarily mean that the browse pages are bad. It’s probably just that the trafficon those pages tends to be pretty removed from the purchasing decision.

Compare that to the search page, which has a ton of traffic but also sells a ton of items. Onceyou’ve gone to the trouble of typing a search, you’re a lot more likely to be in buying mode.

But wait: furniture is expensive. So maybe this is still worth digging into.

Sure, we won’t sell many more items total, but the ones we do sell will cost more.

If the average order on Etsy is $40—which by the way it’s not, I am making up financial detailsin this talk—let’s guess that the average order we’ll create will be ten times that. Let’s guess$400.

From there, we can start to answer what changing the state of the world would imply.

You get a formula like this. We have a certain number of visitors to the site, some percentageof them convert into purchases, and each purchase is worth some average amount. And thenwe multiply that by how much we’re going to improve matters.

I just talked through estimating the first three terms. Now we’re going to take some wildguesses at the fourth.

And if you do that with this project, it looks like this. I’ve picked a half percent through a twopercent increase in sales, which based on our past experience might be even a little nuts. A2% increase in conversions on a page like this would be relatively unheard of.

But let’s say we really hit it out of the park and managed a 2% increase. That would meanEtsy would sell $1000 more per day.

There’s one thing to consider here, which is that Etsy doesn’t keep all of the money when anitem sells. Etsy only takes 3.5% of the purchase price as a fee. So if we include that it addssome additional context.

The net benefit to Etsy for this feature starts to look pretty bad. If two engineers work on thisfor a month and they have a designer for half that time, then this feature would take a longtime to earn back their salaries. Not to mention HR overhead, managerial overhead, theincremental electricity required to run the feature, etc.

Another thing we can look at is how long we’d have to run this experiment to expect to get astatistically significant result. In this example the experiment would have to run for the rest ofthe decade. That is not ideal.

I think in this case it’s clear that we should spend time on something else. And the work I didto determine this only took me about an afternoon. I didn’t waste a month of my life on it, so Icount that as a victory.

Let’s go through another example. This time we’ll model sending emails to people.

Like most e-commerce sites Etsy has a cart with a couple of steps. You add something toyour cart, you fill out your credit card and shipping address, and so on.

People can quit at any step along the way. The idea here would be to wait five days and thensend people an email asking if they really meant to buy that thing they started buying.

The equation in this case looks a little different. We have a set of people who are eligible eachday. They have an average value of stuff in their carts. And we hope to reactivate apercentage of them.

So if there are about 20,000 people eligible for this email every day, and their averagepurchase is $40, our model looks like this.

I plugged in some guesses for how many people would complete the purchase, starting at 1%on the lower end. I think 1% is a reasonable guess because these people are already prettyfar down the checkout funnel.

On the low end of things, we’d sell $8,000 more per day.

Again we have to relate that to how much the company actually makes. And this time aroundit looks much better: about $100K per year in revenue.

In this case our exploration found a project that looks like it’s worth doing.

I actually built that feature while I was at Etsy. It wound up being between a half and onepercent of total sales. That might not sound like a lot, but remember that huge percentagegains are really hard to come by. This was a really big deal, as far as these things go.

So I hope you can see that even crude arithmetic can make one project look like a turkey andanother worth doing, when they both sound cool without the arithmetic.

If I can extrapolate for a few minutes,

The metaphor that I have in my head when I think about this kind of project validation isArchimedes with his lever. (Not to be grandiose or anything.) Archimedes said that given aplace to stand, he could move the Earth.

Picking products based on data is "finding your place to stand” before you apply your effort.

Your ability to move the needle is a function of volume and the audience. This is therelationship you're really trying to reason about.

By volume I mean how many people there are, and by audience I mean the characteristics ofthose people. How often they buy. How much they spend. etc.

In the furniture landing page example I gave you there was a lot of volume, but the audiencewas made up of people that didn’t buy very much. That made it an unattractive project.

To be successful at this kind of work, it really helps to be able to recall your core businessmetrics without digging them up.

Obviously these are the ones you care about for an e-commerce site like Etsy. You wouldcare about different metrics for other kinds of sites.

A really good way to keep those numbers top of mind is to make it impossible for people notto see them. At Etsy we had this toolbar that employees could see. I wrote some code to stickpage-specific business metrics into it.

So people would go about their business on the site every day, and they’d just be involuntarilyexposed to relevant numbers. There was no escape.

Remember we’re making educated guesses when we prioritize. Just getting to within an orderof magnitude of the right answer is your goal.

Without doing this, this your hunches are unlikely to even be that accurate.

I don’t want to come across as saying this is the one true way to choose projects. I do notbelieve that this method is always practical or appropriate.

If your company is brand new, you won’t have any data, so you can’t do this. Everything youdo will probably be some kind of crap shoot.

More mature companies should have tons of data, and could theoretically do this for everyproject. But even then that’s probably still not what you should do.

I don’t think there’s anything wrong with working on features that aren’t going to move theneedle.

As long as you know that you’re working on a feature that isn’t going to move the needle.

What I think is a tragedy is to work on something thinking it will have a big impact, when youshould have known better.

So I’d like to leave you with just a couple of thoughts.

As I’ve tried to demonstrate, a lot of product ideas sound awesome.

I think that “sounding awesome” is a completely unremarkable feature of a product idea.

I think that it’s pretty common for engineers to assume that people are doing this work forthem. That was certainly my implicit assumption when I started my career. And most of theengineers I’ve worked with have had the same notion.

But people might not be doing anything like this when they’re assigning you projects. Youshould make sure that they are. Or you should do it yourself.

I don’t encourage open rebellion or anything. But I think that engineers and product managersshould realize that not only are they capable of doing the work, they might be the peopleMOST capable of doing the work. If your data tooling isn’t sophisticated, engineers might bethe only people who can get some of these numbers.

I've met so many engineers in my career that "just want to build cool stuff." That’s what I wanttoo! And personally I think having an impact is pretty cool.

(h/t John Miles White)