Sofia, October 2010
Experimental marketing:why and how
Andrey Sebrant,Director, Product marketing
A few facts about Yandex
Over 50 million users a month, around 15 million a day
Over 100 million search queries a day
Over 64% of search traffic in Russia
Over 2 600 employees in 7 offices across three countries
Over $300 million in revenues last year
2
A very competitive environment
What do we rely on:
Dream team
Big math
Computing super power
Last but not least: a very efficient marketing
3
“Digital” means a lot
Of course we operate in a digital media
We collect so many data that the entire marketing becomes digital in its core
4
It takes many skills to be a pilot, much more than just ability to read devices.On the other hand, if you don’t know how to read them, the landing may be too hard.
Marketing is an experimental quantitative science
We study the world (people, products and markets)by creating models and testing them in experiments
Models are based on already known facts, observations, intuition and fantasy
Experiments or tests are carried out using standard scientific methods
Marketing studies happiness of users and customers. We have to predict what makes them happy even if they do not realize where the happiness is.
5
Product is the king
On the Net this is true as nowhere else:
- everything is free- all competitors are a few clicks away
6
Case study: Yandex bar
Hypothesis: short search window results in fewer searches because search queries are getting longer
Experiment: 50% of new bars have longer window. Tracking: search activity, search queries length
186%
104% 105% 103%100%
No. of searches Query length,
words
Query length,
chars
Churn Rate
Test bars/control bars
Yes, size matters, but not because of the query length…
7
Case study: Churn rate
Loss of users may look scary
Don’t panic! Analyse this
It turns out that users are similar to isotopes; they decay like in nuclear physics
10000
60000
110000
160000
210000
260000
310000
360000
1 2 3 4 5 6 7
Number of users registered during
week 1 and visiting the service at
least once during each next week
8
Case study: Churn rate
Loss of users may look scary
Don’t panic! Analyse this
It turns out that users are similar to isotopes; they decay like in nuclear physics
10000
60000
110000
160000
210000
260000
310000
360000
1 2 3 4 5 6 7
Number of users registered during
week 1 and visiting the service at
least once during each next week
10000
100000
1000000
1 2 3 4 5 6 7
9
y = 59125e-0,06x
y = 1E+06e-1,341x
1
10
100
1000
10000
100000
1000000
1 2 3 4 5 6 7 8 9 10
More about radioactive decay
The same percent of users quit service every week
Decay rate is characterized by half life, the time taken to decrease the initial amount of users by half
If the process is described by the sum of two exponents, there are two groups of users with different half lives
In many practical cases the churn
rate curve is the sum of two
exponential decay curves
10
Translation from mathematics to marketing
Two groups mean not two isotopes in our case:we observe loyal (long half live) and casual users
Thus, we get quantitative metrics for loyalty which we can measure at the early stages of a new (or re-launched) service.
Why is this so important?
11
Real-time obsession makes us blind to long-term effects
The internet is too fast
We all are obsessed with instantaneous measurements- «At the focus group they say…»- «usability tests show…»- CTR is dropping!…
And we often forget about life-long relationships with and love of our users ;)
12
Source data: tables of registrations and visits
Every week you need a report on how many users registered in previous weeks are still using a service. Also, it’s a useful alerting tool.
Week Week 01, % Week 02, % Week 03, % Week 04, %
19.07.2010 -25.07.2010 100,00 45,93 35,86 32,58
12.07.2010 -18.07.2010 100,00 51,23 40,36 34,91
05.07.2010 -11.07.2010 100,00 61,93 49,25 44,54
28.06.2010 -04.07.2010 100,00 59,55 50,17 43,97
21.06.2010 -27.06.2010 100,00 61,13 48,75 43,46
13
Churn rate is a good metric for evaluation interface changes
You can measure the effect of interface updates on users’ loyalty. Ideally, the new interface should be tested on a small percent of users.
Control group: new users registered in the old interface
Test group: new users registered in the new interface
Metrics to monitor: ratio of percent of loyal users, ratio of half lives
And then you can predict the future!
14
0
5000
10000
15000
20000
25000
30000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Churn rate and the budgets (usability vs. advertising)
Too much of the ad budgets are a waste
Loyal attracted = 5%,Half life = 3
Ad campaign Most advertising campaigns are too far from the ideal, because they mostly attract casual users and very few loyal ones
Loyal attracted = 100%,Half life is eternity
15
No math, just common sense
Before pouring expensive usersinto the website,
plug the holes!
16
But if you like math…
Site audience in a month n depends on the audience of the previous month:
NCAA nn )1()1()(
($)newnat FNN
)1;0(
($);
c
cmaх
G
GCC
New users come on their own
and thanks to promotion
activities:
Decrease in churn rate
costs money. But usually
it’s one-time expense:
17
Enter the segmentation
Different groups of users
Different ads
Different sources of new users
Test and measure responses in each group,test and measure efficiency of every source and ad
And think it all over!
18
19
Not only digital things matter
This still is the most important tool for a marketeer
20
Not only digital things matter
Thank you!Спасибо за внимание
Andrey Sebrante-mail: [email protected]: @asebrantFB: www.facebook.com/asebrant