evaluating long-term

audience effect

how to know the impact of changes

on audience reach

The speakers

Dmitry Kharitonov Digital Director, RBC

Andrey Sverdlov Digital Analytics

Consultant, AT Internet

what’s the problem?

The problem

• You’re a digital publisher having earnings

from advertising.

• You’re addicted to audience volume in terms

of reach and page views.

• One day you’re going to update the site

significantly.

• if you change your site will it affect the

audience metrics and how?

How can you be sure?

• if the difference is based on product change

or just noise?

How can you trust the data?

Common approach – A/B test

There’s number of ways to test direct response

as page views per visit, CTR, clicks and other

actions.

We have a lack of tools to measure impact on

audience reach.

towards the solution

with Andrey

Audience-centric approach in testing

Key idea: look at development of daily share of

returning visitors.

Spoiler: share of visitors of tested option B as a

metric shows this development.

Idea background

New visitors always have base proportion, returning shares play a key role.

Day Audience A total Audience B total Share of B

new returning new returning new returning

1 9 500 500 5.00%

1 200 8 300 63 437 4.98% 5.00%

2 9 300 400 4.12%

1 100 8 200 58 342 5.01% 4.00%

3 9 700 700 6.73%

1 500 8 200 79 621 5.00% 7.04%

Idea background

New visitors always have base proportion, returning shares play key role.

Day Audience A total Audience B total Share of B

new returning new returning new returning

1 9 500 500 5.00%

1 200 8 300 63 437 4.98% 5.00%

2 9 300 400 4.12%

1 100 8 200 58 342 5.01% 4.00%

3 9 700 700 6.73%

1 500 8 200 79 621 5.00% 7.04%

mathematics behind

you can’t trust the numbers without it

Fundamentals

Share of B is a random value.

Share_of_B = B / (B + A) / Base_proportion – 100%

Example:

Day 1

B = 500

B+A = 10 000 (total)

Base proportion = 5%

Share_of_B = 500 / 10 000 / 5% – 100%

= 0%

Day 2

B = 400

B+A = 9 700 (total)

Base proportion = 5%

Share_of_B = 400/ 9 700 / 5% – 100%

= -17,6%

Fundamentals

Share of B is a random number.

and We don’t know it’s distribution function.

Still we can apply methods of statistical

analysis.

Hypothesis

Zero hypothesis: tested option B is the same as

A = no effect on reach.

Right-hand alternative: option B attracts more

returning visitors than A = B is better than A.

Metric to test hypothesis

Daily share of visitors that have seen option B

-4,0%

-2,0%

0,0%

2,0%

4,0%

6,0%

8,0%

10,0%

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75

Audience 7 Moy. mobile sur pér. (Audience)

Zero hypothesis: proportion of A and B has not

changed = no effect on reach.

Right-hand hypothesis: proportion has skewed

to option B = B is better than A.

Hypothesis in terms of metric

What we are testing now?

We’re testing A and B against 1 number –

share of B.

We can make decision based on this number

combined with range of other metrics: page

views number, clicks on links, revenue, CTR,

etc.

we’ve got data, so what?

Can we trust the numbers?

Ok, now we know B is X% better than A.

But can we trust this value?

We need to prove statistical significance of

result.

Statistical significance proof

Build sample distribution.

It’s very much like normal distribution.

If result is out of 3*sigma interval

it is significant.

Otherwise it’s just noise.

We can also use Student’s T-test.

Statistical significance proof: bonus

Empirical Rambler values:

significant results start from 2-3% in audience

reach. Disclaimer: values may vary for your sites.

It is not possible to calculate these border

values with only AT Internet tools.

Technical means

What we used

A/B test – nginx w/testing module

new visitors have constant proportion

returning visitors see what they have seen first time

“murmurhash32” algorithm

Measurement – ATI Multivariate Testing tag

Proof – own server logs and processing

Conclusion

Pros&Cons

+ can understand an impact on an audience reach

+ can set up GO / NO GO constraints for changes

+ can mix audience and direct response metrics in tests

- no out-of-the-box tool

- complex computations

- mathematics involved

Outcome

Evaluation of immediate effect is not enough for

publishers.

Traditional test metrics (page views, clicks, CTR, etc.)

could be combined with audience metrics.

It makes possible to evaluate test effect on audience

reach in mid-term.

Not the easiest approach, still very useful.

Example of metrics mix

Audience reach and Impressions

-10,0%

-8,0%

-6,0%

-4,0%

-2,0%

0,0%

2,0%

4,0%

6,0%

8,0%

10,0%

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75

Audience Impressions 7 Moy. mobile sur pér. (Audience) 7 Moy. mobile sur pér. (Impressions)

Q&A

Dmitry, Andrey