Robert Moakler, Data Science Intern, Integral Ad Science at MLconf SEA - 5/01/15

MLCONF SEATTLE — MAY 1, 2015

A large scale online natural experiment Measuring causal impact of display ads

Robert Moakler — [email protected] | [email protected]

@ MLconf Seattle 2015


The $100+ billion question!

Does online advertising really work?

$104.57 $120.05

$140.15

$160.18 $178.45

$196.05

$213.89 Digital ad spending!% change!

2012 2013 2014 2015 2016 2017 2018!Source: www.emarketer.com, “Global Ad Spending Growth to Double This Year”

20.4% 14.8% 16.7% 14.3%

11.4% 9.9% 9.1%







Do online ads cause you to take some action?


Measuring causal impact!Option 1: Randomized A/B test •  Pros

–  If setup correctly, gives unbiased causal estimates

•  Cons –  Control ads cost as much as real

ones –  Planned before campaign starts –  Coordination of multiple media

partners –  Too many levers to test them all

Campaign Ad PSA


Measuring causal impact!Option 2: Observational study •  Pros

–  Cheap –  Flexible

•  Cons –  Enormous amount of selection bias


Confounding in digital advertising campaigns!•  Why is there selection bias in observational techniques?

–  Online ads are targeted to specific segments of the population based on particular demographics, user interests and behaviors, etc.

–  Targeting ads to specific populations makes comparing users that have received an ad to those that did not very problematic; estimates of causal impact will be overestimated.



W User

features

A Served

ads

Y Convert



W User

features

A Served

ads

Y Convert

Unless we know what information targeters are using, we will never be able to fully adjust for selection bias.


Viewability!•  Web page layout, ad placement details, and user browsing behavior

and setup can all impact the way in which ads are seen online. –  Some ads are served far down on the page (below the fold) –  Ads can be loaded in hidden tabs or windows –  Users may not stay on a page long enough for it to finish loading


Viewability!


Viewability!


Viewability!


Viewability as a natural experiment!Introduce a mediating variable — viewability

W User

features

A Served

ads

Y Convert

V Viewable

ad


Methodology!

Conversion (Y=1)

Web page visit

Effect window

T 0 Untreated user

Treated user

Viewable ad (V=1)

Unviewable ad (V=0)


Methodology!

Conversion (Y=1)

Web page visit

Effect window

T 0 Untreated user

Treated user

Viewable ad (V=1)

Unviewable ad (V=0)

Parameter of interest


Data!•  Seven display advertising campaigns run during the 4th quarter of

2014 –  Diverse industries such as auto insurance, beauty products, finance, and

online marketing –  3 million - 29 million impressions –  2,000 - 2 million conversions


Using viewability as a natural experiment!Compared to the naïve analysis of comparing users that were served and not served ads, we find a drastic decrease in estimated lift when utilizing viewability.


Validation!•  How do we know a reduction in lift means our new estimates are

correct? •  Use negative control tests

–  Use the impressions of one campaign to predict an unrelated conversion


Validation!•  How do we know a reduction in lift means our new estimates are

correct? •  Use negative control tests

–  Use the impressions of one campaign to predict an unrelated conversion

W User

features

A Served

ads

Y Convert

V Viewable

ad

Y-

Unrelated outcome


Validation!Focusing on Campaign B from the previous example, we measure the ads’ impact on unrelated outcomes


Bias in the natural experiment!•  We don’t see zero effect on many of our negative controls

–  There can be other factors that affect viewability and conversion that we don’t account for




W User

features

A Served

ads

Y Convert

V Viewable

ad

W’ User

features




W User

features

A Served

ads

Y Convert

V Viewable

ad

W’ User

features

Parameter of interest


Validation!Returning to Campaign B from the previous example, we measure the ads’ impact on irrelevant outcomes


Summary!•  Viewability enables a natural experiment

–  Combines the benefits of A/B tests and observational analysis

–  Adjustment for viewability features is easier than adjusting for targeting features

–  Results in a large reduction in bias

•  Negative controls allow for validation of models when the true value being estimated is unknown –  As the true effect of a natural experiment is usually unknown, negative controls provide a

method for validation


Versatility!•  Features that can be used in a natural experiment can be found in data sets from a

wide array of industries –  Viewability of stories in a user’s news feed

–  Listening to songs on shuffle

–  Winning bids in online advertising real-time bidding systems

•  Valid negative controls naturally exist in many industries –  Purchasing unrelated products

–  Clicking unrelated links


Acknowledgments!Integral Ad Science Daniel Hill Ekaterina Eliseeva Gijs Joost Brouwer Kiril Tsemekhman

NYU Stern Foster Provost UC Berkley Alan Hubbard


Thanks! Robert Moakler — [email protected] | [email protected]

Date post:	15-Jul-2015
Category:	Technology
Upload:	sessionsevents
View:	152 times
Download:	3 times

Robert Moakler, Data Science Intern, Integral Ad Science at MLconf SEA - 5/01/15

Technology