Measuring the Effects of Advertising: The Digital Frontier*Randall A. Lewis, Google, Inc.Justin M. Rao, Microsoft ResearchDavid Reiley, Google, Inc.* Opinions expressed are our own, not our huge employers
Introduction
advertising is a $200+ billion per-
year industry
(~1.5-2.0% of GDP)
supports “free” services that
constitute a majority of American’s leisure
time
yet effects of advertising are
poorly understood
~$100,000 per month
~$25,000,000 per year
~$3-5,000,000 per year
~$10-25 CPM
~$1-8 CPM
I have no idea
impact of these ads?
digital measurement era two key advances
1) ad delivery and purchase data can be linked at individual level
2) ad delivery can be randomized essential exogenous variation to measure causal effects
plan for the talk
what methods should we use to measure adfx?
where can things go wrong?
what type of precision can we
expect?
we’ll go through a case study
what metrics are used to measure
adfx?
what metrics are should be used to
measure adfx?
what metrics are should be used to
measure adfx?
specific biases that can arise in online
settings
specific biases that can arise in online
settings
what’s in reach? what’s out of reach?
what’s in reach? what’s out of reach?
how does computational
advertising help?
Estimating the Causal Effects of Advertising
a few useful facts on advertising
average American sees ~$1.35 worth
of ads per day
universe of advertisers needs to
net $1.35 in marginal profits
universe of advertisers needs to
net $1.35 $5-6 in marginal profits incremental sales
(to break-even)
any given campaign will be a small
fraction of daily ad exposure
with this in mind…
25 display advertising
field experiments run at Yahoo!
experiment: exposure
determined by flip of coin
results here taken from
Lewis and Rao (2012)
data sharing gives us sales records paired with ad
delivery
Notation:: an individual=sales for person (online + offline)=1 if is treated with firm’s ad=0 if is treated with placebo ad: vector of covariates (we’ll ignore for now, all results go through by just adding “condition on ”)
Regression: average sales difference between exposed (E) and unexposed (U) groups
Experimental study:E: treatment groupU: control group
Observational study:E: endogenously exposedU: pseudo control
=s.d. of sales at individual level=treatment-control (sales impact)
let’s calibrate with medians from the 19
retail sales experiments
retailers ranging from budget to high-
end
(weekly)= (weekly)cost= $0.14 per customer (20-100) ads @$1-5 CPMROI goal=25% increase sales by $0.35 (based on margins)
goal is to increase baseline sales by 5%
problem: standard deviation is 10x the
mean
(for a successful
campaign!)
“not for scatter plots”
some observations
need very large samples to
distinguish a profitable campaign
from zero effect
even experiments with 1M subjects in each group can be
underpowered
even bigger concern
selection bias in observational
studies
omitted factors with partial
full order of magnitude more than treatment
effect
selection bias can swamp true adfx
and this is a concern
ads are by design not randomly
delivered!
(or why would be here!)
can an observational method ever be sure all selection effects have been eliminated down to such a low fraction of variance in dependent variable?
experiments are unbiased, but very noisy
observational methods can be dangerous
Advertising Experiment Case Study
1,577,256 customers of a retailer matched
to Y! ID
(allows tracking of offline sales too)
81% saw retailers ads,
19% held out
81%: 48 display ads per person on
average
treatment group had +5% sales
93% of impact occurred offline
total effect not significant at
10% level
power calculations: 21% chance of
rejecting ad had zero effect if true effect was break-
even
throw out the control group and use observational
techniques comparing exposed
vs. unexposed
would find statistically
significant and negative effect of
the adROI ~-700%
The Click and Related Metrics
CTR (click thru rate) has become second
nature
why?
cleanly defined
easily measurable
occur relatively frequently
an ad-click cannot occur with an ad
but a click can!
ads can crowd out “organic” clicks
need “baseline arrival rate”
can be solved with experiments
what does CTR miss?
brand advertising
offline sales
it is an intermediate metric
cost-per-acquisition (CPA)
advertising/bidding
move upstream from the click
sale with “qualifying link” to
advertisement
but what qualifies?
last ad? last click? first click? weighted
average?
known as the “attribution
problem”
exchanges use ad-hoc rules to “solve” the problem from an
accounting perspective
rules are inherently flawed
and widespread re-targeting has
complicated matters
attribution problem more generally
large firms advertise:- out-of-home- television- online display- online search
large firms advertise:- out-of-home- television- online display- online search
large firms advertise:- out-of-home- television- one point in global
spend- online search
large firms advertise:- out-of-home- television- cross-channel
effects?- online search
large firms advertise:- out-of-home- television- global concavity?- online search
not obvious what to conclude from “one
point”
Activity Bias
activity bias: a temporal selection bias present on the
webLewis, Rao and Reiley (2011)
WWW
in an observational study:
exposed generally more active on web
during campaign period than unexposed
why?
unexposed didn’t see campaign ad
so…
either they did not fit target dimensions
(classic heterogeneity bias)
or they are browsing less on average
(activity bias)
why does browsing activity matter
browsing ad exposure
browsing online purchases
browsing sign-ups
browsing clicks
2 examples
2 examples
without control group what we have
concluded?
activity bias difficult to control for
cross-validation could be promising
Local vs. Global Optimization
“a principled way to find the best match between a given user in a given context and a suitable advertisement“
Andrei Broder, 2008
traditional media: who
computational adv: what
automated targeting
replacing demographic
“portfolio building”
great things about computational advertising:
shifts conversation from “who to hit” to
“what did I get”
if machines beat humans
improved
performance
key parameters that govern spend still
set by humans(in general, for now)
bid, valuation, budget
bid, valuation, budget
crowd-in, crowd-out, etc. still in play
still hard!
maximizing conversions!=
maximizing profit
maximizing incremental
conversions=maximizing profit
careful not to just “advertise to the
purchasers”
Moving Forward
cost of experiments decreasing
ease of experimentation
increasing
ghost ads
90: treatment10: control
same power as:
10: treatment90: control
pre-experiment matching
linked data sources
cross-channel effects
what’s still out of reach?
long-run effects have a fundamental
difficulty
any study on adfx
Campaign
8/6/122:23pm
Campaign
8/6/122:23pm
Campaign
8/6/122:23pmAdfx
measured
Campaign
8/6/122:23pmAdfx
measured
why not just…
Campaign
8/6/122:23pmAdfx
measured
two problems(at least)
1) value to acting on information
2) statistical challenges increase with time
more data !=more precise
inference
why?
with time: cumulative effect
increases
with time: per time period effect decays
(otherwise )
datasignal
signal-to-noise
Campaign
8/6/122:23pmAdfx
measuredjudgment call
Campaign
8/6/122:23pmAdfx
measurednecessary decision
Conclusion
1) advertising and sales linked at individual
2) randomization of ad-delivery possible
whowhat
performance metrics part of culture of
advertising
many concerns still make measuring adfx very difficult