Post on 08-Jul-2020
transcript
Estimating online ad effectiveness
Kay H. BrodersenHal R. VarianGoogle,Inc.
January 7, 2015
A note about the paper
• Created using knitr
• Allows you to merge LATEXand R commands
• Runs R each time you change paper and caches results
• Inserts R code and output into paper
• Voila: reproducible research
Estimating online ad effectiveness
1. Apply treatment: change ad spend, bid, budget, etc.
1.1 Treat in some places but not others1.2 Treat in some times but not others
2. Compare to counterfactual: what would have happenedwithout experiment?
3. Of course, counterfactual can not be observed, so it must beestimated
Procedure
Train a model to predict metric of interest such as websitevisits or revenue.
Test the model on out-of-sample data to evaluateperformance.
Treat Apply model to periods or places where treatmentwas applied to estimate counterfactual
Compare actual and (estimated) counterfactual outcomes.
Hypothetical example of train-test-treat
Jun Jul Aug Sep
5000
6000
7000
8000
9000
1000
011
000
date
visi
ts
Train Test Treat Test
Bayesian Structural Time Series
We will do this in a time series context using BSTS (available fromCRAN.) BSTS combines:
Kalman filter. Accounts for seasonality and trend
Spike-and-slab regression. Automated selection of predictors
Bayesian model averaging. Avoids overfitting.
Described in Scott-Varian [2013,2014], Brodersen et. al. [2013].Related to “interrupted regression”, “synthetic controls”.
Predictors selected by BSTS
Law_and_Government__Public_Safety
Health__Vision_Care
Internet_and_Telecom__Service_Providers
Shopping__Photo_and_Video_Services
Computers_and_Electronics__Electronics_and_Electrical
Inclusion Probability
0.0 0.4 0.8
Prediction from BSTS
Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01
5000
7000
9000
1100
0
cbin
d(y,
y.p
red)
ActualCounterfactualInterval
Research strategy?
1. Use a parametric model for impact of ad campaign?• Benefit: Can use all the data to estimate• Cost: Restrictive functional form: a parallel shift
2. Use alternative estimation technique?
3. Use different models for seasonality and trend
1. Use parallel shift for ad impact
Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01
−10
000
1000
2000
3000
Index
pred
err
or
Boxcar variable
Boxcar indicator variable for campaign
Computers_and_Electronics__Electronics_and_Electrical
Sports
Online_Communities__File_Sharing_and_Hosting
Shopping__Photo_and_Video_Services
campaign
Inclusion Probability
0.0 0.4 0.8
2. Alternative estimation: linear model
Drop Kalman filter, just use simple linear model
• July 4 holiday dummy
• Top two categories from Google Trends as regressors
Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01
5000
7000
9000
1100
0
cbin
d(y,
y.p
red,
y.c
ount
erfa
ctua
l)
ActualPredictedCounterfactual
Deseasonalized the data first
Deseasonalize by fitting model with holiday regressor +day-of-week dummies. (Explain spike.)
Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01
−20
00−
500
500
1500
y.ds
Alternative approaches
1. Make no adjustment for seasonality (since predictor alreadyhas appropriate seasonality)
2. Deseasonalize both predictor and outcome• Use boxcar regressor• Use extrapolation
3. Alternative seasonality: detrend first
Deseasonalize by fitting model with holiday regressor + day ofweek dummies. (Explain spike.)
Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01
−20
00−
500
500
1500
cbin
d(y.
ds, y
.pre
d)
ActualCounterfactual
Use weekly data
Jun 15 Jul 01 Jul 15 Aug 01 Aug 15
6500
7500
8500
9500
cbin
d(y.
wee
k, y
.pre
d1, y
.pre
d2)
actualboxcarphoto
Summary
method estimate
1 bsts-extrap 1830.432 bsts-boxcar 1362.883 bsts-boxcar-all-predictors 1279.054 bsts-boxcar-top-predictors 1327.065 lm-boxcar 1434.576 lm-extrap 1289.197 not deseasonalized 1393.418 deseasonalized-boxcar 1300.679 deseasonalized-extrap 1298.37
10 week-boxcar 1248.61
What about revenue?
• Ad clicks may cannibalize search clicks
• May want to look at total number of clicks (i.e., visitors)
• But ad clicks may be worth more or less than search clicks, soreally want revenue (or profit)
• Can model ad revenue, search revenue separately or together
Examine a different advertiser using a different set of possiblepredictors
BSTS: predictor selection for visits
Movie.Rentals...Sales.Queries
Movie.Rentals...Sales.Matched
Movie.Rentals...Sales.Impressions
Movie.Rentals...Sales.Clicks
campaign
Inclusion Probability
0.0 0.2 0.4 0.6 0.8 1.0
BSTS: Visits actual and counterfactual
Uses the BSTS extrapolation model and a linear regression
Jan 14 Jan 21 Jan 28 Feb 04 Feb 11 Feb 18
9000
011
0000
1300
00
Index
cbin
d(y,
y.p
red.
top
− b
oxca
r, y.
pred
.bst
s)
ActualLinear modelBSTS
BSTS: predictor selection for revenue
Movie.Rentals...Sales.Revenue
Music.Clicks
Movie.Rentals...Sales.Impressions
Movie.Rentals...Sales.Clicks
campaign
Inclusion Probability
0.0 0.2 0.4 0.6 0.8 1.0
BSTS: Revenue actual and counterfactual
Uses the BSTS extrapolation model and a linear regression
Jan 14 Jan 21 Jan 28 Feb 04 Feb 11 Feb 18
1000
0014
0000
1800
00
Index
cbin
d(y,
y.p
red.
top
− b
oxca
r, y.
pred
.bst
s)
ActualLinear modelStructural time series
Revenue cannibalization
When campaign begins ad revenue increases significantly, organicrevenue falls somewhat.
6080
100
120
sear
ch
Jan 14 Jan 21 Jan 28 Feb 04 Feb 11 Feb 18
Index
020
6010
0
ads
cbind(search, ads)