EXPERIENCES ON INCREMENTAL RESPONSEMODELLINGSAS User Forum Finland
Helsinki May 24th 2017
Jaakko Riihimäki
Senior Data Analyst, Customer Insights & Analysis
Telia Finland Oyj
OUTLINE
• Telia Company
• Incremental Response Modelling
in Marketing
• Optimising Outbound Marketing
Campaign
• References
TELIA COMPANY
TELIA COMPANY PROVIDES
COMMUNICATION SERVICES
HELPING MILLIONS OF PEOPLE
TO BE CONNECTED AND
COMMUNICATE, DO BUSINESS
AND BE ENTERTAINED. BY
DOING THAT WE FULFILL OUR
PURPOSE TO BRING THE
WORLD CLOSER – ON THE
CUSTOMER’S TERMS
TELIA COMPANY IS THE LEADING NEW GENERATION OPERATOR IN THE NORDICS AND BALTICS…
SEK BILLION
84.2 NET SALES
25.8 EBITDA
15 CAPEX
21,000 EMPLOYEES
FOCUS ON NORDICS &
BALTICS
December 31 2016 figures refer to continuing operations, i.e. the group excluding the former segment region Eurasia
INCREMENTAL RESPONSEMODELLINGIN MARKETING
EXAMPLE: MARKETING ACTION FOR PRODUCTUPDATE
• Given observations from a marketing action, build a conditional probability model for an
update. Response variable: update / no update
P(update|𝐗, 𝐲) 𝐗 = explanatory variables, 𝐲 = response variable
• For example, in binary classification problem: logistic regression, a neural network model with
a logistic output layer and a prior that favours smooth solutions, a Gaussian process with a
probit likelihood function…
Marketing
action
Probability
model:
P(update|𝐗, 𝐲)
Targeted
marketing
action with
the model
EXAMPLE: MARKETING ACTION FOR PRODUCTUPDATE
• Given observations from a marketing action, build a conditional probability model for an
update. Response variable: update / no update
P(update|𝐗, 𝐲) 𝐗 = explanatory variables, 𝐲 = response variable
• For example, in binary classification problem: logistic regression, a neural network model with
a logistic output layer and a prior that favours smooth solutions, a Gaussian process with a
probit likelihood function…
• Instead of modelling the probability for an update, we should model the probability of an
update because of the marketing action!
Marketing
action
Probability
model:
P(update|𝐗, 𝐲)
Targeted
marketing
action with
the model
DESIGN OF EXPERIMENT FOR INCREMENTALIMPACT
• Given observations from a marketing action and from a control group, build a conditional
probability model for an update given the marketing action. Response variable: update / no
update conditioned to the marketing action.
P(update|𝐗, 𝐲, 𝐭) 𝐗 = explanatory variables
𝐲 = response variable
𝐭 = indicator whether the marketing action was received or not
Marketing
actionIncremental
response
model:
P(update|𝐗, 𝐲, 𝐭)
Targeted
marketing
action with
the modelControl
group
INCREMENTAL RESPONSE MODELLING
Different approaches for incremental response modelling (or uplift modelling)
INCREMENTAL RESPONSE MODELLING
Different approaches for incremental response modelling (or uplift modelling)
• Single probability model P update 𝐗, 𝐲, 𝐭
• Simulate the incremental impact of a marketing action using the model P update 𝐗, 𝐲, 𝐭
INCREMENTAL RESPONSE MODELLING
Different approaches for incremental response modelling (or uplift modelling)
• Single probability model P update 𝐗, 𝐲, 𝐭
• Simulate the incremental impact of a marketing action using the model P update 𝐗, 𝐲, 𝐭
• Two separate probability models: one for the target group and one for the control group
• Subtract the predicted probabilities obtained using the control group from the predicted probabilities obtained
using the target group: P𝑡 update 𝐗, 𝐲 − P𝑐 update 𝐗, 𝐲
INCREMENTAL RESPONSE MODELLING
Different approaches for incremental response modelling (or uplift modelling)
• Single probability model P update 𝐗, 𝐲, 𝐭
• Simulate the incremental impact of a marketing action using the model P update 𝐗, 𝐲, 𝐭
• Two separate probability models: one for the target group and one for the control group
• Subtract the predicted probabilities obtained using the control group from the predicted probabilities obtained
using the target group: P𝑡 update 𝐗, 𝐲 − P𝑐 update 𝐗, 𝐲
INCREMENTAL RESPONSE MODELLING
Different approaches for incremental response modelling (or uplift modelling)
• Single probability model P update 𝐗, 𝐲, 𝐭
• Simulate the incremental impact of a marketing action using the model P update 𝐗, 𝐲, 𝐭
• Two separate probability models: one for the target group and one for the control group
• Subtract the predicted probabilities obtained using the control group from the predicted probabilities obtained
using the target group: P𝑡 update 𝐗, 𝐲 − P𝑐 update 𝐗, 𝐲
• Algorithms:
• For example in binary classification where 𝑦 ∈ {0,1} instead of a response variable 𝐲 use a transformed variable 𝐳that consists of responses 𝐲 from the target group unchanged and responses from the control group inverted 1 −𝐲. With certain assumptions: P𝑡 update 𝐗, 𝐲 − P𝑐 update 𝐗, 𝐲 = 2P "update due to a marketing action" 𝐗, 𝒛 − 1
INCREMENTAL RESPONSE MODELLING
Different approaches for incremental response modelling (or uplift modelling)
• Single probability model P update 𝐗, 𝐲, 𝐭
• Simulate the incremental impact of a marketing action using the model P update 𝐗, 𝐲, 𝐭
• Two separate probability models: one for the target group and one for the control group
• Subtract the predicted probabilities obtained using the control group from the predicted probabilities obtained
using the target group: P𝑡 update 𝐗, 𝐲 − P𝑐 update 𝐗, 𝐲
• Algorithms:
• For example in binary classification where 𝑦 ∈ {0,1} instead of a response variable 𝐲 use a transformed variable 𝐳that consists of responses 𝐲 from the target group unchanged and responses from the control group inverted 1 −𝐲. With certain assumptions: P𝑡 update 𝐗, 𝐲 − P𝑐 update 𝐗, 𝐲 = 2P "update due to a marketing action" 𝐗, 𝒛 − 1
• Semi-supervised style solutions where in addition to a single uplift model, two additional models for the target
and control groups are built. An algorithm defines the training observations to each model and information flows
via training samples between the models.
EVALUATING INCREMENTAL IMPACT
• To evaluate incremental impact (or uplift), we need to measure the number of updates both
from the target and control groups but for a single observation only one of them is known
EVALUATING INCREMENTAL IMPACT
• To evaluate incremental impact (or uplift), we need to measure the number of updates both
from the target and control groups but for a single observation only one of them is known
• One solution is to measure updates at different times but the measurement times can affect
the number of updates
• Need to build a model for ”adjusting” the effect of different measurement times
EVALUATING INCREMENTAL IMPACT
• To evaluate incremental impact (or uplift), we need to measure the number of updates both
from the target and control groups but for a single observation only one of them is known
• One solution is to measure updates at different times but the measurement times can affect
the number of updates
• Need to build a model for ”adjusting” the effect of different measurement times
• Alternative solution: evaluate incremental impact for a group of observations
• The assumption: Similarly modelled obervations behave similarly
• Example: Uplift% for the highest decile = Update% for the observations ranked at the highest decile in the target
group – Update% for the observations ranked at the highest decile in the control group
• Cumulative uplift% can be computed at each decile
EXAMPLE: AREA UNDER THE UPLIFT CURVE (AUUC)
• A point at 100% gives the total
uplift% in success probability if
the whole target group is
contacted
• A diagonal line connecting
points corresponding to 0% and
100% describes the random
selection for the marketing
action
• One measure to summarise the
model performance: the Area
Under the Uplift Curve (AUUC)
MODEL SELECTION USING AUUC
• Model performance can be
summarised using the Area Under
the Uplift Curve (AUUC)
• Models can be compared, for
example, by computing the
differences in AUUC
• To assess the model performance
with respect to different data
partitions, use cross-validation
OPTIMISINGOUTBOUNDMARKETINGCAMPAIGN
EXAMPLE: OUTBOUND MARKETING
• Outbound marketing campaigns
can be optimised with incremental
response modelling
• Relevant communication to
customers
• Reduce unnecessary marketing
costs
EXAMPLE: OPTIMISING MARKETING CAMPAIGN
• In addition to incremental response
modelling, the marketing profits and
costs can be included into the model
• Return on Investment (ROI) can be
used as a measure to summarise a
marketing campaign
• ROI can be optimised, for example,
with respect to the number of
customers contacted given a follow-up
time period (as illustrated)
REFERENCES
REFERENCES
THANK YOU!