Big Data @ PersuasionAPI

Maurits Kaptein

Co-founder / Chief Scientist Science Rockstars

www.persuasionapi.com

Big Data?

Big data is not really defined.

“Datasets that are larger than ‘common’ machines can handle”

What I will and won’t talk about

Yes: What are the challenges that are associated with big data

Yes: How did we solve them in PersuasionAPI (high level)

No: Algorithms

No: Infrastructure / Technical details

3 Key Challenges

• Focus on meaningful data• So much data, but which is useful?

• Move from Analytics to Advice• No reports in hindsight but direct responses

• Inability to run analysis on all of the data• Need for summaries / online learning

Challenge 1:What is meaningful?

What is meaningful

Depends obviously on what your aim is as a company.

We help companies increase conversion (Click-through, sales, etc.)

Persuasion plays a big role:

8Beta Launch presentations Q2 2012

6 Principles of Persuasion

8

8

9Beta Launch presentations Q2 2012

Persuasion Online

9

Should we use all the strategies we can think off?

At the same time?For the same product?

Comparing many strategies with single strategies

Should we use all the strategies we can think of?

No, we are better of selecting a specific one.

Should we use the same strategies for everyone?

Strategies not equally effective for everyone?

Large differences based on personality traits

14Beta Launch presentations Q2 2012

2 Scenarios:

14

Effect of using a strategy

Avera

ge

Individuals

+-

Individuals

Effect of using a strategy

Avera

ge

+-

Should we use the same strategies for everyone?

No, people are distinct in their reactions to different strategies.

Challenge 1:Meaningful data

Identify Persuasive Strategies

Select distinct strategies

Adapt to individuals

Data:{ userId : “zcvx2312”, strategyId : 4, implementation: 32, estimatedSucces : 0.23, certainty : 0.013}

Challenge 2:Moving from analysis to advice

Choose not to produce reports after logging responses…

But rather summarize all the data to be available for direct recommendations.

19Beta Launch presentations Q2 2012

Persuasion Profile:

•A persuasion profile is a collection of the estimates of the effect of persuasion principles for each individual user

Normal Page:

A1 (Scarcity):

A2 (Authority):

A3 (Consensus):

Effect

19

Normal Page:

A1 (Scarcity):

A2 (Authority):

A3 (Consensus):

Effect

20Beta Launch presentations Q2 2012

We log the success of each attempt

• Based on the dynamic image and the link we can monitor the success of each page served to a user.

• We will keep updates of the average performance of your served page variations, and of the performance for each client.

Normal Page:

A1 (Scarcity):

A2 (Authority):

A3 (Consensus):

Effect

20

21Beta Launch presentations Q2 2012

We improve the personal profile

• Based on the response of each client we will update our advice for that user

• The new advice is a combination of the response of that client, as well as that of other clients

Normal Page:

A1 (Scarcity):

A2 (Authority):

A3 (Consensus):

Effect

21

Normal Page:

A1 (Scarcity):

A2 (Authority):

A3 (Consensus):

Effect

22Beta Launch presentations Q2 2012

User navigates, we improve

And so on, for each individual client...

Real time analytics is most effective in predicting behavior

Normal:

A1:

A2:

A3:

Effect

First page served:

Normal:

A1:

A2:

A3:

Effect

Second page served:

Normal:

A1:

A2:

A3:

Effect

Third page served:

22

23Beta Launch presentations Q2 2012

Competing Principles

23

24Beta Launch presentations Q2 2012

Example of adjusted page

1: Log Client ID (e.g. via dynamic image, cookie, etc)

2. Link(s) to log success of the Sales Strategy

3. Hooks to log non-responsiveness to a Sales Strategy

24

Challenge 2:We provide “advice” stating which Strategy to Use for your current customer.

In between page views…

Challenge 3:How do we deal with all the data?

Problem 1: Impossible fitting to all of the data in memory

Move fully to “online” learning:Handle datapoint for datapoint

Do not focus on ( theta | data ) but rather on ( theta | prior(s) )• Summarize all meaningful info in the priors.

Find out what data you need and don’t need to make an impact on the bottom line.• E.g. no demographic data

Use M/R jobs for re-estimating

Problem 2: Individual level estimates are needed fast

Use hierarchical models:Aggregated level => Input for new users

User level => Start model for known users

Apply shrinkage Link the two levels

Use user-level model in isolation if necessaryAnalytical updates thus very fast.

Challenge 3:How do we deal with all the data:

Use online learning and split different levels of the model

Slide with the towell example

30Beta Launch presentations Q2 2012

Results

30

Increase in email click through: >100%(at the 5th reminder)Increase in e-commerce revenue:

>25%

My Big Data considerations:

Focus on meaningful data: Persuasion at an individual level.

Move from analytics to real time response: Provide real-time advice

Inability to analyze all of the data: Use online learning and hierarchical models.

End.

Thanks!

Contact us at:maurits@sciencerockstars.com

+31 621262211

www.sciencerockstars.com