A/B Testing

Dan SirokerCo-Founder & CEO

If data can help win elections, what can it help you do for your business?

‘05‘06‘07‘08’09’10

‘05‘06‘07‘08’09’10

$0

$1

$2

$3

$4

$5

$6

$7

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4

Mill

ion

s

$0

$1

$2

$3

$4

$5

$6

$7

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4

Mill

ion

s

Dan

Pete Eric Elliot

Team: 4Customers: 644

Annual Run Rate: $0.1m

$0

$1

$2

$3

$4

$5

$6

$7

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4

Mill

ion

s

Dan

Pete Eric Elliot

Ricky

Jeff P

Team: 10 (+150%)Customers: 1,508 (+134%)

Annual Run Rate: $1.2m(+982%)

Ryan M

Mat

Meng Xiao

Julio

$0

$1

$2

$3

$4

$5

$6

$7

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4

Mill

ion

s

Dan

Pete Eric Elliot

Ricky

Jeff P

Ryan M

Mat

Meng Xiao

Julio

Sonesh

Mik

Jodie

Sandy

Brendan

Tommy

Dogan

Tyle

Arthu

Greg

Alan

Cara

Cristian

Lauren

Katy

Ryan L

Dae-Ho

Ian

Steve

Jeff Z

Amina

Byron

Shanann

Dan G

Sophie

Jon

Joanna

Jonathan

Travis

Ashley

Cyrus

Eddie

Team: 42 (+320%)Customers: 3,460 (+129%)

Annual Run Rate: $7.6m (+533%)

$0

$1

$2

$3

$4

$5

$6

$7

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4

Mill

ion

s

$0

$1

$2

$3

$4

$5

$6

$7

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4

Million

s

$0 $1 $2 $3 $4 $5 $6 $7 $8 $9

$10 $11 $12 $13 $14 $15 $16 $17 $18 $19

$20 $21 $22 $23 $24 $25 $26 $27 $28

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4 2013 Q1

Q2 Q3 Q4

Mill

ion

s

Actual Projection

$0 $1 $2 $3 $4 $5 $6 $7 $8 $9

$10 $11 $12 $13 $14 $15 $16 $17 $18 $19

$20 $21 $22 $23 $24 $25 $26 $27 $28

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4 2013 Q1

Q2 Q3 Q4

Mill

ion

s

Actual Projection

Cyrus

Eddie

Dan

Pete Eric Elliot

Ricky

Jeff P

Ryan M

Mat Meng XiaoJulio

Sonesh

Mik

Jodie

Sandy

Brendan

Tommy

Dogan

Tyle

Arthu

Greg

Alan

Cara

Cristian

Lauren

Katy

Ryan L

Dae-Ho

Ian

Steve

Jeff Z

Amina

Byron

Shanann

Dan G

Sophie

Jon

Joanna

Jonathan

Travis

Ashley

Team: 42

$0 $1 $2 $3 $4 $5 $6 $7 $8 $9

$10 $11 $12 $13 $14 $15 $16 $17 $18 $19

$20 $21 $22 $23 $24 $25 $26 $27 $28

2010 Q1

Q2 Q3 Q4 2011 Q1

Q2 Q3 Q4 2012 Q1

Q2 Q3 Q4 2013 Q1

Q2 Q3 Q4

Mill

ion

s

Actual Projection

Cyrus

Eddie

Dan

Pete Eric Elliot

Ricky

Jeff P

Ryan M

Mat Meng XiaoJulio

Sonesh

Mik

Jodie

Sandy

Brendan

Tommy

Dogan

Tyle

Arthu

Greg

Alan

Cara

Cristian

Lauren

Katy

Ryan L

Dae-Ho

Ian

Steve

Jeff Z

Amina

Byron

Shanann

Dan G

Sophie

Jon

Joanna

Jonathan

Travis

Ashley

Bo

Lisa

Miyo

Robin

James

David

Marius

Steve

Brian

Amelia

Luke

Adam

Adrian

Bradley

Brett

Jen

Ash

Richar

Lucas

Aaron

Gary

Etienne

Andrea

Neha

Chrix

Nick

Nick

Chris

Jamie

Helen

Tammie

Evan

Grace

Benson

Westly

Marco

Daniel

Kasra

Omari

Eric

Mustafa

Waylon

Fabian

Cate

Saum

Khattaab

Junan

Anshu

Bryce

Lacy

Natarajan

Katie

Andy

Ranjit

Grant

SG

Andy

Pedro

Danielle

Tai

Laura

Darwish

Amanda

Nikhil

Ryan

Hemant

Megan

Oren

Courtney

Kent

Ural

Toby

Oliver

Julia

Jaron

Jonathan

Adam

Asia

Pamela

Martjin

Ben

Becca

Michael

Christop

Dan

Eduar

Yonat

Chris

Chris

Rama

Team: 130 (+210%)

How?

Limited Life Experience

Limited Life Experience+

Overgeneralization

Limited Life Experience+

Overgeneralization=

Advice-Paul Buchheit

Lesson #1:

Define success.

Lesson #1:

Define success .

Lesson #1:

Define quantifiable success metrics.

Visitvisitors per day

Emailsign up

rate$

$ per recipient

Visit Email $sign up

rate $ per recipientvisitors per day

Media

Button

Button:

1. Sign Up

2. Learn More

3. Join Us Now

4. Sign Up Now

Media:

1. Get Involved Image

2. Family Image

3. Change Image

4. Barack’s Video

5. Springfield Video

6. Sam’s Video

Splash page experimentVariations:

Button: “Sign Up”

Button: “Learn More”

Button: “Join Us Now”

Button: “Sign Up Now”

Button:

1. Sign Up

2. Learn More

3. Join Us Now

4. Sign Up Now

Media:

1. Get Involved Image

2. Family Image

3. Change Image

4. Barack’s Video

5. Springfield Video

6. Sam’s Video

Variations:

Splash page experiment

Media: “Get Involved”

Media: “Family”

Media: “Change”

Media: “Barack’s Video”

Media: “Springfield Video”

Media: “Sam’s Video”

Button:

1. Sign Up

2. Learn More

3. Join Us Now

4. Sign Up Now

Media:

1. Get Involved Image

2. Family Image

3. Change Image

4. Barack’s Video

5. Springfield Video

6. Sam’s Video

Variations:

Splash page experiment

Splash page experiment

Splash page experiment

Splash page experiment

Email Subscriptions Volunteers Amount

Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment

Email Subscriptions Volunteers Amount

Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment

Email Subscriptions Volunteers Amount

Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment

Lesson #1:

Define quantifiable success metrics.

Lesson #2:

Less is more. Reduce choices.

vs.

+15% conversion rate

+10% $ per visitor

vs.

+20% submissions

vs.

+16.8% $ per visitor

vs.

+43.8% purchases

Lesson #2:

Less is more. Reduce choices.

Lesson #3:

Words matter. Focus on your call to action.

Donation button experiment

Variations

Donation button experiment

VariationsNever

Signed UpSigned Up,

Never DonatedPreviously Donated

0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%

vs.

+14.6% clicks

vs.+15.75% $ per visitor

Lesson #3:

Words matter. Focus on your call to action.

Lesson #4:

Seek the global maximum.

Source: http://blog.intercom.io/criticism-and-two-way-streets/

vs.

Source: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html

vs.

+0.5% retention+1.4% engagement

vs.

+600% engagement

vs.

+49% conversion rate from 240 A/B tests

Lesson #4:

Seek the global maximum.

Lesson #5:

Start today.

Source: http://techcrunch.com/2013/04/10/optimizely-series-a/

Source: http://techcrunch.com/2013/04/10/optimizely-series-a/

5,000+ Happy Customers!

Lesson #5:

Start today.

1. Define quantifiable success metrics.

2. Less is more. Reduce choices.

3. Words matter. Focus on your call to action.

4. Seek the global maximum.

5. Start today.

Lessons Learned

One more thing...

A/B Testing

Dan SirokerCo-Founder & CEO

If data can help win elections, what can it help you do for your business?