with Scott Tousley

Growth Models: The Science of Converting Users into Customers

WORKSHOP

@sjtousley

@sjtousley

@sjtousley

Process > Model > Experiments

GROWTH PROCESS

GROWTH MODEL

EXPERIMENTS

1

2

3

@sjtousley

Q1

Q2Q3

Q4

W1

W13

W3

W5

W8W9

W11

Annual

W12

W10

W7W6

W2W4

W13

W1

W11

W9

W6W5

W3

W2

W4

W7W8

W12W10

W1W2

W3W4

W5W6

W7W8

W9W10

W11W12W13

W13

W12

W11

W10W9

W8W7

W6W5

W4W3 W2 W1

@sjtousley

Q1“zoom out”

W1

W13

W3

W5

W8W9

W11

W12

W10

W7W6

W2W4

W13

W1

W11

W9

W6W5

W3

W2

W4

W7W8

W12W10

W1W2

W3W4

W5W6

W7W8

W9W10

W11W12W13

W13

W12

W11

W10W9

W8W7

W6W5

W4W3 W2 W1

Q2“zoom out”

Q3“zoom out”

Q4“zoom out”

North Star Metric

@sjtousley

W1

W13

W3

W5

W8W9

W11

Weekly Active Team

W12

W10

W7W6

W2W4

W13

W1

W11

W9

W6W5

W3

W2

W4

W7W8

W12W10

W1W2

W3W4

W5W6

W7W8

W9W10

W11W12W13

W13

W12

W11

W10W9

W8W7

W6W5

W4W3 W2 W1

User Activation Rate

Team Activation Rate

Week 12 Retention New Users

Acquired

@sjtousley

W1

W13

W3

W5

W8W9

W11

Weekly Active Team

W12

W10

W7W6

W2W4

W13

W1

W11

W9

W6W5

W3

W2

W4

W7W8

W12W10

W1W2

W3W4

W5W6

W7W8

W9W10

W11W12W13

W13

W12

W11

W10W9

W8W7

W6W5

W4W3 W2 W1

User Activation Rate

Team Activation Rate

Week 12 Retention New Users

Acquired

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

ExperimentExperiment

Experiment

Experiment

Experiment

@sjtousley

W1

W13

W3

W5

W8W9

W11

Weekly Active Team

W12

W10

W7W6

W2W4

W13

W1

W11

W9

W6W5

W3

W2

W4

W7W8

W12W10

W1W2

W3W4

W5W6

W7W8

W9W10

W11W12W13

W13

W12

W11

W10W9

W8W7

W6W5

W4W3 W2 W1

User Activation Rate

Team Activation Rate

Week 12 Retention New Users

Acquired

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

ExperimentExperiment

Experiment

Experiment

Experiment

@sjtousley

AARRR

@sjtousley

Pirate Metrics

Activation

Retention

Referral

Revenue

Acquisition …. people sign up for your product/service

…. people experience “aha moment”

…. people continually come back to you

…. people tell your friends about you

…. people pay you money

@sjtousley

Activation

Retention

Referral

Revenue

Acquisition …. Create Facebook account

…. Add 7 friends in 10 days

…. Sign in once per day

…. Email Facebook invite to a friend

…. Browse news feed, see ads, FB makes $$$

North Star Metric

Case Study: Facebook

@sjtousley

Activation

Retention

Referral

Revenue

Acquisition …. Create Spotify account

…. Listen to 100 songs in 5 days

…. Listen for at least 1 hour per week

…. Share song with your friend

…. Upgrade to Spotify Premium for $10/month

North Star Metric

Case Study: Spotify

@sjtousley

Activation

Retention

Referral

Revenue

Acquisition …. Create free HubSpot CRM account

…. 2 people view 1 contact record

…. 2 people view 1 contact record per week

…. Tag coworker on contact record (via @dannos)

…. Upgrade to paid HubSpot products

North Star Metric

Case Study: HubSpot CRM

@sjtousley

Case Study: HubSpot CRM

Weekly Active Team = New Teams Activated + Existing Teams Retained - Existing Teams Lost

New Teams Activated = New Teams Acquired * Team Activation Rate

New Teams Acquired = Teams from SEO + Teams from Paid + Teams from Affiliate + ……..

Teams from Paid = Number of impressions * ad conversion rate * landing page conversion rate

Landing page conversion rate = Copywriting + Visual Design + User Experience

North Star

Metric

Zoom out metrics

Zoom in experiments

@sjtousley

Weekly Active Team = New Teams Activated + Existing Teams Retained - Existing Teams Lost

New Teams Activated = New Teams Acquired * Team Activation Rate

New Teams Acquired = Teams from SEO + Teams from Paid + Teams from Affiliate + ……..

10,000 = 5,000 + ???? + 1,500

Case Study: HubSpot CRMNorth Star

Metric

Zoom out metrics

@sjtousley

@sjtousley

W1

W13

W3

W5

W8W9

W11

Weekly Active Team

W12

W10

W7W6

W2W4

W13

W1

W11

W9

W6W5

W3

W2

W4

W7W8

W12W10

W1W2

W3W4

W5W6

W7W8

W9W10

W11W12W13

W13

W12

W11

W10W9

W8W7

W6W5

W4W3 W2 W1

User Activation Rate

Team Activation Rate

Week 12 Retention New Users

Acquired

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

Experiment

ExperimentExperiment

Experiment

Experiment

Experiment

@sjtousley

How do you run experiments?

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

Get the templates!

@sjtousley

Take out your laptops

@sjtousley

scotttousley.com/blog/athens

Experiment Spreadsheet Template

@sjtousley

@sjtousley

@sjtousley

@sjtousley

@sjtousley

@sjtousley

@sjtousley

@sjtousley

@sjtousley

Experiment Document Template

@sjtousley

@sjtousley

@sjtousley

Experiment Presentation Template

@sjtousley

@sjtousley

Insert your text

Problem

@sjtousley

Insert your text

Problem: Today’s UX

@sjtousley

Insert your text

Experiment Design: Proposed UX

@sjtousley

Insert your text (this is optional)

Minimum Viable Experiment (MVE)

@sjtousley

Insert your text

Results

@sjtousley

Next Steps

Insert your text

Case StudyHubSpot CRM: Upgrade Intent

Disclaimer

All data is fake! However, all learnings are very real.

@sjtousley

Inside the product, users have one option if they want to talk to sales -- book a meeting in the future. However, what if users want to talk to sales now?

In addition, we have a poor PQL-to-purchase funnel of only 2%.

Problem

@sjtousley

Problem: Today’s UX

@sjtousley

Problem: Today’s UX

@sjtousley

Problem: Today’s UX

@sjtousley

Problem: Today’s UX

@sjtousley

Problem: Today’s UX

@sjtousley

Problem: Today’s UX

@sjtousley

Problem: Today’s UX

@sjtousley

Problem: Today’s UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

Experiment Design: Proposed UX

@sjtousley

MVE (Minimum Viable Experiment)

Control

@sjtousley

MVE (Minimum Viable Experiment)

Variant

@sjtousley

ResultsTotal PQLs: 219 Total Conversations: 37

17%

CONTROL

@sjtousley

ResultsTotal PQLs: 160 Total Conversations: 74

46%

VARIANT

@sjtousley

ResultsTotal PQLs: 160 Total Conversations: 74

46%

NEW

`

TL;DR

We’ve nearly tripled the amount of sales conversations.

17% → 46%

@sjtousley

Results

Link to data - https://docs.google.com/spreadsheets/d/1Tb8SRferibKjBZ8T-ORE569NN5VTLr530dO3sKK0VY0/edit#gid=0

@sjtousley

Next Steps

We have a few easy next experiments we could attempt:

● Embed Meetings into the app (versus booking outside of app)● Embed live chat into the app (versus chatting outside of app)● Track the entire funnel per action ● Turn follow-up email into plain text and see how that impacts email conversion rates● Implement these same options on the website … not just in the app

In addition, we have a few more challenging experiments we could tackle:

● When user clicks “Call now” it uses the Calling tool instead of them connecting via their own cell phone

● We use our own live chat product instead of an external live chat tool. This might help with product virality as well.

Exercises

@sjtousley

@sjtousley

@sjtousley

@sjtousley

Step 1) Divide into teams of 10

@sjtousley

Step 2) Choose a company for your team

@sjtousley

Step 3) Fill out the AARRR funnel

- How would you describe the acquisition stage?

- How would you describe the activation stage?

- How would you describe the retention stage?

- How would you describe the referral stage?

- How would you describe the revenue stage?

@sjtousley

Step 4) Brainstorm what you think is their high level growth model

- Use your A.A.R.R.R. funnel as a guide to build your model

- Keep it simple! (for now)

- Everything is a math equation

- Raise your hand if you feel stuck

- ***Break it down until you get to something “actionable”***

@sjtousley

Step 5) Choose a metric you can run an experiment on

- Break down your equation enough to where you can test the copy, ad impressions, etc.

@sjtousley

Step 6) Play around with predictive model to get an idea of different metrics you can

influence!

@sjtousley

Step 7) Remember the metric you identified

1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share

@sjtousley

Step 8) Create list of ideas together

@sjtousley

Step 9) Pick your favorite idea you want to prioritize

@sjtousley

1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share

@sjtousley

Step 10) Write experiment document

1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share

@sjtousley

@sjtousley

Step 7) Remember the metric you identified

1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share

@sjtousley

Step 11) Analyze (fake) results

1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share

@sjtousley

@sjtousley

@sjtousley

Step 12) Share results

1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share

@sjtousley

@sjtousley

@sjtousley

Summary

1. Identify the metric

2. Ideate list of experiment ideas to improve metric

3. Prioritize ideas with P.I.E.

4. Write experiment document & turn experiment live

5. Analyze results in experiment document

6. Share results with team

1

2

3

4

5

6

Identify

Ideate

PrioritizeDocument

Analyze

Share

BONUS: Real-world experiment results

@sjtousley

7 Lessons From Experiments that Grew Blog Traffic by 241%

@sjtousley

Lesson 1: Giveways 2x blog post shares

Version A Version B

SUCCESS

@sjtousley

Lesson 2: Inserting “Click-to-Tweets” boosts shares

SUCCESS

@sjtousley

Lesson 3: Twitter was our most valuable “share” source

@sjtousley

Lesson 4: Our “share retention” lowers over time

@sjtousley

Lesson 5: Influencer shares old articles spikes traffic

SUCCESS

@sjtousley

Lesson 6: Headline formulas can scale blog traffic

SUCCESS

@sjtousley

Lesson 7: Social traffic is not correlated with user signups

FAIL

@sjtousley

8 Lessons from Email A/B Experiments

@sjtousley

A/B Test #1: Zeigarnik Effect

2x 4.4%8.5%CTR CTR

SUCCESS

@sjtousley

A/B Test #2: Scarcity Principle

+120%

SUCCESS

@sjtousley

A/B Test(s) #3: Customizing Email Format SUCCESS

Customizing subscriber emails drove an over 50% increase in reader engagement.

● horizontal image (versus mini thumbnail left-aligned) led to 5% higher CTR

● changing the hyperlinked blog title to a visualized email header led to 21% higher CTR

● using “read more” button over hyperlinked text led to 6% greater CTR

● including author headshot in email led to 6% greater CTR

● shorter subject lines (1-3 words) generated 16% greater open rate than longer ones (full titles, more than 5 words, etc)

@sjtousley

A/B Test #4: Holiday Christmas Poem Email

FAIL

14 sales meetings

@sjtousley

A/B Test #5: A.I.D.A. copy had 25% higher CTR

SUCCESS

Attention

Interest

Desire

Action

@sjtousley

A/B Test #6a: Writing references to popular culture

7 sales meetings

FAIL

@sjtousley

A/B Test #6b: Writing references to popular culture

SUCCESS

407 sales meetings!

@sjtousley

FAIL

A/B Test #7: Nurture healthy user (added 30 contacts) to meeting

@sjtousley

A/B Test #8: Sending automated plain-text emails from sales reps

SUCCESS

@sjtousley

New User Onboarding A/B Test @ Duolingo

@sjtousley

New User Onboarding A/B Test @ Duolingo

@sjtousley

New User Onboarding A/B Test @ Duolingo

@sjtousley

New User Onboarding A/B Test @ Duolingo

SUCCESS

+20% Daily Active Users (DAU)

@sjtousley

A/B Tests of Landing Page Scarcity

@sjtousley

A/B Tests of Landing Page Scarcity

@sjtousley

A/B Tests of Landing Page Scarcity

@sjtousley

A/B Tests of Landing Page Scarcity

@sjtousley

Hypothesis: Users aren’t importing their contacts because they don’t realize they CAN import their contacts. It’s a discoverability problem.

Objective: Increase signup-to-import conversion rate.

Results: This button resulted in a +4% increase in new users importing contacts, which in turn resulted in a +3% increase in new user retention.

Next steps: Determine how much more opportunity there is to increase import CVR. Are we nearing a local maximum of CVR for new user -> import success? Also nail down alternative paths for users to get their data into their CRM. How are the existing paths performing? Are there any additional paths that we can add?

Product A/B Test: Adding “Import” button increases retention by 3%

SUCCESS

@sjtousley

“Smoke Test”

Hint: Use this method to learn something new when there isn’t enough volume for statistically significant A/B test!

User Intent Research: People want videos to learn how to use CRM

@sjtousley

Landing Page A/B Test: Increased # of PQLs to generate

Hypothesis: Adding a Smart CTA to the HubSpot Sales Free homepage, targeting current CRM/Sales Free users, will generate new PQLs.

Objective: Increase number of PQLs

Results: In the past 30 days, this generated 63 new PQLs, closing 5 new business deals at average MRR value of $72, totalling $4,320 in ARR. Avg. PQL-to-close rate is ~8%

Lessons Learned: Adding smart content on product pages is an easy, low-effort way to generate new PQLs from current users.

Next steps: Use predictive model to calculate potential future impact scaling this across other product page.

@sjtousley

● https://blog.hubspot.com/sales/100k-email-templates-follow-up

● https://blog.bufferapp.com/social-media-experiments

● https://patreonhq.com/thats-not-a-hypothesis-25666b01d5b4?gi=c819bff3b75d

● https://darius.com/increase-funnel-conversion-with-psych-7378d51c4caf

● https://www.widerfunnel.com/ab-testing-for-long-term-success/

● https://apptimize.com/blog/2016/02/this-is-how-you-find-your-apps-aha-moment/

● https://www.appcues.com/blog/this-one-in-app-message-increased-sales-demos-by-50

● https://www.nickkolenda.com/copywriting-tips

● https://www.groovehq.com/blog/conversion-rate-optimization-wins-of-2015

● https://okdork.com/why-content-goes-viral-what-analyzing-100-millions-articles-taught-us/

More Examples on Growth Experiments

@sjtousley

The tools you need

● Google Spreadsheets to analyze North Star and “zoom out” metrics growth

● Airtable or Trello as project management tool (experiment backlog)

● Google Presentations to visualize your experiments

● Google Optimize to run A/B tests on your website

● Google Analytics to analyze results on your website

● Atlassian’s Wiki as central place to share “What did we learn this week?”

● Statistical significance calculator to double check results

● HubSpot (or similar tool) to run email A/B tests with marketing automation

● Hotjar for heatmaps, clickmaps, and visitor recordings

● Marketing tools like SEMRush, Buzzsumo, Hunter.io, Ahrefs, etc

Q&A