Innovate Faster with the Modern Growth Stack

Brion Hickey

GrowthCisco

Principal Product ManagerAmplitude

Carter Foxgrover

Senior Product MarketerOptimizely

David Isquick

Brion Hickey

GrowthCisco

Principal Product ManagerAmplitude

Carter Foxgrover

Senior Product MarketerOptimizely

David Isquick

• We are recording today’s presentation• Slides will be available after the webinar is complete• Please submit questions via the text box

Housekeeping

Growth - Tech Stack

About Me

Over 120 companies worldwide Then Now

USA

USA EMEA

USA EMEAAPAC

Came home.Got a cat.

Named himMeatball.

Optimization > GrowthTest stuff > Knowing your end users

Transitional Period

Growth

Philosophy

Growth

Philosophy

Strategy

Growth

Philosophy

Strategy

Technologies

Growth

Philosophy

Strategy

Technologies

Tactics

Growth

Philosophy

Strategy

Technologies

Tactics

Growth Tools

Customer Journey

Marketing Analytics (auditing)

Customer Journey

Growth Tools

Product Analytics / BehaviorsMarketing Analytics (auditing)

Customer Journey

Growth Tools

Product Analytics / BehaviorsMarketing Analytics (auditing)

Experimentation framework

Customer Journey

Growth Tools

Product Analytics / BehaviorsMarketing Analytics (auditing)

Experimentation framework

User Testing

Customer Journey

Growth Tools

Marketing Analytics

Marketing Analytics

“It was an awful integration”

Marketing Analytics

“It was an awful integration”

“The reports can’t be right”

Marketing Analytics

“It was an awful integration”

“The reports can’t be right”

“The pixel isn’t firing”

Marketing Analytics

“It was an awful integration”

“The reports can’t be right”

“The pixel isn’t firing”

“I don’t trust (marketing analytics company here)”

Marketing Analytics

ObservePoint

Auditing Marketing Analytics

● “It was an awful integration”○ Specifically addressing the integrity of

the integration○ What tags are missing○ What pages need to be fixed

Auditing Marketing Analytics

● “The reports can’t be right”● “The Pixel isn’t firing”○ Complete inventory of tags firing

(working as expected) ○ List of requests sent for each page ○ Duplicate tags, requests

Auditing Marketing Analytics

● “I don’t trust (insert any marketing analyticscompany here)”○ Output of a initial score/ grade ○ Comprehensive list of elements to fix ○ Update score, gain trust

In-Product Behavior Analytics

Product Analytics / BehaviorsMarketing Analytics (auditing)

Customer Journey

Amplitude

Text HereAmplitude

Retention

NorthStar

Metrics

Text HereAmplitude

Retention

A ha Moment

NorthStar

Metrics

Text HereAmplitude

Retention

A ha Moment

Product Market

Fit

NorthStar

Metrics

Text HereAmplitude

Retention

A ha Moment

Product Market

Fit

NorthStar

Metrics

Text HereAmplitude

Retention

A ha Moment

Product Market

Fit

NorthStar

MetricsWhat toA/BTest

But wait...

Real End User Testing

Product Analytics / BehaviorsMarketing Analytics (auditing)

User Testing

Customer Journey

Usertest.io

User Testing

● It’s not expensive anymore / $15● Reporting drop off. Why?● Initiated user testing.● Our buttons didn’t work.○ Wait what? ○ Ad blockers

User Testing

● It’s not expensive anymore / $15● Reporting drop off. Why?● Initiated user testing.● Our buttons didn’t work.○ Wait what? ○ Ad blockers

Dude, you had 1 job :|

User Testing● Turn it into a working cadence ○ Weekly, monthly○ Stay consistent ○ Especially before and after redesigns/feature releases

● It take a lot of time, so what○ Create summary video ○ Invaluable feedback

Get Organized

● Start to document findings● Access to content isn’t hard● Easy for engineers to consume ○ Screenshots ○ Behaviors ○ Time stamp ○ How to reproduce○ Expected behaviors

Optimizely

Marketing

Customer Journey

Optimizely

Product Marketing

Customer Journey

Optimizely

Product Marketing

Experimentation framework

Customer Journey

Code Level (Full Stack)

Optimizely

Product Marketing

Experimentation framework

Customer Journey

Code Level (Full Stack)

Personalization Recommendations Mobile Native

Optimizely

I can change

I can change I’m listening

I did this

I did this

Product Analytics / BehaviorsMarketing Analytics (auditing)

Experimentation framework

User Testing

Customer Journey

Growth Tools

Stay In Touch Brion.tw Brionh@gmail.com

Build Better Products

Account Owner | email_id@amplitude.com | phone number

Product Analytics for The Modern Growth Stack

Corporate BackgroundAMPLITUDE

● Launched Jan 2014

● Backed by Benchmark, Battery & IVP

● Now power over 20k products for over 5k

companies across the globe

● Analyzed over 2.5T user actions

● #1 Retention Playbook in industry

● Leader in self-serve Product Analytics

The Fastest Moving Growth & Product Teams use AmplitudeOUR CUSTOMERS

Consumer Media B2B Ecommerce Finance Healthcare

Product Analyticsfor Growth

2

Science, not Art

Quarterly releases and NPS surveys

aren’t enough. Modern growth teams

run like optimization machines:

● Mine data for insights

● Rapidly test ideas for impact

● Improve experiences every day

ADAPT OR DIE

Growth Teams Need 3 Types of Product Insights to InnovateWHAT IS PRODUCT ANALYTICS FOR GROWTH?

How do CUSTOMERS

use our products?

Examples

Do some users binge watch shows? If so which ones? What shows are more likely to be binge watched?

Growth Teams Need 3 Types of Product Insights to InnovateWHAT IS PRODUCT ANALYTICS FOR GROWTH?

How do CUSTOMERS

use our products?

What FEATURES

do we prioritize right now?

Examples

Do some users binge watch shows? If so which ones? What shows are more likely to be binge watched?

Which features increased IBM Cloud Usage and revenue amongst our customers? How can we double down on them?

Growth Teams Need 3 Types of Product Insights to InnovateWHAT IS PRODUCT ANALYTICS FOR GROWTH?

How do CUSTOMERS

use our products?

What FEATURES

do we prioritize right now?

How are our METRICS

improving?

Examples

Do some users binge watch shows? If so which ones? What shows are more likely to be binge watched?

Did purchase frequency increase amongst shoppers who received coupon code notifications last week?

Which features increased IBM Cloud Usage and revenue amongst our customers? How can we double down on them?

Most Tools are not Designed to Correlate Product & ImpactWHAT IS PRODUCT ANALYTICS FOR GROWTH?

Product Engineering InfraMarketingBusiness Intelligence

Without Amplitude, growth teams have to rely solely on analysts/ data scientists to answer complex questions

Relying only on Analysts Creates Data BreadlinesWHAT IS PRODUCT ANALYTICS FOR GROWTH?

BI and Query Tools

Production Database

Growth

Marketing

Executives

Analysts

Sales

Finance

Without Amplitude, Growth Teams can’t explore user behavior on their own and spin cycles waiting for analytics facetime

4-8 weeks delay

Why Amplitude

3

Amplitude is your Growth Team’s Personal Data AssistantGROWTH/ PRODUCT DATA SCIENCE AS A SERVICE

Product Growth

Execs

Sales Mrkting

Analysts

Data Scientists

Engineers

Growth Teams can investigate and explore real time data for 90% of their needs while also sharing their insights with the entire organization

Same day answers

Amplitude has unrivalled Depth of Behavioral Analytics THREE LEVELS OF NEW AGE ANALYTICS

LEVEL 1: KPI Monitoring

How many people are using X feature?

LEVEL 2: Problem Diagnosis

What is the drop-off of our onboarding flow?

LEVEL 3: Behavioral Analysis

What user behaviors are causing feature adoption?

VP Growth @

Case Study: Intuit QuickBooks

4

Intuit QuickBooks has become their fastest growing unitMINI CASE STUDY: 3 BUSINESS UNITS AT INTUIT NOW USE AMPLITUDE

Intuit team initially had poor KPIs for their invoicing productMINI CASE STUDY: HOW QUICKBOOKS TEAM LAUNCHED “CONNECT”

How do CUSTOMERS

use our products?

Customers were creating invoices in Quickbooks but not emailing them. Fewer invoices were getting paid

Behavior Problem

Their product team quickly diagnosed the behavior issue MINI CASE STUDY: HOW QUICKBOOKS TEAM LAUNCHED “CONNECT”

How do CUSTOMERS

use our products?

What FEATURES

do we prioritize right now?

Amplitude helped them discover that this was because customers who preferred invoicing through Gmail were not able to get value from Quickbooks. Gmail has >1B DAUs so they prioritized a Quickbooks Gmail plugin

Customers were creating invoices in Quickbooks but not emailing them. Few invoices were getting paid on time.

FeatureInsight

Behavior Problem

They launched their new feature “Connect” w/ great impactMINI CASE STUDY: HOW QUICKBOOKS TEAM LAUNCHED “CONNECT”

How do CUSTOMERS

use our products?

What FEATURES

do we prioritize right now?

How are our METRICS

improving?

Quickbooks Gmail has dramatically increased # invoices paid on time and decreased # clicks needed to get paid opening up a new market opportunity for Intuit

Customers were creating invoices in Quickbooks but not emailing them. Fewer invoices were getting paid

FeatureInsight

Resolution and Impact

Behavior Problem

Amplitude helped them discover that this was because customers who preferred invoicing through Gmail were not able to get value from Quickbooks. Gmail has >1B DAUs so they prioritized a Quickbooks Gmail plugin

Intuit has built a data driven culture for GrowthMINI CASE STUDY: 3 BUSINESS UNITS AT INTUIT NOW USE AMPLITUDE

Connect: Intuit’s Fastest Growing Product

Optimizely

Virtuous CycleAnalytics + Experimentation

Senior Product MarketerOptimizely

David Isquick

Everyone plays a part in growth

Driving Awareness of Those Products

Building Great Products and Experiences

Understanding Product and Marketing Performance

Improving Product and Marketing Performance

Product-driven Marketing-Driven Analytics and Research Driven

Experimentation-Driven

Growth

Enable Rapid Experimentation by Everyone

Built for business teams

See results in real-time Act with confidenceand product

development teams

Always-onexperimentation

Analyze

HypothesizeExperiment

Virtuous CycleAnalytics meets Experimentation

Analyze

HypothesizeExperiment

Virtuous CycleAnalytics meets Experimentation

• Core Metrics• Funnels• Retention• LTV

Analyze

HypothesizeExperiment

Virtuous CycleAnalytics meets Experimentation

• Core Metrics• Funnels• Retention• LTV

• Product improvements

• Marketing improvements

Analyze

HypothesizeExperiment

Virtuous CycleAnalytics meets Experimentation • Website A/B tests

• Product experiments

• Core Metrics• Funnels• Retention• LTV

• Product improvements

• Marketing improvements

Analyze

HypothesizeExperiment

Virtuous CycleIn action

Analyze booking conversion funnel

Hypothesis: Adding More Context Will Help Drive More Bookings

Analyze

HypothesizeExperiment

Virtuous CycleIn action

Analyze booking conversion funnel

Adding a step to funnel to provide more context will increase conversion

Analyze

HypothesizeExperiment

Virtuous CycleIn action

Analyze booking conversion funnel

Split traffic and analyze initial results in terms of conversion

Adding a step to funnel to provide more context will increase conversion

Increase revenue with counterintuitive changes

Adding a step to the booking funnel produced a 5% increase in revenue

Questions?