Product Optimization

Webinar 14-July-2016

VENKATRAMAN RMDirector of Products OLA CABS

Experience:Manager, Product Management | PayPalProduct Marketing Manager | SynaptrisSenior Product Manager | Info Edge India Ltd

WHAT WOULD YOU TAKE AWAY TODAY?

• PM Role Aspirants – Understanding Product Management processes

pertaining to important area in Product Life cycle – Product Optimization

• PMs – Framework to structure your backlog and roadmap – Comprehensive way to approach optimization

In-depth learning of concepts like creating backlog, A/B testing, Customer Driven Innovation etc

NOT IN TODAY’S SCOPE

TYPICAL PRODUCT ROADMAP POST MVP LAUNCH

MVP DESCOPED

ITEMSGROWTH OPTIMIZATIO

N

TECH STACK IMPROVEME

NTS

WHAT IS PRODUCT OPTIMIZATION?

• Incremental improvements to the product to– Deliver more of measurable value to your

customers– Improve Core (and/or Secondary) Product metrics

THESE ARE NOT CONSIDERED OPTIMIZATION Product Flow Revamps Big UI Refreshes Product extensions

TYPICAL PRODUCT LIFE CYCLE

Optimization< 50%

Optimization> 70%

Optimization< 30%

STEP 0 – DEFINE YOUR OPTIMIZATION THEME(S)

Agree on focus metric(s)

• Conversion Rate • Transaction cost %• Improve User experience

All themes to be quantified by a metric

• Abstract tracks such as ‘Improve User experience’ can be quantified by parameters such as ‘Net Promoter Score’ or ‘Feedback score’

Set Targets and challenge yourself and the TEAM

• E.g. Conversion rate improvement by 1% this quarter• Targets help keep focus ON

STEP 1: DATA, DATA, DATA

• What data you need? – Baseline?– Funnel? – Drill downs?

• Validate Data for integrity

• Gaps? – Prioritize instrumentation stories & bug fixes

to roadmap

Product Optimization Cycle

Analyze

Identify

Size

Hypothesize

Prioritize

Build

Test

Learn

* Data Integrity* Go to your Customer

Go back to Customer

* Pre-post* A/B Test

Validation

Effort-Confidence-Impact

Example Analysis – Conversion Funnel (View Item to Place Order)

100%

80%

70%

View Item

Add to Cart

Select/Add shipping address Payment options

58%

45%Pay

39% Order confirmed

Did not like the item?Enough info not available to make a choice?

Did user divert to different page to discover more things?Cant find cart after diversion?

Guest vs. Member?Issue with Enter address page?

Didn’t find payment method of choice?Not a serious buyer?

Integration issues with payment provider?Buyer Card not working?

Questions

• Which is the biggest opportunity in the funnel?

• What are possible hypothesis why user dropped after viewing an item?

• Which part of the funnel is easier to improve?

• Crashes & Errors?

• Can I create separate funnels for different types of users? For different product categories?

SIMPLE PRIORITIZATION FRAMEWORK

Feature Impact (1-9) Effort (1-9) 1-High; 9-Low effort

Confidence factor (1-9)

Net scoreI*E*C

F1 8 8 7 448F2 5 6 9 270F3 6 3 5 90F4 3 1 3 6

How about extending the above?- Adding weights to each of the factors

Why not I + E + C instead of I*E*C?

When can ordering be compromised? - Context variables and real issues

Measure

A/B Test

•Measure under similar circumstances•Measure in isolation from other projects•Phased rollout based on results•Preparation time required for every project/test

Pre/Post measurement

•Hard to control external factors•Cannot separate out impact of parallel projects•Almost impossible to measure small improvements

Impacted User

segment Drill down analysis

OPTIMIZATION AS BOTTOM UP INNOVATION

• Data can provide Deeper Insights which is otherwise unavailable

• Engineers and QA teams know more than what you think – Edge use cases, dead-end flows, bugs, vulnerabilities. TAP THEIR KNOWLEDGE

RECAP

• How to define your Optimization Theme – Metric focus

• Importance of having Data & Quality Data

• Optimization Cycle

• Prioritization Framework (Impact-Effort-Confidence Factor)

• Measurement techniques - A/B Testing, Pre/Post

Questions?

DescriptionLevel 1

DescriptionLevel 2

DescriptionLevel 3

DescriptionLevel 4

Goal