PERSONALIZATION10 LESSONS LEARNED from NETFLIX SUCCESS

(and how to apply them to your video services)

Personalization10 Lessons Learned from Netflix’s Success (and how to apply them to video services)

Greetings!

Kauser Kanji Pancrazio AuteriManaging Editor, VOD Professional CTO, Contentwise

21

Today’s objectives

Share the findings of observing how Netflix uses personalization as a competitive advantage in engaging and retaining users and planning content acquisition

Show how you can connect the dots and take advantage of those findings for your online video services

So what is personalization?

ph. Thinkstock

Not like this Personalized like this

It’s about the pleasure of choice and the abundance of options

It’s understanding needs and narrowing the options to make choosing a pleasant experience

Personalization can go wrong

It’s not just high-tech

Brian & Doreen remember customers’ taste, curate and organize shelves, plan promotions, update the display windows,

listen to requests and recommend products and let customers browse their shop in Somerset, UK

Netflix

Hulu

Amazon

Home page (web)

Pure S-VOD

Free, ad-based VOD Upsell S-VOD Hulu+

S-VOD Prime T-VOD

So what’s the problem?

Broad range of user types and tastes

Fragmentation of content sources and applications: too many places to look at

Content availability can be very dynamic over time

Most UI solutions rely on drill-down and user efforts

User’s attention span and screen real estate are very limited

You name it!

See the opportunity?

Users feel more engaged

Consumption increases

Habits formation occurs

Your service becomes a destination for unified discovery

People talk about your brand with passion

Subscribers perceive the value and the abundance of what you offer

Things can happen when people have a truly

personalized experience

Let’s start!

10 lessons learned from Netflix

The information, interpretations, advice and recommendations contained in this presentation are not endorsed in any way by Netflix and are based on information publicly shared by Netflix or its employees.

10 lessons learned from Netflix

1. Set objectives and pick metrics

10 lessons learned from Netflix

Netflix: maximize member satisfaction and month-to-month subscription retention

Example Metrics

1. Set objectives and pick metrics

10 lessons learned from Netflix

Example Metrics

Canceled subscriptions per month Interactive sessions resulting in a playback Played minutes per user per month Fully watched playbacks Interaction time before starting a playback Returning users

1. Set objectives and pick metrics

10 lessons learned from Netflix

2. Consider UX as mission-critical

10 lessons learned from Netflix

2. Consider UX as mission-critical

secret sauce best practices

innovative functionalities well-tuned business rules

UX Engine Users

Delivery infrastructureAudiovisual quality

Customer support

Content offer

Editorial curation

10 lessons learned from Netflix

2. Consider UX as mission-critical

UX Engine UsersCode Rules

Widgets Behaviors

Configurations Management tools

=

10 lessons learned from Netflix

2. Consider UX as mission-critical

Changes in UI behavior can have a dramatic impact on key metrics. Multiscreen? Make sure behavior is consistent across devices

Pay special attention tocross-screen consistency of

Welcome screens Frequent user actions

User “lost” actions

Leverage the UX Engine to control UI behavior

across all screens

10 lessons learned from Netflix

3. Personalize UX as much as possible

10 lessons learned from Netflix

3. Personalize UX as much as possible

At Netflix, more than 75% of views come from some sort of

recommendation or personalized ranking

Views %

0 25 50 75 100

from personalized ranking

source: Netflix

10 lessons learned from Netflix

3. Personalize UX as much as possible

User’s attention span is very limited

The first 8-12 seconds are criticalWeinreich et al. - ACM 2008

Desired outcomes - find something to watch - engage in some sort of exploration

10 lessons learned from Netflix

3. Personalize UX as much as possibleScreen real estate is very limited too

Ideally user should find relevant content

in the first screen

10 lessons learned from Netflix

3. Personalize UX as much as possible

A B C D E F

Alphabetic By release year

By ingestion order Canned categories

Personalized order

Even “computed” lists such as Most popular Most viewed

Recently added and My list!

Ineffective sort criteria Effective criteria

10 lessons learned from Netflix

3. Personalize UX as much as possible

Netflix personalizes in both directions

There are two “folds”

1. ranking of items in a carousel 2. ranking of carousels in the layout

2

Real estate “above-the-fold”

1

10 lessons learned from Netflix

3. Personalize UX as much as possibleFeatured content

Resume play + My list

{Most likely actions

Popular Top picks for you Recently added

Main genres Pseudo-genres

Because you watched… Friends watching…

Watch again

Displayed in a personalized order"

"Some of them disappear for

a while if never “touched” "or because of A/B Testing

10 lessons learned from Netflix

3. Personalize UX as much as possible

Personalized reordering may be disorienting for some people

Indeed some Netflix users complain about this;

but it seems to be effectiveand we’ll show you

how to handle it

Aha!

My Aha! moment with NETFLIX

10 lessons learned from Netflix

4. Understand user’s lifestyle and context

10 lessons learned from Netflix

4. Understand user’s lifestyle and context

Netflix mines usage data to extract behavior patterns

Personalization may be affected by context elements

Device type Time of the day Day of the week Season of the year User is at-home or out-of-home Geo-location (traveling, commuting, weekend-home…) Local weather Popular news Other users in close proximity (phones/wearables)

10 lessons learned from Netflix

5. Use interaction data then ask for feedback

Priority on high-value usage eventsPlayback start/stop/resume View asset details Add to personal list

Other interactionsTrick-play control Search Sharing Navigation paths…

Ask for feedback

Like Dislike Love it!

5-stars

10 lessons learned from Netflix

6. Let users know how the service is adapting to their tastes

10 lessons learned from Netflix

Promote trust in the system

Encourage users to give feedback

Better personalization

6. Let users know how the service is adapting to their tastes

10 lessons learned from Netflix

Use meaningful labels referring to past behavior

user can recognize

Because you watched Breaking Bad

Because of your interest for Time Travel

Because you loved Kill Bill Vol.1

6. Let users know how the service is adapting to their tastes

10 lessons learned from Netflix

7. Ensure metadata captures content nuances and is consistent

10 lessons learned from Netflix

7. Ensure metadata captures content nuances and is consistent

Actors, Directors, Writers

TopicsCharacters Themes Moods

Genres

Keywords - MicrotagsLocations Time Periods

Synopsis Release Year

Country Studio

Duration

Language

“NETFLIX QUANTUM THEORY”

A set of best practices for manual micro-tagging of video content

Social acceptability of the lead character

10 lessons learned from Netflix

7. Ensure metadata captures content nuances and is consistent

Let users search for content you don’t have

10 lessons learned from Netflix

7. Ensure metadata captures content nuances and is consistentWith richer content metadata

you can use analytics tounderstand content performance

and drive content acquisition (or even original production)

And add meaning to user profiles

Movie Iron Man 3

Data from Gracenote-TMS

TV Series Breaking Bad

Data from Gracenote-TMS

10 lessons learned from Netflix

8. Give reasons to come back often

Refresh catalog frequently - OR - Let the UX Engine do it for you (virtually)

10 lessons learned from Netflix

8. Give reasons to come back oftenRe-shuffle top items to periodically

change the ones above-the-foldItems outside the first screen are still highly relevant for the user

User perceives novelty and will be keen to return more often

10 lessons learned from Netflix

9. Run frequent UI experiments

Identify the UI elements on the path to the key goalsRoll-out the variations and look at 2-5 metrics

Run the experiments for two weeks or until statistical validityDesign and plan experiments not to interfere with each other

Experiments consume interaction events: make sure there is enough activity to feed all of the active variations

There is no “perfect way” and there are many types of users: experiments and adaptation seem to be the most effective ways

10 lessons learned from Netflix

10. Close the loop, base decisions upon data

Netflix was the only network that said “We believe in you. We’ve run our data, and it tells us that our audience would watch this series. We don’t need you to do a pilot”

Kevin Spacey, actor and producer

10. Close the loop, base decisions upon data

Listen to Kevin saying this (video)

10 lessons learned from Netflix

10. Close the loop, base decisions upon data

Netflix proved to be agile and effective in rolling out variations and track several metrics across hundreds of client platforms

Netflix team is very disciplined on reporting UI events. This enables full visibility in analytics and higher ROI

Yes. At Netflix they go nuts for analytics! And they look to be right

Netflix uses analytics to heavily influence the content acquisition policy

10 Lessons from Netflix - Recap1. Set objectives, pick metrics and share them with the team 2. Consider UX as mission-critical 3. Personalize UX as much as possible 4. Understand user’s lifestyle and context 5. Use interaction data then ask for feedback 6. Let users know your service is adapting to their tastes 7. Ensure metadata captures content nuances and is consistent 8. Give reasons to come back often 9. Run frequent UI experiments 10. Close the loop and base your decisions upon data

We need a way to turn these lessons into practicetouching all the stakeholders in our projects

Netflix solutions are applicable (and applied) at… Netflix

Other services may include S-VODas well as Linear TV, DVR,

Transactional VOD, Pay TV, Pay-per view, music videos, sports highlights,

Advertising or User-generated Content…

"

UIDO

A set of checklists to guide you while introducing

personalization in your video service

UX IX

DX OX

User Experience Integrator Experience

Developer Experience Operator Experience

What you deliver How you start

How you build it Tools to manage"

UIDO

UX User Experience What you deliver

UX User Experience What you deliver

Content types ✓ Movies ✓ Series ✓ Episodes ✓ Extras ✓ Music videos ✓ Playlists ✓ News ✓ Sports events ✓ Sports highlights ✓ Scheduled programs ✓ Channels ✓ _____________________

Aggregates ✓ Collections ✓ Seasons ✓ Channel bundles ✓ Movie bundles ✓ Sports Team bundles ✓ Sports League bundles ✓ __________________

UX User Experience What you deliver

Key UX features ✓ Manually curated collections ✓ Search results ✓ Search suggestions while you type (single/multi-type) ✓ Search refine with smart filters (facets) ✓ Similar content ✓ Personalized picks for user ✓ Critics-based feed (Rotten Tomatoes, Metacritic…) ✓ Series you watch (with next-episode) ✓ VOD bookmarking (resume playback) ✓ User’s list ✓ Predictive browsing (surfacing folders) ✓ Personalized pseudo-genres

Reference UI "

Showing most of the personalization use cases supported by ContentWise

UX User Experience What you deliver

Key UX features (cont’d)

✓ Social graph (e.g. friends, followers) ✓ Sharing actions ✓ Content can be embedded ✓ Co-watching (blended profiles) ✓ Profile explanation with content metadata ✓ User can rate content (stars, like, dislike, love, etc.)

UX User Experience What you deliver

For kids ✓ Parental ratings ✓ Kids mode ✓ Specialized metadata (e.g. Commonsense) ✓ Editorial curation ✓ Curation by parents ✓ Analytics for parents

UX User Experience What you deliver

Content sources ✓ Linear schedule (line-ups) ✓ Start-over TV system ✓ VOD Catalog ✓ Local DVR ✓ Network DVR ✓ Reverse EPG (catch-up) ✓ ______________

UX User Experience What you deliver

Device types ✓ Phone ✓ Tablet ✓ PC ✓ TV ✓ Watch

Access models ✓ S-VOD ✓ T-VOD ✓ Ad-VOD ✓ Free-Linear ✓ Pay-Linear ✓ PPV

Profile types ✓ Personal ✓ Household ✓ Main account powers ✓ Blended ✓ Personas templates ✓ Personal on device

Access locations ✓ At-home, OOH ✓ On-net, off-net

UX User Experience What you deliver

Entitlements ✓ S-VOD packages ✓ Rented movies ✓ Purchased movies ✓ Purchased seasons ✓ Purchased episodes ✓ Subscribed channels ✓ Subscribed bundles (e.g. Channel + S-VOD) ✓ ______________________

Explaining a recommendationBecause you liked

these other movies

Affinity between the user’s taste and the

recommended movie (using the tag structure)

ContentWise Reference UI

OX Operator Experience How to manage

OX Operator Experience How to manage

✓ Managing UI Elements with UX Engine ✓ Creating and updating editorial lists ✓ Generating and curating pseudo-genres ✓ Accessing analytics ✓ Content planning using analytics ✓ Managing variations and experiments for A/B Testing ✓ Understanding the impact of business rules on key metrics

Personalized pseudo-genres

INTENSE ACTION MOVIES

mood genre type

2000s AUSTRALIAN THRILLER MOVIES

prodcountry

genre typereleaseyear

AMERICAN DRAMA MOVIES STARRING TOM HANKS

Cont

entW

ise

Refe

renc

e U

I

The magic of richer metadata

MOVIES FROM FEMALE DIRECTORS

person roletype

MOVIES STARRING A ROCKSTAR

type

gender from semantic

enrichment

looking into actors

person role from semantic

enrichmentCo

nten

tWis

e Re

fere

nce

UI

Curation of Pseudo-genresMetadata fields considered for

labels

Status of the pseudo-genre

Type: Editorial

or Computed

ContentWise Management Console

Driving from the UX Engine

Rendered by UI code

Configured by UX Engine

ContentWise Management Console

UX Engine - Personalized Order of Carousels

Content planning - Choosing items to retire

and are “expiring”

Find movies with a small number of “estimated”

residual views

Automatically create a business rule

The rule can be used in A/B Testing to anticipate the impact of removing these movies from the catalog.

ContentWise Management Console

A/B/C Testing

Biz rule #1

Biz rule #N

Variation A

Biz rule #1

Biz rule #N

Variation B

Experiment

Group A

Group B

Control Group

Results Metrics

Normal behavior

Launching an experiment

This is an experiment

Treated user base

Variations

Monitoring experiment results

MetricsVariations Affected users

ContentWise Management Console

DX Developer Experience How to build

DX Developer Experience How to build

UI Element ✓ Carousel ✓ Group of carousel ✓ Item ✓ Item attribute ✓ __________

DX Developer Experience How to build

UI Eventsto be reported

✓ Item displayed ✓ Item selected ✓ Item details accessed ✓ User rating submitted ✓ Explanation displayed ✓ Tuned-in ✓ Tuned-out ✓ Playback started ✓ Playback resumed ✓ Playback paused ✓ Playback completed ✓ Playback stopped

✓ Item saved to list ✓ Item removed from list ✓ Preview playback started

IX Integrator Experience How to start

IX Integrator Experience How to start

✓ Content model map ✓ Event model map ✓ User ID map ✓ Data refresh policy ✓ Bulk ingestion automation ✓ Delta updates automation ✓ Client applications map ✓ UI elements to be managed from UX Engine

Thank you!

pancrazio.auteri@contentwise.tvkauser.kanji@vodprofessional.com

www.contentwise.tvwww.vodprofessional.com

For more information, please visit our website or contact us

Digital TV. Personalized