Date post: | 15-Jan-2015 |
Category: |
Data & Analytics |
Upload: | kinshuk-mishra |
View: | 771 times |
Download: | 2 times |
May 6th, 2014
Real-Time Personalization with Ad Tech @Spotify
Kinshuk Mishra [email protected] @_kinshukmishra
1
About Me• @Spotify since 2011 • Data and Backend engineering • With the Monetization team
May 12, 2014
Spotify in numbersStarted in 2006, available in 56 markets 20+ million songs, 20,000 added daily 24+ million active users, 6+ million subscribers 1.5 billion playlists
May 12, 2014
Monetization at Spotify• $1 billion paid to rights holders since launch
• 1 in every 4 Spotify users is a paying subscriber
• Ad revenue pays for free tier music streaming
• Build platform to power artist and music label promotions
What is our mission?
• Power the free tier with ads customers love. – Delight users, brands and artists with standout
promotional experiences. – Deliver the right message for every moment.
5
Delight users, brands and artists with standout promotional experience
6
• Ad formats need to be
- Native
- Actionable
- Scalable
- Engaging
7
Spotlight
8
Album promotion
9
Sponsored Genre
10
Sponsored Genre
11
!(Right message for every moment)
Things we had to do
• Make ads context aware.
• Create measurable ad formats.
• Build tools to help brands understand their audience.
• Build flexible targeting infrastructure.
• Analyze user behavior to improve ad relevancy.
12
Why is real time interesting?
• True assessment of the context - current view, last track, etc.
• Immediate feedback - ad clicked, followed a playlist, etc.
• Increased ad opportunities - hyperlocal, current mood, etc.
13
source : http://nativemobile.com/tag/real-time-bidding
How to serve right message for every moment ?
• Fix the WTFs for ads in the music context - Realtime short window activity analysis
• Improve ad relevancy based on user behavior - (Realtime + Batch) long window activity analysis
14
Some quick wins
• Fix the WTFs for ads in the music context - Fix the “Not Safe For Kids” ads problem. - Fix the relevancy of ads in your music session.
• Improve ad relevancy based on user behavior - Fix ad relevancy based on user’s musical taste.
15
Targeting Architecture16
Desired overall design 17
Our requirements
• Scalable log collection • Capability to process logs in batch and realtime mode and aggregate user activity
• Capability to store user profiles and enable serve time lookups
• Ability to update and add new features to existing user profiles
Bigdata @Spotify and choices
• 700 node hadoop cluster • 400 GB service logs daily • 4.5 TB user data daily • 7500 hadoop jobs daily • 64 TB data generated daily • YARN Map-Reduce, Giraph, Storm, Spark, etc.
!
19
Overall design and tech choices20
Source : http://tfosuccess.com/day-106-107-the-storm-arrived/
What is Storm?
• Real time stream processing • Like Hadoop without HDFS • Like Map/Reduce with many reducer steps • Fault tolerant and guaranteed message processing
21
Storm @Spotify
• storm-0.8.0 • 22 node cluster • 15+ topologies • 200k tuples/second • ads, recommendation, analytics, monitoring, etc.
22
source: http://storm.incubator.apache.org/
Use-case
Use real-time session genre information to control
ad serving in real-time
23
Realtime activity analysis • Kafka -> Storm -> Memcached -> Targeting Backend
• Soft realtime
• Easy to scale
24
Getting data25
Getting data across the globe26
Genre Topology27
May 12, 2014
Measuring effectiveness• Build ad quality scores from lower-level metrics
• Positive response, avoidance, etc.
• A/B test hypotheses, evaluate against quality scores
• Build quality score optimization into real-time system
May 6th, 2014
Want to join the band?https://www.spotify.com/us/jobs/ or https://twitter.com/Spotifyjobs !
Kinshuk Mishra [email protected] @_kinshukmishra