Date post: | 08-Sep-2014 |
Category: |
Technology |
Upload: | toby-matejovsky |
View: | 2,209 times |
Download: | 4 times |
Magic Scaling SprinklesExperience operating a high-volume, low-latency ad buying platform at Tapad
@tobym@TapadEng
Who am I?Toby MatejovskyFirst engineer hired at Tapad 3+ years agoScala developer
@tobym
What are we talking about?
One of the key components that allows Tapad’s realtime ad-buying platform to hit 350,000 TPS.
Outline• What Tapad does• Why Aerospike is a good fit• Operational experience• What’s next
What Tapad Does(Real-time bidding)
The Tapad Difference.A Unified View.
Ad exchangeWant to show an ad to device 123?
Tapad
Sure, show this ad for $2 CPM
No thanks
Great, you won. Ad was displayed!
How about to device XYZ?
95% response time: ~30 ms
Why Aerospike?FastSafeScale outExpiration/eviction
Super fast key-value store350,000 reads per secondon 7 nodes
99% of reads are under 1 millisecond
SafeReplication factorXDR (cross datacenter replication)SSD-backed
Scale outLinear scalability, just add a node*
*will revisit this during the next section
Expiration and evictionOld data expires automaticallyOldest data is evicted if the database is running out of space
This is desired behavior in ad-tech world
Operational experience with Aerospike at TapadConfigurationMigrationsEvictionUsage
Tapad’s Aerospike Configuration100% keys in memory
100% data in SSD storage
Replication factor 2
512-byte block size
Need lots of free space in memory and storage for defrag (high-water mark)
Migrations and partitionsNew node requires data migration, means degraded performanceNetwork partition may trigger some data migration
EvictionAwesome feature, not intuitive if objects’ TTLs are not nicely distributed
UsageBlocking and non-blocking clients availableLZ4-compressed protobufHot key error
What’s next?Smaller minimum block sizeReplace Redis (UDFs)Multiple keys to reference the same record