Magic Scaling SprinklesExperience operating a high-volume, low-latency ad buying platform at Tapad

@tobym@TapadEng

Who am I?Toby MatejovskyFirst engineer hired at Tapad 3+ years agoScala developer

@tobym

What are we talking about?

One of the key components that allows Tapad’s realtime ad-buying platform to hit 350,000 TPS.

Outline• What Tapad does• Why Aerospike is a good fit• Operational experience• What’s next

What Tapad Does(Real-time bidding)

The Tapad Difference.A Unified View.

Ad exchangeWant to show an ad to device 123?

Tapad

Sure, show this ad for $2 CPM

No thanks

Great, you won. Ad was displayed!

How about to device XYZ?

95% response time: ~30 ms

Why Aerospike?FastSafeScale outExpiration/eviction

Super fast key-value store350,000 reads per secondon 7 nodes

99% of reads are under 1 millisecond

SafeReplication factorXDR (cross datacenter replication)SSD-backed

Scale outLinear scalability, just add a node*

*will revisit this during the next section

Expiration and evictionOld data expires automaticallyOldest data is evicted if the database is running out of space

This is desired behavior in ad-tech world

Operational experience with Aerospike at TapadConfigurationMigrationsEvictionUsage

Tapad’s Aerospike Configuration100% keys in memory

100% data in SSD storage

Replication factor 2

512-byte block size

Need lots of free space in memory and storage for defrag (high-water mark)

Migrations and partitionsNew node requires data migration, means degraded performanceNetwork partition may trigger some data migration

EvictionAwesome feature, not intuitive if objects’ TTLs are not nicely distributed

UsageBlocking and non-blocking clients availableLZ4-compressed protobufHot key error

What’s next?Smaller minimum block sizeReplace Redis (UDFs)Multiple keys to reference the same record

Thank You@tobym@TapadEng

Toby Matejovsky, Director of Engineeringtoby@tapad.com@tobym