22
XXX Horizontally scaled data processing architecture Using Redis @
XXX
HorizontallyscaleddataprocessingarchitectureUsingRedis
@
About me
Ophir Hordan Member of Jfrog swamp Leading Bintray development
Bintray – Leading Distribution platform – Support distribution of Maven,
Debian, RPM, Docker, nuget…. – Use CDN for fast download – Runs in multiple site around the
world
Some Numbers ___________
Packages 300k
Downloads 900m/Month
Services
– Download binaries – Web application – REST API
– Backend micro services
The Layout
UI & REST API
MongoDB
Download Server
CouchDB
Index Calculation
Async Process
Event Processing
Event Processing
No single point of failure
True distribution
Quiet period
Data Sharding
and Twemproxy aka nutcracker Twemproxy
The flow Service
Twemproxy
Micro Service
Twemproxy
DB0InboundData
DB1Queuemanagementandquiteperiod
DB2WorkingDBandretriescounter
-ZsetbyEmestemp-Hashfordatapayload
HashkeysValuewithdatapayload
-Hashfordatapayload-HashforretrycounEngandrecoverydata
Redis structure LUAScanallkeysMovetodb1
zrangeByScore(currentEmestampminus5seconds)
Service Pod Microservice Redis
Outbound Twemproxy
Redis
Pod
Pod
Inbound Twemproxy
Microservice
Things we learned on the way… • Multiple values in single key is problematic to
monitor • aof file disk flush settings has big impact on
Redis performance • No official package of Redis and Twemproxy • Twemproxy only support single Redis db • Redis can easily hold Millions of keys • Protocol buffer works well as Redis document • Twemproxy doesn't have fail over • Twemproxy has limited sharding support (was
good enough for us)
QUESTIONS?
