Post on 28-Nov-2014
description
transcript
GDS: Genium Data Store Real Time, Low Latency, Reliable!
Iuliia Proskurnia!EMDC!KTH!2013!
2!
3!
4!
3900 companies! 39 countries! over 1500 corporate products!
USE CASE!
Write events!Retrieve ranges of records!
5!
Fault-Tolerant?!
Consistent?!
Fast?!
Scalable?!
6!
Approaches
! Consensus based!
! ...!! Total Order Multicast!
! Symmetric!
! Token Site !
Uniform Reliable Total Order
◦ Validity !
◦ Uniform Integrity !
◦ Uniform Agreement !
◦ Uniform Total Order !
!
7!
8!
Genium INET Message Bus �Uniform Reliable Total Order Multicast
! Similar to Amoeba protocol!
! However... Fault Tolerant!!
9!
GDS: Genium Data Store
! Uses Genium INET Message Bus abstraction!
! Clients, Sequencer, Data store!
! Rewinders and sequencer replication!
! Active replication!
Client!
Data store node!
Data store node!
10!
GDS high level abstraction
LEDS!
11!
LEDS
! Column based!
! BLOBS!
! Appends!
! Range Queries!
! Not Distributed!
! Not fault-tolerant!
12!
Properties
! Consistent!
! Failure Resilient!! Replication!
! Rewinders!
! Cite Replication!
Total Order!
13!
Possible Failure Scenarios C
lient
Fai
lure!
Sequ
ence
r Fai
lure!
���8VHU
6HT
'6
���
>Q@
>Q@
:RUN
>Q��@
&XUUHQW�0HVVDJH��>Q��@ 5HZLQG�0HVVDJH��>Q@
���8VHU
6HT
'6
��� 5HZLQG�0HVVDJH��>Q@
6HT6WDQG%\
>Q@
>Q@
>Q@
>Q��@
3ULPDU\�3�
���
3
>Q@
>Q@
>Q@
5HZLQG�0HVVDJH��>Q@
14!
Scalability
! Natural Load Balancing!
! Partitioning (manual)!
15!
Evaluation
! Inserts (throughput/latency)!
! Range Queries (throughput)!
! Range transmission failure!
16!
Set Up
17!
Writes �Throughput
18!
Writes Limits
19!
Writes �Latency
20!
Range Queries �Throughput
21!
Range Queries �Scalability
8 Concurrent Users!
22!
Range Queries �Link Failure
23!
Summary
! uniform reliable total order multicast !
! scales fine!
! low latency!
! consistent, fault-tolerant!
24!
Future Work
! Generality!
! Send compressed chunks!
! Automated partitioning!
! Long-running tests!
25!
Comments? Questions?
Thesis Writing Process!
26!
Single record read�without load
27!
Single record read�with load (10 000 inserts)
28!
Single record read�scalability
29!
Discussion