Just-RightConsistency!
Valter Balegas,NunoPreguiça,MarcShapiro,AnnetteBieniusa,ChristopherS.Meiklejohn,CarlaFerreira&manyothers
1
Agenda
• Geo-replication;• Consistency/availabilitytrade-off;• Just-RightConsistency!• … andhowtouseit.
2
Geo-replication
3
A
CB
Consistency
High availability
Low latency
Geo-Replication
4
A
CB
H
D
EF
G
IConsistency →High latency
Consistency/availabilitytrade-off
• Weakconsistencyisdifficulttogetright:• Notwell-definedsemantics;• Difficulttoprogram;• Unpredictableerrorsinproductionduetouncoordinatedexecutions.
• Strongconsistencyperformsworse,butissafe:• Well-definedsemantics;• Application’slogicisprotectedfromconcurrencyerrors;• Scalableandlow-latencyinsomedeployments;• Cross-replicacoordinationaffectsperformanceinthewide-area.
5
Consistency/availabilitytrade-off
• Whatwereallycareaboutisapplicationcorrectness.
• Serializingoperationsisa(overlyconservative)wayofmaintainingcorrectness.
• Example:• Operationsonabankaccountcanbeappliedconcurrentlyaslongasmoneydoesnotdisappear,getduplicated,orusedwithoutpermission.
6
Just-RightConsistency!
• Ensureapplication correctness withminimalcoordination.
• Howtodoit?
7
BankingApplication:
• Allreplicasmustconvergetothesamestate.• Accountbalanceisequaltoinitbalance+∑ deposit()�
� – ∑withdrawals()�� ;
• Themoneycannottemporarilydisappearinatransfer.• Accountbalancemustbenon-negative.
8
Problem1:Statedivergence
9
• Concurrentupdatesleadtostatedivergence.
Clt1
DC1
DC2
Clt2
a:5
a:5
a:2
a:3
a:?
a:?
withdraw(3,a)
withdraw(2,a)
write(a)
write(a)
Replicateddatatypes
10
updt(-3,a)
updt(-2,a)
Clt1
DC1
DC2
Clt2
withdraw(3,a)
withdraw(2,a)
a:5
a:5
a:2
a:3
a:0
a:0
• Ensurestateconvergencewithoutlosingupdates.
Problem2:Atomicoperations
11
Clt1
DC1
DC2
Clt2
transfer(3,ac,as)
balance(ac) balance(as)2 1 2+1!=6balance(ac+s) 6
ok
updt(-3,ac)updt(+3,as)
ac:5,as:1
ac:5,as:1 ac:2,as:1 ac:2,as:4
ac:2,as:4
• ACIDtransactionsarenothighlyavailable.
Transactionalconsistency
12
Clt1
DC1
DC2
Clt2
transfer(3,ac,as)
balance(ac) balance(as)5 1 5+1==6balance(ac+s) 6
ok
updt(-3,ac)updt(+3,as)
ac:5,as:1
ac:5,as:1 ac:2,as:4
ac:2,as:4
• Ensureatomicitywithweakerisolation.• Snapshotreads;atomicupdates.
Problem3:Asynchronousreplication
13
• Doesnotpreservetheexecutionorderofoperationsattheorigin.
Clt1
DC1
DC2
Clt2
transfer(3,ac,as) ok notify(clt2,’updated_as’)
balance(as) 1
ac:5,as:1
ac:5,as:1
balance(as) 1
notify()
Causalconsistency
14
• Executionorderofoperationsatdifferentreplicasrespectthehappens-beforerelationship.
Clt1
DC1
DC2
Clt2
transfer(3,ac,as) ok notify(clt2,’updated_as’)
balance(as) 1
ac:5,as:1
ac:5,as:1
balance(as) 4
notify()
Problem4:Invariantviolation
15
• Concurrentexecutionsmightbreakthecorrectnessofapplications.à Usecoordination?
Clt1
DC1
DC2
Clt2
withdraw(3,a) ok
ok
a:5
a:5
withdraw(3,a)
a:2
a:2
updt(-3,a)
updt(-3,a) a:-1
a:-1
Minimizecoordination
• Usecoordinationonly whennotsafetoexecuteoperationsasynchronously.
16
Clt1
DC1
DC2
Clt2
withdraw(3,a) ok
fail
a:{5,4}
a:{5,1}
withdraw(3,a)
a:{2,1}
updt(-3,a)
req(a,3)grant(1,a)
a:{2,3}
a:{2,0}
Performance
17
0 20 40 60 80
100 120 140 160 180 200
0 20 40 60 80 100 120
Late
ncy
[ms]
Time [s]
Latencyofoperationsforasinglesite,ina3DCdeployment.
OtherapplicationInvariants
• Bidirectionalrelationshipà usetransactions;• Referentialintegrityà automaticrepair;• Overdraftà compensations;• Sequentialidentifiersà fallbacktocoordination.
18
Just-RightConsistency!
• Ensureapplication correctness withminimalcoordination.
• Howtodoit?• Convergentdata-types;• Transactions;• Causality;• Maintainapplication-levelinvariants.
• Toolstosupportapplicationdevelopment.
19
JRC!Tools
• Helpprogrammersverifyapplicationcorrectnessusingasoundapproach:
• Staticprogramanalysis.
• Detectconcurrencyconflicts.
• Modifyapplicationsandtestagain.
• Publicly-availabletoolsontheway.
20
JRCToolsvideo:http://tiny.cc/JRC-TOOLS-VIDEO
AntidoteDB
21
http://syncfree.github.io/antidote/
• Just-RightConsistencydatabase.• Geo-/Partial-replication.• SQL-likeinterfaceontheway:
• Well-knownandwidelyadopted;• Out-of-the-boxsupportformaintainingmanycommonclassesofinvariants.
AntidoteDB performanceinFMKe Benchmark
22
0
5
10
15
20
25
30
35
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
Late
ncy
(ms)
Throughput (tx/s)
1DC-Normal2DC-Nornal3DC-Normal
Latencyofoperationsformultiplesiteswith4nodespersite.
Questions?
23
• JRCToolsvideo:http://tiny.cc/JRC-TOOLS-VIDEO
• AntidoteDB:http://syncfree.github.io/antidote/
• FMKe Benchmark• http://tiny.cc/fmke
Backupslides
24
25
Conflictdetectionalgorithm
ValterBalegasetal.– NOVALINCS,DI,FCT,UniversidadeNOVAdeLisboa@SyncFreeMonth36 25
enroll(“Mario”,”World Cup”)enrolled(“Mario”, World Cup”) := true
remove_tournament(“WorldCup”)tournament(“World Cup”) := false
R1 R2InitialState
tournament(“World Cup”)
player(“Mario”)
R1State
tournament(“World Cup”)
player(“Mario”)
enrolled(“Mario”, WC”)
R2State
tournament(“World Cup”)
player(“Mario”)
MergedState
player(“Mario”)
enrolled(“Mario”, WC”)
INV=enrolled(p,t)⇒ player(p)∧ tournament(t)
T⇒ F
a1,b1
Referentialintegrityexample
a1a2a3
b1
b3
b2
26
a1,b1
Referentialintegrityexample
a1a2a3
b1
b3
DC1
DC2
a1,b2add
b2
27
a1,b1
Referentialintegrityexample
a1a2a3
b1
b2
b3
DC1
DC2
a1,b2add
rem
b2
28
Referentialintegrityexample
a1a2a3
b1
b3
DC1
DC2
a1,b2add
Danglingpointer
a1,b1a1,b2
29
rem b2
Preventiverepairexample
a1a2a3
b1
b3
DC1
DC2
a1,b2add
rem
a1,b1a1,b2
30
b2
b2
Preventiverepairexample
a1a2a3
b1
b3
DC1
DC2
a1,b2add
rem
a1,b1a1,b2
31
b2
b2
*,b2
Preventiverepairexample
a1a2a3
b1
b3
DC1
DC2
a1,b2add
a1,b1
32
*,b2rem b2
Scalability
33