YilongGeng,Shiyu Liu,Zi Yin,Balaji Prabhakar,MendelRosenBlumStanfordUniversity
incollaborationwithAshishNaik,AminVahdatGoogleInc.
SynchronizingClocksinaDatacenterto10sofNanoseconds
Clocksynchronizationisuseful• Boostsperformanceofdatabasesbymaintainingcausalityandexternal
consistency• Enablesschedulingtasksandresourceswithprecisetiming• Getsridofthe“clockless” assumptioninbuildingdistributedsystems,
enablesbrand-newsystemsandapplications
Clocksynchronizationverychallenging• PTPandPPSrequirecompatiblehardwareandareexpensive• ManyPTPcompatibleswitchesstillperformpoorlyunderload
Background
Syncingclockswithprobes
Probe
Probe
𝑇𝑋# 𝑅𝑋%
𝑇𝑋%𝑅𝑋#
t
𝑡# = 𝑡 + Δ𝑡# 𝑡% = 𝑡 + Δ𝑡%
ProbefromAtoB:
• Receivetime=transmittime+delay
• 𝑅𝑋% − 𝛥𝑡% = 𝑇𝑋# − 𝛥𝑡# + 𝑃𝑟𝑜𝑝𝑜𝑔𝑎𝑡𝑖𝑜𝑛𝑎𝑛𝑑𝑞𝑢𝑒𝑢𝑒𝑖𝑛𝑔𝑑𝑒𝑙𝑎𝑦
• 𝛥𝑡% − 𝛥𝑡# = 𝑅𝑋% − 𝑇𝑋# − 𝑃𝑟𝑜𝑝𝑜𝑔𝑎𝑡𝑖𝑜𝑛𝑎𝑛𝑑𝑞𝑢𝑒𝑢𝑒𝑖𝑛𝑔𝑑𝑒𝑙𝑎𝑦
• 𝛥𝑡% − 𝛥𝑡# < 𝑅𝑋% − 𝑇𝑋#
ProbefromBtoA:
• 𝛥𝑡% − 𝛥𝑡# > 𝑇𝑋% − 𝑅𝑋#
Eachprobeisaboundontheclock
Clockboundsovertime𝛥𝑡
%−𝛥𝑡
#(𝜇𝑠)
𝑡#(𝑠𝑒𝑐)
SyncingclockswithSVMs
Codedprobes
Network
Secondpacketdelayedmore
Firstpacketdelayedmore
Likelynoqueueing delay
Secondpacket
Firstpacket
10us >>10us
<<10us
~10us
Codedprobes
Thenetworkeffect-- exposingtheerror
A
B
C
Ifmyclockisat10,B’sclockmustbeat10:15
Ifmyclockisat10:15,C’sclockmustbeat10:05
Ifmyclockisat10:05,A’sclockmustbeat9:50
Guys,weareoffby10minutes!
2?
2?
6?
-10?
5?
15?
3.3
3.3
3.3
NetFPGA verification
• SingleNetFPGA actsas4independentNICssharingthesameclock
• DifferentNetFPGAs syncedwithI/Opins
NetFPGA verification
SingleNetFPGA DifferentNetFPGAs
w/onetowrkeffect
w/networkeffect
w/o networkeffect
w/networkeffect
Meanofabs.error(ns) 40.0 11.0 38.6 13.6
99th percentile ofabs.error(ns) 94.3 22.7 89.6 29.4
10-minute experimentat40%networkloadwithK=10
Robusttohighnetworkload
0 10 20 30 40 50 60 70 80
Network load (%)
0
5
10
15
20
25
30
35
40
Err
or(n
s)
mean99th percentile
Synchronizationerrorstaysunder40nsat80%load
Probebasedclocksynchronization• Workswithsimpleswitches• OnlyneedswidelyusedtimestampingcapableNICs
3Keyideastoachievenanosecondprecision• Support vectormachines• Codedprobes• Networkeffect
Distributedimplementation• Lightweight:~5Mb/sbandwidth andverysmallCPUoverhead• Runs inrealtime
Summary
