38
אכסניאAkh-san-ya\?aksnaja?\ n.(ancientAramaic,fromAncientGreekxénos)1:Hospitality,lodging;2:Host.
אכסניאAkh-san-ya\?aksnaja?\ n.(ancientAramaic,fromAncientGreekxénos)1:Hospitality,lodging;2:Host.
פותחים בכבוד אכסניא:AncientJewishcustomtobeginpublicspeakingbyhonouringorthankingthehosts.
WhoAmI?• Lifeintechbusiness:– 10yrsfinancialservicesIT– 10+yrsconsulWng&training– Somestartupsontheway
• Avid(ifnotverygood)icehockeyplayer• Long-Wmeloverofgreatengineering….whenusedtomakearealdifference
• AtomicInc:– ConsulWng– Training
(not24601)
ALi\leHistory
Summer2015
• FintechX:“Helpuscontainerize!”– Hint:Itisharderthanyou
think…andworthit– Culture/process>technology
• QuesWon:Networking?• Answer:ScienWficmethod
ALi\leHistory
Summer2015
• FintechX:“Helpuscontainerize!”– Hint:Itisharderthanyou
think…andworthit– Culture/process>technology
• QuesWon:Networking?• Answer:ScienWficmethod
Summer2016• GoodpracWcedemands:
1. RedotestswithnewopWonsandversions
2. Maketestsavailable3. Explainitallwell
WhatIs“Ultra-Low”Latency?
1. h\p://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt
“every100msofdelaycosts1%ofsales”[1]
WhatIs“Ultra-Low”Latency?
“extra0.5sinsearchpagegeneraWonWmedroppedtrafficby20%”[2]
1. h\p://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt2. h\p://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
“every100msofdelaycosts1%ofsales”[1]
WhatIs“Ultra-Low”Latency?
“extra0.5sinsearchpagegeneraWonWmedroppedtrafficby20%”[2]
1. h\p://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt2. h\p://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
Not.Even.Close.
“every100msofdelaycosts1%ofsales”[1]
NetworkingWorkloads
• NetworkedWorkloads:“thingsthatdoworkandmusttalk”
• Sameprinciplesforallworkloads:– VMs– Cloud– Serverless– Containers
NetworkingOpWonsDirectMetalmacvlanBridge/vSwitch(noNAT)net=hostSR-IOV
OverlayFlannelWeaveDockerOverlayCalico(IPIP)
WorkloadAwarenessDockerbridge(NAT)
FabricAwarenessCalico(NaWve)
OurTests
WhatWeTested• netperf⇒netserver• UDP&TCPround-robin• Sizes:300,500,1024,2048• NoorchestraWon=complete
control• 50000iteraWons
– Lawoflargenumbers• Latency(Avg,%iles),CPU• DifferenRals,notabsolutes
HowWeTested• .net
– Becauseithadtobemetal– Wickedsmartteam
• Completetestrun– Networkchanges– HardwarevariaWons,errors
h\ps://github.com/deitch/[email protected]
LocalNetworkingSummary• SR-IOVhorriblelatencybutgreatCPU– Holdthatthought…
• net=hostonparwithmetal• macvlanclosestvirtualizedtometal• Restinsamerange:– Latency:5-10𝓊-secoverhead– CPU:negligibledifference
• Calico(IPIP&naWve)&Dockeroverlayslightlymoreperformant
• WatchoutforverylargeTCPpackets
RemoteNetworkingSummary
• Weave(sleeve)addslatencyandCPU– Reasonfor“fastdatapath”
• Again,macvlanbestvirtualized• Alltherest:– Latency:within50𝓊-secofeachother,exceptSR-IOVwithverylargeTCPpackets
– CPU:similar,butkeepaneyeonFlannel(UDP)
SR-IOVSR-IOVdoesnotautomaRcallymeanbeXer
• Switchinnetworkcard• TradeshostCPUforcardprocessor• Qualityvariesdrama5cally– EvenMellanoxfarworselocally
• My2¥:SR-IOVfallsfurtherbehinddueto:– SpeedofiteraWon– Open-source– Sowware+CPU
Headaches(andThanks)• Headaches
– WeaveSYN-(nothing)– etcdis“touchy”– PacketL3networkispowerfulbut…unique
• Macvlan,weave,flannel:allrequiredpingsformac• Se{ngupbridgew/oNAT,Calico,macvlanwas“different”
– SR-IOViscomplicatedandflaky,especiallyMellanox– netperfwithUDPpacketscangetstuck(Calico-ipip)– Andawholelotmore(askmeoffline)
• Andthanks:
– BryanBoreham,AdamHarrisonatweave.works– ZacSmith,Adam,Aaron,Andy,Lucas,everyoneatPacket
Whatelsecouldwedo?Ø OtherhardwaretypesØ OthernetworkfabricsØ Dockermacvlannetworkdriver(experimental)Ø IpvlanØ OtherpacketsizesØ KernelandnetworkstacktuningØ Distant(andVPN)networksØ Othertrafficpa\ernsØ Otherhost-to-hostencrypWonØ Awholelotmore…
Conclusions• SR-IOV:mostoftheWme,justnotworthit• Performance:– Metal(+net=host):alwaysperformsbest– Directnetwork++:macvlanisyourfriend– Others:Roughlysimilar,carefulofWeave(sleeve)
• What’syourusecase?– ULL:Metal/net=host>macvlan>calico>overlay– Everythingelse:Focusonyourarchitectureandskills
Pickintelligently:easier,notsimple
Conclusions• SR-IOV:mostoftheWme,justnotworthit• Performance:– Metal(+net=host):alwaysperformsbest– Directnetwork++:macvlanisyourfriend– Others:Roughlysimilar,carefulofWeave(sleeve)
• What’syourusecase?– ULL:Metal/net=host>macvlan>calico>overlay– Everythingelse:Focusonyourarchitectureandskills
Pickintelligently:easier,notsimple
QuesWonsandhelp:@[email protected]
