35
| Date post: | 19-Feb-2017 |
| Category: |
Technology |
| Upload: | deitcher |
| View: | 184 times |
| Download: | 4 times |
WhoAmI?• Lifeintechbusiness:– 10yrslarge-scalemission-criKcalIT– 10+yrsconsulKng&training– Somestartupsontheway
• Avid(ifnotverygood)icehockeyplayer• Long-Kmeloverofgreatengineering….whenusedtomakearealdifference
• AtomicInc:– GeneralistpracKKoner
• Networkjustoneelement– Product:engineering:operaKons
(not24601)
ALiVleHistory
Summer2015
• FintechX:“Containerizeus!”– Hint:Itisharderthanyou
think…andworthit– Culture/process>technology
• QuesKon:Networking?• Answer:ScienKficmethod
ALiVleHistory
Summer2015
• FintechX:“Containerizeus!”– Hint:Itisharderthanyou
think…andworthit– Culture/process>technology
• QuesKon:Networking?• Answer:ScienKficmethod
Fall2016• GoodpracKcedemands:
1. RedotestswithnewopKonsandversions
2. Maketestsavailable3. Explainitallwell
WhatIs“Ultra-Low”Latency?
1. hVp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt
“every100msofdelaycosts1%ofsales”[1]
WhatIs“Ultra-Low”Latency?
“extra0.5sinsearchpagegeneraKonKmedroppedtrafficby20%”[2]
1. hVp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt2. hVp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
“every100msofdelaycosts1%ofsales”[1]
WhatIs“Ultra-Low”Latency?
“extra0.5sinsearchpagegeneraKonKmedroppedtrafficby20%”[2]
1. hVp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt2. hVp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
Not.Even.Close.
“every100msofdelaycosts1%ofsales”[1]
NetworkingOpKonsDirectMetalmacvlanBridge/vSwitch(noNAT)net=hostSR-IOV
OverlayFlannelWeaveDockerOverlayCalico(IPIP)
WorkloadAwarenessDockerbridge(NAT)
FabricAwarenessCalico(NaKve)
OurTests
WhatWeTested• netperf⇒netserver• UDP&TCPround-robin• Sizes:300,500,1024,2048• NoorchestraKon=complete
control• 50000iteraKons
– Lawoflargenumbers• Latency(Avg,%iles),CPU• DifferenQals,notabsolutes
HowWeTested• .net
– Becauseithadtobemetal– Wickedsmartteam
• Completetestrun– Networkchanges– HardwarevariaKons,errors
hVps://github.com/deitch/[email protected]
LocalNetworkingSummary• SR-IOVhorriblelatencybutgreatCPU– Holdthatthought…
• net=hostonparwithmetal• macvlanclosestvirtualizedtometal• Restinsamerange:– Latency:5-10𝓊-secoverhead– CPU:negligibledifference
• Calico(IPIP&naKve)&Dockeroverlayslightlymoreperformant(marginoferror?)
• WatchoutforverylargeTCPpackets
RemoteNetworkingSummary
• Weave(sleeve)addslatencyandCPU– Reasonfor“fastdatapath”
• Again,macvlanbestvirtualized• Alltherest:– Latency:within50𝓊-secofeachother,exceptSR-IOVwithverylargeTCPpackets
– CPU:similar,butkeepaneyeonFlannel(UDP)
SR-IOVSR-IOVdoesnotautomaQcallymeanbeWer
• Switchinnetworkcard• TradeshostCPUforcardprocessor• Qualityvariesdrama3cally– EvenMellanoxfarworselocally
• My2€:SR-IOVfallsfurtherbehinddueto:– SpeedofiteraKon– Open-source– Sosware+CPU
Whatelsecouldwedo?Ø OtherhardwaretypesØ OthernetworkfabricsØ Othernetworkoverlayversions(wehavethedata…)Ø DockermacvlannetworkdriverØ ipvlanØ IPv6Ø KernelandnetworkstacktuningØ DistantnetworksØ OthertrafficpaVerns(mulKcastvsunicast)Ø Otherhost-to-hostencrypKonØ OtherkernelversionsØ OtherOSes(Illumos-based?)Ø Awholelotmore…
Headaches(andThanks)• Headaches
– WeaveSYN-(nothing)– etcdis“touchy”– PacketL3networkispowerfulbut…unique
• Macvlan,weave,flannel:allrequiredpingsformac• Sexngupbridgew/oNAT,Calico,macvlanwas“different”
– SR-IOViscomplicatedandflaky,especiallyMellanox– netperfwithUDPpacketscangetstuck(Calico-ipip)– Andawholelotmore(askmeoffline)
• Andthanks:
– BryanBoreham,AdamHarrisonatweave.works– ZacSmith,Adam,Aaron,Andy,Lucas,everyoneatPacket
Conclusions• SR-IOV:mostoftheKme,justnotworthit• Performance:– Metal(+net=host):alwaysperformsbest– Directnetwork++:macvlanisyourfriend– Others:Roughlysimilar,carefulofWeave(sleeve)
• What’syourusecase?– ULL:Metal/net=host>macvlan>calico>overlay– Everythingelse:Focusonyourarchitectureandskills
Pickintelligently:easier,notsimple
Conclusions• SR-IOV:mostoftheKme,justnotworthit• Performance:– Metal(+net=host):alwaysperformsbest– Directnetwork++:macvlanisyourfriend– Others:Roughlysimilar,carefulofWeave(sleeve)
• What’syourusecase?– ULL:Metal/net=host>macvlan>calico>overlay– Everythingelse:Focusonyourarchitectureandskills
Pickintelligently:easier,notsimple
QuesKonsandhelp:@[email protected]
