BufferbloatBufferbloat

Why is the Internet slow?Why is the Internet slow?

Stephen HemmingerStephen Hemmingerstephen@networkplumber.orgstephen@networkplumber.org

Limelight Networks May 2015

BufferbloatBufferbloat

● What is the problemWhat is the problem● What causes the problemWhat causes the problem● SolutionsSolutions

– DemonstrationDemonstration● Current statusCurrent status

– The GoodThe Good– The BadThe Bad– The UglyThe Ugly

Network PerformanceNetwork Performance

ThroughputMbits/sec

LatencyRound Trip Time

Fairness

How bad is it?How bad is it?

Why is worse now?Why is worse now?

● No longer in FTP universeNo longer in FTP universe● TCP Initial Window Size = 10TCP Initial Window Size = 10● Advanced TCP algorithmsAdvanced TCP algorithms● Network Offload packet trainsNetwork Offload packet trains● Bad benchmarksBad benchmarks● Router/switch memory is cheapRouter/switch memory is cheap● Everyone fears dropping packetsEveryone fears dropping packets

Head of Line blockingHead of Line blocking

Queuing Theory BasicsQueuing Theory Basics

averagetimeinqueueutilization

servicerate1 utilization

Graphic courtesy Sprint, Apricot 2004

From Fred Baker: Bufferbloat!

Netalyzr: DownstreamNetalyzr: Downstream

Netalyzr: UpstreamNetalyzr: Upstream

TCP throughput dynamicsTCP throughput dynamicsIn

crea

sing

Mea

sura

ble

Thr

ough

put

Increasing TCP Window

“knee” “cliff”

Bottleneck Capacity

QueueDepth

meanthroughput effectivewindowmeanroundtriptime

From Fred Baker: Bufferbloat!

Blame LinuxBlame Linux

● Windows XPWindows XP– Maximum window 64kMaximum window 64k

● Windows 7Windows 7– Bandwidth limit to 80 mbitsBandwidth limit to 80 mbits

● AndroidAndroid– Receive window limitedReceive window limited

Blame the customerBlame the customer

● Customers call Customers call supportsupport

● Applications are Applications are using more using more bandwidthbandwidth

● Block and chargeBlock and charge

Why Queueing?Why Queueing?

Priority Queue: TheoryPriority Queue: Theory

Priority Queue: RealityPriority Queue: Reality

Hierarchical Token BucketHierarchical Token Bucket

Random Early DetectRandom Early Detect

Explicit Congestion NotificationExplicit Congestion Notification

PACKETS

Ideal Active Queue ManagementIdeal Active Queue Management

● FairFair– All flows get some bandwidthAll flows get some bandwidth

● SimpleSimple– No tuningNo tuning

● Easy to deployEasy to deploy– No special hardware, no protocol changesNo special hardware, no protocol changes

● ReasonableReasonable– Won't create multi-second latencyWon't create multi-second latency

Stocastic Fair QueueStocastic Fair Queue

CodelCodel

PIEPIE

CakeCake

Priority

Fair Queue

CoDel

Back to Reality

Benchmark issuesBenchmark issues

● BadBad– Bytes/secBytes/sec– Packes/secPackes/sec– LatencyLatency

● GoodGood– Throughput + LatencyThroughput + Latency– Multiple connectionsMultiple connections– Real not simulatedReal not simulated

Better toolsBetter tools

● DSLreport'sDSLreport's– http://www.dslreports.com/speedtest/http://www.dslreports.com/speedtest/

● ICSI netalyzrICSI netalyzr– http://netalyzr.icsi.berkeley.edu/http://netalyzr.icsi.berkeley.edu/

● Real time Response Under LoadReal time Response Under Load– https://github.com/tohojo/netperf-wrapperhttps://github.com/tohojo/netperf-wrapper– Flent: The Flexible Network TesterFlent: The Flexible Network Tester

DSLReports speedtestDSLReports speedtest

5 sec delay!!

FIFO – the defaultFIFO – the default

Høiland-Jørgensen T., Battling Bufferbloat

Ping 1sec!

Upload

Download

Stochastic Fair QueueStochastic Fair Queue

Høiland-Jørgensen T., Battling Bufferbloat

Controlled Delay - codelControlled Delay - codel

Høiland-Jørgensen T., Battling Bufferbloat

Fair Queue Controlled DelayFair Queue Controlled Delay

Høiland-Jørgensen T., Battling Bufferbloat

Bufferbloat statusBufferbloat status

Bufferbloat: The GoodBufferbloat: The Good

● Industry awarenessIndustry awareness● Standards progressStandards progress● Active researchActive research● Open Source SolutionsOpen Source Solutions

Once you start looking,Once you start looking,Bufferbloat can be everywhereBufferbloat can be everywhere● EdgeEdge

– Home routerHome router● ProviderProvider● NetworkNetwork● AcceleratorsAccelerators

– CachesCaches– Load balancers,...Load balancers,...

● ServersServers

How is Bufferbloat Solved on Linux?How is Bufferbloat Solved on Linux?

● Queuing disciplinesQueuing disciplines– Codel, PIE, ...Codel, PIE, ...

● Linux internal Linux internal – BQL, FQ, PacingBQL, FQ, Pacing

● Enable good defaultsEnable good defaults– ecnecn– fq_codel fq_codel

Bufferbloat: the BadBufferbloat: the Bad

● Legacy equipmentLegacy equipment– OEM focus on cost not AQMOEM focus on cost not AQM

● Congestion on backboneCongestion on backbone– It's not a bug it's a feature?It's not a bug it's a feature?– Problem is political not technicalProblem is political not technical

Bufferbloat: the UglyBufferbloat: the Ugly

● WirelessWireless– AggregationAggregation– MulticastMulticast– Bad benchmarksBad benchmarks– Proprietary firmwareProprietary firmware

Questions?

Thank youStephen Hemminger@networkplumber

stephen@networkplumber.org

Bufferbloat resources

Bufferbloat.net: http://bufferbloat.netEmail Lists: http://lists.bufferbloat.net

CeroWrt: http://www.bufferbloat.net/projects/cerowrt Other talks: http://mirrors.bufferbloat.net/Talks

Jim Gettys Blog – http://gettys.wordpress.com A big thanks to the bloat mailing list, Jim, Kathie, Van, Dave, Eric, ISC, and all the other the CeroWrt/OpenWrt contributors