Bufferbloat is all wet?Bufferbloat is all wet?

Stephen HemmingerStephen Hemmingerstephen@networkstephen@networkplumber.orgplumber.org

@networkplumber@networkplumber

Linuxcon Europe Oct 2014

BufferbloatBufferbloat

● What is the problemWhat is the problem● What causes the problemWhat causes the problem● SolutionsSolutions

– DemonstrationDemonstration● Current statusCurrent status

– The GoodThe Good– The BadThe Bad– The UglyThe Ugly

Signs of BufferbloatSigns of Bufferbloat

● You might have bufferbloat ifYou might have bufferbloat if– ThroughputThroughput

● Download 10% of advertisedDownload 10% of advertised● Upload doesn't workUpload doesn't work

– LatencyLatency● large (> 200ms)large (> 200ms)● VariableVariable

– And no packet loss!!!And no packet loss!!!

How bad is it?How bad is it?

Queuing Theory BasicsQueuing Theory Basics

averagetimeinqueueutilization

servicerate1 utilization

Graphic courtesy Sprint, Apricot 2004

From Fred Baker: Bufferbloat!

Network PerformanceNetwork Performance

ThroughputMbits/sec

LatencyRound Trip Time

Fairness

Netalyzr: DownstreamNetalyzr: Downstream

Netalyzr: UpstreamNetalyzr: Upstream

Classical TCPClassical TCP

TCP throughput dynamicsTCP throughput dynamicsIn

crea

sing

Mea

sura

ble

Thr

ough

put

Increasing TCP Window

“knee” “cliff”

Bottleneck Capacity

QueueDepth

meanthroughput effectivewindowmeanroundtriptime

From Fred Baker: Bufferbloat!

Blame LinuxBlame Linux

● Windows XPWindows XP– Maximum window 64kMaximum window 64k

● Windows 7Windows 7– Bandwidth limit to 80 mbitsBandwidth limit to 80 mbits

● AndroidAndroid– Receive window limitedReceive window limited

Blame the customerBlame the customer

● Customers call Customers call supportsupport

● Applications are Applications are using more using more bandwidthbandwidth

● Block and chargeBlock and charge

Back to Reality

Benchmark issuesBenchmark issues

● BadBad– Bytes/secBytes/sec– Packes/secPackes/sec– LatencyLatency

Realtime Response Under LoadRealtime Response Under Load

● Use existing toolUse existing tool– NetperfNetperf

● Combine multiple streamsCombine multiple streams– DownloadDownload– UploadUpload– Latency measurementLatency measurement

FIFO – the defaultFIFO – the default

Høiland-Jørgensen T., Battling Bufferbloat

Ping 1sec!

Upload

Download

Stochastic Fair QueueStochastic Fair Queue

Høiland-Jørgensen T., Battling Bufferbloat

Controlled Delay - codelControlled Delay - codel

Høiland-Jørgensen T., Battling Bufferbloat

Fair Queue Controlled DelayFair Queue Controlled Delay

Høiland-Jørgensen T., Battling Bufferbloat

Bufferbloat statusBufferbloat status

Router

Where is Bufferbloat?Where is Bufferbloat?

BackboneISP ISP

SenderReceiver

ApplicationApplication

TCP

QDisc

TCP

Router

Bufferbloat Solved?Bufferbloat Solved?

● Queuing disciplinesQueuing disciplines– Codel, PIE, ...Codel, PIE, ...

● Linux internal Linux internal – BQL, FQ, PacingBQL, FQ, Pacing

● Default qdiscDefault qdiscSysctl ­w sys.net.core.default_qdisc=fq_codelSysctl ­w sys.net.core.default_qdisc=fq_codel

Yes, many times overYes, many times over

Bufferbloat: The GoodBufferbloat: The Good

● Industry awarenessIndustry awareness● Standards progressStandards progress● Active researchActive research● Open Source Open Source

SolutionsSolutions

Bufferbloat: the BadBufferbloat: the Bad

● Legacy equipmentLegacy equipment– OEM focus on cost not AQMOEM focus on cost not AQM

● Congestion on backboneCongestion on backbone– It's not a bug it's a feature?It's not a bug it's a feature?– Problem is political not technicalProblem is political not technical

Bufferbloat: the UglyBufferbloat: the Ugly

● WirelessWireless– AggregationAggregation– MulticastMulticast– Proprietary firmwareProprietary firmware

Questions?

Dave T<dave.taht@bufferbloat.net>

Thank youStephen Hemminger@networkplumber

stephen@networkplumber.org

Bufferbloat resources

Bufferbloat.net: http://bufferbloat.netEmail Lists: http://lists.bufferbloat.net

IRC Channel: #bufferbloat on chat.freenode.net CeroWrt: http://www.bufferbloat.net/projects/cerowrt Other talks: http://mirrors.bufferbloat.net/Talks

Jim Gettys Blog – http://gettys.wordpress.com RRUL Test- https://github.com/tohojo/netperf-wrapper

A big thanks to the bloat mailing list, Jim, Kathie, Van, and Eric, ISC, the CeroWrt contributors, OpenWrt, the Internet Civil Engineering Institute, and the Google Fiber for their interest and support in the work!