Linux TCP/IP Tuning

transcript

Analyzing TCP Performance

Sr. Staff EngineerLinux Kongress 2004

2004-09-09

Stephen Hemminger

Agenda

■ Introduction■ TCP for muggles■ Engineering Process■ Problem examples■ Network Tools■ Wrapup

Outside of scope

■ Non TCP protocols■ SCTP, multicast, etc

■ Queuing theory - “no math”■ Hardware and product comparisons

My Background

■ Did TCP back in the “old school”■ BSD 4.2, Ethernet■ SMP Unix versions of OSI, Netware, Appletalk, ...■ Plan9 Hypercube communication

■ Linux■ Incorporation of TCP research in 2.6 kernel■ Performance tests for LWE■ Wizard gap

Limits of my knowledge

■ Only worked with current Linux (2.4/2.6)■ Will mention tools here that I have not used

extensively■ Involved in development of Linux not deployment

or research

Agenda

TCP for “muggles”

■ connection establishment■ slow start■ windows■ congestion control■ silly window

Connection establishment

SYN+ACK

Data 1(10)

Ack 11

connect

Client Server

accept

ethereal

tcpdump trace

13:28:21.745624 IP 172.20.1.60.38052 > 216.239.39.99.http: S 1765497548:1765497548(0)win 5840 <mss 1460,sackOK,timestamp 1563951453 0,nop,wscale 7>

13:28:21.831935 IP 216.239.39.99.http > 172.20.1.60.38052: S 227058185:227058185(0)ack 1765497549 win 8190 <mss 1460>

13:28:21.832035 IP 172.20.1.60.38052 > 216.239.39.99.http: . ack 1 win 584013:28:21.832321 IP 172.20.1.60.38052 > 216.239.39.99.http: P 1:126(125) ack 1 win 584013:28:21.939237 IP 216.239.39.99.http > 172.20.1.60.38052: . ack 126 win 3146013:28:21.972448 IP 216.239.39.99.http > 172.20.1.60.38052: P 1:485(484) ack 126 win 3146013:28:21.972529 IP 172.20.1.60.38052 > 216.239.39.99.http: . ack 485 win 643213:28:21.973016 IP 172.20.1.60.38052 > 216.239.39.99.http: F 126:126(0) ack 485 win 6432

Flow control

Data 1011 (1400)

ACK 1010 (5000)

Ack 6010 (1000)

read (1000)

Data 3811 (1400)

Data 2411 (1400)Data 5211 (800)

Ack 6010 (0)

Retransmission

Data 1

Ack 1Ack 1

Data 2

Multiple ack's = fast retransmit

Tcptrace

http://tcptrace.org

Tool to convert captured data into graphs■ Time sequence graph■ Throughput■ RTT

Lots more than time to cover here!

http://xplot.org■ Takes plot command scripts■ Mouse

■ Zoom – drag with the left button■ Zoom out – click the left button ■ Scroll – drag with middle button■ Dump – shift-left button produces postscript

■ Shift-middle and shift-right also

Time Sequence Graph

Windows & Buffering

■ Used to isolate TCP from application read/write■ Used for congestion control■ Upper bound determined by system parameters

Congestion window

■ slow start■ Window normally starts small■ Grows in response to ack

■ congestion control■ Packet loss = congestion

Silly Window

Data (2000)

Ack [10]write8k bytes

Ack [2000]

Read8k bytes

“Hey, I am not going to try and send this data now give me a bigger window first”

OK, thanks

Model of TCP networks

Network

Send Window

Sender

Receive Window

Receiver

BDP = Bandwidth (bytes/sec) * Delay (secs/unit)

BDP - Bandwidth Delay Product

■ BDP = amount of data in transit■ Examples

■ DSL/Cable modem (international)

1,000,000 bit/sec * 1/8 byte/bit * 500 ms = 62500 bytes

■ Gigabit across US

1,000,000,000 bit/sec * 1/8 byte/bit * 70 ms = 8,75 Mbytes

0.1 1 10 100 10000.1

Delay (ms)

Bandwidth Delay Product (BDP)

8K1M64K

Broadband

ResearchLAN

Internet

■ Router queues■ Delays

■ Speed of light (70ms coast/coast)■ Slow routers

■ Packet correlation, sizes■ DoS

Extensions for larger windows

■ TCP Selective Acknowlegement (SACK) RFC2018

■ Don't have to retransmit everything

■ Window scaling (RFC1323)■ Window size multiplied by 2n

■ Protection Against Wrapped Sequence (PAWS)■ Timestamp inside each packet

TCP options negotiation 1

IP 172.20.1.60.32820 > 216.239.39.99.http: S 3599527174:3599527174(0) win 5840<mss 1460,sackOK,timestamp 2519711 0,nop,wscale 2>

IP 216.239.39.99.http > 172.20.1.60.32820: S 3820474812:3820474812(0) ack 3599527175 win 8190 <mss 1460>IP 172.20.1.60.32820 > 216.239.39.99.http: . ack 1 win 5840IP 172.20.1.60.32820 > 216.239.39.99.http: P 1:126(125) ack 1 win 5840

Window scale by 4

But server doesn't support scaling

TCP options negotiation 2

IP 172.20.1.60.32823 > 65.172.181.13.http: S 4120108902:4120108902(0) win 5840 <mss 1460,sackOK,timestamp 3036627 0,nop,wscale 2>

IP 65.172.181.13.http > 172.20.1.60.32823: S 2295773021:2295773021(0) ack 4120108903 win 5792

Window scale by 4

Your scaling is okay, but don't scale mine

Linux TCP window tuning

■ Send window - net.ipv4.tcp_wmem■ three values : initial default max

■ default is 4K 16K 128K■ also limited by net.core.wmem_max

■ Receive window – net.ipv4.tcp_rmem

■ three values : initial default max■ default is 4K 85K 170K

■ also limited by net.core.rmem_max

Linux TCP window tuning

■ Overall memory – net.ipv4.tcp_mem■ three values : low pressure max■ automatic value based on system memory

■ Application window – net.ipv4.tcp_app_mem

■ reserved space to handle slow applications

■ Some firewalls and routers are buggy■ Corrupt window scale change N to 0■ Forget to track state, or read RFC wrong■ Connections will hang because initial window looks

like a silly window■ 1% of the net is buggy..

■ Linux 2.6.9 chooses window scale based on maximum possible receive window

■ Default tcp_rmem => window scale of 2■ Buggy devices will see ¼ of the real window

Agenda

Performance Engineering process

■ Define what your goal■ Capture information■ Analyze and form hypothesis■ Prototype to validate hypothesis

■ If successful■ Make changes on production system■ Report problems or patches to others

Goal setting

■ Know what is possible:■ bus bandwidth, network latency, etc.

■ Know your application■ Compare with similar applications

TCP performance testing

■ Goal: Improve TCP performance over high bandwidth * delay links

■ Plan:■ New TCP congestion control■ Validate and test

Testing TCP over WAN

■ Want to test performance of TCP over high BDP links

■ Can't afford a 10Gbit trans-continental link■ Proposal: emulate network delay over 1Gbit

Ethernet

Existing network emulation tools

■ Dummynet

http://info.iet.unipi.it/~luigi/ip_dummynet/I don't want to setup separate FreeBSD machine

■ NISTnethttp://snad.ncsl.nist.gov/itg/nistnet/

Only on 2.4 and not ready to be in main tree

http://developer.osdl.org/shemminger/netem■ Started out as simple delay only hack■ Grown up to do all the functionality of NISTnet

Ethernet (eth0)

Current TCP research

■ Alternative TCP congestion■ Vegas■ Westwood■ Binary Increase Congestion Control (BIC)

■ Research community based around Web100

TCP Reno

■ Standard default in 2.4/2.6■ Adjusts congestion window based on packet loss■ Slow start – window grows slowly■ Additive Increase window on each Ack■ Multiplicative Decrease on loss

TCP Vegas

■ Original work by Larry Peterson■ Patches existed for 2.2, 2.4 and part of web100■ sysctl net.ipv4.tcp_cong_avoid

■ Measure bandwidth based on RTT■ Adjust congestion window on bandwidth■ Avoids packet loss

TCP Westwood

■ Work by Caludio Casetti■ Patches for 2.4 by Angelo Dell'Aera■ sysctl net.ipv4.tcp_westwood

■ Focused on wireless ■ packet loss != congestion

■ Measure bandwidth based on RTT■ Use normal Reno till congestion then adjust

congestion window based on bandwidth

Binary Increase Congestion Control (BIC)

■ Work by Lisung Xu■ Patches for Web100 (2.4)■ sysctl net.ipv4.tcp_bic

■ Designed for best high speed networks■ Modification of Reno■ Use additive increase when congestion window

is large■ Binary search increase when window is small

Tuning

■ Default tcp parameters not big enough ■ Need bigger send and receive window

■ Send window autosized based on rtt already■ Receive window autosizing was done in Web100

Receiver Tuning

■ Patches from John Heffner■ sysctl net.ipv4.tcp_moderate_rcvbuf

■ Dynamic Right Sizing (DRS)■ adjust receive window based on RTT■ If application doesn't set window then do it for them■ Window will grow from default to max

Receiver auto-tuning

0 50 100 150 2000

Default

Auto Tuned

Delay (ms)

Throughput vs Delay (initial run)

0 50 100 150 200

Delay (ms)

RenoVegas

WestwoodBic

What's happening

■ NAPI■ Driver API to allow avoiding interrupts■ Trades off latency for overall performance

■ E1000 driver■ Uses NAPI for transmit

Answer: Transmit ring gets full and driver flow blocks

Solution: set TxDescriptors=1000

Thorughput vs Delay (rerun)

0 25 50 75 100 125 150 175 2000

Westwood

Delay (ms)

Performance still slow

■ Vegas and Westwood are terrible■ Not at full link speed■ Performance falling off with delay

Vegas trace with 100ms delay

Vegas detail

Westwood (70ms)

Westwood detail

BIC trace (100ms)

BIC detail (100ms)

How to squeeze out more performance

■ Large MTU (4k) + 63%■ LAN driver not-module up to 10%■ Turn off timestamps + 4%■ Bind IRQ to processor varies

Congestion more work

■ Vegas doesn't use available window■ Does it under estimate bandwidth?

■ Westwood■ Another bandwidth problem

■ BIC■ When does it make into binary mode?■ What is holding back window?

■ Netem■ Higher resolution? Packet groups?

Agenda

Other tools

■ Information about■ ISP connection■ Sockets open

■ Testing infrastructure■ More data capture■ Monitoring

Tools: basic

■ Network path information■ Ping – send icmp echo

■ Measure of round trip time and loss■ Can be blocked by firewall

■ Traceroute – use IP source routing■ Usually blocked now

■ Pathcapture (pcap)■ Bandwidth and delay measurement

Tools: Network interface

■ ifconfig■ Basic statistics, packets sent/received/errors

■ ip -stats link■ Alternate newer, may have more info

■ SNMP■ Remote access to same information■ Slightly more work

Tools: Sockets

■ Netstat■ TCP statistics■ Open sockets

■ Ss■ More statistics available (rtt, etc)

■ Recvmsg■ Application can see TCP info (cmsg)

Tools: test servers

■ SYN testtelnet syntest.psc.edu 7960

■ TCP bandwidthhttp://www.epm.ornl.gov/~dunigan/java/misc/tcpbw.html

http://dslreports.com

■ ANL network confighttp://miranda.ctd.anl.gov:7123

■ Path MTUhttp://www.ncne.org/jumbogram/mtu_discovery.php

Tools: testing

■ Ttcp■ Basic send /receive throughput

■ Iperf■ Longer running tests and turnaround

■ Netperf■ Includes cpu and other statistics

■ Dbs■ Multiclient testing

Tools: monitoring

■ Ntop■ Measure of network activity by service■ Nice web interface

■ Mailgraph■ Long term mail statistics

■ Web server activity log analysis

Tools: data capture

■ Tcpdump■ Filter packets by protocol, address, etc■ Decode many protcols

■ Ethereal■ GUI interface

■ RMON■ Remote monitoring

■ Kismet■ Wireless activity

Tools: generators

■ Pktgen■ Kernel level packet generation■ Can generate maximum hardware packet rate

■ Network packet generator■ Application level

Tools: simulation

■ Ns■ Describe overall system■ Event based simulation■ Used for protocol analysis

■ SSFnet■ More detailed models of real hardware

Tools: client simulator

■ Web■ SPECweb, Apache (as), httpload

■ NFS■ Nfsstone

■ FTP■ Dkftpbench

Conclusion

■ Data capture can provide clues of:■ Application problems■ Device problems■ TCP/IP problems

■ Nothing is ever simple

Linux TCP/IP Tuning

Documents