15-744: Computer NetworkingL-4 TCP

TCP BasicsTCP reliabilityCongestion control basicsTCP congestion controlAssigned reading[JK88] Congestion Avoidance and Control[CJ89] Analysis of the Increase and Decrease Algorithms for Congestion Avoidance in Computer Networks[FF96] Simulation-based Comparisons of Tahoe, Reno, and SACK TCP[FHPW00] Equation-Based Congestion Control for Unicast Applications

Key Things You Should Know AlreadyPort numbersTCP/UDP checksumSliding window flow controlSequence numbersTCP connection setup

OverviewTCP reliability: timer-driven

TCP reliability: data-driven

Congestion sources and collapse

Congestion control basics

TCP congestion control

TCP modeling

Introduction to TCPCommunication abstraction:ReliableOrderedPoint-to-pointByte-streamFull duplexFlow and congestion controlled

Protocol implemented entirely at the endsFate sharing

Sliding window with cumulative acksAck field contains last in-order packet receivedDuplicate acks sent when out-of-order packet received

Evolution of TCP19751980198519901982TCP & IPRFC 793 & 7911974TCP described byVint Cerf and Bob KahnIn IEEE Trans Comm1983BSD Unix 4.2supports TCP/IP1984Nagels algorithmto reduce overheadof small packets;predicts congestion collapse1987Karns algorithmto better estimate round-trip time1986Congestion collapseobserved1988Van Jacobsons algorithmscongestion avoidance and congestion control(most implemented in 4.3BSD Tahoe)19904.3BSD Renofast retransmitdelayed ACKs1975Three-way handshakeRaymond TomlinsonIn SIGCOMM 75

TCP Through the 1990s1993199419961994ECN(Floyd)Explicit CongestionNotification1993TCP Vegas (Brakmo et al)real congestion avoidance1994T/TCP(Braden)TransactionTCP1996SACK TCP(Floyd et al)Selective Acknowledgement1996HoeImproving TCP startup1996FACK TCP(Mathis et al)extension to SACK

Whats Different From Link Layers?Logical link vs. physical linkMust establish connectionVariable RTTMay vary within a connectionReorderingHow long can packets live max segment lifetimeCant expect endpoints to exactly match linkBuffer space availabilityTransmission rateDont directly know transmission rate

Timeout-based RecoveryWait at least one RTT before retransmittingImportance of accurate RTT estimators:Low RTT unneeded retransmissionsHigh RTT poor throughputRTT estimator must adapt to change in RTTBut not too fast, or too slow!Spurious timeoutsConservation of packets principle more than a window worth of packets in flight

Initial Round-trip EstimatorRound trip times exponentially averaged:New RTT = a (old RTT) + (1 - a) (new sample)Recommended value for a: 0.8 - 0.90.875 for most TCPsRetransmit timer set to b RTT, where b = 2Every time timer expires, RTO exponentially backed-offLike EthernetNot good at preventing spurious timeouts

Jacobsons Retransmission TimeoutKey observation:At high loads round trip variance is highSolution:Base RTO on RTT and standard deviation or RRTTrttvar = * dev + (1- )rttvardev = linear deviation Inappropriately named actually smoothed linear deviation

Retransmission AmbiguityABACKSampleRTTOriginal transmissionretransmissionRTOX

Karns RTT EstimatorAccounts for retransmission ambiguityIf a segment has been retransmitted:Dont count RTT sample on ACKs for this segmentKeep backed off time-out for next packetReuse RTT estimate only after one successful transmission

Timestamp ExtensionUsed to improve timeout mechanism by more accurate measurement of RTTWhen sending a packet, insert current timestamp into option4 bytes for seconds, 4 bytes for microsecondsReceiver echoes timestamp in ACKActually will echo whatever is in timestampRemoves retransmission ambiguityCan get RTT sample on any packet

Timer GranularityMany TCP implementations set RTO in multiples of 200,500,1000msWhy?Avoid spurious timeouts RTTs can vary quickly due to cross trafficMake timers interrupts efficient

Delayed ACKSProblem:In request/response programs, you send separate ACK and Data packets for each transactionSolution:Dont ACK data immediatelyWait 200ms (must be less than 500ms why?)Must ACK every other packetMust not delay duplicate ACKs

OverviewTCP reliability: timer-driven

TCP reliability: data-driven

Congestion sources and collapse

Congestion control basics

TCP congestion control

TCP modeling

TCP FlavorsTahoe, Reno, Vegas differ in data-driven reliabilityTCP Tahoe (distributed with 4.3BSD Unix)Original implementation of Van Jacobsons mechanisms (VJ paper)Includes:Slow start Congestion avoidanceFast retransmit

Fast RetransmitWhat are duplicate acks (dupacks)?Repeated acks for the same sequenceWhen can duplicate acks occur?LossPacket re-orderingWindow update advertisement of new flow control windowAssume re-ordering is infrequent and not of large magnitudeUse receipt of 3 or more duplicate acks as indication of lossDont wait for timeout to retransmit packet

Fast RetransmitTimeSequence NoDuplicate AcksRetransmissionX

Multiple LossesTimeSequence NoDuplicate AcksRetransmissionXXXXNow what?

TahoeTimeSequence NoXXXX

TCP Reno (1990)All mechanisms in TahoeAddition of fast-recovery Opening up congestion window after fast retransmitDelayed acksHeader prediction Implementation designed to improve performanceHas common case code inlinedWith multiple losses, Reno typically timeouts because it does not receive enough duplicate acknowledgements

RenoTimeSequence NoXXXXNow what? timeout

NewRenoThe ack that arrives after retransmission (partial ack) should indicate that a second loss occurredWhen does NewReno timeout?When there are fewer than three dupacks for first lossWhen partial ack is lostHow fast does it recover losses?One per RTT

NewRenoTimeSequence NoXXXXNow what? partial ackrecovery

SACKBasic problem is that cumulative acks provide little informationAck for just the packet receivedWhat if acks are lost? carry cumulative alsoNot usedBitmask of packets received Selective acknowledgement (SACK)How to deal with reordering

SACKTimeSequence NoXXXXNow what? sendretransmissions as soonas detected

Performance IssuesTimeout >> fast rexmitNeed 3 dupacks/sacksNot great for small transfersDont have 3 packets outstandingWhat are real loss patterns like?

OverviewTCP reliability: timer-driven

TCP reliability: data-driven

Congestion sources and collapse

Congestion control basics

TCP congestion control

TCP modeling

CongestionDifferent sources compete for resources inside networkWhy is it a problem?Sources are unaware of current state of resourceSources are unaware of each otherIn many situations will result in < 1.5 Mbps of throughput (congestion collapse)

Causes & Costs of CongestionFour senders multihop pathsTimeout/retransmit

Q: What happens as rate increases?

Causes & Costs of CongestionWhen packet dropped, any upstream transmission capacity used for that packet was wasted!

Congestion CollapseDefinition: Increase in network load results in decrease of useful work doneMany possible causesSpurious retransmissions of packets still in flightClassical congestion collapseHow can this happen with packet conservationSolution: better timers and TCP congestion controlUndelivered packetsPackets consume resources and are dropped elsewhere in networkSolution: congestion control for ALL traffic

Other Congestion Collapse CausesFragmentsMismatch of transmission and retransmission unitsSolutionsMake network drop all fragments of a packet (early packet discard in ATM)Do path MTU discoveryControl trafficLarge percentage of traffic is for controlHeaders, routing messages, DNS, etc.Stale or unwanted packetsPackets that are delayed on long queuesPush data that is never used

Where to Prevent Collapse?Can end hosts prevent problem?Yes, but must trust end hosts to do right thingE.g., sending host must adjust amount of data it puts in the network based on detected congestionCan routers prevent collapse?No, not all forms of collapseDoesnt mean they cant help Sending accurate congestion signalsIsolating well-behaved from ill-behaved sources

Congestion Control and AvoidanceA mechanism which:Uses network resources efficientlyPreserves fair network resource allocationPrevents or avoids collapseCongestion collapse is not just a theoryHas been frequently observed in many networks

OverviewTCP reliability: timer-driven

TCP reliability: data-driven

Congestion sources and collapse

Congestion control basics

TCP congestion control

TCP modeling

ObjectivesSimple router behavior DistributednessEfficiency: Xknee = Sxi(t)Fairness: (Sxi)2/n(Sxi2)Power: (throughputa/delay)Convergence: control system must be stable

Basic Control ModelLets assume window-based controlReduce window when congestion is perceivedHow is congestion signaled?Either mark or drop packetsWhen is a router congested?Drop tail queues when queue is fullAverage queue length at some thresholdIncrease window otherwiseProbe for available bandwidth how?

Linear ControlMany different possibilities for reaction to congestion and probingExamine simple linear controlsWindow(t + 1) = a + b Window(t)Different ai/bi for increase and ad/bd for decreaseSupports various reaction to signalsIncrease/decrease additivelyIncreased/decrease multiplicativelyWhich of the four combinations is optimal?

Phase plotsSimple way to visualize behavior of competing connections over timeEfficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2

Phase plotsWhat are desirable properties?What if flows are not equal?Efficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2Optimal pointOverloadUnderutilization

Additive Increase/DecreaseBoth X1 and X2 increase/decrease by the same amount over timeAdditive increase improves fairness and additive decrease reduces fairnessT0T1Efficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2

Multiplicative Increase/DecreaseBoth X1 and X2 increase by the same factor over timeExtension from origin constant fairnessT0T1Efficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2

Convergence to EfficiencyxHEfficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2

Distributed Convergence to Efficiency

Convergence to FairnessxHEfficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2xH

Convergence to Efficiency & FairnessxHEfficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2xH

IncreaseEfficiency LineFairness LineUser 1s Allocation x1User 2s Allocation x2xL

ConstraintsDistributed efficiencyI.e., Window(t+1) > Window(t) during increaseai > 0 & bi 1Similarly, ad < 0 & bd 1Must never decrease fairnessa & bs must be 0ai/bi > 0 and ad/bd 0Full constraintsad = 0, 0 bd < 1, ai > 0 and bi 1

What is the Right Choice?Constraints limit us to AIMDCan have multiplicative term in increase (MAIMD)AIMD moves towards optimal point

OverviewTCP reliability: timer-driven

TCP reliability: data-driven

Congestion sources and collapse

Congestion control basics

TCP congestion control

TCP modeling

TCP Congestion ControlMotivated by ARPANET congestion collapseUnderlying design principle: packet conservationAt equilibrium, inject packet into network only when one is removedBasis for stability of physical systemsWhy was this not working?Connection doesnt reach equilibriumSpurious retransmissionsResource limitations prevent equilibrium

TCP Congestion Control - SolutionsReaching equilibriumSlow startEliminates spurious retransmissionsAccurate RTO estimationFast retransmitAdapting to resource availabilityCongestion avoidance

TCP Congestion ControlChanges to TCP motivated by ARPANET congestion collapseBasic principlesAIMDPacket conservationReaching steady state quicklyACK clocking

AIMDDistributed, fair and efficientPacket loss is seen as sign of congestion and results in a multiplicative rate decrease Factor of 2TCP periodically probes for available bandwidth by increasing its rateTimeRate

Implementation IssueOperating system timers are very coarse how to pace packets out smoothly?Implemented using a congestion window that limits how much data can be in the network.TCP also keeps track of how much data is in transitData can only be sent when the amount of outstanding data is less than the congestion window.The amount of outstanding data is increased on a send and decreased on ack(last sent last acked) < congestion windowWindow limited by both congestion and bufferingSenders maximum window = Min (advertised window, cwnd)

Congestion AvoidanceIf loss occurs when cwnd = WNetwork can handle 0.5W ~ W segmentsSet cwnd to 0.5W (multiplicative decrease)Upon receiving ACKIncrease cwnd by (1 packet)/cwndWhat is 1 packet? 1 MSS worth of bytesAfter cwnd packets have passed by approximately increase of 1 MSSImplements AIMD

Congestion Avoidance Sequence PlotTimeSequence No

Congestion Avoidance BehaviorTimeCongestionWindowPacket loss+ TimeoutGrabbingback BandwidthCutCongestionWindowand Rate

Packet ConservationAt equilibrium, inject packet into network only when one is removedSliding window and not rate controlledBut still need to avoid sending burst of packets would overflow linksNeed to carefully pace out packetsHelps provide stability Need to eliminate spurious retransmissionsAccurate RTO estimationBetter loss recovery techniques (e.g. fast retransmit)

TCP Packet PacingCongestion window helps to pace the transmission of data packetsIn steady state, a packet is sent when an ack is receivedData transmission remains smooth, once it is smoothSelf-clocking behaviorPrPbArAbReceiverSenderAs

Reaching Steady StateDoing AIMD is fine in steady state but slowHow does TCP know what is a good initial rate to start with?Should work both for a CDPD (10s of Kbps or less) and for supercomputer links (10 Gbps and growing)Quick initial phase to help get up to speed (slow start)

Slow Start Packet PacingHow do we get this clocking behavior to start?Initialize cwnd = 1Upon receipt of every ack, cwnd = cwnd + 1ImplicationsWindow actually increases to W in RTT * log2(W)Can overshoot window and cause packet loss

Slow Start Example

Slow Start Sequence PlotTimeSequence No...

Return to Slow StartIf packet is lost we lose our self clocking as wellNeed to implement slow-start and congestion avoidance togetherWhen timeout occurs set ssthresh to 0.5wIf cwnd < ssthresh, use slow startElse use congestion avoidance

TCP Saw Tooth BehaviorTimeCongestionWindowInitialSlowstartFast Retransmitand RecoverySlowstartto pacepacketsTimeoutsmay stilloccur

How to Change WindowWhen a loss occurs have W packets outstandingNew cwnd = 0.5 * cwndHow to get to new state?

Fast RecoveryEach duplicate ack notifies sender that single packet has cleared networkWhen < cwnd packets are outstandingAllow new packets out with each new duplicate acknowledgementBehaviorSender is idle for some time waiting for cwnd worth of dupacksTransmits at original rate after waitAck clocking rate is same as before loss

Fast Recovery TimeSequence NoSent for each dupack afterW/2 dupacks arriveX

NewReno ChangesSend a new packet out for each pair of dupacksAdapt more gradually to new windowWill not halve congestion window again until recovery is completed Identifies congestion events vs. congestion signalsInitial estimation for ssthresh

Rate Halving RecoveryTimeSequence NoSent after everyother dupackX

Delayed Ack ImpactTCP congestion control triggered by acksIf receive half as many acks window grows half as fastSlow start with window = 1Will trigger delayed ack timerFirst exchange will take at least 200msStart with > 1 initial windowBug in BSD, now a feature/standard

OverviewTCP reliability: timer-driven

TCP reliability: data-driven

Congestion sources and collapse

Congestion control basics

TCP congestion control

TCP modeling

TCP ModelingGiven the congestion behavior of TCP can we predict what type of performance we should get?What are the important factorsLoss rateAffects how often window is reducedRTTAffects increase rate and relates BW to windowRTOAffects performance during loss recoveryMSS Affects increase rate

Overall TCP BehaviorTimeWindowLets concentrate on steady state behavior with no timeouts and perfect loss recovery

Simple TCP ModelSome additional assumptionsFixed RTTNo delayed ACKsIn steady state, TCP losses packet each time window reaches W packetsWindow drops to W/2 packetsEach RTT window increases by 1 packetW/2 * RTT before next lossBW = MSS * avg window/RTT = MSS * (W + W/2)/(2 * RTT) = .75 * MSS * W / RTT

Simple Loss ModelWhat was the loss rate?Packets transferred = (.75 W/RTT) * (W/2 * RTT) = 3W2/81 packet lost loss rate = p = 8/3W2W = sqrt( 8 / (3 * loss rate))BW = .75 * MSS * W / RTTBW = MSS / (RTT * sqrt (2/3p))

TCP FriendlinessWhat does it mean to be TCP friendly?TCP is not going awayAny new congestion control must compete with TCP flowsShould not clobber TCP flows and grab bulk of linkShould also be able to hold its own, i.e. grab its fair share, or it will never become popularHow is this quantified/shown?Has evolved into evaluating loss/throughput behaviorIf it shows 1/sqrt(p) behavior it is okBut is this really true?

TCP PerformanceCan TCP saturate a link?Congestion controlIncrease utilization until link becomes congestedReact by decreasing window by 50%Window is proportional to rate * RTTDoesnt this mean that the network oscillates between 50 and 100% utilization?Average utilization = 75%??Nothis is *not* right!

TCP Congestion ControlOnly W packets may be outstandingRule for adjusting WIf an ACK is received: W W+1/WIf a packet is lost:W W/2SourceDest

Single TCP FlowRouter without buffers

Summary Unbuffered LinktWMinimum window for full utilizationThe router cant fully utilize the linkIf the window is too small, link is not fullIf the link is full, next window increase causes dropWith no buffer it still achieves 75% utilization

TCP PerformanceIn the real world, router queues play important roleWindow is proportional to rate * RTTBut, RTT changes as well the windowWindow to fill links = propagation RTT * bottleneck bandwidthIf window is larger, packets sit in queue on bottleneck link

TCP PerformanceIf we have a large router queue can get 100% utilizationBut, router queues can cause large delaysHow big does the queue need to be?Windows vary from W W/2Must make sure that link is always fullW/2 > RTT * BWW = RTT * BW + QsizeTherefore, Qsize > RTT * BWEnsures 100% utilizationDelay?Varies between RTT and 2 * RTT

Single TCP FlowRouter with large enough buffers for full link utilization

Summary Buffered LinktWMinimum window for full utilizationWith sufficient buffering we achieve full link utilizationThe window is always above the critical thresholdBuffer absorbs changes in window sizeBuffer Size = Height of TCP SawtoothMinimum buffer size needed is 2T*CThis is the origin of the rule-of-thumbBuffer

Example10Gb/s linecardRequires 300Mbytes of buffering.Read and write 40 byte packet every 32ns.Memory technologiesDRAM: require 4 devices, but too slow. SRAM: require 80 devices, 1kW, $2000.Problem gets harder at 40Gb/sHence RLDRAM, FCRAM, etc.

Rule-of-thumbRule-of-thumb makes sense for one flowTypical backbone link has > 20,000 flowsDoes the rule-of-thumb still hold?

If flows are synchronizedAggregate window has same dynamicsTherefore buffer occupancy has same dynamicsRule-of-thumb still holds.t

If flows are not synchronizedB0

Central Limit TheoremCLT tells us that the more variables (Congestion Windows of Flows) we have, the narrower the Gaussian (Fluctuation of sum of windows)Width of Gaussian decreases with Buffer size should also decreases with

Required buffer sizeSimulation

Important LessonsHow does TCP implement AIMD?Sliding window, slow start & ack clockingHow to maintain ack clocking during loss recovery fast recovery

Modern TCP loss recoveryWhy are timeouts bad?How to avoid them? fast retransmit, SACK

How does TCP fully utilize a link?Role of router buffers

Next LectureTCP Vegas/alternative congestion control schemesREDFair queuingCore-stateless fair queuing/XCPAssigned reading[BP95] TCP Vegas: End to End Congestion Avoidance on a Global Internet[FJ93] Random Early Detection Gateways for Congestion Avoidance[DKS90] Analysis and Simulation of a Fair Queueing Algorithm, Internetworking: Research and Experience[SSZ98] Core-Stateless Fair Queueing: Achieving Approximately Fair Allocations in High Speed Networks[KHR02] Congestion Control for High Bandwidth-Delay Product Networks

EXTRA SLIDESThe rest of the slides are FYI

Integrity & DemultiplexingPort numbersDemultiplex from/to processServers wait on well known ports (/etc/services)ChecksumIs it sufficient to just checksum the packet contents?No, need to ensure correct source/destinationPseudoheader portion of IP hdr that are criticalChecksum covers Pseudoheader, transport hdr, and packet bodyUDP provides just integrity and demux

TCP HeaderSource portDestination portSequence numberAcknowledgementAdvertised windowHdrLenFlags0ChecksumUrgent pointerOptions (variable)DataFlags:SYNFINRESETPUSHURGACK

TCP Flow ControlTCP is a sliding window protocolFor window size n, can send up to n bytes without receiving an acknowledgement When the data is acknowledged then the window slides forwardEach packet advertises a window sizeIndicates number of bytes the receiver has space forOriginal TCP always sent entire windowCongestion control now limits this

Window Flow Control: Send SideSent but not ackedNot yet sentwindowNext to be sentSent and acked

Window Flow Control: Receive SideAcked but notdelivered to userNot yetackedReceive bufferwindow

TCP PersistWhat happens if window is 0?Receiver updates window when application reads dataWhat if this update is lost?TCP Persist stateSender periodically sends 1 byte packetsReceiver responds with ACK even if it cant store the packet

Connection EstablishmentA and B must agree on initial sequence number selectionUse 3-way handshakeABSYN + Seq ASYN+ACK-A + Seq BACK-B

Sequence Number SelectionWhy not simply chose 0?Must avoid overlap with earlier incarnation

Connection SetupCLOSEDSYNSENTSYNRCVDESTABLISTENactive OPENcreate TCBSnd SYN create TCBpassive OPENdelete TCBCLOSEdelete TCBCLOSEsnd SYNSENDsnd SYN ACKrcv SYNSend FINCLOSErcv ACK of SYNSnd ACKRcv SYN, ACKrcv SYNsnd ACK

Connection Tear-downNormal terminationAllow unilateral closeTCP must continue to receive data even after closingCannot close connection immediately What if a new connection restarts and uses same sequence number?

Tear-down Packet ExchangeSenderReceiverFINFIN-ACKFINFIN-ACKData writeData ack

Connection Tear-downCLOSINGCLOSEWAITFINWAIT-1ESTABTIME WAITsnd FINCLOSEsend FINCLOSErcv ACK of FINLAST-ACKCLOSEDFIN WAIT-2snd ACKrcv FINdelete TCBTimeout=2mslsend FINCLOSEsend ACKrcv FINsnd ACKrcv FINrcv ACK of FINsnd ACKrcv FIN+ACK

Detecting Half-open Connections

(CRASH)CLOSEDSYN-SENT (!!) SYN-SENT SYN-SENTSYN-SENT

(send 300, receive 100) ESTABLISHED(??)ESTABLISHED (Abort!!) CLOSED

TCP BTCP A

Observed TCP ProblemsToo many small packetsSilly window syndromeNagels algorithmInitial sequence number selectionAmount of state maintained

Silly Window SyndromeProblem: (Clark, 1982)If receiver advertises small increases in the receive window then the sender may waste time sending lots of small packetsSolutionReceiver must not advertise small window increases Increase window by min(MSS,RecvBuffer/2)

Nagels AlgorithmSmall packet problem:Dont want to send a 41 byte packet for each keystrokeHow long to wait for more data? Solution:Allow only one outstanding small (not full sized) segment that has not yet been acknowledged

Why is Selecting ISN Important?Suppose machine X selects ISN based on predictable sequenceFred has .rhosts to allow login to X from YEvil Ed attacksDisables host Y denial of service attackMake a bunch of connections to host XDetermine ISN pattern a guess next ISNFake pkt1: [, guessed ISN]Fake pkt2: desired command

Time Wait IssuesWeb servers not clients close connection firstEstablished Fin-Waits Time-Wait ClosedWhy would this be a problem?Time-Wait state lasts for 2 * MSLMSL is should be 120 seconds (is often 60s)Servers often have order of magnitude more connections in Time-Wait

TCP ExtensionsImplemented using TCP optionsTimestampProtection from sequence number wraparoundLarge windows

Protection From WraparoundWraparound time vs. Link speed1.5Mbps: 6.4 hours10Mbps: 57 minutes45Mbps: 13 minutes100Mbps: 6 minutes622Mbps: 55 seconds < MSL!1.2Gbps: 28 secondsUse timestamp to distinguish sequence number wraparound

Large WindowsDelay-bandwidth product for 100ms delay1.5Mbps: 18KB10Mbps: 122KB > max 16bit window45Mbps: 549KB100Mbps: 1.2MB622Mbps: 7.4MB1.2Gbps: 14.8MBScaling factor on advertised windowSpecifies how many bits window must be shifted to the leftScaling factor exchanged during connection setup

Maximum Segment Size (MSS)Exchanged at connection setupTypically pick MTU of local linkWhat all does this effect?EfficiencyCongestion controlRetransmissionPath MTU discoveryWhy should MTU match MSS?