FAST TCP Steven Low CS/EE netlab.CALTECH.edu Oct 2003.

transcript

FAST TCP

Steven Low

CS/EEnetlab.CALTECH.edu

Oct 2003

FAST Protocols for Ultrascale Networks

netlab.caltech.edu/FAST

Internet: distributed feedback control system TCP: adapts sending rate to congestion AQM: feeds back congestion information

Rf (s)

Rb’(s)

l ctyc

)()(1)( tan)(

)()(1-2

tqtttT

wx iid

TCP AQM

Theory

Calren2/Abilene

Chicago

Amsterdam

Geneva

SURFNet

StarLight

WAN in LabCaltech

research & production networks

Multi-Gbps50-200ms delay

Experiment

Students Choe (Postech/CIT) Hu (Williams) J. Wang (CDS) Z.Wang (UCLA) Wei (CS)

Industry Doraiswami (Cisco) Yip (Cisco)

Faculty Doyle (CDS,EE,BE) Low (CS,EE) Newman (Physics) Paganini (UCLA)

Staff/Postdoc Bunn (CACR) Jin (CS) Ravot (Physics) Singh (CACR)

Partners CERN, Internet2, CENIC, StarLight/UI, SLAC, AMPATH, Cisco

People

155Mb/s

slowstart

equilibrium

FASTrecovery

FASTretransmit

timeout

10Gb/s

Implementation

netlab.caltech.edu

Outline

Motivation Network model FAST TCP

Equilibrium Stability Experiments

TCP/IP

Applications

TCP/AQM

Transmission

WWW, Email, Napster, FTP, …

Ethernet, ATM, POS, WDM, …

netlab.caltech.edu

High Energy Physics Large global collaborations

2000 physicists from 150 institutions in >30 countries 300-400 physicists in US from >30 universities & labs

SLAC has 500TB data by 4/2002, world’s largest database Typical file transfer ~1 TB

At 622Mbps: ~ 4 hrs At 2.5Gbps: ~ 1 hr At 10Gbps: ~15min Gigantic elephants!

LHC (Large Hadron Collider) at CERN, to open 2007 Generate data at PB (1015B)/sec Filtered in realtime by a factor of 106 to 107

Data stored at CERN at 100MB/sec Many PB of data per year To rise to Exabytes (1018B) in a decade

netlab.caltech.edu

HEP high speed network

… that must change

netlab.caltech.edu

HEP Network (DataTAG)

NLNLSURFnet

GENEVA

UKUKSuperJANET4ABILEN

ABILENE

ESNETESNET

CALREN

ItItGARR-B

NewYork

FrFrRenater

STAR-TAP

STARLIGHT

Triangle

2.5 Gbps Wavelength Triangle 2002 10 Gbps Triangle in 2003

Newman (Caltech)

netlab.caltech.edu

Performance at large windowsns-2 simulation

10Gbps

capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps; 100 ms round trip latency; 100 flowsJ. Wang (Caltech, June 02)

txq=100 txq=10000

Linux TCP Linux TCP FAST

average utilization

capacity = 1Gbps; 180 ms round trip latency;1 flowC. Jin, D. Wei, S. Ravot, etc (Caltech, Nov 02)

DataTAG Network:CERN (Geneva) – StarLight (Chicago) – SLAC/Level3 (Sunnyvale)

txq=100

netlab.caltech.edu

Outline

TCP/IP

Applications

TCP/AQM

Transmission

netlab.caltech.edu

Congestion Control

~ W packets per RTT Lost packet detected by missing ACK Congestion signal: delay and loss

Source

Destination

data ACKs

netlab.caltech.edu

Congestion control

Example congestion measure pl(t) Loss (Reno) Queueing delay (Vegas)

netlab.caltech.edu

TCP/AQM

Congestion control is a distributed asynchronous algorithm to share bandwidth

It has two components TCP: adapts sending rate (window) to congestion AQM: adjusts & feeds back congestion information

They form a distributed feedback control system Equilibrium & stability depends on both TCP and AQM And on delay, capacity, routing, #connections

xi(t)TCP: Reno Vegas

AQM: DropTail RED REM/PI AVQ

netlab.caltech.edu

Network model

Rb’(s)

TCP Network AQM

lieR lis

lif link uses source if

lieR lislib link uses source if R

netlab.caltech.edu

for every RTT

{ if W/RTTmin – W/RTT < then W ++

if W/RTTmin – W/RTT > then W -- }

queue size

Vegas model

i dtqtxtT

x )()( if )(

else 0ix

i dtqtxtT

x )()( if )(

Gl:))((1

llcl ctypl

Link queueing delay

E2E queueing delay

netlab.caltech.edu

Vegas model

Rb’(s)

TCP Network AQM

netlab.caltech.edu

Outline

TCP/IP

Applications

TCP/AQM

Transmission

netlab.caltech.edu

Methodology

Protocol (Reno, Vegas, RED, REM/PI…)

Equilibrium Performance

Throughput, loss, delay

Fairness Utility

Dynamics Local stability Cost of stabilization

))( ),(( )1(

txtpGtp

txtpFtx

netlab.caltech.edu

Network Links l of capacities cl

Sources sL(s) - links used by source sUs(xs) - utility if source rate = xs

121 cxx 231 cxx

netlab.caltech.edu

Summary: duality model

subject to

)( max0

Flow control problem (Kelly, Malloo, Tan 98)

TCP/AQM Maximize utility with different utility functions

Primal-dual algorithm

))( ),(( )1(

tRxtpGtp

txtpRFtx T

VegasDropTail, RED, REM

Result (L 00): (x*,p*) primal-dual optimal iff 0 ifequality with ** lll pcy

netlab.caltech.edu

Example utility functions

1 )1( :General

log : Vegas

1 :2-Reno

3/2tan23

:1-Reno

netlab.caltech.edu

Game interpretation

lllssss

xpRxxU

)( max0

Source s:

lslslp

Link l:

sllsss tpRUtx )()1( 1'

slslll ctxtptp )()()1(

netlab.caltech.edu

Synchronous convergence

Theorem (L & Lapsley 99)

Provided R has full row rank & Us strictly concave:

Gradient projection algorithm of dual problem

Converges to optimal primal-dual solutions if

Limit point: unique Pareto optimal Nash equilibrium

netlab.caltech.edu

Asynchronous convergence

Sources and links update & compute at different times with different frequencies using delayed info

Theorem (L & Lapsley 99)

Converges in asynchronous environment with smaller

netlab.caltech.edu

Equilibrium of VegasNetwork

Link queueing delays: pl

Queue length: clpl

Sources

Throughput: xi

E2E queueing delay : qi

Packets buffered:

Utility funtion: Ui(x) = i di log x Proportional fairness

iiii dqx

netlab.caltech.edu

Validation (L. Wang, Princeton)

Source rates (pkts/ms)# src1 src2 src3 src4 src51 5.98 (6) 2 2.05 (2) 3.92 (4)3 0.96 (0.94) 1.46 (1.49) 3.54 (3.57)4 0.51 (0.50) 0.72 (0.73) 1.34 (1.35) 3.38 (3.39)5 0.29 (0.29) 0.40 (0.40) 0.68 (0.67) 1.30 (1.30) 3.28

(3.34)

# queue (pkts) baseRTT (ms)1 19.8 (20) 10.18 (10.18)2 59.0 (60) 13.36 (13.51)3 127.3 (127) 20.17 (20.28)4 237.5 (238) 31.50 (31.50)5 416.3 (416) 49.86 (49.80)

netlab.caltech.edu

Methodology

Protocol (Reno, Vegas, RED, REM/PI…)

Equilibrium Performance

Throughput, loss, delay

Fairness Utility

Dynamics Local stability Cost of stabilization

))( ),(( )1(

txtpGtp

txtpFtx

netlab.caltech.edu

Theorem (Low et al, Infocom’02) Reno/RED is locally stable if

Stability: Reno/RED

Rb’(s)

TCP Network AQM

TCP: Small Small c Large N

RED: Small Large delay

netlab.caltech.edu

Stability: scalable control

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Theorem (Paganini, Doyle, L, CDC’01) Provided R is full rank, feedback loop is locally stable for arbitrary delay, capacity, load and topology

netlab.caltech.edu

Stability: Stabilized Vegas

)()(1)( tan)(

1 )()(1-

tTx iid

tqtxi ii

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Theorem (Choe & L, Infocom’03) Provided R is full rank, feedback loop is locally stable if

),( max aTx ii

netlab.caltech.edu

Stability: Stabilized Vegas

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Theorem (Choe & L, Infocom’03) Provided R is full rank, feedback loop is locally stable if

),( max aTx ii

netlab.caltech.edu

Stability: FAST

)()(1)( tan)(

1 )()(1-

tTx iid

tqtxi ii

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Application Stabilized TCP with current routers Queueing delay as congestion measure has right scaling Incremental deployment with ECN

netlab.caltech.edu

Outline

TCP/IP

Applications

TCP/AQM

Transmission

netlab.caltech.edu

Window control algorithm

Theorem (Jin, Wei, L ‘03) In absence of delay Mapping from w(t) to w(t+1) is contraction Global exponential convergence Full utilization after finite time Utility function: i log xi (proportional fairness)

netlab.caltech.edu

Network

(Sylvain Ravot, caltech/CERN)

netlab.caltech.edu

FAST BMPS

Internet2Land Speed

Record

Baltim

unnyvale

#flows

FAST Standard MTU Throughput averaged over > 1hr

netlab.caltech.edu

Aggregate throughput

1 flow 2 flows 7 flows 9 flows 10 flows

Average utilization

88%FAST Standard MTU Utilization averaged over > 1hr

1hr 1hr 6hr 1.1hr 6hr

netlab.caltech.edu

Aggregate throughput

Average utilization

92%FAST Standard MTU Utilization averaged over 1hr

txq=100 txq=10000

SCinet Caltech-SLAC experiments

SC2002 Baltimore, Nov 2002

Acknowledgments

PrototypeC. Jin, D. Wei

TheoryD. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)

Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S.

Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J.

Navratil, J. Williams StarLight: T. deFanti, L. Winkler

Major sponsorsARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF

netlab.caltech.edu

Dynamic sharing: 3 flowsFAST Linux

Dynamic sharing on Dummynet capacity = 800Mbps delay=120ms 3 flows iperf throughput Linux 2.4.x (HSTCP: UCL)

netlab.caltech.edu

Dynamic sharing: 3 flowsFAST Linux

HSTCP STCP

Steady throughput

netlab.caltech.edu

FAST Linux

throughput

STCPHSTCP

Dynamic sharing on Dummynet capacity = 800Mbps delay=120ms 14 flows iperf throughput Linux 2.4.x (HSTCP: UCL)

netlab.caltech.edu

FAST Linux

throughput

STCPHSTCP

Room for mice !

netlab.caltech.edu

Outline

TCP/IP

Applications

TCP/AQM

Transmission

netlab.caltech.edu

Network model

TCP Network AQM

))( ),(( )1(

tRxtpGtp

txtpRFtx T

Reno, Vegas

DT, RED, …

liRli link uses source if 1 IP routing

netlab.caltech.edu

Motivation

cppRxxU

max)( max min

subject to )( maxmax

:Primal

netlab.caltech.edu

Motivation

Can TCP/IP maximize utility?

cppRxxU

max)( max min

subject to )( maxmax

:Primal

Shortest path routing!

netlab.caltech.edu

TCP-AQM/IP

Theorem (Wang, et al 03)

Primal problem is NP-hard

Proof Reduce integer partition to primal problem

Given: integers {c1, …, cn}Find: set A s.t.

netlab.caltech.edu

TCP-AQM/IP

Theorem (Wang, et al 03)

Primal problem is NP-hard

Achievable utility of TCP/IP?

Stability? Duality gap?

Conclusion: Inevitable tradeoff between

achievable utility routing stability

netlab.caltech.edu

Ring networkdestination

Single destination Instant convergence of

TCP/IP Shortest path routing

Link cost = pl(t) + dl

price static

TCP/AQM

IPr(0)

… r(t), r(t+1) , …

routing

netlab.caltech.edu

TCP/AQM

IPr(0)

… r(t), r(t+1) , …

Stability: r ?

Utility: V ?r* : optimal routing

V* : max utility

netlab.caltech.edu

rTheorem (Infocom 2003)

“No” duality gap Unstable if = 0

starting from any r(0), subsequent r(t) oscillates between 0 and 1

link cost = pl(t) + dl

Stability: r ?

Utility: V ?

netlab.caltech.edu

Theorem (Infocom 2003)

Solve primal problem asymptoticallyas

Stability: r ?

Utility: V ?

netlab.caltech.edu

Theorem (Infocom 2003)

large: globally unstable small: globally stable medium: depends on r(0)

Stability: r ?

Utility: V ?

netlab.caltech.edu

General network

Conclusion: Inevitable tradeoff between

achievable utility routing stability

random graph20 nodes, 200 links Achievable

utility

netlab.caltech.edu

FAST TCP: motivation, architecture, algorithms, performance. submitted for publication, July 1, 2003

-release: August 2003Inquiry: fast-support@cs.caltech.edu

FAST Project Review Caltech, Oct 27-28, 2003

FAST TCP Steven Low CS/EE netlab.CALTECH.edu Oct 2003.

Documents