Date post: | 20-Dec-2015 |
Category: |
Documents |
View: | 217 times |
Download: | 3 times |
High Performance Networking with
Little or No Buffers
Yashar Ganjali
High Performance Networking GroupStanford University
[email protected]://www.stanford.edu/~yganjali
Joint work with:Guido Appenzeller, Ashish Goel, Tim Roughgarden, Nick McKeown
May 5, 2005
May 5, 2005 2
MotivationNetworks with Little or No
Buffers Problem
Internet traffic is doubled every year Disparity between traffic and router
growth (space, power, cost) Possible Solution
All-Optical Networking Consequences
Large capacity large traffic Little or no buffers
May 5, 2005 3
Which would you choose?
DSL Router 1 DSL Router 2
$504 x 10/100 Ethernet
1.5Mb/s DSL connection
1Mbit of packet buffer
$554 x 10/100 Ethernet
1.5Mb/s DSL connection
4Mbit of packet buffer
Bigger buffers are better
May 5, 2005 4
What we learn in school
Packet switching is good Long haul links are expensive Statistical multiplexing allows efficient
sharing of long haul links Packet switching requires buffers Packet loss is bad Use big buffers Luckily, big buffers are cheap
May 5, 2005 5
Statistical Multiplexing
Observations1. The bigger the buffer, the lower the packet loss.2. If the buffer never goes empty, the outgoing line
is busy 100% of the time.
May 5, 2005 6
What we learn in school
Queueing Theory
kkXP
EX
1
11
M/M/1
X
Buffer size
Loss rate
Observations1. Can pick buffer size for a given loss rate.2. Loss rate falls fast with increasing buffer size.3. Bigger is better.
May 5, 2005 7
What we learn in school
Moore’s Law: Memory is plentiful and halves in price every 18 months. 1Gbit memory holds 500k packets
and costs $25.
Conclusion: Make buffers big. Choose the $55 DSL router.
May 5, 2005 8
Why bigger isn’t better
Network users don’t like buffers Network operators don’t like buffers Router architects don’t like buffers We don’t need big buffers We’d often be better off with smaller
ones
May 5, 2005 9
Backbone Router Buffers
Universally applied rule-of-thumb: A router needs a buffer size:
• 2T is the two-way propagation delay• C is capacity of bottleneck line
Context Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines.. Usually referenced to Villamizar and Song: “High
Performance TCP in ANSNET”, CCR, 1994. Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design
CTB 2
CRouterSource Destination
2T
May 5, 2005 10
Review: TCP Congestion Control
Only W packets may be outstanding
Rule for adjusting W If an ACK is received: W ← W+1/W If a packet is lost: W ← W/2
May 5, 2005 11
Review: TCP Congestion Control
Only W packets may be outstanding
Rule for adjusting W If an ACK is received: W ← W+1/W If a packet is lost: W ← W/2
Source Dest
maxW
2maxW
t
Window size
May 5, 2005 12
Buffer Size in the Core
ProbabilityDistribution
B
0
Buffer Size
W
May 5, 2005 13
Backbone router buffers
It turns out that The rule of thumb is wrong for a core
routers today
Required buffer is instead of CT 2n
CT 2
May 5, 2005 14
2T C
n
Simulation
Required Buffer Size
May 5, 2005 15
Validation
Theoretical results validated by: Thousands of ns2 simulations Network lab (Cisco routers) at University
of Wisconsin Stanford University dorm traffic Internet2 experiments
Ongoing work with network operators and router vendors…
May 5, 2005 16
Impact on Router Design
10Gb/s linecard with 200,000 x 56kb/s flows Rule-of-thumb: Buffer = 2.5Gbits
• Requires external, slow DRAM Becomes: Buffer = 6Mbits
• Can use on-chip, fast SRAM• Completion time halved for short-flows
40Gb/s linecard with 40,000 x 1Mb/s flows Rule-of-thumb: Buffer = 10Gbits Becomes: Buffer = 50Mbits
May 5, 2005 17
How small can buffers be?
Imagine you want to build an all-optical router for a backbone network…
…and you can build a few dozen packets in delay lines.
Conventional wisdom: It’s a routing problem (hence deflection routing, burst-switching, etc.)
Our belief: First, think about congestion control.
May 5, 2005 18
TCP with ALMOST No Buffers
Utilization of bottleneck link = 75%
May 5, 2005 19
Two Concurrent TCP Flows
May 5, 2005 20
TCP Throughput with Small Buffers
TCP Throughput vs. Number of Flows
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 200 400 600 800 1000Number of Flows
Th
rou
gh
pu
t
May 5, 2005 21
TCP Reno Performance
Buffer Size = 10; Load = 80%
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000
Bottleneck Capacity Mbps
Th
rou
gh
pu
t
May 5, 2005 22
The chasm between theory and practice
M/M/1
11
X
kkXP
EX
1
= 50%, EX = 1 packet = 75%, EX = 3 packets
= 50%, P[X>10] < 10-3
= 75%, P[X>10] < 0.06
Theory (benign conditions) Practice
Typical OC192 router linecard buffers over 2,000,000 packets
Can we make the traffic arriving at the routersPoisson “enough” to get most of the benefit?
May 5, 2005 23
Ideal Solution
If packets are spaced out perfectly; and The starting times of flows are chosen
randomly; We only need a small buffer for contention
resolution.
May 5, 2005 24
Pacing We need to break bursts
Modify TCP: Instead of sending packets when your receive ACKS send packets with a fixed rate of CWND/RTT.
Rely on network properties:• Access links throttle the flows to low rate• Core:Acess > 1000:1• TCP’s window size is limited today.
If these properties make the flow look like poisson with only 5-10 packets of buffering we can get 70-80% throughput.
May 5, 2005 25
What we know so farabout very small buffers
ArbitraryInjectionProcess
If Poisson Processwith load < 1
CompleteCentralized
Control
Any rate > 0need unbounded
buffers
Theory Experiment
Need buffersize of approx:
O(logD + logW)i.e. 20-30 pkts
D=#of hopsW=window size
TCP Pacing:Results as goodor better than forPoisson
Constant fractionthroughput withconstant buffers
[Leighton 1999]
May 5, 2005 26
CWND: Reno vs. Paced TCP
May 5, 2005 27
TCP Reno: Throughput vs. Buffer Size
May 5, 2005 28
Paced TCP: Throughput vs. Buffer Size
May 5, 2005 29
Early resultsCongested core router with 10 packet buffers.Average offered load = 80%RTT = 100ms; each flow limited to 2.5Mb/s
router
source
server
source
10Gb/s
>10Gb/s >10Gb/s
May 5, 2005 30
Slow access links, lots of flows, 10 packet buffers
router
source
server
source
10Gb/s
5Mb/s 5Mb/s
Congested core router with 10 packet buffers.RTT = 100ms; each flow limited to 2.5Mb/s
May 5, 2005 31
Conclusion
We can reduce 1,000,000 packet buffers to 10,000 today.
We can probably reduce to 10-20 packet buffers: With many small flows, no change needed With some large flows, need pacing in the
access routers or at the edge devices. Need more experiments.
May 5, 2005 32
Experiments
Performance measurement with Small (thousands of packets); and Tiny (tens of packets) buffers
Metrics: Link utilization (goodput/throughput) Drops Buffer occupancy Etc.
Data Gathered for minutes to days High load (50-70% utilization) is better
Thank you!Thank you!
Questions?Questions?