31
CSE 222A: Computer Communication Networks Alex C. Snoeren Lecture 9: Hybrid Networks Thanks: Nathan Farrington
CSE 222A: Computer Communication Networks Alex C. Snoeren
Lecture 9:Hybrid Networks
Thanks: Nathan Farrington
Lecture 9 Overview
● Project discussion ● Overview optical technology
● Helios paper discussion
2 CSE 222A – Lecture 9: Hybrid Networks
The Case for Optics
3
● Electrical Packet Switch
● $400/port ● 10 Gb/s fixed rate ● 12 W/port ● Requires transceivers ● Per-packet switching ● For bursty, uniform traffic
● Optical Circuit Switch
● $400/port ● Rate free ● 240 mW/port ● No transceivers ● 1-ms switching time ● For stable, pair-wise traffic
3 CSE 222A – Lecture 9: Hybrid Networks
Optical Circuit Switch
Lenses Fixed Mirror
Mirrors on Motors
Glass Fiber Bundle
Input 1 Output 2 Output 1
Rotate Mirror 1. Full crossbar switch 2. Does not decode packets 3. Needs external scheduler
4 CSE 222A – Lecture 9: Hybrid Networks
Electrical Packet Switch 1 2 3 4 5 6 7 8
WDM MUX WDM DEMUX
Optical Circuit Switch
Superlink
10G WDM Optical Transceivers
No Transceivers Required 80G
Wavelength Division Mux
5 CSE 222A – Lecture 9: Hybrid Networks
Bisection Bandwidth
10% Electrical (10:1
Oversubscribed)
100% Electrical
Helios Example 10% Electrical + 90%
Optical
Cost $6.3 M
Power 96.5 kW
Cables 6,656
Example: N=64 pods * k=1024 hosts/pod = 64K hosts total; 8 wavelengths
N pods, k-ports each
k switches, N-ports each
Bisection Bandwidth
10% Electrical (10:1
Oversubscribed)
100% Electrical
Helios Example 10% Electrical + 90%
Optical
Cost $6.3 M $62.3 M
Power 96.5 kW 950.3 kW
Cables 6,656 65,536
Example: N=64 pods * k=1024 hosts/pod = 64K hosts total; 8 wavelengths
N pods, k-ports each
k switches, N-ports each
Bisection Bandwidth
10% Electrical (10:1
Oversubscribed)
100% Electrical
Helios Example 10% Electrical + 90%
Optical
Cost $6.3 M $62.2 M $22.1 M 2.8x Less
Power 96.5 kW 950.3 kW 157.2 kW 6.0x Less
Cables 6,656 65,536 14,016 4.7x Less
Example: N=64 pods * k=1024 hosts/pod = 64K hosts total; 8 wavelengths
Fewer Core Switches
N pods, k-ports each
Less than k switches, N-ports each
10G 10G 10G 80G 80G 80G
Pod 1 -> 2: • Capacity = 10G • Demand = 10G • Throughput = 10G Pod 1 -> 3: • Capacity = 80G • Demand = 80G • Throughput = 80G
OCS EPS
Setup a Circuit
Pod 1 Pod 2 Pod 3
9 CSE 222A – Lecture 9: Hybrid Networks
10G 10G 10G 80G 80G 80G
Pod 1 -> 2: • Capacity = 10G • Demand = 10G • Throughput = 10G Pod 1 -> 3: • Capacity = 80G • Demand = 80G • Throughput = 80G
OCS EPS
Traffic Patterns Change
Pod 1 Pod 2 Pod 3
10 CSE 222A – Lecture 9: Hybrid Networks
10G 10G 10G 80G 80G 80G
Pod 1 -> 2: • Capacity = 10G • Demand = 10G 80G • Throughput = 10G Pod 1 -> 3: • Capacity = 80G • Demand = 80G 10G • Throughput = 10G
OCS EPS
Traffic Patterns Change
Pod 1 Pod 2 Pod 3
11 CSE 222A – Lecture 9: Hybrid Networks
10G 10G 10G 80G 80G 80G
Pod 1 -> 2: • Capacity = 10G • Demand = 10G 80G • Throughput = 10G Pod 1 -> 3: • Capacity = 80G • Demand = 80G 10G • Throughput = 10G
OCS EPS
Pod 1 Pod 2 Pod 3
Break a Circuit
12 CSE 222A – Lecture 9: Hybrid Networks
10G 10G 10G 80G 80G 80G
Pod 1 -> 2: • Capacity = 10G • Demand = 10G 80G • Throughput = 10G Pod 1 -> 3: • Capacity = 80G • Demand = 80G 10G • Throughput = 10G
OCS EPS
Pod 1 Pod 2 Pod 3
Setup a Circuit
13 CSE 222A – Lecture 9: Hybrid Networks
10G 10G 10G 80G 80G 80G
Pod 1 -> 2: • Capacity = 80G • Demand = 80G • Throughput = 80G Pod 1 -> 3: • Capacity = 80G • Demand = 80G 10G • Throughput = 10G
OCS EPS
Pod 1 Pod 2 Pod 3
14 CSE 222A – Lecture 9: Hybrid Networks
10G 10G 10G 80G 80G 80G
Pod 1 -> 2: • Capacity = 80G • Demand = 80G • Throughput = 80G Pod 1 -> 3: • Capacity = 10G • Demand = 10G • Throughput = 10G
OCS EPS
Pod 1 Pod 2 Pod 3
15 CSE 222A – Lecture 9: Hybrid Networks
10G 10G 10G 80G 80G 80G
OCS EPS
Pod 1 Pod 2 Pod 3
Pod Switch Manager
Pod Switch Manager
Pod Switch Manager
Circuit Switch Manager
Topology Manager
16 CSE 222A – Lecture 9: Hybrid Networks
Outline of Control Loop1. Estimate traffic demand 2. Compute optimal topology for maximum throughput 3. Program the pod switches and circuit switches
17 CSE 222A – Lecture 9: Hybrid Networks
1. Estimate Traffic DemandQuestion: Will this flow use more bandwidth if we give it
more capacity?
1. Identify elephant flows (mice don’t grow) Problem: Measurements are biased by current topology
2. Pretend all hosts are connected to an ideal crossbar switch
3. Compute the max-min fair bandwidth fixpoint
Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan, Nelson Huang, and Amin Vahdat. Hedera: Dynamic Flow Scheduling for Data Center Networks. In NSDI’10.
18 CSE 222A – Lecture 9: Hybrid Networks
2. Compute Optimal Topology1. Formulate as instance of max-weight perfect matching
problem on bipartite graph 2. Solve with Edmonds algorithm
1
2
3
4
1
2
3
4
Source Pods Destination Pods
a) Pods do not send traffic to themselves b) Edge weights represent interpod
demand c) Algorithm is run iteratively for each
circuit switch, making use of the previous results
19 CSE 222A – Lecture 9: Hybrid Networks
Example: Compute Optimal Topology
20 CSE 222A – Lecture 9: Hybrid Networks
Example: Compute Optimal Topology
21 CSE 222A – Lecture 9: Hybrid Networks
Example: Compute Optimal Topology
22 CSE 222A – Lecture 9: Hybrid Networks
Traditional Network Helios Network
100% bisection bandwidth (240 Gb/s)
23 CSE 222A – Lecture 9: Hybrid Networks
Hardware● 24 servers
◆ HP DL380 ◆ 2 socket (E5520)
Nehalem ◆ Dual Myricom 10G NICs
● 7 switches ◆ One Dell 1G 48-port ◆ Three Fulcrum 10G 24-
port ◆ One Glimmerglass 64-
port optical circuit switch ◆ Two Cisco Nexus 5020
10G 52-port
24 CSE 222A – Lecture 9: Hybrid Networks
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
25 CSE 222A – Lecture 9: Hybrid Networks
Hash Collisions TCP/IP Overhead
190 Gb/s Peak 171 Gb/s Avg
Traditional Network
26 CSE 222A – Lecture 9: Hybrid Networks
160 Gb/s Peak 43 Gb/s Avg
Helios Network (Baseline)
27 CSE 222A – Lecture 9: Hybrid Networks
Port Debouncing
0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0
Time (s)
1. Layer 1 PHY signal locked (bits are detected) 2. Switch thread wakes up and polls for PHY status
• Makes note to enable link after 2 seconds 3. Switch thread enables Layer 2 link
28 CSE 222A – Lecture 9: Hybrid Networks
Without Debouncing
160 Gb/s Peak 87 Gb/s Avg
29 CSE 222A – Lecture 9: Hybrid Networks
Without EDCTh
roug
hput
(Gb/
s)
Time (s)
160 Gb/s Peak 142 Gb/s Avg
Software Limitation
Thro
ughp
ut (G
b/s)
Time (s)
27 ms Gaps
30 CSE 222A – Lecture 9: Hybrid Networks
For Next Class…
● Read and review FairCloud paper
● Keep moving on term projects!
31 CSE 222A – Lecture 9: Hybrid Networks
