ACM/IEEE ANCS, Oct 29, 2012

Ji-Yong Shin

Cornell University

In collaboration with

Emin Gün Sirer (Cornell), Hakim Weatherspoon (Cornell) and Darko Kirovski (MSR)

On the Feasibility of Completely Wireless Datacenters

Conventional Datacenter

… Top of Rack Switch

Aggregate Switch

Core Switch

…

Conventional Datacenter

… Top of Rack Switch

Aggregate Switch

Core Switch

…

Conventional Datacenter

…

Going Completely Wireless

• Opportunities

– Low maintenance : no wires

– Low power: no large switches

– Low cost: all of the above

– Fault tolerant: multiple network paths

– High performance: multiple network paths

Which wireless technology?

60GHz Wireless Technology • Short range

– Attenuated by oxygen molecules

• Directional

– Narrow beam

• High bandwidth

– Several to over 10Gbps

• License free

– Has been available for many years

6

Why now? • CMOS Integration

- Size < dime

- Manufacturing cost < $1

[Pinel ‘09]

7 m

m

5 mm

Rx Tx

60 GHz Antenna Model • One directional

– Signal angle between 25° and 45°

– Maximum range < 10 m

– No beam steering

• Bandwidth < 15Gbps

– TDMA (TDD)

– FDMA (FDD)

• Power at 0.1 – 0.3W

How to integrate to datacenters?

Designing Wireless Datacenters

• Challenges

– How should transceivers and racks be oriented?

– How should the network be architected?

– Interference of densely populated transceivers?

Completely Wireless Datacenters • Motivation

• Cayley Wireless Datacenters – Transceiver placement and topology

• Server and rack designs

– Network architecture • MAC protocols and routing

• Evaluation – Physical Validation: Interference measurements

– Performance and power

• Future

• Conclusion

Transceiver Placement: Server and Rack Design

• Rack • Server

Intra-rack space

Inter-rack space

2D View

3D View

3-way switch (ASIC design)

How do racks communicate with each other?

Cayley Network Architecture: Topology

Masked Node Problem and MAC • Most nodes are hidden terminals to others

– Multiple (>5) directional antennae => Masked node problem

– Collisions can occur

• Dual busy tone multiple access [Hass’02] – Out of band tone to preserve channels

– Use of FDD/TDD channels as the tone

12

Cayley Network Architecture: Routing • Geographical Routing • Inter rack

– Diagonal XYZ routing • Turn within rack

– Shortest path turning

• Within dst rack to dst server – Up down to dst story – Shortest path to dst server

1818181818181818181818181818181818181818S 1818181818181818181818181818181818181818 1818181818181818181818181818181818181818

1818181818181818181818181818181818181818D18181818181818181818181818181818181818181818181818181818181818181818181818181818

181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818

1818181818181818181818181818181818181818

Completely Wireless Datacenters • Motivation

• Cayley Wireless Datacenters – Transceiver placement and topology

• Server and rack designs

– Network architecture • MAC protocols and routing

• Evaluation – Physical validation: Interference measurements

– Performance and power

• Future

• Conclusion

Hardware Setup for Physical Validation

• Use of a conservative platform

• Real-size datacenter floor plan setup

• Validation of all possible interferences

Intra-rack communications Inter-rack communications

Physical Validation: Interference Evaluation (Signal angle θ = 15° )

-80

-75

-70

-65

-60

-55

-50

-45

-40

10 9 8 7 6 5 4 3 2 1

RSS

(d

B)

Server ID of Rx

Intra-Rack Space (Tx on server 0)

Error free Default noise

Physical Validation: Interference Evaluation (Signal angle θ = 15° )

-80

-75

-70

-65

-60

-55

-50

-45

-40

15 14 13 12 11 10

RSS

(d

B)

Server ID of Rx on Rack C

Non-Adjacent Inter-Rack Space (Tx on Rack D)

Error free Default noise

Tx: server 2

-80

-75

-70

-65

-60

-55

-50

-45

-40

10 9 8 7 6

RSS

(d

B)

Server ID of Rx on Rack A

Orthogonal Inter-Rack Space (Tx on Rack D)

Error free Default noise

Tx: server 0

-80

-75

-70

-65

-60

-55

-50

-45

-40

15 14 13 12 11 10

RSS

(d

B)

Server ID of Rx on Rack B

Diagonal Inter-Rack Space (Tx on Server 2 of Rack D)

Error free Default noise

Edge of signal: can be eliminated

-80

-75

-70

-65

-60

-55

-50

-45

-40

10 9 8 7 6

RSS

(d

B)

Server ID of Rx on Rack A

Orthogonal Inter-Rack Space (Tx on Rack D)

Error free Default noise

Tx: server 0 Tx: server 1

-80

-75

-70

-65

-60

-55

-50

-45

-40

10 9 8 7 6

RSS

(d

B)

Server ID of Rx on Rack A

Orthogonal Inter-Rack Space (Tx on Rack D)

Error free Default noise

Tx: server 0 Tx: server 1

Tx: server 2

-80

-75

-70

-65

-60

-55

-50

-45

-40

15 14 13 12 11 10

RSS

(d

B)

Server ID of Rx on Rack C

Non-Adjacent Inter-Rack Space (Tx on Rack D)

Error free Default noise

Tx: server 2 Tx: server 3

-80

-75

-70

-65

-60

-55

-50

-45

-40

15 14 13 12 11 10

RSS

(d

B)

Server ID of Rx on Rack C

Non-Adjacent Inter-Rack Space (Tx on Rack D)

Error free Default noise

Tx: server 2 Tx: server 3

Tx: server 4

Potential Interference:

can be blocked using conductor

curtains

Evaluation

• Performance: How well does a Cayley datacenter perform and scale?

– Bandwidth and latency

• Failure tolerance: How well can a Cayley datacenter handle failures?

– Server, story, and rack failure

• Power: How much power does a Cayley datacenter consume compared to wired datacenters

• Simulate 10K server datacenter – Packet level: routing, MAC protocol, switching delay, bandwidth

• Conventional datacenter (CDC)

– 3 Layers of oversubscribed switches (ToR, AS, CS) • (1, 5, 1), (1, 7, 1) and (2, 5, 1) • Latency: 3-6us switching delay • Bandwidth: 1Gbps server

• FAT-tree: Equivalent to CDC (1,1,1) • Cayley wireless datacenter

– 10Gbps bandwidth – 1 Transceiver covers 7 to 8 others – Signal spreading angle of 25° – Low latency Y-switch (<< 1us)

Evaluation Setup

Top of Rack

Aggregate

Core

10

10

2

(1,5,1)

Evaluation Setup

• Uniform random

– Src and dst randomly selected in entire datacenter

• MapReduce

– Src sends msg to servers in same row of rack

– Receiver sends msg to servers in same column of rack

– Receivers send msg to servers inside same pod with 50% probability

Cayley datacenters have the most bandwidth

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Uniform Rand Hops: CDC < 6, Cayley > 11

MapReduce Hops: CDC < 6, Cayley > 8

Maximum Aggregate Bandwidth Normalized to Fat-tree

fat-tree CDC 151 CDC 171 CDC 251 Cayley

Bandwidth • Burst of 500 x 1KB packets per server sent

Latency • Uniform random benchmark

• MapReduce benchmark

Cayley datacenters typically performs the best

0

50

100

150

200

100 200 300 400 500

Late

ncy

(u

s)

Packet Injection Rate (Packet/Second/Server)

Uniform Random (4KB Packet)

fat-tree CDC 251

CDC 171 CDC 151

Cayley

0

2000

4000

6000

8000

10000

100 200 300 400 500

Late

ncy

(u

s)

Packet Injection Rate (Packet/Second/Server)

Uniform Random (16KB Packet)

0

200

400

600

100 200 300 400 500

Late

ncy

(u

s)

Packet Injection Rate (Packet/Second/Server)

MapReduce (4KB Packet)

0

500

1000

1500

2000

2500

100 200 300 400 500

Late

ncy

(u

s)

Packet Injection Rate (Packet/Second/Server)

MapReduce (16KB Packet)

Fault Tolerance

Cayley datacenters are extremely fault tolerant

0

20

40

60

80

100

0 10 20 30 40 50 60 70 80 90

Pre

serv

ed

co

nn

ect

ivit

y (%

)

Failed components (%)

Preserved connectivity among live nodes

Node

Story

Rack

25% 55% 77% 99%

Power Consumption to Connect 10K Servers

• Conventional datacenter (CDC) *

– Depending on the oversubscription rate 58KW to 72KW

• Cayley datacenter – Transceivers consume < 0.3W

– Maximum power consumption: 6KW

• Less than 1/10 of CDC power consumption

Switch Type Typical Power

Top of rack switch (ToR) 176W

Aggregation switch (AS) 350W

Core switch (CS) 611W

* Cost and spec of Cisco 4000, 5000, 7000 series switches

Discussion and Future Work • Only scratched the surface

– How far can wireless datacenters go with no wires?

• Need larger experiment/testbed

– Interference and performance of densely connected datacenter?

• Scaling to large datacenters (>100K servers)?

• Scaling to higher bandwidth (> 10Gbps)?

Conclusion • Completely wireless datacenters can be feasible

• Cayley wireless datacenters exhibit

– Low maintenance

– High performance

– Fault tolerant

– Low power

– Low cost

References • S. Pinel, P. Sen, S. Sarkar, B. Perumana, D. Dawn, D. Yeh,

F. Barale, M. Leung, E. Juntunen, P. Vadivelu, K. Chuang, P. Melet, G. Iyer, and J. Laskar. 60GHz single-chip CMOS digital radios and phased array solutions for gaming and connectivity. IEEE Journal on Selected Areas in Communications, 27(8), 2009.

• Z.J. Hass and J. Deng. Dual busy tone multiple access (DBTMA)-a multiple access control scheme for ad hoc networks. IEEE Transactions on Communications, 50(6), 2002.

• PEPPM. Cisco Current Price List. http://www.peppm.org/Products/cisco/price.pdf, 2012.

27