Ýmir VigfússonGregory ChocklerYoav Tock

Rx for Data Center Communication Scalability

Hussam Abu-LibdehRobert BurgessKen BirmanHaoyuan Li

Mahesh Balakrishnan

IBM Research, Haifa Labs

Cornell University Microsoft Research,Silicon Valley

Useful

– IPMC is fast, and widely supported

– Multicast and pub/sub often used implicitly

– Lots of redundant traffic in data centers

[Anand et al. SIGMETRICS ’09]

Rarely used

– IP Multicast has scalability problems!

IP Multicast in Data Centers

IP Multicast in Data Centers

• Switching hierarchies

IP Multicast in Data Centers

• Switches have limited state space

Switch model (10Gbps) Group capacity

Alcatel-Lucent OmniSwitch OS6850-4 260

Cisco Catalyst 3750E-48PD-EF 1,000

D-Link DGS-3650 864

Dell PowerConnect 6248P 69

Extreme Summit X450a-48t 792

Foundry FastIron Edge X 448+2XG 511

HP ProCurve 3500yl 1,499

IP Multicast in Data Centers

IP Multicast in Data Centers

• NICs also have limited state space

E.g. 16 exact addresses 512-bit Bloom filter

IP Multicast in Data Centers

IP Multicast in Data Centers

• Kernel has to filter out unwanted packets!

• Packet loss triggers further problems

–Reliability layer may aggravate loss

–Major companies have sufferedmulticast storms

IPMC has dangerous

scalability issues

IP Multicast in Data Centers

Key ideas

• Treat IPMC groups as a scarce resource– Limit the number of physical IPMC groups– Translate logical IPMC groups into either physical

IPMC groups or multicast by iterated unicast.

• Merge similar groups together

Dr. Multicast

• Transparent: Standard IPMC interface to user, standard IGMP interface to network.

• Robust: Distributed, fault-tolerant service.

• Optimizes resource use: Merges similar multicast groups together.

• Scalable in number of groups: Limits number of physical IPMC groups.

Dr. Multicast

Dr. Multicast

• Library maps logical IPMC to physical IPMC or iterated unicast

• Transparent to the application

– IPMC calls intercepted and modified

• Transparent to the network

– Ordinary IPMC/IGMP traffic

• Transparent: Standard IPMC interface to user, standard IGMP interface to network.

• Robust: Distributed, fault-tolerant service.

• Optimizes resource use: Merges similar multicast groups together.

• Scalable in number of groups: Limits number of physical IPMC groups.

Dr. Multicast

Dr. Multicast

• Per-node agent maintains global group membership and mapping

– Library consults local agent

• Leader agent periodically computes new mapping (see later).

• State reconciled via gossip

Library Layer Overhead

• Experiment measuring sends/sec at one sender• Sending to r addresses realizes roughly 1/r ops/sec• Insignificant overhead when mapping logical IPMC group to

physical IPMC group.

Network Overhead and Robustness

• Experiment on 90 Emulab nodes

Nodes introduced 10 at a time.Total network traffic grows linearly.

Average traffic received per-node.Robust to major correlated failure

Half of thenodes die

• Transparent: Standard IPMC interface to user, standard IGMP interface to network.

• Robust: Distributed, fault-tolerant service.

• Optimizes resource use: Merges similar multicast groups together.

• Scalable in number of groups: Limits number of physical IPMC groups.

Dr. Multicast

Optimization questions

BLACK

Multicast

Users GroupsUsers Groups

Optimization Questions

Assign IPMC and unicast addresses s.t. Min. receiver filtering Min. network traffic Min. # IPMC addresses … yet deliver all messages to interested parties

Optimization Questions

Assign IPMC and unicast addresses s.t. receiver filtering network traffic # IPMC addresses (hard)M

)1(

• Knob to control relative costs of CPU filtering and of duplicate traffic.

• Both and are part of administrative policy.

M

MCMD Heuristic

Groups in `user-interest’ space

GRAD STUDENTS FREE FOOD

(1,1,1,1,1,0,1,0,1,0,1,1)(0,1,1,1,1,1,1,0,0,1,1,1)

MCMD Heuristic

Groups in `user-interest’ space

Grow M meta-groups around the groups greedily while cost decreases

MCMD Heuristic

Groups in `user-interest’ space

Grow M meta-groups around the groups greedily while cost decreases

MCMD Heuristic

Groups in `user-interest’ space

Unicast

Unicast

224.1.2.3

224.1.2.4

224.1.2.5

• Social:

– Yahoo! Groups

– Amazon Recommendations

– Wikipedia Edits

– LiveJournal Communities

– Mutual Interest Model

Data sets/models

Users Groups

MCMD Heuristic

• Total cost on samples of 1000 logical groups.

– Costs drop exponentially with more IPMC addresses

• Social:

– Yahoo! Groups

– Amazon Recommendations

– Wikipedia Edits

– LiveJournal Communities

– Mutual Interest Model

• Systems:

– IBM Websphere

Data sets/models

Users Groups

MCMD Heuristic

• Total cost on IBM Websphere data set (simulation)

– Negligible costs when using only 4 IPMC addresses

• Transparent: Standard IPMC interface to user, standard IGMP interface to network.

• Robust: Distributed, fault-tolerant service.

• Optimizes resource use: Merges similar multicast groups together.

• Scalable in number of groups: Limits number of physical IPMC groups.

Dr. Multicast

Group Scalability

• Experiment on Emulab with 1 receiver, 9 senders• MCMD prevents ill-effects when the # of groups scales up

IPMC is useful, but has scalability problems

Dr. Multicast treats IPMC groups as scarce and sensitive resources

– Transparent, backward-compatible

– Scalable in the number of groups

– Robust against failures

– Optimizes resource use by merging similar groups

• Enables safe and scalable use of multicast

Dr. Multicast