an introduction to
Distem – http://distem.gforge.inria.fr/ 1 / 17
Imagine you want to build a race car
Distem – http://distem.gforge.inria.fr/ 2 / 17
But you want it to work well
on both dry tracks and wet tracks
Distem – http://distem.gforge.inria.fr/ 3 / 17
You could use a simulation :
model the car and the trackand compute how they would interact
Distem – http://distem.gforge.inria.fr/ 4 / 17
Or you could use a real world experiment :
build a car, and try it on a real circuit
But sometimes, rain is unlikely
Distem – http://distem.gforge.inria.fr/ 5 / 17
Or you could use a real world experiment :
build a car, and try it on a real circuit
But sometimes, rain is unlikely
Distem – http://distem.gforge.inria.fr/ 5 / 17
You could move to another race track . . .But what if you could change the weather ?
That’s what Distem does :Take a platform, and alter it to suit
your experimentation needs
Distem – http://distem.gforge.inria.fr/ 6 / 17
You could move to another race track . . .But what if you could change the weather ?
That’s what Distem does :Take a platform, and alter it to suit
your experimentation needsDistem – http://distem.gforge.inria.fr/ 6 / 17
An emulator for distributed systems
Take your real application
Run it on a cluster
And use Distem to alter the platformso it matches the experimental conditions you need
Heterogeneous nodesLong distance networksFaults, perf. variations
Grid, Cloud, P2P features. . .
Distem – http://distem.gforge.inria.fr/ 7 / 17
Ideal complement to other toolsSimGrid
Toolkit for the simulation ofdistributed applications in
heterogeneous environments
Grid’5000Large-scale platformfor experiments ondistributed systems
1700 nodes, 7000 coresFully reconfigurable
Grid’5000
Distem can run on Grid’5000
Distem – http://distem.gforge.inria.fr/ 8 / 17
Ideal complement to other toolsSimGrid
Toolkit for the simulation ofdistributed applications in
heterogeneous environments
Grid’5000Large-scale platformfor experiments ondistributed systems
1700 nodes, 7000 coresFully reconfigurable
Grid’5000
Distem can run on Grid’5000Distem – http://distem.gforge.inria.fr/ 8 / 17
What can Distem do for you ?
I Introduce heterogeneity in an homogeneous clusterHow does your app perform when some nodes are slower ?
I Emulate complex network topologiesHow does your app perform on a Grid ? on a slower network ?
I Inject faults and performance variations during the experimentHow does your app behave when a node crashes ?When the available CPU time decreases ?When the available network bandwidth increases ?
Distem – http://distem.gforge.inria.fr/ 9 / 17
Introducing heterogeneity
I Distem splits real nodes into several virtual nodes
I with a different number of coresI with different CPU performance
I Each virtual node can be used as a real Linux system
0 1 2 3 4 5 6 7
VN 1 VN 2 VN 3 Virtual node 4
CPU cores
CP
Upe
rform
ance
Distem – http://distem.gforge.inria.fr/ 10 / 17
Emulating network topologies
I Emulate several local networks linked togetherI Control available bandwidth and latency on each interface
n3
n1
n2
←5 Mbps, 10ms
10 Mbps, 5ms→if0
←1 Mbps, 30ms
1 Mbps, 30ms→
if0
←100 Mbps, 3ms
100 Mbps, 1ms→
if0
n4
n5
←4 Mbps, 12ms
6 Mbps, 16ms→if1
←10 Kbps, 200ms
20 Kbps, 100ms→
if0
←200 Kbps, 30ms
512 Kbps, 40ms→ if0
Distem – http://distem.gforge.inria.fr/ 11 / 17
Distem internals
Distem uses modern Linux featuresto steal resources from applications :
LXC, cgroups, cpufreq, iptables, traffic control
Node 1
Node 2
Node 3
Switch
Node 1
Node 2
Node 3
Switch
Distem – http://distem.gforge.inria.fr/ 12 / 17
User interfaces
I Command-line interface – distem command,,, Easy to use/ Harder to automate/ No access to more advanced features
I Ruby library, Easy to automate, Access to all features,// Easy to use if you known Ruby
I REST API, Can be used from any language/ Requires REST knowledge
Distem – http://distem.gforge.inria.fr/ 13 / 17
Overview of a Distem instance
User’s machine
Uses the command line interface,
the Ruby client library
or a REST client
Pnode 2
distemd
Pnode 3
distemd
Coordinator& Pnode 1
Starts and controls other Pnodes
Keeps the global state of the platform
Is the gateway to Pnodes and Vnodes
distemdREST
REST
VnodesDistem – http://distem.gforge.inria.fr/ 14 / 17
Example experiment : SCP vs RsyncI Transfer 50 files (total : 5 MB)I Available bandwidth : 10 MbpsI Varying network latency
0 20 40 60 80 1000
10
20
Emulated latency (ms)
Tran
sfer
time
(s)
SCPRsync
Distem – http://distem.gforge.inria.fr/ 15 / 17
Distem scales to thousands of virtual nodes
I Executing a command with ClusterShell and TakTuk
0 10,000 20,000 30,000 40,0000
200
400
600
Virtual nodes
Exe
cutio
ntim
e(s
)
ClusterShellTakTuk
Distem – http://distem.gforge.inria.fr/ 16 / 17
More information ?
http://distem.gforge.inria.fr/
Distem – http://distem.gforge.inria.fr/ 17 / 17