INRIA meeting with NII 1October 7, 2003

Grand Large

ACI MD GdX

Grid eXplorer(GdX)

A research tool for exploring Grid/P2P issuesFranck CappelloINRIA Grand-Large

LRI, University of Paris sud.fci@lri.fr

Research Program ACI “Data Mass”

Pierre SensINRIA Regal

LIP6, University of Paris 6.Pierre.Sens@lip6.fr

Pascale PrimetINRIA RESO

LIP, ENS de Lyon.Pascale.Primet@ens-lyon.fr

Olivier RichardINRIA Apache

ID, IMAG.olivier.richard@imag.fr

Christohpe CérinLARIA,

University of Amiens.cerin@laria.u-picardie.fr

www.lri.fr/~fci/GdX

INRIA meeting with NII 2October 7, 2003

Grand Large

ACI MD GdX

Outline

• Motivating a large scale instrument for Grid

• A large scale instrument for exploring Grid issues

in reproducible experimental conditions

• Concluding remarks

INRIA meeting with NII 3October 7, 2003

Grand Large

ACI MD GdX

Open issues in Grid/P2P

• Security• Data Storage/consultation/movement• Multi users/ Multi applications scheduling• Coordination (virtual, ephemeral infrastructure)• Programming• Fault Tolerance!• Scalability• Performance• Easy/efficient deployment techniques• Application characterization techniques• Etc.

INRIA meeting with NII 4October 7, 2003

Grand Large

ACI MD GdX

Fundamental components of Grids• Systems

– nodes, OS, – distributed systems mechanisms (resource discovery, storage, scheduling,

etc.), – middleware, runtimes,

– Fault (crash, transient)– Workload (multiple users/multiple applications)– Heterogeneity (resource diversity, performance)– Malicious users/behaviors

• Networks – routers, links, topology,– protocols, – Theoretical features: synchronous, pseudo synchronous or asynchronous

– Disconnection– Packet loss– Congestion

Static

Static

Dyn.

Dyn.

INRIA meeting with NII 5October 7, 2003

Grand Large

ACI MD GdXWhat are the current approaches

for studying Systems and Networks?• Theoretical models:

– Scheduling, load balancing, performance, etc.– congestion, routing, packet loss, topology, traffic, etc. Difficulty to model dynamic behaviors and system complexity

• Simulators: – SimGrid, SimGrid2, GridSim, MicroGrid, Bricks– NS, NS2, Cnet, Real, etc.

Strong limitations (scalability, != than execution of real codes, validation)

• Experimental testbed:– For Grid Most testbed are for production, each testeb is specific– Long tradition in Network (Arpanet, Magic, Geant, Renater, VTHD)

Shared testbed not fully decoupled (cost) from production network, experimental conditions difficult to reproduce, representativeness?)

We have no way to test: a) ideas independently, at a significant scale, b) with realistic parameters and behaviors!

INRIA meeting with NII 6October 7, 2003

Grand Large

ACI MD GdX

Case Study 1:XtremWeb-Auger

Understanding the origin of very high cosmic rays:• Aires: Air Showers Extended Simulation

– Sequential, Monte Carlo. Time for a run: 5 to 10 hours (500MhzPC)

PC worker Aires

PC worker

air shower

Server

Internetand LAN

PC Worker

PC Client

Air shower parameter database (Lyon, France) XtremWeb

EstimatedPC number ~ 5000• Trivial parallelism

• Master Worker paradigm

INRIA meeting with NII 7October 7, 2003

Grand Large

ACI MD GdX

Internet

IclusterGrenoblePBS

MadisonWisconsinCondor

U-psudnetwork

LRICondor Pool

AutresLabos

lri.fr

XW ClientXW Coordinator

Application :

AIRES (Auger)

Deployment:

• Coordinator at LRI

• Madison: 700 workers

Pentium III, Linux

(500 MHz+933 MHz)

(Condor pool)

• Grenoble Icluster: 146 workers

(733 Mhz), PBS

• LRI: 100 workers

Pentium III, Athlon, Linux

(500MHz, 733MHz, 1.5 GHz)

(Condor pool)

Case Study 1: XtremWeb-Auger

INRIA meeting with NII 8October 7, 2003

Grand Large

ACI MD GdX

Case Study 1: XtremWeb-Auger

No way to reproduce the same experimental conditions(Configuration BUT ALSO the Dynamic of the system)How to compare fundamental mechanisms then (scheduling), At large Scale (100 000 nodes)?

3000 tasks

INRIA meeting with NII 9October 7, 2003

Grand Large

ACI MD GdXCase study 2: MPICH-V: Fault tolerant MPI for the Grid

Node

Network

Node

Dispatcher

Node

Channel MemoryCheckpoint

server

PC client MPI_send() PC client MPI_recv()

Programmer’s view unchanged:

Problems: 1) volatile nodes (any number at any time) 2) firewalls (PC Grids) 3) non named receptions ( should be replayed in the

same order as the one of the previous failed exec.)

INRIA meeting with NII 10October 7, 2003

Grand Large

ACI MD GdX

~4,8 Gb/s~4,8 Gb/s

~1 Gb/s

Icluster-Imag, 216 PIII 733 Mhz, 256MB/node5 subsystems with 32 to 48 nodes, 100BaseT switch 1Gb/s switch mesh between subsystemsLinux, PGI Fortran or GCC compiler Very close to a typical Building LANSimulate node Volatility

Very close tothe LRI network!

Case study 2: MPICH-V: Fault tolerant MPI for the Grid

INRIA meeting with NII 11October 7, 2003

Grand Large

ACI MD GdX

MPICH-V (CM but no logs)

MPICH-V (CM with logs)

MPICH-V (CM+CS+ckpt)

MPICH-P4

Number of faults during execution

Total executiontime (sec.)

Base exec.without ckpt.and fault

0 1 2 3 4 5 6 7 8 9 10610650700750800850900950

100010501100

~1 fault/110 sec.

MPICH-V vs. MPICH-P4 Execution time with faults(Fault injection)

Interesting but, what about MPICH-V on 10 000 nodes?

Case study 2: MPICH-V: Fault tolerant MPI for the Grid

INRIA meeting with NII 12October 7, 2003

Grand Large

ACI MD GdX

New generation of research tools

• Emulab/NIST Net/Modelnet– Network Emulator with actual PC+ routers emulator (dummynet)– Reproducible experimental conditions – Real applications and protocols

• PlanetLab– A real platform distributed over the

Internet (planet wide)– Real life conditions (not reproducible)– Real applications and protocols

log(abstraction)

mathsimulationemulationlive nk WANiLab

NSSSFNetQualNetJavaSim Mathis

formula

DummyNetEmuLabModelNetWAIL

HENPAbileneCalRENWAILPlanetLabCAIRNNLR

WANiLab

• WANiLab– Cluster with real routers – Reproducible experimental conditions – Real applications and protocols

INRIA meeting with NII 13October 7, 2003

Grand Large

ACI MD GdXExperiments range demandfor Grid/P2P: “GRIDinLAB”

• Virtual Grids: (1 Grid node on 1 PC)Emulation of a Grid/P2P systems at 1:1 scale

•Execute real applications on Grid nodes (possibly slowdown the CPU for heterogeneity emulation)• Use actual routers or emulators (Dummynet)• Inject traces (congestion, workload, fault)

• Emulation of Grids:(10 or 100 Grid nodes on 1 PC)Emulation of a Grid/P2P systems at 10/100:1 scale

• Execute real applications or core of applications• Use network emulators• Inject synthetic traces

• Large scale simulation of Grids:(1000 Grid nodes on 1 PC)Simulation of Large Grid/P2P systems at 1000:1

• Simulate Application• Simulate network• Simulate dynamic conditions

INRIA meeting with NII 14October 7, 2003

Grand Large

ACI MD GdX“GRIDinLAB” in the methodology spectrum

log(abstraction)

log(cost)

mathsimulationemulationlive nt WANiLab

NSSSFNetQualNetJavaSim

Mathis formulaOptimizationLinear modelNonlinear modelStocahstic model

DummyNetEmuLabModelNetWAILHENP

AbileneCalRENWAILPlanetLabCAIRNNLR

WANiLab

Real Wan Routers

Emulation drivensimulation

“GRIDinLAB”

Credits

: WANiLAB

Large scalesimulation

Grid eXplorer

INRIA meeting with NII 15October 7, 2003

Grand Large

ACI MD GdX

Outline

• Motivating a large scale instrument for Grid

• A large scale instrument for exploring Grid

issues in reproducible experimental

conditions

• Concluding remarks

INRIA meeting with NII 16October 7, 2003

Grand Large

ACI MD GdX

Grid eXplorer

A “GRIDinLAB” instrument for CS researchers (Not a production facility)

For• Grid/P2P researcher community• Network researcher community

Addressing specific issues of each domainEnabling research studies combining the 2 domainsEase and develop collaborations between the two communities.

INRIA meeting with NII 17October 7, 2003

Grand Large

ACI MD GdX

Grid eXplorer

An experimentalconditions database orgeneration

An experimentalPlatform:A cluster+HP Network+Software

A tool set for conductingexperiments &measurements,result analysis

INRIA meeting with NII 18October 7, 2003

Grand Large

ACI MD GdX

Grid eXplorer: the big picture

A set of sensors

An experimentalConditions data base

Emulator CoreHardware + Soft

forEmulationSimulation

A set of toolsfor analysis

Validation onReal life testbed

INRIA meeting with NII 19October 7, 2003

Grand Large

ACI MD GdXGrid eXplorer (GdX)

research project1) Build the instrument: “Design and develop, for the

community of Computer Science Researchers, an emulation platform for Large Scale Distributed Systems (Grid, P2P and other distributed systems)”

2) Use the Instrument for “a set of research experiments investigating the impact of Large Scale in distributed systems and especially related to large data sets (security, reliability, performance).”

• 1K CPU cluster (may be only 500 depending on the budget) • configurable network (Ethernet, Myrinet, others?) • configurable OS (kernel, distribution, etc.)• A set of emulation/simulation tools (existing + new ones)• Multi-users• Located/managed by IDRIS

INRIA meeting with NII

INRIA meeting with NII 20October 7, 2003

Grand Large

ACI MD GdX

IMAG, ID (UMR 5132), Laboratoire d’Informatique et Distribution, Université de Grenoble

LaRIA (UPRES EA 2083), Laboratoire de Recherche en Informatique d’Amiens, Université de Picardie Jules Verne

LRI (UMR 8623), Laboratoire de Recherche en Informatique, Université de Paris-sud

LAAS-CNRS (UPR 8001), Laboratoire d'Analyse et d'Architecture des Systèmes

LORIA (UMR 7503), Laboratoire lorrain de recherche en informatique et ses applications

LIP-ENS Lyon (URM 5668), Laboratoire de l'Informatique du Parallélisme

LIFL (ESA 8022), Laboratoire d’Informatique Fondamentale de Lille

INRIA Sophia Antipolis, UNSA, I3S-CNRS

LIP6 (UMR 7606), Laboratoire d'Informatique de Paris 6

LABRI (UMR 5800), Laboratoire Bordelais de Recherche en Informatique

IBCP (UMR5086), Institut de Biologie et Chimie des Protéines

CEA, Direction des Technologies de l'Information (Saclay)

IRISA, Institut de Recherche en Informatique et Systèmes Aléatoires

Laboratories involved in GdX 13 Labs

INRIA meeting with NII 21October 7, 2003

Grand Large

ACI MD GdXAlain Lecluse (IBCP),Alexandre Genoud, (Projet OASIS, INRIA Sophia Antipolis)Antoine Vernois, (IBCP)Arnaud Contes, (Projet OASIS, INRIA Sophia Antipolis)Aurélien Bouteiller, (LRI),Bénedicte Legrand (LIP6)Brice Goglin (doctorant), (INRIA LIP RESO),Brigitte Rozoy (LRI)Cécile Germain (LRI) Christophe Blanchet, (IBCP)Christophe Cérin, (Amiens, Laria)Christophe Chassot, (LAAS-ENSICA),Colette Johnen (LRI)CongDuc Pham, (LIP)Cyril Randriamaro, (LaRIA)Denis Caromel, (Projet OASIS, INRIA Sophia Antipolis)Eddy Caron, (LIP/ENS Lyon), Emmanuel Jeannot, (Loria)Eric Totel (Supélec Rennes)Fabrice Huet, (Projet OASIS, INRIA Sophia Antipolis)Faycal Bouhaf (DEA)(INRIA LIP RESO), Franck Cappello, (LRI)Françoise Baude, (Projet OASIS, INRIA Sophia Antipolis)Frédéric Desprez, (LIP/INRIA Rhône-Alpes), Frédéric Magniette, (LRI)Gabriel Antoniu, (IRISA/INRIA Rennes), George Bosilca, (LRI)Georges Da Costa, (ID-IMAG),Géraud Krawezik (LRI)Gil Utard, (LaRIA)Gilles Fedak, (LRI)Grégory Mounié (ID-IMAG)Guillaume Auriol, (LAAS-ENSICA),Guillaume Mercier, (LaBRI),Guy Bergère, (LIFL, GrandLarge INRIA Futur)Haiwu He, (LIFL, GrandLarge INRIA Futur)Isaac Scherson, (LIFL, GrandLarge, INRIA Futur)Jens Gustedt (LORIA & INRIA Lorraine) Joffroy Beauquier (LRI)Johanne Cohen, (Loria)Kavé Salamatian (LIP6), Lamine Aouad (LIFL, GrandLarge, INRIA Futur)

Laurent Baduel, (Projet OASIS, INRIA Sophia Antipolis)Laurent Dairaine, (LAAS)Luc Bougé, (IRISA/ENS Cachan Antenne de Bretagne), Luciana Arantes (LIP6), Ludovic Mé, (Supélec Rennes)Luis Angelo Estefanel, (ID-IMAG)Marin Bertier (LIP6), Mathieu Goutelle, (KIP)Mathieu Jan, (IRISA) Michel Diaz, (LAAS-ENSICA),Michel Koskas (Amiens, Laria)Nicolas Lacorne, (IBCP)Nicolas Larrieu (LAAS-ENSICA), Nicolas Viovy (CEA-DSM-LSCE)Oleg Lodygensky, (LRI)Olivier Richard (ID-IMAG), Olivier Soyez, (LaRIA)Pascal Berthou, (LAAS-ENSICA),Pascale Primet (LIP), Pascale Vicat-Blanc Primet, (INRIA LIP RESO),Patrick Sénac, (LAAS-ENSICA),Philippe d'Anfray (CEA-DTI/SISC), Philippe Gauron, (LRI)Philippe Owezarski (LAAS)Pierre Fraigniaud, (LRI)Pierre Lemarini, (LRI)Pierre Sens (LIP6 / INRIA),Pierre-André Wacrenier, (LaBRI),Raymond Namyst, (LaBRI),Samir Djilali, (LRI)Sébastien Tixeuil (LRI)Serge Petiton, (LIFL, GrandLarge INRIA Futur)Stéphane Vialle (Supélec) Tanguy Pérennou (LAAS)Thierry Gayraud, (LAAS-ENSICA),Thierry Priol, (IRISA)Thomas Hérault, (LRI)Timur Friedman (LIP6)Vincent Danjean, (LaBRI),Vincent Néri (LRI)

84 members

INRIA meeting with NII 22October 7, 2003

Grand Large

ACI MD GdX

4 Research Topics

The 4 research topics and their leaders:

-Infrastructure (Hardware + system), Olivier Richard (ID-IMAG, Grenoble)-Emulation, Pierre Sens (LIP6, Paris 6)-Network, Pascale Primet (LIP,Lyon)-Applications. Christophe Cérin (Laria, Amiens)

INRIA meeting with NII 23October 7, 2003

Grand Large

ACI MD GdXExperiences Infrastructure Emulation Network ApplicationI.1 Platform X X X XI.2 Virtual Grid X X

I.3 Virt. Techniques X X

I.4 Emul driven Simul X

I.5 Network. X X X

I.6 Heterogeneity emul X

I.7 Communication X

I.8 Internet Emul. X X X

II.1 Engineering tech. X X X

II.2 Mobile objects X X

II.3 Fault tolerance X X

II.4 DHT. X

II.5 Data base X X

II.6 Scheduling X X

II.7 Comm. Optimizat. X

II.8 Data sharing X X

II.9 Uni and multicast X X

II.10 Cellul. automaton X X

II.11 Bioinformatique X

II.12 P2P storage X X

II.13 Reliability X X X

II.14 Security X X X

II.15 NG. Internet X X X

II.16 Grid coupled sys. X

INRIA meeting with NII 24October 7, 2003

Grand Large

ACI MD GdX

Grid eXplorer (GdX) current status:

• First stage: Building the Instrument

– First GdX meeting was on September 16, 2003.

– Hardware design meeting planned for October 15.

– Hardware selection meeting on November 8

– Choosing the nodes (single or dual?)

– Chossing the CPU (Intel IA 32, IA64, Athlon 64, etc.)

– Chossing the experimental Network (Myrinet, Ethernet, Infiniband, etc.)

– Choosing the general experiment production architecture (parallel OS architecture,

user access, batch scheduler, result repositoty)

– Chossing the experimental database harware

– Etc.

INRIA meeting with NII 25October 7, 2003

Grand Large

ACI MD GdX

Example: Nearest Neighbor Scheduling (distribute a task set among a large number of nodes)

•Three phases Self

stabilizing algo (sand

heap):

—Negotiation

—Distribution

—Execution

Anti clock wise

X

Y

X=X - X-Y 2

Y=Y + Y-Y 2

If X>Y

Negotiation rule: Distribution rule:

Tasks and Parametersfollow the negotiationroute, immediately

Execution rule:

Execution startsImmediately whenA local load balanceIs reached

INRIA meeting with NII 26October 7, 2003

Grand Large

ACI MD GdX

Simulation/Emulation tools

INRIA meeting with NII 27October 7, 2003

Grand Large

ACI MD GdXNearest Neighbor Schedulingwith a 3D visualization tool

10K tasks on 900 nodes in mesh

•Negotiation (red movie)

•Distribution (blue movie)

•Execution (green movie)

•Observation results:

—Symmetry for the negotiation phase

—Asymmetry for Distribution and Execution phases.

—Waves

Several hours to get 1 movie parallel simulation is required!

INRIA meeting with NII 28October 7, 2003

Grand Large

ACI MD GdX

Outline

• Motivating a large scale instrument for Grid

• A large scale instrument for exploring Grid issues

in reproducible experimental conditions

• Concluding remarks

INRIA meeting with NII 29October 7, 2003

Grand Large

ACI MD GdX

www.lri.fr/~fci/GdX

INRIA meeting with NII 30October 7, 2003

Grand Large

ACI MD GdX

Grid eXplorer (GdX)

• A long term effort

A medium term milestone: 2 years

for a fully functional prototype

• For Grid and Network researcher communities,

• Many scientific issues (large scale emulation, experimental

conditions injection, distance to reality, etc.)

• A Cluster of 1K CPU + Experimental condition database +

Simulators/Emulators + Visualization tools