Grid eXplorer (GdX) An Instrument for eXploring the GRID

F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposalF. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

Grid eXplorer Grid eXplorer (GdX)(GdX)

AnAn InstrumentInstrument for eXploring the for eXploring the GRIDGRID

F. Cappello, O. Richard, P. SensF. Cappello, O. Richard, P. Sens(([email protected]@lri.fr, , [email protected]@imag.fr, , pierre.senspierre.sens

@[email protected]))

LRI, ID-Imag, Lip6, etc.

mailto:[email protected]





What we know about What we know about Grid/P2P?Grid/P2P?

Infrastructure features:Infrastructure features:

• Large scale distributed systemsLarge scale distributed systems

• Dynamic infrastructures (network, nodes)Dynamic infrastructures (network, nodes)

• Dynamic workloadsDynamic workloads

• Heterogeneous (parallel & uniproc. nodes)Heterogeneous (parallel & uniproc. nodes)

• High Computation/Communication perf. High Computation/Communication perf. RatioRatio



Application features:Application features:

• Large data sets (data bases, physics Large data sets (data bases, physics

instruments, generation of very large results)instruments, generation of very large results)

• Huge computations (in terms of #operations)Huge computations (in terms of #operations)

• Long run timesLong run times

• Large number of data operations (movements, Large number of data operations (movements, transactions, etc.)transactions, etc.)



Utilization:Utilization:

• A large number of users (researcher, industry, A large number of users (researcher, industry, individual, etc.) individual, etc.)

• A large set of applications (research, business)A large set of applications (research, business)

• High throughput (file download, multi-High throughput (file download, multi-parameters, computation, etc.)parameters, computation, etc.)

• Real time (Sensors, Entertainment, HPC, etc.)Real time (Sensors, Entertainment, HPC, etc.)


What are open issues?What are open issues?• Security (GSI, CAS)Security (GSI, CAS)• Data Storage/consultation/movementData Storage/consultation/movement• Multi users/ Multi applications schedulingMulti users/ Multi applications scheduling• Coordination (virtual, ephemeral infrastructure)Coordination (virtual, ephemeral infrastructure)• ProgrammingProgramming• Fault Tolerance!Fault Tolerance!• ScalabilityScalability• PerformancePerformance• Easy/efficient deployment techniquesEasy/efficient deployment techniques• Application characterization techniquesApplication characterization techniques• Etc.Etc.


What is Grid today?What is Grid today?

• Middleware: Globus, Legion, Netsolve, Unicore, Middleware: Globus, Legion, Netsolve, Unicore, DIET, Condor, XtremWeb, Boinc, NWS DIET, Condor, XtremWeb, Boinc, NWS they are actually working!they are actually working!

• Testbed: DataGRID, TeraGRID, Etoile, Grads, Testbed: DataGRID, TeraGRID, Etoile, Grads, XW, Boinc XW, Boinc Difficult to build (debug, human factor, etc.)Difficult to build (debug, human factor, etc.)

• Dedicated applications (SETI, Kazaa, Jabber, Dedicated applications (SETI, Kazaa, Jabber, etc.)etc.) they are working already (at large scale!), they are working already (at large scale!), BUT they address much less issues!BUT they address much less issues!


Why Grid is so difficult?Why Grid is so difficult?

Grids are very complex systems (we still have Grids are very complex systems (we still have problems with large scale parallel computers!, problems with large scale parallel computers!, we have less control on Grid resources! ) we have less control on Grid resources! )

Many issues are addressed simultaneouslyMany issues are addressed simultaneously

We need a methodology enabling the study We need a methodology enabling the study of Grid issues, independently but realistically. of Grid issues, independently but realistically.


What are the current What are the current approaches?approaches?• Simulators: SimGRID, MicroGRID, etc.Simulators: SimGRID, MicroGRID, etc.

they have strong limitations (scalability,they have strong limitations (scalability,!= than execution of real codes, validation)!= than execution of real codes, validation)

• Experimental testbed (is there any?)Experimental testbed (is there any?) Most testbed are for production, each testeb is Most testbed are for production, each testeb is

specific, representativeness?specific, representativeness?

We have no way to test: We have no way to test: ideas independently, at a significant scale,ideas independently, at a significant scale, with realistic parameters and behaviors! with realistic parameters and behaviors!


What is missing?What is missing?• A full fledge scientific environmentA full fledge scientific environment

(reproducible realistic experimental conditions)(reproducible realistic experimental conditions)

Probes measuring the performance of real resources Probes measuring the performance of real resources and networks (Ganglia, NWS, la grenouille)and networks (Ganglia, NWS, la grenouille)

Fully experimental testbed, (GRID 5000 would Fully experimental testbed, (GRID 5000 would remove this lack)remove this lack)

Not enough, we need Not enough, we need instrumentsinstruments with with parametrisable reproducible experimental parametrisable reproducible experimental conditions conditions


MethodologyMethodology

• When physicists can’t measure a phenomenonWhen physicists can’t measure a phenomenon because a) the item to measure is not because a) the item to measure is not

reachable or b) the phenomenon is hidden by reachable or b) the phenomenon is hidden by others, others,

They build instruments!They build instruments!

And they observe…And they observe…


Grid eXplorerGrid eXplorer

An experimentalconditions database orgeneration An experimental

hardwareplatform

A tool set for conductingexperiments &measurements


Grid eXplorer:Grid eXplorer:Analogy with physic Analogy with physic

instrumentsinstrumentsA set of sensorsInside real platform

A database ofReal platform

A Hardware Platform forEmulationSimulation

A tool set for ObservationResults Analysis

Validation on real platforms


Grid Grid eXeXplorer (GdX): plorer (GdX): InstrumentInstrument for Grid exploration: for Grid exploration:

• An instrument for exploring Grid middleware, An instrument for exploring Grid middleware,

algorithms, performance and applications under algorithms, performance and applications under

reproducible experimental conditionsreproducible experimental conditions

• A tool set for emulation/simulation of large scale A tool set for emulation/simulation of large scale

distributed systemsdistributed systems

• 1K 1K CPUCPU clusters + clusters + configurableconfigurable network & OS network & OS

• For Grid, P2P, etc.For Grid, P2P, etc.

• Potentially connected to Grid testbedsPotentially connected to Grid testbeds



Grid eXplorer (GdX): Grid eXplorer (GdX): eXperimental conditions databaseeXperimental conditions database

• A set of sensors (Nodes, Networks): A set of sensors (Nodes, Networks):

Academic Networks (x K nodes, GRID 5000)Academic Networks (x K nodes, GRID 5000)

ADSL (la grenouille ADSL (la grenouille 60 K nodes) 60 K nodes)

• A common format for tracesA common format for traces

• A tool set for accessing, managing tracesA tool set for accessing, managing traces

• Tools for trace analysisTools for trace analysis



Grid eXplorer (GdX): Grid eXplorer (GdX): Hardware platform for eXperiments:Hardware platform for eXperiments:

• 1K CPU clusters 1K CPU clusters

• configurable network configurable network

• configurable OSconfigurable OS

• Multi-usersMulti-users

• Potentially located/managed by IDRIS or CEAPotentially located/managed by IDRIS or CEA



Grid eXplorer (GdX): Grid eXplorer (GdX): Tool set:Tool set:

• Emulators (folding 10K virtual nodes on 1K nodes)Emulators (folding 10K virtual nodes on 1K nodes)

• Parallel simulators (difficult!)Parallel simulators (difficult!)

• Virtual GRID environment (1k virtual nodes on 1k Virtual GRID environment (1k virtual nodes on 1k

nodes)nodes)

• Measurement tools,Measurement tools,

• Visualization tools.Visualization tools.



Grid eXplorer (GdX): Grid eXplorer (GdX): eXperiments:eXperiments:

• Real scale pseudo-emulation (1:1 scale)Real scale pseudo-emulation (1:1 scale)

• Large scale emulation (10:1 scale)Large scale emulation (10:1 scale)

• Large scale simulation (100:1 scale)Large scale simulation (100:1 scale)

• Application test (bottleneck discovery, performance Application test (bottleneck discovery, performance

evaluation, optimization)evaluation, optimization)

• Infrastructure test (bottleneck discovery, performance Infrastructure test (bottleneck discovery, performance

evaluation, optimization)evaluation, optimization)

• Connection to Grid 5000, International Grids, etc. Connection to Grid 5000, International Grids, etc.



Grid eXplorer (GdX): Grid eXplorer (GdX): Instrument related projects:Instrument related projects:

• Instrument projectsInstrument projects– Cluster management (ID Imag),Cluster management (ID Imag),– Sensor event data base (LRI-fci, la grenouille),Sensor event data base (LRI-fci, la grenouille),– Event analysis (ID Imag),Event analysis (ID Imag),– Emulators (LRI-fci, etc.),Emulators (LRI-fci, etc.),– Experimental conditions injection,Experimental conditions injection,– Result visualization (???),Result visualization (???),– Result analysis,Result analysis,– Validation methodology,Validation methodology,– User validation (physicists, biologists, chUser validation (physicists, biologists, cheemists, etc.)mists, etc.)



Grid eXplorer (GdX): Grid eXplorer (GdX): Grid/P2P Grid/P2P rresearesearchch projects: projects:

• GridGrid– DataGRDataGRID technology (???)ID technology (???)– GRIDRPC (LIP, Desprez?)GRIDRPC (LIP, Desprez?)– Code coupling (IRISA?)Code coupling (IRISA?)

• P2PP2P– Large scale desktop Grids (LRI-fci)Large scale desktop Grids (LRI-fci)– Large scale storage (Laria, Irisa?)Large scale storage (Laria, Irisa?)– Large scale data consistLarge scale data consisteencncyy (Lip6-P. s (Lip6-P. sensens))– Large scale fault tolerance (LRI-Beauquier)Large scale fault tolerance (LRI-Beauquier)

• NetworkNetwork– Emulation of latencyEmulation of latency, , pacpackket losset loss, bandwidth, bandwidth– Emulation of topologyEmulation of topology



Grid eXplorer (GdX): Grid eXplorer (GdX): User applications:User applications:

• Bio Grid (Genomining, IBBMC, etc.)Bio Grid (Genomining, IBBMC, etc.)

• Medi GRID (???)Medi GRID (???)

• Geo GRID (IFP, CGG)Geo GRID (IFP, CGG)

• Power GRID (EDF)Power GRID (EDF)

• Industry GRID (EADS, Alcatel Space)Industry GRID (EADS, Alcatel Space)

• Academic GRID (Orsay, Berkeley, SDSC, Suitzerland, etc.)Academic GRID (Orsay, Berkeley, SDSC, Suitzerland, etc.)

• Industry Partners (GridXpert???, IBM???)Industry Partners (GridXpert???, IBM???)



Grid eXplorer (GdX)Grid eXplorer (GdX)Project organisationProject organisation

• A Virtual laboratoryA Virtual laboratory

• A director (or a group),A director (or a group),

• A scientific director, an administrative directorA scientific director, an administrative director

• A scientific council (with physicists),A scientific council (with physicists),

• A Researcher groupA Researcher group

• A group of engineers, A group of engineers,

• A user groupA user group

• A set of scientific eventsA set of scientific events



Grid eXplorer Grid eXplorer GdXGdX

• A long term effortA long term effort

– A medium term milestone: 4 yearsA medium term milestone: 4 years

A fully functional prototypeA fully functional prototype

• Many scientific issues (large scale emulation, experimental Many scientific issues (large scale emulation, experimental

conditions injection, distance to reality, etc.)conditions injection, distance to reality, etc.)

• A tool for Grid users or potential usersA tool for Grid users or potential users

• A tool for Grid/P2P developers A tool for Grid/P2P developers

Date post:	01-Jan-2016
Category:	Documents
Upload:	cairo-walker
View:	30 times
Download:	1 times

Grid eXplorer (GdX) An Instrument for eXploring the GRID

Documents