Henri Bal

Vrije Universiteit AmsterdamFaculty of Sciences

DAS-1 DAS-2 DAS-3

Outline

• Introduction• DAS-1, DAS-2, DAS-3 systems• Impact of DAS• Collaborations with

- VL-e- SURFnet- Grid’5000

• New users

Distributed ASCI Supercomputer

• Joint infrastructure of ASCI research school• Clusters integrated in a single

distributed testbed • Long history (10 years)

and continuity

DAS is a Computer Science grid

• Motivation: CS needs its own infrastructure for- Systems research and experimentation- Distributed experiments- Doing many small, interactive experiments

• DAS is simpler and more homogeneous than other grids- Single operating system - “A simple grid that works’’

Typical heterogeneous Grid

DAS-1 (1997-2002)

VU (128) Amsterdam (24)

Leiden (24) Delft (24)

6 Mb/sATM

Configuration

200 MHz Pentium Pro64-128 MB memory2.5 GB local disk

Myrinet interconnectBSDI => Redhat Linux

DAS-2 (2002-2007)VU (72) Amsterdam (32)

Leiden (32) Delft (32)

SURFnet1 Gb/s

Utrecht (32)

Configuration

two 1 GHz Pentium-3s>= 1 GB memory20-80 GB disk

Myrinet interconnectRedhat Enterprise LinuxGlobus 3.2PBS => Sun Grid Engine

DAS-3 Procedure

• NWO proposal (ASCI, VL-e, MultimediaN), Sep’04• Funding from NWO/NCF, Apr’05• Steering group + implementation group• European tender (with Stratix and TUD/GIS)• Selected ClusterVision, Apr’06

• Optical wide-area network bySURFnet (GigaPort-NG project)

VU (85)

TU Delft (68) Leiden (32)

UvA/MultimediaN (46)

UvA/VL-e (40)

DAS-3 dual AMD Opterons4 GB memory250-1500 GB diskMore heterogeneous: 2.2-2.6 GHz Single/dual core nodesMyrinet-10G (exc. Delft)Gigabit EthernetScientific Linux 4Globus, SGE

SURFnet6

10 Gb/s lambdas

Performance

DAS-1 DAS-2 DAS-3

# CPU cores 200 400 792

SPEC CPU2000 INT (1 core) 78.5 454 1445

SPEC CPU2000 FP (1 core) 69.0 329 1858

1-way latency MPI (s) 21.7 11.2 2.7

Max. throughput (MB/s) 75 160 950

Wide-area bandwidth (Mb/s) 6 1000 40000

Outline

• Introduction• DAS-1, DAS-2, DAS-3 systems• Impact of DAS• Collaborations with

- VL-e- SURFnet- Grid’5000

• New users

Usage of DAS

• ~200 users, 34 Ph.D. theses• Clear shift of interest:

Cluster computing

Distributed computing

Grids & peer-to-peer computing

Virtual laboratories for e-Science

Opening workshops• DAS-1 opening (1998):

- only 1 talk (VU) about distributed computing

• DAS-2 opening (2002):- only 1 talk (VU) about local computing

• DAS-3 opening (NOW):- talks about grids, e-Science, optical networks,

international embedding, distributed multimedia computing, P2P/gossiping, co-allocating Grid Scheduler

Impact of DAS

• Major incentive for VL-e 20 M€ BSIK funding- Virtual Laboratory for e-Science

• Collaboration SURFnet on DAS-3- SURFnet provides multiple 10 Gb/s light paths

• Collaboration with French Grid’5000- Towards a European scale CS grid?

Grid’5000

VL-e: Virtual Laboratory for e-Science

Some VL-e results

• VLAM-G workflow system• KOALA scheduler (see Mohamed’s talk)• Zorilla peer-to-peer system (see Drost’s talk)• Java-centric grid computing (see demo)

- Ibis: Grid communication library- Satin: divide&conquer on grids- JavaGAT: Gridlab Application Toolkit

• Applications of Ibis, Satin, JavaGAT:- Automatic grammar learning (UvA)- Protein identification (AMOLF)- Brain image analysis (VUmc)

Dordrecht1

Breda1

Tilburg1

Sub network 4: Blue Azur

Subnetwork 3: Red

Sub network 1: Green

Subnetwork 2: Dark blue

Subnetwork 5: Grey

Zwolle1

Enschede1

Nijmegen1

Dwingeloo1

Amsterdam2

Den Bosch1

Utrecht1

Maastricht1

Eindhoven1

Rotterdam1

Groningen1

Delft

LeidenHilversum

Amsterdam1

Den Haag

SURFnet6

• 4 DAS-3 sites happen to be on Subnetwork 1 (olive green)

• Only VU needs extra connection

Source: Erik-Jan Bos

Adding 10G waves for DAS-3

Delft

Amsterdam1Amsterdam2

Amsterdam3

G5 G5

G5 G5

WSS <-

split

select

select

split

WSS ->

WSS <-

WSS ->

WSS <-

split

select

select

split

WSS ->

WSS <-

WSS ->

G3

Delft

G5G3

G3G3

GMD

GMD

G5

Utrecht

G1

Delft

G1

G1G1

GMD

GMD

G9

G9

G9

G9

GMD

GMD

Hilversum

G6

Delft

G6

G6

G6

GMD

GMD

G5

G5

Leiden

G4

G4

G4

G4

GMD

GMD

Den Haag

G5 G5

G5 G5 G5 G5

G5G5

• Band 5 added at all participating nodes

• WSS added for reconfigurability

• “Spur” to connect VU• Full photonic mesh

possible

• Source: Erik-Jan Bos

• Also see De Laat’s talk

“spur” – hardwired connections

WSS

UvA

Leiden

Delft

MultiMedian

VU

International grid experiments

• Distributed supercomputing on GridLab- Real speedups on a very

heterogeneous testbed

• Crossgrid experiments- N-body simulations (astronomy)- Grid-based interactive

visualization of medical images

Grid’5000

Grenoble

RennesLyon

Toulouse

Sophia

Orsay

Bordeaux

1000

500

500

500

500

500

500

500

500

Grid’5000

• Optical 10 Gb/sec link between Paris and Amsterdam, funded by VL-e

• See Cappello’s talk (and keynote @ CCGrid2007)

Connecting DAS-3 and Grid’5000

Users of DAS-3• ASCI, VL-e• MultimediaN

- See MultimediaN talk + demo• NWO projects

- StarPlane, JADE-MM, GUARD-G, VEARD, GRAPE Grid, SCARIe, AstroStream, CellMath,MesoScale, …..

• NCF projects (off-peak hours)

New users

• DAS is intended primarily for computer scientists from ASCI, VL-e, MultimediaN

• How do we define “Computer Scientist”?• Need a light-weight admission test …

Proposed test

There are only 10 types of people in the world:

Those who understand binary and those who don't.

= Pass as Computer ScientistSource:

Acknowledgements• Lex Wolters• Dick Epema• Cees de Laat• Frank Seinstra• Erik-Jan Bos• Henk van der Vorst

• Andy Tanenbaum• Bob Hertzberger• Henk Sips• Aad van der Steen• Many others

• NWO• NCF• SURFnet• VL-e• MultimediaN• VU, UvA, TUD, Leiden

• ASCI• ClusterVision• TUD/GIS• Stratix

Special acknowledgement ….

• Reasons why DAS is ``a simple grid that works’’

• 1) simple, homogeneous design

• 2) Kees Verstoep