GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid2009-04-03: Napoli
ENEA Grid e CRESCO in GRISU’: uno strumento per la scienza dei materiali
Presentazione: S. Raia, S. MiglioriCollaborazioni : M. Celino, M. Gusso, P. Morvillo, A. Marabotti, L. Cavallo, F. Ragone.
Outline:• ENEA Grid / Cresco infrastructures• Computational Capability• Some examples of Material Science Codes running on ENEA
ENEA Grid infrastructures [1]ENEA
-12 research centers in Italy -A Central Computer and Network Service (INFO)- 6 Computer Centres: Casaccia, Frascati, Bologna,Trisaia, Portici, Brindisi- Multiplatform resources for serial & parallel computation and graphical post-processing.
- Others computer resources in ENEA: departments & individuals
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
Main features:- Access from any kind of connection- Sharing data in world wide areas (geographical file system AFS) - Access to the data from any kind of digital device client - Running any kind of programs - Access to National and International GRIDS
ENEA Grid infrastructures [2]
For each computational site, ENEA manages Local Networks (LAN) whereas the computational centres (WAN) are interconnected via “Consortium GARR” Network
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
O.S. #CPU/Core Gflops …where
AIX >300 3000 Frascati(258), Bologna(24), Portici(18),Brindisi(2)
Linux x86 32/64 >3000 25000 Frascati(140), Casaccia(54), Portici(2700),Trisaia(20), Brindisi(84)
Linux IA64 (Altix) 64 300 Casaccia
IRIX 26 40Frascati(8), Casaccia(4),
Portici(1),Trisaia(8), Brindisi(1), Bologna(5)
Solaris 8 10 Trisaia(4), Casaccia(2), Bologna(2)
Windows 32/64 46 100 Frascati(4), Portici(34),Trisaia(4), Brindisi(4)
Mac OS X 14 60 Frascati(1), Trisaia(13)
ENEA Grid computatonal resources [4]
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
ENEA-GRIDwww.afs.enea.it/project/eneagrid
Data archivesColleagues & 3D
Softwarecatalogs
ComputersApplication
Fabric
Connectivity
Resource
Collective
User
ENEA Grid Architecture & resources integration [3]
*A choice of mature components for reliability and easiness of support and maintenance:
Distributed File System: AFSJob and resources manager: LSF MulticlusterUnified GUI access: Java and Citrix TechnologiesQuality monitoring system: PatrolLicence Servers
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
* Integration with department and individual resources Distributed File System: AFS Licence pool sharing
* Possible integration with other institutions
Portici LAN
Interconnessione InfiniBand4XDDR SERVER GPFS
4 Nodi IBM 3650
IB FC
Sistema Dischi ad alta velocità
2 GByte/s160 TByte
IBM/DDN 9550
Sistema backup300 TByte
IBM Tape Library TS3500 con 4 drive
SERVER BACKUP3 Nodi IBM 3650
FCIB
GARR(WAN)
Sezione 1(Grande Memoria)
42 Nodi SMP IBM x3850-M2 con 4 Xeon Quad-Core Tigerton E7330 (32/64 GByte RAM 2.4GHz/ 1066MHz/6MB L2) per un totale di 672 core Intel Tigerton
Sezione 3(Speciale)
4 Nodi blades IBM QS21 con 2 Cell BE Processors 3.2 Ghz each.6 Nodi IBM x3755, 8 Core AMD 8222 con una scheda FPGA VIRTEX54 Nodi IBM x 3755, 8 core AMD 8222 con schede NVIDIA Quadro FX 4500 X2 4 Nodi windows 8 core 16 Byte RAM
Doppia Intercconnessione a 1 Gbit1 Rete a 1 Gbit di gestione
Sezione 2(Alto Parallelismo)
256 Nodi blades IBM HS21 con 2 Xeon Quad-Core Clovertown E5345 (2.33GHz/1333MHz/8MB L2), 16 GB RAM per un totale di 2048 core Intel Clovertown
4x10 Gbits
2000 Mbits
35 Nodi di ServizioServer di :
•Front-end•insallazione•AFS•…
CRESCO (Portici site) computatonal resources [5]
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
• Multicores Platforms (Cresco Linux, sp AIX) : 8, 16, 32, 48 smp nodes • GPFS / Infiniband for high parallelism codes • Hybrid Parallelism (Cresco smp nodes with IB interconnections)• Distributed parallel Jobs across WAN or heterogeneous platforms (Parallel Jobs arrays (AFS ‘sysname’ mechanism))• Nvidia / Cell processor / FPGA accelerators (…next)
Some Codes running in ENEA env.:CPMD
Gromacs Pc Gamess
Gaussian…. …. others: Abinit, Amber, etc …
ENEA Grid / Cresco Capability for Material Science codes:
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
Gromacs
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles.It is primarily designed for biochemical molecules like proteins and lipids that have a lot of complicated bonded interactions, but it can be used also for research on non-biological systems, e.g. polymers.
Dimeric protein + water + ions: ~77000 atoms
Timescale of simulation: 5 ns
Simulation made on 16 cores on Cresco1,
Time required: 40 h.
Peptide + water + ions: ~2500 atomsTimescale of simulation: 40 nsTime required: 24 h (on 8 cores smp Intel Xeon)
PC GAMESS / Firefly
PC GAMESS/Firefly: computational Chemistry code for ab initio and DFT calculations. Developed for high performance calculation on Intel processors (x86, AMD64, and EM64T).
http://classic.chem.msu.su/gran/gamess/index.html
Used on CRESCO for studying electronic proprieties and absorption spectrum of molecules in the photovoltaic field.
On Cresco: Performances improvement of 370% (16 cores) compared to runs on PC Desktop quad core.
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
modello TDDFT B3LYP/6-311G*
N. cores (MPI procs) Elapsed Time ( min.)
16 313.7
32 200.2
48 142.7
64 125.1
80 114.0
96 103.1
112 93.9
128 90.0
Starting from the basic laws of quantum mechanics, Gaussian is used to predicts the energies, molecular structures, and vibrational frequencies of molecular systems, along with numerous molecular properties derived from these basic computation types.
It is used to study molecules and reactions under a wide range of conditions, including both stable species and compounds which are difficult or impossible to observe experimentally.
On CRESCO is used to study proteins folding: • On shared memory• Embarrassing parallel: array of parallel jobs on 8 cores smp nodes.
Gaussian 03: electronic structure program.
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
8 / 16 MPI procs
8 / 16 MPI procs
8 / 16 MPI procs
.
.
.
.
Parallel_job[1]
Parallel_job[2]
Parallel_job[N] Inp.[N]
Inp.[2]
Inp.[1]
.
.
.
.
.
.
.
.
CPMD: Car-Parrinello Molecular Dynamics
Extented tests in order to: 1- Scaling performances up to 1024 cores 2- Exploiting Dual-level Parallelism 3- Outer loop parallelization: TASKSGROUPS
Total tasks MPI : Threads TASKS
GROUPSIterations / sec
1024 1024:1 8 0.0068
1024 1024:1 NO 0.0060
1024 512:2 8 0.0244
1024 512:2 NO 0.0089
1024 256:4 8 0.0253
1024 256:4 NO 0.0131
It is a plane wave/pseudopotential implementation of Density Functional Theory, particularly designed for ab-initio molecular dynamics.
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
Conclusions …
References in ENEA …
Some Example Codes running in ENEA env.:CPMD
Gromacs Pc GamessGaussian
Some Example Codes running in ENEA env.:CPMD
Gromacs Pc GamessGaussian
Salvatore Raia | GRISU' Open Day su Scienza ed Ingegneria dei Materiali e Grid | 2009-04-03: Napoli
