The ENEA experience in Materials Science

Massimo Celino

ENEA – C. R. CasacciaVia Anguillarese 30100123 Rome, Italy

Email: massimo.celino@enea.it

Hands on Training School on Molecular and Materials Science Grid ApplicationsICTP - Trieste31 march 2010

Computational MAterials Science and Technology LabCMAST Laboratory : www.afs.enea.it/project/cmast

ENEA staff

M. Celino, S. Giusepponi, M. Gusso, R. Grena, P. Morvillo, G. Gianese, V. Rosato

S. Migliori, S. Raia, G. Bracco, S. Podda, A. Santoro, C. Sciò, A. Rocchi, A. Quintiliani

Molecular modeling activities

Support and maintenance of the ENEA GRID infrastructure

• Based on:

–OpenAFS (shared filesystem)

–LSF Multicluster (resource manager)

ENEA GRID

CASACCIACASACCIA

FRASCATIFRASCATI

S.TeresaS.Teresa

SaluggiaSaluggia

IspraIspra

BOLOGNABOLOGNA

PORTICIPORTICI

TRISAIATRISAIA

BRINDISIBRINDISI

ManfredoniaManfredonia

ENEA-GRID Computational & 3D Centers(more then 3600 core)

90>3000

400

140

30

#CPU/Core

45

New 192 core AMD 6 core

New 224 core

Intel Nehalem

ENEA GRID

Parallel platforms in ENEA GRID

Hardware

• CRESCO HPC, Portici (NA) #125 in Top500 June 2008 (#2 in Italy) 17.1 Tflops, 300 hosts, 2720 cores, InfiniBand 4xDDR

• Other resources: ~100 hosts ~650 cpu

– AIX: IBM SP5 256 cpu (12 p575 1.5GHz, 16 cpu + 1 p595 1.9 Ghz, 64 cpu, 1.5 Tflops); SP4, 96 cpu, Frascati (Rome)

– SGI Altix 350 (IA64) 32 cpu Casaccia (Rome) & Onyx– Cray XD1 24 cpu Casaccia (Rome)– Linux clusters 32/x86_64; Apple cluster; Windows servers

Software

• Commercial codes (fluent, ansys, abaqus, nastran,...)

• Research codes and open source (CPMD, MCNP, OpenFoam,...)

• Computing environment s (Matlab, IDL, ...)

HPC CRESCO HALL

HPC CRESCO PLATFORM

Section 2(MPP)

256 Nodes blades IBM HS21 with 2 Xeon Quad-Core Clovertown E5345 (2.33GHz/1333MHz/8MB L2), 16 GB RAM total 2048 cores Intel ClovertownNew 28 Nodes (224 core) Nehalem 2.4 GHz

Portici LAN

SERVERS GPFS

4 Nodes IBM 3650

IBFC

High speed storage

2 GByte/s160 TByte IBM/

DDN 9550

Backup system

300 TByte IBM Tape Library TS3500 with 4 drives

SERVERS BACKUP

3 Nodes IBM 3650

FCIB

GARR(WAN)

Section 1(Large memory)42 Nodes SMP IBM x3850-M2 with 4 Xeon Quad-Core Tigerton E7330 (32/64 GByte RAM 2.4GHz/ 1066MHz/6MB L2) total 672 cores Intel Tigerton

Section 3(Special)

4 Nodes blades IBM QS21 con 2 Cell BE Processors 3.2 Ghz each.6 Nodes IBM x3755, 8 Core AMD 8222 FPGA VIRTEX54 Nodes IBM x 3755, 8 core AMD 8222 with NVIDIA Quadra FX 4500 X2 4 Nodes Windows 8 core 16 Byte RAM

4x10 Gbits 1000+1000 Mbits

35 Nodes for services

Servers for :• Front-end• installation

s• AFS

New 10 Nodes for tools

Tools:• Faro• Jobrama• Amaca

Interconnections InfiniBand

4XDDR

Fluent

OpenFoam

Heart combustion

Commercial code

User Code

SOME CRESCO SPEEDUP

Open Source

User Code:HEART code for combustion

Born in 1998 with 6 Italian geographic clusters

Main aim • Give to ENEA researchers an unified environment to use in an easy way heterogeneous computers and applications software• Build a homogeneous ICT infrastructure but with a distributed and delegated control • Integrate the ENEA-GRID with National and International GRIDS

Sensor nets

ENEA-GRIDwww.afs.enea.it/project/eneagrid

Data archivesColleagues & 3D

Softwarecatalogs

ComputersApplication

Fabric

Connectivity

Resource

Collective

User

THE ENEA GRID

NETWORKNETWORK

DATA ACQUISITIONDATA ACQUISITION DATA ANALYSISDATA ANALYSIS

Cell Centered Data Base Cell Centered Data Base ““CCDB”CCDB”

IMAGINGIMAGINGINSTRUMENTSINSTRUMENTS

COMPUTATIONALCOMPUTATIONALRESOURCESRESOURCES

MULTI-SCALEMULTI-SCALEDATABASESDATABASES

ADVANCEDADVANCEDCOMPUTERCOMPUTERGRAPHICSGRAPHICS

ENEA GRID INFRASTRUCTURE

AFS Geographical cross platform & File System

AIX SGI SUN CUDA Win Linux Cell Be FPGA

LSF

User programs and commercial codes

LSF multi-cluster as integrator

Graphical User Interface

Application Portal (ICA-Protocol)

ICAWEB

Qu

ality

of s

erv

ice

Mon

itorin

g, A

uto

matic

recovery

Accou

ntin

g

ENEA GRID Architecture

New

Application Portal (NX based)

newnew

FARO (Fast Access to Remote Objects)

JOBRAMA (job monitoring)

AMACA (AFS Memorize And Check Application)

Visual tool able to scan the state of OpenAFS core components and report their condition, while signaling potential criticalities

HOMEMADE ENEA SERVICES

From 1998 to 2009 Citrix for UNIX

AFS Geographical cross platform& File System

Load Leveler LSF

Graphic User Interface LSF multi-cluster as integrator

Telnet

User programs & commercial code

AFS Geographical cross platform& File System

Load Leveler LSF

Graphic User Interface LSF multi-cluster as integrator

Telnet

User programs & commercial code

ICAWEB

From 2008 to …… WEB FreeNX for UNIX

COMMERCIAL SOFTWARE

OpenSource SOFTWARE

ENEA-GRID SOFTWARE EVOLUTION

FARO

FARO – Fast Access to Remote Objects

Built upon the user

FARO is the result of a software integration process where all the components interoperate in order to guarantee web access to remote resources, in a fast, secure and reliable way.

With FARO, researchers need only a reasonably modern web browser (with Java support), and every remote resource they use (from Xterm to desktops of remote Windows machines) will instantaneously reach their location.

Demo accepted in 5th EGEE USER FORUM (due in 11-15 April 2010)

• Backend communicates with Job manager (LSF is currently implemented, but other schedulers can be integrated)• Frontend (web-applications) aggregates data and shows results

JOBRAMA

The SPAGO approach

AIX

IRIXMacOS

CRAY

Altix Any Linux

Also, the service provider retains its autonomy!

Design Principle - Employs one or more proxy to execute grid commands on Wns without gLite.

- Shared file systems allow data sharing between proxies and WNs.

ENEA GRID

Demo accepted in 5th EGEE USER FORUM (due in 11-15 April 2010)

FARO – CPMD

Interface towards the european GRID

Allows to submit CPMD jobs to the GRID.

The job is automatically executed on a GRID cluster supporting CPMD jobs

CMAST Virtual Lab home page

CMAST Virtual Lab home page

ENEA-GRID access

ACTIVITIES

CODES

• Hydrogen storage: MgH2

• Metallic membranes • Organic-inorganic adhesion• Liquid and amorphous AX2 systems

• Icosahedral order in undercooled metals• semiconductors• Tungsten and its alloys• CdS quantum dots• Materials under pressure• etc

• CPMD, CP2K, Quantum Expresso• GROMACS• Homemade for intermetallics and

semiconductors

PARTNERS

• University of Salerno, Rome “Tor Vergata”, Rome “La Sapienza”, Sassari, Camerino, Napoli, Perugia, etc

• CNR Avellino

• University of Lille, Strasbourg, Berlin, Zurich, Bath, Barkatullah

• ST Microelectronics• Ylichron• Numonyx• Tecnomarche• …

• Several ENEA groups

CMAST Virtual Lab

Hydrogen storage

The remarkable ability of magnesium to store significant quantities of hydrogen has fostered intense research efforts in the last years in view of its future applications where light and safe hydrogen-storage media are needed. Magnesium material, characterized by light weight and low cost of production, can reversibly store about 7.7 wt% hydrogen (MgH2).

However, further research is needed since Mg has a high operation temperature and slow absorption kinetics that prevent for the moment the use in practical applications. For these reasons a detailed study of the interface between Mg and MgH2 is needed to characterize the dynamics of hydrogen at the

interface. Thanks to Amelia Montone, ENEA TEPSI Project

Hydrogen desorption

Starting configuration

Mg surface

MgH2 surface

InterfaceLx= 6.21 Å

Ly= 50.30 Å

Lz= 15.10 Å

MgH2:

60 Mg atoms + 120 H atoms

Idrogeno

Magnesio

Mg: 72 atoms

Catalyst, T= 400 K

Hydrogen desorption

T= 700 K

0

100

200

300

400

500

0 200 400 600 800 1000

CRESCO: ~ 8 sec – 150 PE ~ 3 sec – 576 PE

CPMD

Peptide on carbon nanostructures

PEPTIDEPeptide sequence HWSAWWIRSNQS taken from the exeperimental work of Wang et al (Nat Mater 2003)

Modeling of the initial peptide folding by MD simulations in water and energy monimization starting from a totally unfolded atomic structureGROMACS code with OPLS-AA force field

GRAFENE

PEPTIDE

Rigid docking between folded peptide and carbon surface by means of genetic

algorithm

Peptide on carbon nanostructures

Peptide on carbon nanostructures

MD relaxation in water of the best peptide-carbon surface complexes to adjust geometrical matching and to further optimize the binding energy

Car Parrinello molecular dynamics to characterize the electronic structure and better define the contact regions

Peptide on carbon nanostructures

Peptide on carbon nanostructures

Liquid and amorphous AX2 materials

Initial configuration

Final configuration:• few homopolar bonds Si-Si• most of the Si atoms have tetrahedral coordination

Diffusion coefficient

Total neutron structure factor

First Sharp Diffraction Peak in AX2 liquids

500 atoms studied with about 600 cores: 8 sec per time step

Liquid and amorphous AX2 materials

C. Massobrio, M. Celino, P. S. Salmon, R. A. Martin, M. Micoulaut, A. Pasquarello, “Atomic structure of the two intermediate phase glasses SiSe4 and GeSe4”, Phys. Rev. B 79, 174201 (2009)

Even in simple liquids the short range order may differ from that of crystalline solids.

Undercoolings as large as 20% of the melting temperature is observed for a great variety of different metals.

Icosahedral short range order is postulated in the melts to explain the large undercoolings of pure metals.

A commons approach is to characterize the liquid metal by starting from the analysis of the single icosahedra and its deformations.

Icosahedral short range order

13 atoms

55 atoms

147 atoms

The potential energy, by using for example the Lennard-Jones interatomic potential, for the 13 atom icosahedra cluster is about 8% smaller than corresponding fcc and hcp structure.

Icosahedral short range order

∑ +=i

iB

iRc EEE

( ) 21

1/22

−= ∑ −−

j

rrqiB

oijeE ξ( )∑ −−=j

rrpiR

oijAeE 1/

Classical Molecular Dynamics simulations of 4000 copper atoms

Constant temperature and constant pressure simulations

Interatomic potential:

Molecular dynamics

Cleri, Rosato, PRB 48 (1), pp. 22-33, 2003

Experimental melting temperatureTm=1356 K

Temperature of interest:Liquid: T= 1450 K Undercooled: T=1150 K

<5% of difference with experimental densities

MD: D=3.90 10-5 cm2/sec at T=1400 K

Exp: D=3.97 10-5 cm2/sec at Tm=1356 K

Molecular dynamics temperature in the NPT ensemble

Molecular dynamics

Neutron Structure Factor

Room pressure

Pair correlation function

Bond angle distribution

Temperatures MD simulations:Liquid: T= 1450 K Undercooled: T=1150 K

Experimental results:

T= 1142 K ; P= 0.3 GPaT= 1167 K ; P= 0.7 GPaT= 1253 K ; P= 1.4 GPaT= 1333 K ; P= 3.3 GPa

T= 1150 K

ExperimentsMD

Experiments:F. Coppari et al. J. Phys. 121 (2008) 042009

Bond angle distribution

Radial distribution function

High pressure

Common neighbour analysis

Three indeces jkl specifies the local environment of a pair of atoms:

j = the number of neighbors common to both atoms

k = the number of bonds between the common neighbors

l = the number of bonds in the longest continuous chain formed by the k bonds between common neighbors

For example:

555 icosahedral order

421 fcc order

421 and 422 hcp order

Ref: Clarke and Jónsonn (PRE 93)

ICOSAHEDRAL cluster has the central atom characterized by 12 nearest neighbor atoms with 555 environment

N555 = 12

Icosahedral symmetry

N555= 12T= 1150 K

P= 0.3 GPa Perc: 0.40

P= 3.3 GPa Perc: 0.72

T= 1150 K

P= 0.3 GPa Perc: 0.14

P= 3.3 GPa Perc: 0.29

T= 1450 K

N555

Icosahedral symmetry

N555

T= 1150 K

FCC symmetry

Opposite behaviour for the FCC symmetry

M. Celino, F. Coppari, A. Di Cicco, “Pressure effects on icosahedral short range order in undercooled copper” Solid State Science 12 (2010) 179-182