e-Science Technologies in the Simulation of Complex Materials

L. Blanshard, R. Tyer, K. Kleese

S. A. French, D. S. Coombes, C. R. A. CatlowB. Butchart, W. Emmerich – CSH. Nowell, S. L. Price – Chem

eMaterials

Combinatorial Computational Catalysis

Polymorphismprediction of prediction of polymorphspolymorphs – – a drug substance may exist a drug substance may exist as two or more crystalline as two or more crystalline phases in which the phases in which the molecules are packed molecules are packed differently. differently.

Acid Sites in Zeolites

explore which sites are involved in explore which sites are involved in catalysiscatalysis – – used in used in diverse diverse industries including petroleum, industries including petroleum, chemical, polymers, chemical, polymers, agrochemicals, and environmental. agrochemicals, and environmental.

N

CH3

NO2

NO2

H

CH3

OH

H

Combinatorial Computational Catalysis

Acid Sites in Zeolites

explore which sites are involved in explore which sites are involved in catalysiscatalysis – – used in used in diverse diverse industries including petroleum, industries including petroleum, chemical, polymers, chemical, polymers, agrochemicals, and environmental. agrochemicals, and environmental.

Polymorphismprediction of prediction of polymorphspolymorphs – – a drug substance may exist a drug substance may exist as two or more crystalline as two or more crystalline phases in which the phases in which the molecules are packed molecules are packed differently. differently.

N

CH3

NO2

NO2

H

CH3

OH

H

• simulations take too long to run• data are distributed across many sites and systems• no catalogue system• output in legacy text files, different for each program • few tools to access, manage and transfer data• workflow management is manual• licensing within distributed environment

e-Science Issues to Address

Acid Sites in Zeolites

•Determine the extra framework cation position within the zeolite framework.

•Explore which proton sites are involved in catalysis and then characterise the active sites.

•To produce a database with structural models and associated vibrational modes for Si/Al ratios.

•Improve understanding of the role of the Si/Al ratio in zeolite chemistry.

Chabazite: 1T site, 12 Si centres per unit cell, 8 membered ring channels (3.8Å * 3.8Å).

Si/Al – 11 = 4

Si/Al – 5 = 160

Si/Al – 3 = 5760

Si/Al – 2 = 184,320

The number of calculations quickly becomes an issue when realistic Si/Al ratios are considered.

A Si/Al ratio of 2 would require 184,320 calculations at ~100 second each.

= 5120.0 hours = 213 days of cpu time.

The Problem

When substitution of a second Al is considered there are now 4 * (10 * 4) possible structures as symmetry has been broken.

Note this is for a very simple zeolite with 36 ions per unit cell, materials of interest have 296.

A combined MC and EM approach has been developed to model zeolitic materials with low and medium Si/Al ratios. Firstly Al is inserted into a siliceous unit cell and then charge compensate with cations.

MC/EM

-122

91.3

8-1

2290

.88

-122

90.3

8

Initial structures

Latti

ce e

nerg

y (e

V)

-122

95.3

2-1

2295

.22

-122

95.1

2-1

2295

.02

Final structures

Latti

ce e

nerg

y (e

V)

-122

91.3

8-1

2290

.88

-122

90.3

8

Initial structures

Latti

ce e

nerg

y (e

V)

-122

95.3

2-1

2295

.22

-122

95.1

2-1

2295

.02

Final structures

Latti

ce e

nerg

y (e

V)

Name OpSys Arch State Activity LoadAv Mem ActvtyTime

vm1-8@faraday.r IRIX65 SGI Owner Idle 1.192 128 3+03:01:02vm1-14@tyndall.r IRIX65 SGI Unclaimed Idle 0.000 507 0+00:15:09ising2.ri.ac. LINUX INTEL Unclaimed Idle 0.200 501 [?????]vm1-16@strutt1-4 OSF1 ALPHA Owner Idle 1.113 1024 0+0:26:46xp2.ri.ac.uk OSF1 ALPHA Owner Idle 1.113 256 49+12:26:46xp3.ri.ac.uk OSF1 ALPHA Unclaimed Idle 0.000 256 0+00:55:00d8.ri.ac.uk WINNT40 INTEL Unclaimed Idle 0.000 255 0+02:09:45ATLANTIC WINNT51 INTEL Unclaimed Idle 0.008 256 0+01:02:30BABBLE.ri.ac. WINNT51 INTEL Unclaimed Idle 0.252 512 0+00:22:57D500.ri.ac.uk WINNT51 INTEL Owner Idle 0.533 254 0+05:26:06PCDAVIDC.ri.a WINNT51 INTEL Unclaimed Idle 0.000 504 0+03:51:26e-sam.ri.ac.u WINNT51 INTEL Unclaimed Idle 0.001 512 0+03:16:39pcalexey.ri.a WINNT51 INTEL Unclaimed Idle 0.002 256 0+00:35:53

Machines Owner Claimed Unclaimed Matched Preempting

ALPHA/OSF1 18 1 0 1 0 0 INTEL/LINUX 1 0 0 1 0 0 INTEL/WINNT40 1 0 0 1 0 0 INTEL/WINNT51 14 1 0 5 0 0 SGI/IRIX65 22 15 0 7 0 0

Total 56 17 0 15 0 0

RI Condor Pool

We have set up and tested a Condor pool at the RI, which has 50+ heterogeneous nodes from desktop PC’s, machines controlling instruments to main servers of the DFRL.

Name OpSys Arch State Activity LoadAv Mem ActvtyTime

vm1-8@faraday.r IRIX65 SGI Owner Idle 1.192 128 3+03:01:02vm1-14@tyndall.r IRIX65 SGI Unclaimed Idle 0.000 507 0+00:15:09ising2.ri.ac. LINUX INTEL Unclaimed Idle 0.200 501 [?????]vm1-16@strutt1-4 OSF1 ALPHA Owner Idle 1.113 1024 0+0:26:46xp2.ri.ac.uk OSF1 ALPHA Owner Idle 1.113 256 49+12:26:46xp3.ri.ac.uk OSF1 ALPHA Unclaimed Idle 0.000 256 0+00:55:00d8.ri.ac.uk WINNT40 INTEL Unclaimed Idle 0.000 255 0+02:09:45ATLANTIC WINNT51 INTEL Unclaimed Idle 0.008 256 0+01:02:30BABBLE.ri.ac. WINNT51 INTEL Unclaimed Idle 0.252 512 0+00:22:57D500.ri.ac.uk WINNT51 INTEL Owner Idle 0.533 254 0+05:26:06PCDAVIDC.ri.a WINNT51 INTEL Unclaimed Idle 0.000 504 0+03:51:26e-sam.ri.ac.u WINNT51 INTEL Unclaimed Idle 0.001 512 0+03:16:39pcalexey.ri.a WINNT51 INTEL Unclaimed Idle 0.002 256 0+00:35:53

Machines Owner Claimed Unclaimed Matched Preempting

ALPHA/OSF1 18 1 0 1 0 0 INTEL/LINUX 1 0 0 1 0 0 INTEL/WINNT40 1 0 0 1 0 0 INTEL/WINNT51 14 1 0 5 0 0 SGI/IRIX65 22 15 0 7 0 0

Total 56 17 0 15 0 0

RI Condor Pool

But where is PC-CRAC???

-12090

-12070

-12050

ConfigurationsT

E (

eV

)

full

100

50

20

10

5

single

Level of Optimisation

50eV

-12090

-12070

-12050

-12030

-12010

-11990

-11970

-11950

-11930

-11910

-11890

-11870

-11850

TE

(eV

)

full

100

50

20

10

5

single

Level of Optimisation

240eV

MOR

Mordenite – • 1 dimensional channel system• simulation cell contains two unit cells• 296 atoms, with 96 Si centres (referred

to as T sites).• Substituting 8 T sites with 8 Na cations

GulpWinXP

Gulp Files

Workflow

MC_subs

Perlscript

MS Excel

SRB

GulpWinXP

Gulp Files

Workflow II

MC_subs

Perlscript

MS Excel

SRB

Si-zeo structureInteratomic potsInput file

Batch of labelled Gulp files

C++

f90

Scommands

Subset of data in formatted file

Script autobatch sub

Script for cleaning dirs

Extensive use of Condor pools (UCL ~950 nodes in teaching pools). ~150 cpu-years of previously unused compute resource have been utilised in this study. Close collaboration with the NERC e-minerals project has allowed access to this resource.

150,000 calculations have been performed each with varying numbers of particles per simulation box, which means a total of ~75,000,000 particles have been included in our simulations of Mordenite to date.

Condor Stats

Jobs submitted in 1,000 job batches – issue of stability. Shadows – not my game but a pain when Condor Master dies due to too many jobs hitting the queue (guilty feeling as Master was not solely running pool but also being used for science by pool administrator. Maximum number of jobs in queue.

Condor Specifics

Handling of data and analysis becomes RDS.However, keeping the pool full of jobs is also a tedious step when jobs are short, which is the ideal for the UCL pool (re: turning off pool once a day) – drip feeding.

Condor Specifics

Thought in application design is key – many on UCL pool are TOTALLY unsuitable for UCL Condor Pool.

MOR

Mordenite – • 1 dimensional channel system• simulation cell contains two unit cells• 296 atoms, with 96 Si centres (referred

to as T sites).• Substituting 8 T sites with 8 Na cations

0-12085

-12083

-12081

-12079

-12077

-12075

-12073

-12071

-12069

-12067

-12065

ConfigurationsT

ota

l E

ner

gy

(eV

)

5350

5370

5390

5410

5430

5450

5470

5490

5510

5530

5550

Cel

l V

ol.

full_TE

full_Vol

5 per. Mov. Avg. (full_TE)

5 per. Mov. Avg. (full_Vol)

It can be seen that there are two distinct regions, -12079eV to -12076eV and -12075eV to -12073eV, but there is no obvious correlation between total energy and cell volume.

100

100 Configurations

20eV

-12090

-12085

-12080

-12075

-12070

-12065

configurationsT

E

5350

5400

5450

5500

5550

VO

L

TE

VOL

200 per. Mov. Avg. (TE)

200 per. Mov. Avg. (VOL)

However, when 10,000 structures are considered it is clear that the most stable structures correspond to cation placements that do not cause the cell to expand. This requires that the cations sit in the large channel.

0 10000

10000 Configurations

25eV

Energy_eV-12085 -12080 -12075 -12070 -12065 -12060 -12055 -12050 -12...

5350

5400

5450

5500

5550

5600

10000 Configurations

Comparison of Regions

-12079.5eV -12075.04eV

Interrogation ofdatabase

Energy and gnorm within

desired range?

NO

YES

No further analysis

Path to file (held indatabase) selected to

text file

Text file used as inputfor script calling analysis program

for each path.

Analysis

mysql, allows input from a text file, C/C++ program or mysql command line and GUI

Properties: Total energy, cell volume, lattice parameters, T-O distances, T-O-T bond angles, cation-framework oxygen distances, coordination of user specified species etc.

GulpWinXP

Gulp Files

Workflow III

MC_subs

SRB

db

mysql

 

Property Good Bad

Lattice Energy(eV)

< -12070 > -12068

Al-Na average distance (Å)

> 3.6 < 3.4

cell volume(Å3)

< 5420 > 5475

average cation – Oxygen (Å)

> 2.75 < 2.65

 

Building an Ensemble

Validation

Comparison with experiment is very promising showing a large difference in the quality of the fit between ‘good’ set and ‘bad’.

Is the job queued?

Start

Loop through all jobs in database

YES

NO

Queue jobs and update database

Search directory treefor output files

Found files?

Find energyupdate databasechange filename

YES

All jobsqueued?

NO

Set status basedon energy table

Any jobs'IDLE'

NO

YES

EndYESNO

Monitor

Jobs

Analysis

Model/ConfigurationGenerator

Distributed Computing

Portal

SteeringImprove generation / model

strategy

User Input: Structural modelSi/Al, cation types, [H2O] etc.

User Input: Diffraction data, chemical analysis, building units, Si/Al, cation types, [H2O] etc.

Analysis(geometry, energy, fit)

D. Lewis, R. Coates, S. FrenchUCL Chem / RI

Drip Feeding and Interactive Steering using Relational Databases

db

db

db

Workflow IV

SSH CML CML

CML

CML

Workflow service needs to be exposed to outside world as a web service

Since we require new WSDL interfaces for each application it is a perfect opportunity to employ a standard representation for chemical structures.

XML standard in Chemistry is CML (Chemical Markup Language)

We are now doing science that was not possible before the advancements made within e-Science.

Key Achievement

FER

Ferrite – • 2 dimensional channel system• simulation cell contains 115 atoms.• substituting at 4 T sites with 4 Na cations

Only 75 out of 100 configurations optimise

-4400

-4398

-4396

-4394

-4392

-4390

-4388

-4386

1 11 21 31 41 51 61 71

TE

in

eV

1950

1970

1990

2010

2030

2050

2070

2090

2110

Configurations

Vo

l

TE eV

Vol

5 per. Mov. Avg. (Vol)

5 per. Mov. Avg. ( TE eV)

14eV

100 Configurations

Again there are steps in Total Energy and again this time no correlation with volume for the low number of configurations.

-4400

-4398

-4396

-4394

-4392

-4390

-4388

-4386

1 1001 2001 3001 4001 5001 6001 7001 8001

TE

in

eV

1950

1970

1990

2010

2030

2050

2070

2090

2110

2130

2150

Configurations

Vo

l

TE

Vol

200 per. Mov. Avg. (TE)

200 per. Mov. Avg. (Vol)

15eV

10000 Configurations

However, this time when 10,000 structures are considered there are no clear steps in the volume. The volume still increases with decreasing stability but this is due to cell expansion caused by Al to Al interactions.

Only 7500 out of 10000 optimise

Comparison of Regions

Comparison of Regions

MFI

ZSM5 – • 3 dimensional channel system• simulation cell contains 292 atoms• substituting at 4 sites with 4 Na cations

-12215

-12214

-12213

-12212

-12211

-12210

-12209

-12208

-12207

-12206

-12205

1 1001 2001 3001 4001

TE

in

eV

5250

5270

5290

5310

5330

5350

5370

5390

Configurations

Cel

l V

olu

me

TE

Vol

200 per. Mov. Avg. (Vol)

200 per. Mov. Avg. (TE)

10eV

10000 Configurations

There is a step in Total Energy but this time only one and from then the trend is smooth.

When confirmed the lowest energy positions of Al the cation is exchanged for a proton and again energy minimised.

This method will allow us to construct realistic models of low and medium Si/Al zeolites. Such structures can be used for further simulations and aid the interpretation of experimental data.

What Next

BaTiO3

Solid Solutions

Solid Solutions

BaSrTiO3

Solid Solutions

SrTiO3

• upload files as part of workflow to SRB• generate metadata• upload extracted data from files• more extensive use of CML

Ongoing and Future Work

We are now doing science that was not possible before the advancements made within e-Science.

Key Achievement

1. First use of CML schema for defining Web Service port types.2. Calculation of 50,000 configurations of zeolite Mordenite (24,000,000 particles) to gain insight into structure when a realistic ratio of Al substitution is included in model.3. Successfully exposed Fortran codes as OGSI Web Services - prototype application deployed on 80 nodes. The prototype computational polymorph application is being ported to a larger production machine.4. First use of BPEL standard for orchestrating web services in a Grid application.5. Open Source BPEL implementation in development enabling late binding and dynamic deployment of large computational processes.6. Integration of OGSI and BPEL with Sun Grid Engine.7. Development of Graphic User Interface for polymorph application - connects to relational database via EJB interface.8. Infrastructure for metadata and data management9. SRB and dataportal are already being used to hold datasets and being used for transferring the data between different scientists and computer applications.10. Implementation of Condor pool at Ri.

Achievements To Date

Polymorph Prediction

Different crystal structures of a molecule are called polymorphs.

Polymorphs may have considerably different properties(e.g. bioavailability, solubility, morphology)

Polymorph prediction is of great importance to the pharmaceutical industry where the discovery of a new polymorph during production or storage of a drug may be disastrous

Drug molecules are often flexible and this makes the polymorph prediction process more challenging…

N

CH3

NO2

NO2

H

CH3

OH

H

MOLPAK Generation of ~6000 densely packed crystal

structures using rigid molecular probe

DMAREL Lattice energy optimisation

For flexible molecules: conformational optimisation

n feasible rigid molecular probes representing energetically plausible conformers

Data : Unit cell volume, density, lattice energy

Restricted number of structures selected crystal structures and properties stored in

Database

Morphology

n times

n = number of conformers

Polymorph Prediction Workflow

Store data files from simulations in the Store data files from simulations in the Storage Resource BrokerStorage Resource Broker

Storage Resource Broker

We are now doing science that was not possible before the advancements made within e-Science.

Key Achievement