1

Towards Rapid GeoScience Model Development

Bill AppelbeVictorian Partnership for Advanced Computing

Director, Software Development - ACcESS bill@vpac.org

May 5th , 2002

2

Outline

Scientific Software – Why is the software not keeping pace with the models? How can we change the way we build scientific

software? Goals for scientific software development

The Snark Project Philosophy, toolkits, off-the-shelf stuff Snark-itechture Snark Finite Element Particle in Cell (PIC) models Snark Particle Models (SPH) Model Integration Other Collaborations

Timelines, deliverables, availability

3

Scientific Software – Requirements for New Models

Increasing demands on scientific software Must be 3D

o 2D just not realistic

3D requires massive increase in computational requirements

o 10,000x10,000 2D grid (108) is 1012 in 3D!

Therefore must be scaleable-parallel

Parallel programming is hard! Only MPI-parallel programs scale to hundreds

of processors But MPI programs must be crafted by hand

4

Scientific Software – Requirements for Geoscience

Increasing demands to be able to rapidly Quickly test new models Incorporate different rheological models

oRequires changing the solver, not just the data

Incorporate new observational dataModels must be linked and

interoperate

5

Scientific Software – What's Wrong With This Picture

Scientific Software Largely been built "from scratch" in Fortran and C Each team or project builds their own specialized

programs (codes)o To meet their own specific modelso Codes start small, but soon grow by "feature-creep"o Evolve to "hero-codes" – maintenance and extension

require heroic efforts, eventually all thrown away "Cut and paste" programming

Little or no reuse or sharing of modules Limited to simple, low-level libraries (e.g. BLAS)

Little or no change in productivity in decades

6

What Can We Learn from Commercial Software?

Commercial software development An order-of-magnitude increase in

productivity since 1990, by oUsing Generators, Frameworks, extensible

librariesoComponent-based technology oFor example, JBuilder

Why has scientific software development not kept pace?

7

What Inhibits Productivityin Scientific Software?

1. Focus on publications and grants No incentive to build reusable software No publication or grant credit for doing

it

2. Lack of expertise in modern software engineering among computational scientists

3. Until recently, no effective large-scale reusable components

8

What Are We Doing About It in Australia?

1. Form a major national earth sciences consortium – ACcESS

All the major computational earth science research groups involved

ANU, UQ, Monash U, U of Melbourne, CSIRO, VPAC, ...

2. Get long-term Federal funding Mostly for software A fair amount of compute grunt

9

Goals for Geoscience Software

Reduce custom code by an order of magnitude

Thus reduce the time to build and test new models from months to days

Scaleable parallel programs Create a framework for combined

particle and grid codes Model interoperability

10

Goals for Geoscience Software - Can it be done?

YES! At VPAC, we have prototypes

demonstrating all the above First versions should be released by

mid-year

11

The Snark Project - Goals

Create a framework for geoscience software Building on top of "state of the art" tools,

components and librariesRebuild and re-engineer existing

geoscience models into this frameworkIntegrate particle and mesh based

models

12

Snark – Philosophy, toolkits, off-the-shelf stuff

"Productivity is measured by the number of lines of code NOT written"

Careful survey and testing of existing scientific software tools & components See vpac.org/snark

The core component technology that Snark uses is PETSc A powerful extensible library of solvers and

support software from Argonne

13

Using PETSc

Strong support group and documentation Large library of linear and non-linear

solvers, grid support functions, etc. Scaleable parallelism built-in! Continuously being extended We are adding particle libraries

The programmer can focus on "setting up the problem", not the algorithm details

The only drawback is the "learning curve"See ... http://www-fp.mcs.anl.gov/petsc

/

14

Snark – Other tools

Off the shelf tools are available for many ancillary functions Graphics, viewing data sets

oVis5D, VU, .. Grid refinement/triangulation

oLaGRIT, Triangle Domain decomposition

oParmetis

All these tools just hook in to PETSc!

15

Automatic Mesh Generation – Triangle

16

Snark –Top-Level Architecture

Almost all grid and particle models fit into one top-level architecture!

There is an isomorphism between irregular grids and particle ensembles

Grid Point ParticleGrid tools that can

partition and refine irregular grids generate grid communication, etc

work equally well for particle ensembles

17

Snark –Top-Level Architecture

Initialize boundary conditions, grids/particle ensembles

Update matrix Update particle

properties

Solve matrixSolve particle

properties

Timestepper & IO

PETScLibrary

Derived from constitutive laws

Partition acrossprocessors

18

The Snark Engine –Detailed Architecture

Main Routine LaGriT/GMVAppl. Init.

Ax = b setup & loading

Vis5D or VU

PETSc

FEM & Shape Fun.

PIC

Unstr. Mesh

Post-ProcessingEvaluations

Prometheusor FEAP

ParMETIS

Visualization

VPAC codeOther code

TAO

19

Snark – Particle in Cell Models

Based on the Ellipsis PIC code A "state of the art" code for creeping viscoelastic fluids Uses a novel Particle in Cell model

o Coupled Eulerian multigrid solver to Lagrangian particles Used to model problems in earth sciences, e.g. mantle

convection and crustal deformation Limitations of Ellipsis

A classic "hero-code"o 30,000 lines of code, several variantso Not parallel, very difficult to adapt and maintain for new

constitutive lawso 2D and 3D versions, but only for regular rectangular grids

• 3D version has taken person-years to adapt

20

Ellipsis

Very efficient for rheological simulations on large timescales

21

Snark – Particle in Cell Models

The Snark version of ellipsis 3,000 lines of code

oAn order of magnitude reduction

Does both viscoelastic and thermally driven convection

Fully parallel and scaleable

A quick Snark video

22

Snark – Particle in Cell Models (cont)

Current limitations of the Snark version of Ellipsis Particle tracking implemented

o But not integration/update of matrix from particle states• Coming soon!

Only 2Do But 3D version for regular rectangular grids fairly trivial

• 3D and 2D supported by PETSc• Estimate delivery June/July of prototype

o 3D irregular grid with arbitrary topology by end of 2002 Programmer must hand code matrix setup/update for

constitutive laws and boundary conditionso Medium-term solution will evolve to setup/update

generators from operator specifications

23

Snark –The Mesh/Particle Continuum

There is a clear continuum between mesh and particle based codes

PETSc’s support of particle codes is in its infancy To move towards unified particle/mesh codes, we

are putting a “pure particle” code into Snark SPH, Smoothed Particle Hydrodynamics Used extensively in astrophysics Ideally suited to: rock fracture, multi-phase mixed fluids The code framework is readily extensible to incorporate

other particle models such as DEM or LMSheath

24

Snark –The Mesh/Particle Continuum

3D parallel version of SPH built, being extended to Allow specification of arbitrary boundary

conditionso Simulation of Tsunamis and dam fracture

Handle multi-fluidso Student working on simulation of pyroclastic flows

Integrate with PETSc framework of PIC codes Extend to DEM codes

A quick video or two

25

Collaboration

Surface process modeling (ANU)Reconstruction software (Monash)LSM (UQ)Griddles

Linking applications across the grid With NO source language changes

26

Couple to Other Models

The Surface models (ANU, UoM);

27

Couple to Other Models

PlatyPlusPlus (ACRC, Monash U)

28

Couple to Other Models

LSM (QUAKES, UQ)

29

Timelines, Deliverables, Availability

Snark V1 will release midyear There is a website with copious

documentation –vpac.org/snark Programmer must hand code matrix

setup/update for constitutive laws and boundary conditions

Medium-term goal is to enable constitutive laws to be input as operator specifications

Key project Be able to model the evolution of the Tibetan

plateau

30

Thank You!

VPAC's HPC Facility – 128 processor Compaq AlphaServer SC