The Pencil Code: multi-purpose and multi-user maintained

The Pencil Code: multi-purpose The Pencil Code: multi-purpose and multi-user maintainedand multi-user maintained

Axel Brandenburg

(Nordita, Copenhagen)

Axel Brandenburg: The Pencil-Code 2

OverviewOverview

• Pencil formulation (advantages, headaches)

• Structure of code, cvs maintainence

• High-order schemes, tests

• Peculiarities on big linux clusters

• Online data processing


Pencil CodePencil Code

• Started in Sept. 2001 with Wolfgang Dobler

• High order (6th order in space, 3rd order in time)

• Cache & memory efficient

• MPI, can run PacxMPI (across countries!)

• Maintained/developed by many people (CVS!)

• Automatic validation (over night or any time)

• Max resolution so far 10243


Range of applicationsRange of applications

• Isotropic turbulence– MHD (Nils), passive scalar, cosmic rays

• Stratified layers– Convection, radiative transport (Tobi)

• Shearing box• MRI (Nils), Planetesimals (Anders), Interstellar (Tony)

• Sphere embedded in box– Fully convective stars (Dobler), geodynamo (McMillan)


Pencil formulationPencil formulation

• In CRAY days: worked with full chunks f(nx,ny,nz,nvar)

– Now, on SGI, nearly 100% cache misses

• Instead work with f(nx,nvar), i.e. one nx-pencil• No cache misses, negligible work space, just 2N• Communication before sub-timestep• Then evaluate all derivatives, e.g. call curl(f,iA,B)

– Vector potential A=f(:,:,:,iAx:iAz), B=B(nx,3)


A few headachesA few headaches

• All operations must be combined– Curl(curl), max5(smooth(divu)) must be in one go

• rms and max values for monitoring– call max_name(b2,i_bmax,lsqrt=.true.)– call sum_name(b2,i_brms,lsqrt=.true.)

• Similar routines for toroidal average, etc

• Online analysis (spectra, slices, vectors)


CVS maintainedCVS maintained

• pserver (password protected)– Public (check-out only), private (ci/co, 20 people)

• Set of 10 test problems

• Nightly auto-test (different machines, web)

• Before check-in: run auto-test yourself

• Mpi and nompi dummy module for single processor machine (or use lammpi on laptops)


Switch modulesSwitch modules

• magnetic or nomagnetic (e.g. just hydro)

• hydro or nohydro (e.g. kinematic dynamo)

• density or nodensity (burgulence)

• entropy or noentropy (e.g. isothermal)

• radiation or noradiation (see Tobi’s talk)

• dustvelocity or nodustvelocity (planetesimals)


Features, problemsFeatures, problems

• Namelist (can freely introduce new params)

• Upgrades forgotten on no-modules (auto-test)

• SGI namelist problem (see pencil FAQs)

10

Pencil Code check insPencil Code check ins


High-order schemesHigh-order schemes

• Alternative to spectral or compact schemes– Efficiently parallelized– No transpose necessary

• 6th order central differences in space• Non-conservative scheme

– Allows use of logarithmic density and entropy– Copes well with strong stratification and

temperature contrasts


High-order spatial schemesHigh-order spatial schemes

x

fffffff iiiiii

i 60

945459 321123'

2321123''

180

227270490270272

x

ffffffff iiiiiii

i

Main advantage: low phase errors


Wavenumber characteristicsWavenumber characteristics


Higher order – less viscosityHigher order – less viscosity


Less viscosity – also in Less viscosity – also in shocksshocks


High-order temporal schemesHigh-order temporal schemes

),( 111 iiiii utFtww

Main advantage: low amplitude errors

iiii wuu 1

3)()(

0 , uuuu nn

2/1 ,1 ,3/1

1 ,3/2 ,0

321

321

1 ,2/1

2/1 ,0

21

21

1

0

1

1

3rd order

2nd order

1st order

2N-RK3 scheme (Williamson 1980)


Shock tube testShock tube test


Hydromagnetic turbulence Hydromagnetic turbulence and subgrid scale models?and subgrid scale models?

• Want to shorten diffusive subrange– Waste of resources

• Want to prolong inertial range– Focus of essential physics

• Reasons to be worried about hyperviscosity– Shallower spectra– Wrong amplitudes of resulting large scale fields

42

2


Simulations at 512^3Simulations at 512^3

Withhyperdiffusivity

Normaldiffusivity

Biskamp & Muller (2000)


256 processor run at 1024256 processor run at 102433


MHD equationMHD equation

JBuA

t

fuuBJu

)(1

lnD

D3122

sc

t

utD

lnD

AB

BJ

Induction

Equation:

Magn.Vectorpotential

Momentum andContinuity eqns


Vector potentialVector potential

• B=curlA, advantage: divB=0• J=curlB=curl(curlA) =curl2A• Not a disadvantage: consider Alfven waves

z

uB

t

b

z

bB

t

u

00 and ,

uBt

a

z

aB

t

u02

2

0 and ,

B-formulation

A-formulation 2nd der onceis better than1st der twice!


Wallclock time versus processor #


Sensitivity to layout onSensitivity to layout onLinux clustersLinux clusters

yprox x zproc

4 x 32 1 (speed)

8 x 16 3 times slower

16 x 8 17 times slower

Gigabituplink 100 Mbit

link only

24 procsper hub

25

Why this sensitivity to layout?

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5

6 7 8 9 0 1 2 3 4

All processors need to communicatewith processors outside to group of 24

26

Use exactly 4 columns

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

16 17 18 19

20 21 22 23

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

Only 2 x 4 = 8 processors need to communicate outside the group of 24 optimal use of speed ratio between 100 Mb ethernet switch and 1 Gb uplink


Fragmentation over many switches


Animation of uzAnimation of uz


Animation of B vectorsAnimation of B vectors


Animation of B vectorsAnimation of B vectors


Animation of energy spectraAnimation of energy spectra


Saturation behavior explained by Saturation behavior explained by magnetic helicity conservationmagnetic helicity conservation

termssurface2 BJBA dt

d

0BJSteady state,closed box

0 bjBJSmall scale andlarge scalecurrent helicityin balance

2f

21 bB kk


With hyperdiffusivityWith hyperdiffusivity

23231 f

bB kk for ordinaryhyperdiffusion

42k


ConclusionsConclusions• Subgrid scale modeling can be unsafe (some problems)

– shallower spectra, longer time scales, different saturation amplitudes

• High order schemes– Low phase and amplitude errors

– Need less viscosity

• 100 MB link close to bandwidth limit• Comparable to Origin• 2x faster with GB switch• 100 MB switches with GB uplink optimal

Date post:	14-Jan-2016
Category:	Documents
Upload:	elom
View:	45 times
Download:	1 times

The Pencil Code: multi-purpose and multi-user maintained

Documents