SPECTRAL PRECONDITIONERS FOR NONHYDROSTATIC

ATMOSPHERIC MODELS: EXTREME APPLICATIONS

P.K. Smolarkiewicz�, C. Temperton

�, S.J. Thomas

�, A.A. Wyszogrodzki

��National Center for Atmospheric Research, Boulder, Colorado, U.S.A.�European Centre for Medium Range Weather Forecasts, Reading, UK.�Los Alamos National Laboratory, Los Alamos, New Mexico, U.S.A.

Motivation

� We are concerned with DNS/LES of high Reynolds number andlow Mach number flows — i.e., highly turbulent and essentiallyincompressible flows — of inhomogeneous anisotropic fluids withrestoring forces, viz. complex fluids.

� The associated elliptic BVPs are poorly conditioned ( ������ �������

for terrestrial GCMs) nonseparable, containing cross derivatives, andnonsymmetric — due to domain anisotropy, planetary rotation, strat-ification, curvilinear coordinates, irregular lower boundary, etc.

� Such BVPs are difficult — i.e., a universally-effective solutiondoes not exist. Each particular case may call for a user’s interventionin customizing the elliptic solver, to achieve a judicious compromisebetween the accuracy and computational expense.

2

Approach

� An introduction to CG methods (from PDE perspective): Smo-larkiewicz & Margolin. Variational methods for elliptic problems in fluid models.Proc. ECMWF Workshop on Developments in numerical methods for very high

resolution global models 5-7 June 2000; Reading, UK, ECMWF, 137–159.

� Our method of choice is the restarted generalized conjugate resid-ual GCR(

�) algorithm (Eisenstat et al., 1983, SIAM J. Numer. Anal.)

proven successful in geophysical applications.

� An artful preconditioner can dramatically accelerate solver con-vergence!

� We consider spectral methods in the horizontal, with a line-relaxation scheme in the vertical.

3

Continuity of Efforts

� Bernardet (1995, MWR); Elman & O’Leary (1998, JCP)

� Thomas et al. (2003, MWR) reported advantages of spectral pre-conditioning, in the context of the serial code of the Canadian MC2model — a semi-Lagrangian, semi-implicit elastic, nonhydrostaticall-scale research/weather-prediction type model.

� We continue in the context of the massively-parallel, nonhydro-static anelastic, deformable-grid, Eulerian/semi-Lagrangian modelEULAG for multi-scale research of geophysical flows.

4

Preconditioned GCR(�) Scheme, for � ��� ��� � � �

, � � �� ��������� � �� ���� ��������������� ���� ��� �!�"� �� � #�$������� %�& �����('*) %,+

For any initial guess -/.0 , set 12.0 %3& 0 � -/. �4'65 0 , 7�.0 % � ���0 � 12. � ; then iterate:8�9;:4< % �>=�?@= �!� �BA�CED F G;H 9 CEIEJ :LK JMC�HNJPO 9Q 9;:SR %UT = �!� =NV ' � O 9W % ' X 1>Y & � 7ZY �L[

X & � 7 Y � & � 7 Y �L[ =- Y]\ �0 % - Y0 � W 7 Y0 = 1 Y]\ �0 % 1 Y0 � W & 0 � 7 Y � =

J�^�F_D(F Q ` 1 Y]\ � `"acb =d 0 % � ���0 � 1 Y]\ � � =& 0 � d � % ef@ghi�j � �k imln ghoMj �Ep i]o d�k o �rq i dtsu 'wv d�xy 0 =

z�{ j .]| Y~} { % '�X & � d � & � 7 { �L[X & � 7 { � & � 7 { �][ =

7 Y]\ �0 % d 0 � Yh{ j . } { 7 {0 = & 0 � 7 Y]\ � � %�& 0 � d � � Yh{ j . } { & 0 � 7 { � =JMC�O�O 9 =: JM��JtD�� - = 1 = 7 = & � 7 ��� 0 D 9 � - = 1 = 7 = & � 7 ��� .0 =JMC�O�O 9 �

assumingz#�w��� & � � �P� a T , with “ % ” holding if and only if

���>� � %�T .

5

Line-relaxation Preconditioners� Anisotropy in the vertical

��������� � � � % + simple, yet effectivepreconditioners, derivable from the Richardson (1910, Phil. Trans. Roy. Soc.)scheme: d \ � ' d

� �� % � � � d � � � � � d \ � � ' 1 Y]\ � = � � & ' p q � =(1)

where,� �

and� �

are the horizontal and the vertical counterparts of�

,����

isthe iteration parameter (based on spectral properties of

� �), � numbers succes-

sive Richardson iterations,R

numbers the outer iterations of GCR(V), and p q �

denotes cross-derivative terms.

� Eq.(1) leads to a linear problem��� '�� �� � � � d \ � % �) = �) � d � � �� � � � � d � ' 1 Y]\ � � =(2)

readily invertible using a Thomas’ algorithm; alternating implicit discretizationbetween

� �and

� �in fractional steps of

��leads to alternating-direction-implicit

(ADI) preconditioners (Skamarock et al., MWR, 1997).

� The preconditioner in (1) can be improved ( � � T %) by extending the schemeto the diagonally-preconditioned Duffort-Frankel algorithm

� d \ �4' � d� �� % � � � d �(' � � d \ � ' d � � � � � d \ � � ' 1 Y]\ � =

(3)

where� ' � � �

stands for the diagonal coefficient embedded within the matrix rep-resenting

� �on the grid. As

� ���� �, (3)

�block Jacobi preconditioner.

6

Spectral Preconditioning

� The idea is to allow for the “ � -implicitness” in the horizontal,and to converge with � ���� � in the Richardson iteration, so��� � � ���

� ��� ��� � � � �� ��� � � � � �/� ��� � ���

��� � � � � �� ��� � � � � � � ��� � (4)

becomes iteration free.

� Assuming � � � � ��������� ����� �! �#"%$#&�')( � �+*-, /. *10 �32, and same for

�4� � ,leads toh �����6587 ����� �! � �� ����� 9 ����� �! �;:#< �� �����: <

� �� �����>= "%$#&?')( � ��*@, . *80 �32BA �DC(5)

and to the corresponding set of independent linear tridiagonal prob-lems in the Fourier space,

E ����� lFn 7 ����� �! �B 9 ����� �G � � <� < sIHu �� ����� �! � � �� ����� �G � (6)

7

� ISSUES: i) coefficient homogenization within � — to avoidFourier transforms of the coefficients themselves, and multiplyingthe resulting series; ii) massively-parallel implementations.

� APPROACH: custom-programmed tensor-product 2D Fouriertransformations (for either periodic or open boundaries) with a fully-distributed spectral space, in the spirit of the domain-decompositionemployed for the physical space.

Figure 1: Static block distribution (SBD) method for computing tensor-productFourier transforms, Calvin (1996, Parall. Comp.), becomes static local (SLD)as processors array

�1D

8

Anelastic Model: Analytic Formulation

� Prusa & S., JCP 2003; Wedi & S., JCP 2004; Prusa & S. ibid.

� (physical domain) ��� � ��� (computational domain)� � C � � A ���;C� ���;C� � �

(7)

� Assumptions:1) �� and ��� are (topologically) cuboidal, toroidal, or spheroidal;2) coordinates

���;C� �of ��� are orthogonal and stationary;

3)� A �

;4)

� , C 0 �are independent of

.

5) (7) is a diffeomorphic mapping (homeomorphism OK)

40.

32.

24.

16.

8.

0. -2500. -1500. -500. 500. 1500.

cmx,cmn,cnt: 0.1250E+00-0.1250E+00 0.1563E-01

2500.

z (k

m)

cmx,cmn,cnt: 0.1406E+00-0.1563E+00 0.1563E-01

cmx,cmn,cnt: 0.3125E+00-0.2813E+00 0.3125E-01cmx,cmn,cnt: 0.2500E+00-0.2500E+00 0.3125E-01 40.

32.

24.

16.

8.

0. -2500. -1500. -500. 500. 1500. 2500.

w: t = 40.00 hr

z (k

m)

x (km)_

w: t = 48.00 hr

40.

32.

24.

16.

8.

0. -2500. -1500. -500. 500. 1500. 2500.

x (km)_

-2500. -1500. -500. 500. 1500. 2500.

w: t = 32.00 hrw: t = 16.00 hr

40.

32.

24.

16.

8.

0.

Figure 2: Gravity-wave packet past oscillating source.

9

� Anelastic system of Lipps & Hemler (J. Atmos. Sci., 1982)� � ����� � ��� � � �

(8)

: �: �

� � � � � �� �� �/� ��� �� �

� ����� � � � � C(9)

: � �: �

� � ��� � � ��� � C(10)

where ( � Prusa & Smolar. ibid.)� �! � � � � � " � � #%$&# � " � � �

�'# �)(*#� �

: ( : � � #+$&# � ��, � � " �), � : �)( : � � -�

� � A �), � # �#� " � � � � �/.0�

(11)

10

Finite-Difference Approximations

� Smolar. & Prusa, Turbulent Flow Computation, Kluver, 2002

� Each prognostic equation can be written as a Lagrangian evolu-tion equation or Eulerian conservation law:

: �: �

� � �# � , �# � � � � � , � , � � � � , � C

(12)

Where,� , � � � �

,� A ���

or � , and�

the associated rhs.

� Either form is approximated to� � � � < C � , < � with

��� �� � �� � � ��� � � � �"� � �� � � � �"� � �� � �� � � � �"� � �� C(13)

where� � �� is the solution sought at the grid point

� � � � C � � � , ��denotes a two-time-level either advective semi-Lagrangian or flux-form Eulerian NFT transport operator (viz. advection scheme).

11

Elliptic Pressure Equation: exact projection

� The algorithm in (13) forms system implicit for all�

(� � C � � )

because all (principal)� � � are unknown

�� � � � � � � � ��� � � � � � � � � � � � ��� ��� � � � C��� � C (14)

where� � � � C��� � A � � � � � �*� � � � � � � �

� ����� � � ��� � � � � � . � � � � ��� � � � (15)

accounts for the implicit representation of the buoyancy via (10).

� On grids unstaggered for�

(e.g., A or B), (14) can be invertedalgebraically to construct expressions for

� �via (11) that, after sub-

stitution to (8), produce BVP implied by the model discretization:� � �� , � * � , � � . ' �� � � ��� � � ��� � � ��� � � � � � � � � 2�� �� � � � " (16)

��� � ��� � � ��� ��� � � � � � � � � � � A � �defined in (11), and

� � � � � � � � .� Boundary conditions imposed on

� � � � , subject to the integra-bility condition ������� � , � � � � :�� � �

, constrain�� � �

.

� The resulting BVP is solved using a preconditioned GCR(�)

solver. Given updated � �

, and hence the updated� �

, the updated�and

� ,are constructed from

� �using transformations in (11).

12

RESULTS

� Flapping membranes (Wedi & Sm., JCP,193 2004):

Figure 3: Potential flow simulation past 3D undulating boundaries

semi-Lagrangian model option; �� � �� � ��

grid;� � ��� � � ; 7 � , � � � �

; GCR(4)Dirichlet boundaries for

�; 16 PE of IBM SP RS/6000.

For � � � � (&� , � � * � , � � �� � ��� �SP: � < <�� � � ��

, wallclock time 0:18:16LR: � < <�� � � ��

, wallclock time 0:42:24

Table 1: Vorticity errorsfield Max � � Average Std. dev.� �� � � � � � � .������

2.94* � ���

-7.52* � � ���

3.75* � ���

13

� Decaying turbulence in a triply periodic box (Herring & Kerr,

Phys. Fluids A, 5, 1993; Margolin et al., J. Fluid Eng., 124 2002):

Figure 4: Decaying turbulence

Eulerian model option; Statistics after 10 � � ; � � � � � � ��� � ���

10−8

10−7

10−6

10−5

10−4

10−3

10−2

0

10

20

30

40

50

60

70Convergence test

epsilon

mea

n nu

mbe

r of

iter

atio

ns

Figure 5: GCR(2) iterations required for the convergence

14

Table 2: Relative cost of the SP componentsPE FT 3dg-solv. FT

���1 17.36 4.63 18.624 2.37 1.04 2.238 1.38 0.43 1.0816 1.04 0.17 0.6332 1.33 0.06 0.7664 1.79 0.03 0.59

Table 3: Parallel performance of LR and SP preconditioners; 128�

gridPE PX PY X1Y1 X1Y0 X0Y1 X0Y0 X1Y1 X1Y0 X0Y1 X0Y01 1 1 4650 7920 7535 10659 591 519 459 5034 2 2 648 - - - 123 115 111 798 2 4 336 - - - 93 61 53 458 4 2 348 - - - 121 57 62 4616 4 4 214 - - - 54 36 32 2832 4 8 163 - - - 40 29 25 2932 8 4 179 - - - 45 29 28 2564 8 8 92 - - - 47 44 34 36

Table 4: Parallel performance of LR and SP preconditioners; 256�

gridPE PX PY X1Y1 X1Y1 X0Y064 8 8 1553 380 252

128 16 8 961 350 212256 16 16 529 358 212

�Regardless of poorer scaling, SP wins big

�

15

� Mesoscale valley flows (cf. Prusa & Smolar. ibid.):

Figure 6: Valley flow. Vertical velocity contours ink��

cross section at z=9 km (left)and on the vertical ribbon aligned with the center of the valley (right).

Eulerian model option;� � � � � ��� �

grid; � � �� � �

, 7 ��; IBM SP RS/6000.

For � � � � (&� , � � * � , � � � � � � � �SP: � � < � � � � � �

, wallclock time 2:54:33 , 20 PELR: � � < � � � � � �

, wallclock time 1:42:24 , 20 PE

For � � � � (&� , � � * � , � � � � � � � �SP: � � < � � � � � ��

, wallclock time 4:46:21 , 40 PELR: � � < � � � � � ���

, wallclock time 2:42:02 , 40 PE

16

� Held-Suarez climate simulations (Sm. et al. JAS, 58 2001):

Π /φ

Π/λ

Π /φ

Π/λ

cmx, cmn, cnt: -22.0, 14.0, 2.0 22.0 m/s

cmx, cmn, cnt: -20.0, 16.0, 4.0

cmx, cmn, cnt: 258.0, 312.0, 3.0

Contour from 306.0 to 330.0 by 1.0 and from 348.0 to 748.0 by 16.0

25.3 m/s

b'

b

a

a'

Figure 7: Instantaneous solutions of the idealized climate problem after 3 years of simulation.

Eulerian model option;� ��� ��� � �

geospherical grid; � � � � � �

, 7 ��� �

; 24 PE of IBM SP RS/6000.

For � � � � (&� , � � * � , � � �� � � � �SP: Total failure, no convergence in the GCR!LR: � < � ��� � � �

, wallclock time 0:25:04

17

The observed failure of spectrally-preconditioned GCR is not uniqueto large scale flows.

Figure 8: Urban PBL;� �����

contours ink �

cross section at z=10 m (left) and inthe central

���cross section (right).

Eulerian model option;� � � � � � ��� �

grid; � � � � � � � �

, 7 ��� �

; IBM SP RS/6000.

For � � � � (&� , � � * � , � � �� � ��� �SP: 64 PE, � � � � � � �

, wallclock time 3:32:18LR: 200 PE, � < � � � � � � �

, wallclock time 3:55:57

18

REMARKS

� SP preconditioners are a useful option, but not a panacea. Inparticular, the coefficient homogenization appears destructive for thesolver convergence in problems with substantial variability of thecoefficients in the horizontal.

� Depending upon the problem at hand, simpler LR precondition-ers can be much more effective than SP.

� Since SP preconditioners have substantial overhead comparedto LR, it appears counterproductive to use them in problems wherethe main solver converges in several iterations with LR. Conversely,SP may be advantageous in large-time-step integrations where LRrequire numerous iterations of the main solver, or where � � � � .

� SP preconditioners may win big in inherently transient prob-lems, where LR cannot take advantage from slow variability (andthus additivity with solver iterations) of the solution in a portion ofthe spectral range.

� MPP programing effort can be substantial