Spectral Methods for
Mantle Convection with 3D
Viscosity Variations
Robert Moucha
Alessandro Forte
Workshop for Advancing Numerical Modeling of Mantle
Convection & Lithospheric Dynamics
July 10, 2008
Outline
Introduce the variational functional for Stoke’s flow
Spectral formulation of the variational functional
Solving the system – practical considerations
Benchmarks
Introduction
A complete treatment of the mathematical details may be found in Forte & Peltier (1994),
Advances in Geophysics 36.
For the geodynamical impact of lateral viscosity variations see Moucha et al. (2007) , GJI.
Introduction (cont.)
Incompressible fluid Anelastic-liquid Newtonian rheology Infinite Prandtl number approximation
Boussinesq-like fluid approximation
Assumptions utilized in the buoyancy induced flow problem (Stoke's flow) in the mantle.
2410Pr ≈=ρκ
η
Introduction (cont.)
Cartesian representation of non-hydrostatic dynamical equations:
0=∂ kkuConservation of Mass :
ijijki EP ηδσ 21 +−=Stress tensor:
011 =∂+∂+∂ oiiokik φρφρσConservation of Momentum :
( )ijjiij uuE ∂+∂=
2
1Strain-rate tensor:
Introduction (cont.)
The boundary conditions imposed on the flow field at the outer and inner surfaces are:
0ˆ =ii un
0ˆˆ =jijinh σ
Zero Radial Velocity:
Free Slip:
Other boundary conditions imposed on the flow field at the outersurface can also be imposed, such as rigid boundary conditions.
Introduction (cont.)
Different methods exist for obtaining a solution to the coupled system of equations, such as finite-difference/volume (STAG3D), finite-element (TERRA ; CitComS), or spectral methods.
There are numerous spectral methods and pseudo (or hybrid) spectral methods (e.g. Zhang & Christenson, 1993; Čadek et al. 1993; Martinec et al., 1993; Forte & Peltier, 1994).
Advantages of spectral methods:
•Meshless – circumvents challenges with choosing grid types anddomain decomposition
•Quasi-analytical solution
Disadvantages:
•Computationally and storage-wise expensive for 3D viscosity
•Non-Newtonian rheology requires an iterative approach (Čadek et al. 1993
The Variational Formulation
1. Assume that ui satisfies the governing equations and the BCs. Now
consider a flow perturbation δui such that:
0=∂ kk uδ
2. Take the inner product of the flow perturbation δui with the momentum conservation equation:
( ) ( ) ( ) ( ) 0011 =∂+∂+∂−∂ iiioiikkiikik uuuu δφρδφρδσδσ
[ ] [ ] 0ˆˆ 1001 =−+−∂ ∫∫ Skkkkik
Vkikiii dSunundVEu δφρδσδσδφρ
kikiikki Eu δσδσ =∂
Integrating over the volume V occupied by the medium by virtue of Gauss'
theorem, and by symmetry of the stress tensor :
We get:
The Variational Formulation (cont.)
4. Since both the solution ui and the perturbed flow ui + δui satisfy the same BCs on the surface S we therefore have:
[ ] [ ] 0ˆˆ 1001 =−+−∂ ∫∫ Skkkkik
Vkikiii dSunundVEu δφρδσδσδφρ
which implies that δui must be tangential to S; thus
0ˆ =kk un δ
0ˆ =ikik un δσ
The Variational Formulation (cont.)
[ ] 0201 =−∂∫V kikiii dVEEu δηδφρ
5. Because of the incompressibility assumption
011 == kkkiki EPEP δδδ
The Variational Formulation (cont.)
[ ] 0201 =−∂∫V kikiii dVEEu δηδφρ
[ ]∫ ∂−==V
iiijij dVuEEWW 01;0 φρηδ
Rate of energy released by buoyancy
Rate of viscous dissipation energy
Assuming that δη = 0, i.e. η does not depend on the flow velocity ui
(otherwise see Čadek et al., 1993)
The Variational Calculation using spherical harmonics
qp ∇×+∇××∇=
=•∇vvv
v
rru
u 0since
( ) ( ) ( )∑=m
mmYrprp
,
,,,l
llφθφθ
( ) ( ) ( )∑=m
mmYrqrq
,
,,,l
llφθφθ
Expand the flow field u in terms of poloidal (p) and toroidal (q) flow generating scalars as follows:
Represent the poloidal and toroidal flow fields in terms of spherical harmonic basis functions:
We allow for explicit 3-D viscosity variations by also expanding the viscosity:
( ) ( ) ( )∑=m
mmYrr
,
,,,l
llφθηφθη
The Variational Calculation using spherical harmonics (cont.)
[ ]∫ ∂−=V
iiijij dVuEEW 01 φρη
By expressing the tensor inner product EijEij in terms of the so-called contravariant canonical components using generalized spherical harmonics
described in Phinney & Burridge (1973), the viscous dissipation energy integral becomes:
The Variational Calculation using spherical harmonics (cont.)
( )( )( )
( ) ( ) ( )
( )
( )drr
dr
dq
dr
dir
r
p
dr
pd
dr
dq
dr
dir
r
p
dr
pdJs
iqr
p
dr
dpiq
r
p
dr
dp
r
Js
r
p
dr
dp
r
p
dr
dp
r
Jsr
tmtm
JsJsdVEE
t
s
t
s
st
s
mmms
t
s
t
s
t
smmmss
a
b
t
s
t
s
mmstm
J
s
sJm tsV
ijij
2
2
2
2
2
2
2
2
2
2
2
11
2
2121
2
2
2
2
1
, ,
2
2
011
2
022
6
000
1212124
+
Ω+×
−
Ω+ΩΩ
−−
++
−+
ΩΩΩΩ
−+
−
−
ΩΩ
×
−−+++=
∫
∑∑∑∫
−−
+
−=
ll
l
ll
l
ll
ll
ll
l
l
ll
l
l
l
ll
η
πη
[ ]∫ ∂−=V
iiijij dVuEEW 01 φρηWigner 3-j symbols
( ) ( )( )2
21,
2
121
+−=Ω
+=Ω
llll ll
Where;
The Variational Calculation using spherical harmonics (cont.)
( ) ( ) ( ) ( ) ( )
( )∗
+
−=
×+++= ∑
φθ
φθφθ
,
121212,,
21
21
21
21
21
21
21
22
2
11
1
Nm
mNmN
YmmmNNN
YY
l
ll
lll
ll
llllll
lll
A summary of useful aspects of spherical harmonic coupling rules, in the context of fluid dynamics in spheres, can be found in Forte & Peltier (1994, Appendix II).
The Variational Calculation using spherical harmonics (cont.)
( )( )( )
( ) ( ) ( )
( )
( )drr
dr
dq
dr
dir
r
p
dr
pd
dr
dq
dr
dir
r
p
dr
pdJs
iqr
p
dr
dpiq
r
p
dr
dp
r
Js
r
p
dr
dp
r
p
dr
dp
r
Jsr
tmtm
JsJsdVEE
t
s
t
s
st
s
mmms
t
s
t
s
t
smmmss
a
b
t
s
t
s
mmstm
J
s
sJm tsV
ijij
2
2
2
2
2
2
2
2
2
2
2
11
2
2121
2
2
2
2
1
, ,
2
2
011
2
022
6
000
1212124
+
Ω+×
−
Ω+ΩΩ
−−
++
−+
ΩΩΩΩ
−+
−
−
ΩΩ
×
−−+++=
∫
∑∑∑∫
−−
+
−=
ll
l
ll
l
ll
ll
ll
l
l
ll
l
l
l
ll
η
πη
[ ]∫ ∂−=V
iiijij dVuEEW 01 φρη
The Variational Calculation using spherical harmonics
( ) ( )
bar
m
bar
m
dr
rpdrp
,
2
2
,0
==
== l
l
One possible set of radial basis functions that satisfy these boundary conditions are the following modified Fourier basis:
One can chose any set of radial basis functions, but they must satisfy the following poloidal and toroidal (free-slip) boundary conditions:
( ) ( )
( )
−
−=
=∑
ba
arkrf
rfprp
k
k
k
m
k
m
πsin
ll( ) ( )
( )
−
−=
=∑
ba
arkrrg
rgqrq
k
k
k
m
k
m
πcos
ll
( )0
,
=
= bar
m
r
rq
dr
dl
The Variational Calculation using spherical harmonics (cont.)
The functional W will be at a minimum when the following conditions are satisfied:
( ) ( )[ ] 001 =∂−
∂
∂=
∂
∂∫V iiijijt
sn
t
sn
dVuEEpp
Wφρη
( ) ( )[ ] 0=
∂
∂=
∂
∂∫V ijijt
sn
t
sn
dVEEqq
Wη
The Variational Calculation using spherical harmonics (cont.)
( ) ( )( )∫∑∑
∗+
=+a
bn
t
s
mk
m
k
mk
nst
mk
m
k
mk
nst drrgrfr
rssqBpA
2
0
1
0,,,,
)1( ρ
ηl
l
l
l
l
l
0,,,,
=− ∑∑mk
m
k
mk
nst
mk
m
k
mk
nst pDqCl
l
l
l
l
l
Minimizing the functional W will yield the following couples set of algebraic equations:
The Variational Calculation using spherical harmonics (cont.)
The coefficients involve a numerical radial integration of the viscosity and the flow basis functions, for example:
( )( )( )
( ) ( ) ( )
( )
( )drr
r
f
dr
fd
r
f
dr
fdJs
r
f
dr
df
r
f
dr
df
r
Js
r
f
dr
df
r
f
dr
df
r
Jsr
tmtm
JsJsA
n
s
n
kks
nnkk
ss
a
b
nnkk
stm
J
s
sJ
mk
nst
2
2
2
2
2
2
2
2
2
2
2
11
2
2121
2
2
2
2
1
0
2
22
011
4
022
12
000
121212
Ω+×
Ω+ΩΩ
−−
+
+
ΩΩΩΩ
−+
−
−
ΩΩ
×
−−+++=
∫
∑
−−
+
−=
l
l
ll
l
l
l
l
l
l
l
ll
η
η
The Variational Calculation using spherical harmonics (cont.)
( ) ( )( )∫∑∑
∗+
=+a
bn
t
s
mk
m
k
mk
nst
mk
m
k
mk
nst drrgrfr
rssqBpA
2
0
1
0,,,,
)1( ρ
ηl
l
l
l
l
l
0,,,,
=− ∑∑mk
m
k
mk
nst
mk
m
k
mk
nst pDqCl
l
l
l
l
l
=
− 0
d
q
p
CD
BA
bSx =
The Variational Calculation using spherical harmonics (cont.)
( ) ( )( )∫∑∑
∗+
=+a
bn
t
s
mk
m
k
mk
nst
mk
m
k
mk
nst drrgrfr
rssqBpA
2
0
1
0,,,,
)1( ρ
ηl
l
l
l
l
l
0,,,,
=− ∑∑mk
m
k
mk
nst
mk
m
k
mk
nst pDqCl
l
l
l
l
l
=
− 0
d
q
p
CD
BA
bSx =
= 0 for 1D viscosity
S is for super matrix
11 ××× = nnnn bxS
( ) flowflowr ssnn 22 +=
nr = number of radial basis functions
sflow = spherical harmonic degree of flow scalars
where
Currently we use:
nr = 40
sflow = 32
n = 87040 !!
Equivalent Spatial Resolution:
36 km radially
600 km at the surface
56 GB of storage (double precision)
S is for super matrix (cont.)
Very dense matrix
Badly conditioned!
But, consistent!
1510−≈− bSy
row
s
columns100 200 300 400 500
100
200
300
400
5000
1
2
3
4
5
6
7
8
log10|S|
For example;nr=40, sflow = 6
1/cond(S) = 10-7
Residual;
S is for super matrix (cont.)
Very dense matrix
Badly conditioned!
But, consistent!
2110−≈− bSy
log10|diag(1/R)*S|
For example;nr=40, sflow = 6
1/cond(S) = 10-5
Residual;
row
s
columns100 200 300 400 500
100
200
300
400
500-8
-7
-6
-5
-4
-3
-2
-1
0
Row Scaling => maximum in a row
Getting a solution
Use a direct method for solving the system of equations
Use a cluster (130 Opterons (1.6GHz) with 3 GB of RAM/CPU)
Use ScaLapack, which is a linear algebra library for parallel computers that implements block-oriented LAPACK routines
1. Distribute matrix and RHS on a processor grid using 2D block-cyclic distribution
2. Solve the system of equations using LU factorization (PDGETRF) and solver (PDGETRS)
row
s
columns100 200 300 400 500
100
200
300
400
5000
1
2
3
4
5
6
7
8
Getting a solution (cont.)
2D block cyclic distribution in ScaLapack on a 2x2 processor grid
log10|S|
0000 0000 0000
0000 0000 0000
0000 0000 0000
1111 1111 1111
1111 1111 1111
1111 1111 1111
2222 3333 2222 3333 2222 3333
2222 3333 2222 3333 2222 3333
2222 3333 2222 3333 2222 3333
0000 1111
2222 3333
1 2 3 4 5 6 7 8 9
x 104n
S gen.
Solve
Total
Getting a solution (cont.)
Obtaining a solution is expensive!
100
101
102
103
101
102
103
104
Number of Processors
Ela
ps
ed
Tim
e (
s)
S gen.
Solve
Total
Perfect Efficiency
nr=40, sflow = 12, sviscosity = 12n = 13440
nr=40, sflow = 12-32, sviscosity = 12# of processors = 121
Benchmarking
Variational calculations for
n = 87040
nr = 40, sflow = 32, sviscosity=32, sdensity = 12
Compare with a finite-element Stokes’ flow solution in spherical geometry using CitcomS v1.1 (Zhong et al., 2000).
12 spherical caps, each with 65x65x65 nodes.
The buoyancy forces in the mantle are derived from a seismic tomography model using velocity-to-density conversion profile.
Benchmarking (cont.)
The spatial variations in viscosity are expressed as:
where;
and ν(r,θ,φ) are estimated on the basis of homologous-temperature scaling using seismic-tomography derived temperature anomalies for the mantle.
( ) ( ) ( )[ ], , 1 , ,r r rη θ φ η ν θ φ= +
( )10
0
ar
rη η
=
η0 = 1.0x1021 Pa s
Benchmarking (cont.)
Viscosity at 500 km
η0 = 1.0x1021 Pa s
Viscosity at 2740 km
Temperature at 500 km
Temperature at 2740 km
∆T = 1500 K, T0 = 0.5
Benchmarking (cont.)
Spatial Finite-element
Spectral VariationalSpectral Variational
Difference (Spectral – CitComS)
8 (m)
-8
80 (m)
-80
Currently in development Efficient Wigner 3-j symbol storage and look-up
Use B-spline radial basis functions
B-spline radial basis offer local support
Local support reduce the Super matrix into a block-tridiagonal form in the case of a cubic B-spline
Block-tridiagonal form offers a huge speed up by reducing the number of blocks in the Super matrix that need to be calculated
Massive reduction in storage requirements for the Super matrix
A direct block-triadiagonal solver reduces the number of FLOP by over 2 orders of magnitude vs. traditional direct dense matrix solvers (impact on number of communications should be also less)
Degree 64 (~310 km) gives a matrix size of n = 337920
Dense super matrix = 850GB vs. B-spline Block-Tridiagonal compact Matrix = 190 GB
Impact of 3D Viscosity
Impact of 3D Viscosity (cont.)
Impact of 3D Viscosity (cont.)
Impact of 3D Viscosity (cont.)
