Modeling of Seismic Wave Propagation at the Scale of theEarth on a Large Beowulf

[Extended Abstract]

Dimitri KomatitschSeismological Laboratory

California Institute of TechnologyPasadena, California 91125, USA

komatits@gps.caltech.edu

Jeroen TrompSeismological Laboratory

California Institute of TechnologyPasadena, California 91125, USA

jtromp@gps.caltech.edu

ABSTRACTWe use a parallel spectral-element method to simulate thepropagation of seismic waves generated by earthquakes inthe entire 3-D Earth. The method is implemented usingMPI on a large PC cluster (Beowulf) with 151 processorsand 76 Gb of RAM. It is based upon a weak formulationof the equations of motion and combines the flexibility of afinite-element method with the accuracy of a pseudospectralmethod. The finite-element mesh honors all discontinuitiesin the Earth velocity model. To maintain a relatively con-stant number of grid points per seismic wavelength, the sizeof the elements is increased with depth in a conforming fash-ion, thus retaining a diagonal mass matrix. The effects of at-tenuation and anisotropy are incorporated. We benchmarkspectral-element synthetic seismograms against a normal-mode reference solution for a spherically symmetric Earthvelocity model. The two methods are in excellent agreementfor all waves with periods greater than 20 seconds.

1. INTRODUCTIONModeling seismic wave propagation resulting from large

earthquakes at the scale of the entire Earth using fully three-dimensional (3-D) velocity models poses a formidable nu-merical challenge. The effects of an anisotropic astheno-sphere, a slow and very thin crust, sharp fluid-solid discon-tinuities at the inner-core (ICB) and core-mantle (CMB)boundaries, and attenuation must all be accounted for. Inthis article we demonstrate that the spectral-element method(SEM), introduced more than 15 years ago in computationalfluid mechanics [12], can meet this challenge. We reach un-precedented resolution by using a message-passing algorithmon a large cluster of PCs (Beowulf) with 151 processors and76 Gb of RAM.

The SEM has previously been used to accurately modelwave propagation on local and regional scales [13, 8, 11]. Ex-

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SC2001 November 2001, DenverCopyright 2001 ACM 1-58113-293-X/01/0011 ...$5.00.

amples of such local or regional simulations include seismicrisk assessment in sedimentary basins such as Los Angeles,Tokyo or Mexico City, and active seismic experiments inoil fields for the petroleum industry. It has also recentlybeen applied to the problem of global wave propagation ininnovative work by [5] and [4]. They use a so-called ‘mor-tar’ version of the SEM [1], which allows for non-conformingmeshes. This makes mesh design more flexible, but comes ata significant increase in the complexity and cost of the im-plementation, because the mass matrix is no longer diagonalon the non-conforming interfaces, and as a result an iterativesolver has to be used to solve the non-diagonal system. Inthis work we use a classical SEM based upon a conformingmesh that retains a diagonal mass matrix, thus greatly sim-plifying the algorithm and reaching high parallel efficiency,which in turn allows us to perform for the first time 3-Dglobal simulations at unprecedented resolution. Comparedto previous works, we introduce the effect of anisotropy ofseismic velocities in the asthenosphere (the upper 220 kmof the Earth in the upper mantle), and also incorporatethe effect of attenuation of the waves (loss of energy dueto anelastic behavior of the materials). In addition, we alsoemploy a powerful way of handling the fluid region of themodel (the fluid outer core of the Earth, which is in contactwith the solid mantle and the solid inner core) based upona simple and efficient domain decomposition technique.

2. DESIGNING A MESH FOR THE EARTHAs in any finite-element method, a first crucial step to-

wards the accurate simulation of 3-D seismic wave propa-gation resulting from earthquakes is the design of a mesh.A classical spectral-element method (SEM) relies upon amesh of hexahedral finite elements Ωe that are isomorphousto the cube. Tetrahedra that are classical in finite elementmethods are excluded in the SEM because of the tensorisa-tion of the polynomial basis that is required to obtain anexactly diagonal mass matrix, as will be explained in Sec-tion 3. The six sides of each hexahedral element must matchup exactly with the sides of neighboring elements. Such amesh is traditionally called a geometrically conforming meshin the finite-element literature. For reasons of accuracy, agood mesh should honor all the major velocity discontinu-ities in the model, and the size of the elements should reflectthe distribution of wave speeds, such that one maintains arelatively similar number of grid points per seismic wave-

A

A

A

A

A

C C C C

CC

B B

B B

B B

B B

A

C

B

Figure 1: View of the six building blocks that con-stitute the cubed sphere. Analytical relations mapeach of the six faces of the cube to the surface of thesphere. Besides a top and bottom, each block hasfour sides that need to match up exactly with fourother blocks to complete the cube, as indicated bythe arrows. The mesh size also needs to be increasedin the globe as a function of depth to maintain a sim-ilar number of grid points per wavelength through-out the model. This is accomplished in three stages;schematically, these four sides have one of three de-signs: A, B, or C, as illustrated on the right. Whenthe six blocks are fitted together to make the entireglobe, they match perfectly.

length throughout the model. Since wave speed generallyincreases with depth in the Earth, this implies that the el-ements should become gradually larger with depth. Theserequirements make the design of a good mesh for the globechallenging.

The mesh we use is based upon the concept of the quasi-uniform gnomonic projection, or ‘cubed-sphere’ [15, 17, 14]that was first introduced for seismic wave propagation prob-lems by [5]. The key idea is to map each of the six sides ofa cube to the surface of the sphere. An increase in elementsize, to adapt it to the variations of wave speed with depth,can be obtained by first doubling the mesh in one lateral di-rection, and, subsequently, at a greater depth, increasing itssize in the other lateral dimension. Figure 1 illustrates howthis may be accomplished for the entire globe based upon athree-stage doubling as a function of depth. Note that thereare three types of chunks: AB, AC, and BC. In each of thetypes the doubling is performed at different levels, such thatthe final six chunks fit together perfectly to make the entireglobe.

The final mesh used in the simulations is shown in Figure 2and is designed to honor all velocity discontinuities in thespherically-symmetric 1-D standard reference wave velocitymodel for the Earth, which is called the Preliminary Refer-ence Earth Model (PREM) [7]. Each of the six chunks has240 × 240 elements at the free surface and, as a result of thethree doublings with depth, 30 × 30 elements at the ICB.

Figure 2: Mesh used for the simulations presentedin this study. It honors all velocity discontinuitiesin the Earth model. The mesh is doubled in sizethree times with depth. Each of the six chunks has240 × 240 elements at the surface of the Earth and30 × 30 elements at the inner-core boundary. Thetriangle indicates the location of the epicenter of theearthquake, situated on the equator and the Green-wich meridian. Rings of receivers (seismic recordingstations) with a 2-degree spacing along the equa-tor and the Greenwich meridian are shown by thedashes.

Figure 3: To avoid a mesh singularity associatedwith the Earth’s center, we place a cube at the cen-ter of the solid inner core, following the idea intro-duced by [5]. This figure shows the actual meshused. Note that there is a layer of three elementsbetween the inner-core boundary and the centralcube. Note also that element size within the cen-tral cube is not constant; this reflects a match-upwith the angularly equidistant mesh at the inner-core boundary.

To avoid singularities of coordinates at the Earth’s center,[5] introduced the idea of placing a cube around the centerof the inner core. We make use of this idea in the presentstudy. The mesh within this cube needs to match up withthe cubed sphere mesh at the ICB, as shown in Figure 3.

3. THE SPECTRAL-ELEMENT METHOD

3.1 Equations of seismic wave motionThe elastic wave equation for the Earth’s mantle, crust

and solid inner-core may be written in the form

ρ ∂2t s = ∇ · T, (1)

where ρ denotes the 3-D distribution of density and T thestress tensor which is linearly related to the displacementgradient ∇s by Hooke’s law, T = c : ∇s. In its most gen-eral form, e.g., in a triclinic crystal, the fourth-order elastictensor c has 21 independent components (e.g., [2]). Twotypes of boundary conditions must be considered: on thesurface of the Earth the traction n ·T, where n denotes theunit outward normal on the free surface, vanishes, and onthe CMB and the ICB (i.e., the fluid-solid boundaries) thenormal component of velocity n · v and the traction n · Tare continuous.

Spectral-element methods, like finite-element methods, arebased upon an integral or ‘weak’ formulation of the problem.This formulation is obtained by taking the dot product of

the momentum equation (1) with an arbitrary test-vector w,integrating by parts and imposing the stress-free boundarycondition. This gives

∫M

ρw · ∂2t sd3r = −

∫M

∇w : T d3r + M : ∇w(rs) S(t)

−∫

CMB

w · T · nd2r. (2)

To correctly model interactions between the solid mantleand the fluid outer core, we need to impose the continuity oftraction and of the normal velocity at the CMB. We imple-ment the fluid-solid interactions based upon a simple andefficient domain decomposition method: in the mantle weimpose the continuity of traction and in the fluid outer corewe impose the continuity of normal velocity. The equationsin the solid inner core of the Earth are similar to those pre-sented above and are therefore not detailed here (since it isalso a solid region in contact with the fluid outer core).

In the fluid outer core, the equation of motion can bewritten in terms of a scalar potential χ as

κ−1∂2t χ = ∇ · (ρ−1∇χ), (3)

where κ is the bulk modulus of the fluid. That potential isrelated to pressure in the fluid by p = −∂tχ. The weak formof this equation is obtained by multiplying it by a scalartest function w and integrating by parts. At the fluid-solidmatching interfaces (the CMB and the ICB) we need to im-pose the continuity of normal velocity, therefore we replacethe normal component of velocity n · v in the integrals overthe CMB and ICB with the normal component of veloc-ity n · ∂ts in the mantle or inner core, which gives:

∫OC

κ−1w∂2t χ d3r = −

∫OC

ρ−1∇w · ∇χ d3r

−∫

CMB

w n · ∂tsd2r +

∫ICB

w n · ∂ts d2r. (4)

3.2 Interpolation, integration and discretiza-tion

To represent the displacement field on an element requiresthe introduction of grid points in each element. Typically,in a SEM it is optimal (in terms of the precision/cost ra-tio) to use Lagrange polynomials of degree 4 to 10 for theinterpolation of functions [16]. This is much higher thanthe degree-1 or degree-2 approximations classically used infinite-element methods. The control points ξ are chosen tobe the l + 1 Gauss-Lobatto-Legendre points, which are theroots of (1 − ξ2)P ′

l (ξ) = 0, where P ′l denotes the deriva-

tive of the Legendre polynomial of degree l (e.g., [3]). Thereason for this choice is that it leads to an exactly diagonalmass matrix and therefore to fully explicit time schemes,which greatly simplifies the implementation of the method,in particular on a parallel computer. Functions f , such asthe displacement field s and the test vector w, are inter-polated in terms of triple products of Lagrange polynomi-als. This choice of test vector makes the SEM a Galerkinmethod, because its basis functions are the same as thoseused to represent the displacement. The gradient of a func-tion, ∇ f =

∑3i=1 xi∂if , evaluated at the Gauss-Lobatto-

Legendre point x(ξα′ , ηβ′ , ζγ′), may be written in the form

∇ f(x(ξα′ , ηβ′ , ζγ′)) ≈3∑

i=1

xi

[ nα∑α=0

fαβ′γ′′α(ξα′)∂iξ

+

nβ∑β=0

fα′βγ′′β(ηβ′)∂iη +

nγ∑γ=0

fα′β′γ′γ(ζγ′)∂iζ

], (5)

where the ′(ξ) represent the derivatives of the Lagrange in-terpolants at the Gauss-Lobatto-Legendre points, and the∂jξ are the partial derivatives of the Jacobian transforma-tion of the coordinate system to the [−1, 1]3 reference cube.

To solve the weak form of the equations of motion, nu-merical integrations over the elements need to be performed.In a spectral-element method, one uses the Gauss-Lobatto-Legendre integration rule:

∫Ωe

f(x) d3x =

∫ ∫ ∫ 1

−1

f(x(ξ, η, ζ)) J(ξ, η, ζ) dξ dη dζ

≈nα,nβ ,nγ∑α,β,γ=0

ωαωβωγfαβγJαβγ , (6)

where the ωα denote the weights associated with the Gauss-Lobatto-Legendre quadrature (e.g., [3]), and J is the Jaco-bian of the transformation of the coordinate system to the[−1, 1]3 reference cube.

The term on the left hand side of the weak form of theequation of motion (2) is traditionally called the mass ma-trix in finite-element modeling. At the elemental level, thisintegration may be written as

∫Ωe

ρw · ∂2t s d3x

=

∫ ∫ ∫ 1

−1

ρ(x(ξ))w(x(ξ)) · ∂2t s(x(ξ), t) J(ξ) d3ξ

≈nα,nβ ,nγ∑α,β,γ=0

ωαωβωγJαβγραβγ3∑

i=1

wαβγi sαβγ

i (t). (7)

By independently setting factors of wαβγ1 , wαβγ

2 and wαβγ3

equal to zero, since the weak formulation (2) must hold forany test vector w, we obtain an equation for each compo-nent of acceleration sαβγ

i (t) at grid point (ξα, ξβ , ξγ), that wecan subsequently march explicitely in time. The remarkableproperty of equation (7) is that the value of acceleration

at each point of a given element, sαβγi (t), is simply mul-

tiplied by the factor ωαωβωγραβγJαβγ , i.e., as mentionedearlier, the elemental mass matrix is exactly diagonal. Thishas a very important implication in practice, since it greatlysimplifies the message-passing implementation of the SEMalgorithm. Note also that in this respect the SEM signifi-cantly differs from more traditional finite-element methods,in which a sparse (i.e., non diagonal) system would be ob-tained.

The next integral that needs to be evaluated is the stiff-ness matrix (the first term of the right-hand side of equa-

tion (2)). We find∫Ωe

∇w : T d3x ≈nα,nβ ,nγ∑α,β,γ=0

3∑i=1

wαβγi

[ωβωγ

nα′∑α′=0

ωα′Jα′βγe F α′βγ

i1 ′α(ξα′)

+ ωαωγ

nβ′∑β′=0

ωβ′Jαβ′γe F αβ′γ

i2 ′β(ηβ′)

+ ωαωβ

nγ′∑γ′=0

ωγ′Jαβγ′e F αβγ′

i3 ′γ(ζγ′)

],(8)

where Fik =∑3

j=1 Tij ∂jξk. The value of the stress tensor Tis determined by Hooke’s law, T = c : ∇s. This calculationrequires knowledge of the gradient of displacement ∇s at theGauss-Lobatto-Legendre integration points, which is givenby:

∂isj(x(ξα, ηβ , ζγ), t) =[nσ∑

σ=0

sσβγj (t)′σ(ξα)

]∂iξ(ξα, ηβ , ζγ)

+

[nσ∑

σ=0

sασγj (t)′σ(ηβ)

]∂iη(ξα, ηβ , ζγ)

+

[nσ∑

σ=0

sαβσj (t)′σ(ζγ)

]∂iζ(ξα, ηβ , ζγ). (9)

The remaining volume and surface integrals in (2) and (4)are identical in form to other integrals already discussed inthis section.

3.3 Assembly of the system and time marchingIn each spectral element, functions are sampled at the

Gauss-Lobatto-Legendre points. Grid points that lie on thesides, edges, or corners of an element are shared amongstneighbors. Therefore, as in a classical finite-element method,we need to distinguish the local mesh of grid points that de-fine an element from the global mesh of all the grid points inthe model, many of which are shared amongst several spec-tral elements. Note that there are three unknowns (or ‘de-grees of freedom’ in finite-element parlance) per grid pointin the solid regions of the model (the three components ofthe displacement vector), but only one degree of freedomin the fluid outer core (the generalized scalar potential χ).The contributions to the degrees of freedom (i.e., the inter-nal forces computed separately) from all the elements thatshare a common global grid point need to be summed to ob-tain the right global internal forces. In a traditional finite-element method this is referred to as assembling the sys-tem. This assembly stage is a costly part of the calculationon parallel computers, because information from individualelements needs to be shared with neighboring elements. Ina SEM, this is the only operation that involves communica-tions between distinct CPUs (implemented based upon MPIin practice).

To take full advantage of the fact that the global mass ma-trix is exactly diagonal, time discretization of the resultingglobal second-order ordinary differential equation obtainedafter assembling the system is achieved based upon an ex-plicit second-order finite-difference scheme. Such a scheme,

which is a particular case of the general Newmark scheme forsecond-order hyperbolic systems [9], is conditionally stable,and the Courant stability condition is governed by the max-imum value of the ratio between the pressure wave velocityand the size of the grid spacing.

4. PARALLEL IMPLEMENTATION USINGMPI ON A BEOWULF

We implement the method on a cluster of PCs using mes-sage passing (MPI). In our SEM, since the mass matrix isexactly diagonal by construction, the PCs spend most oftheir time doing computations, and communications repre-sent only a small fraction of the time of simulation. There-fore, this is an optimal situation for a parallel implemen-tation, and clusters of PCs are ideal for this application inspite of the high latency of the network connecting them.

The SEM calculations are performed on a Beowulf in theSeismological Laboratory at Caltech. This machine consistsof 76 dual-processor PCs with 1 Gigabyte of memory each.The PCs are connected using standard 100-Mbits Ethernet.The simulations are distributed over 151 processors. Eachof the six chunks that constitute the globe is subdividedamongst 25 processors (corresponding to 25 mesh slices),and the cube at the center of the inner core uses one sepa-rate processor. Figure 4 shows how the slices are designedin the cubed-sphere mesh. Note that inside each of the sixchunks the mesh of slices is derived from a regular Carte-sian topology. However, the message passing topology mustbe different between chunks: each corner of each chunk isshared between three rather than two or four slices. Thiscomplicates the message-passing implementation. We solvethe problem using a three-step sequence of messages: wefirst assemble the contributions between slices inside eachchunk; then between slices located on the edges of differentchunks, excluding the corners of valence 3; then in a laststep we assemble these corners separately.

The mesh shown in Figures 2 and 3 contains a total ofapproximately 2.6 million spectral elements. In each spec-tral element we use a polynomial degree N = 4 for the ex-pansion of the Lagrange interpolants at the Gauss-Lobatto-Legendre points, which means each spectral element con-tains (N + 1)3 = 125 points, and the total global systemof ordinary differential equations, counting common pointson the edges of the elements only once, contains 179 millionpoints (i.e., approximately 539 million degrees of freedomsince we solve for the three components of displacement ateach grid point in the solid regions). After division of themesh into slices, each processor is responsible for 17,000 el-ements. With a polynomial degree N = 4, this correspondsto roughly 1.1 million grid points per processor.

The central cube in the inner core, shown in Figure 3,poses yet another difficulty from a message-passing point ofview. Since it is handled by a separate processor, and sinceit shares grid points with all the other slices, a separatecommunication pattern has to be implemented based upona master-slave programming philosophy: all the slices sendtheir contributions (the internal forces computed locally) tothe central cube, which acts as a master, collecting and sum-ming them, and then sending the result back to the slices,which act as slaves. The number of elements in the centralcube is smaller than in any of the slices, which ensures thatload balancing of the application is unaffected.

Figure 4: In the parallel MPI implementation, eachof the six chunks that makes up the cubed sphere issubdivided in terms of 25 slices of elements (top).Each of these slices resides on a single CPU. Thecentral cube (bottom) is handled by one extra pro-cessor, such that the entire calculation involves 151CPUs. The results on the edges of a slice need tobe communicated to all its neighbors. Note thatthe communication patterns are different for slicesinside a chunk, on the edges of a chunk, and onthe corners of a chunk. The MPI communicationpattern is particularly difficult for the central cube,which is handled by a separate processor that needsto communicate with all the other processors, be-cause every slice of the mesh touches it (here someslices have been removed for clarity). One needsto use a master/slave programming methodology inorder to avoid communication patterns that coulddeadlock.

5. THE GREAT BOLIVIAN EARTHQUAKEOF JUNE 9, 1994

We benchmark the SEM against an independent referencesolution for the anisotropic reference Earth velocity and den-sity model PREM. We use the mesh and source-receivers(i.e., epicenter of the earthquake and seismic recording sta-tions) geometry shown in Figure 2. For simplicity, the epi-center of the earthquake is located on the equator and theGreenwich meridian. Note that because the PREM velocitymodel is spherically symmetric, this is strictly equivalentto using the real location of the event in Bolivia. Seis-mic stations record ground displacement along the equa-tor and the Greenwich meridian at 2-degree intervals. Forcomparison and validation, we use a classical independentreference solution computed based upon a quasi-analyticalnormal-mode summation technique (e.g., [6]). Note thatthe normal-mode method is very accurate, but limited tospherically-symmetric models (i.e., to 3-D simulations in 1-D radial structures such as PREM), whereas our SEM canhandle fully 3-D velocity models.

We simulate a large and very deep earthquake of magni-tude Mw = 8.2 that occured in Bolivia on June 9, 1994, ata depth of 647 km. This earthquake is one of the largestdeep events ever recorded, and has therefore generated a lotof attention in the seismological community. The event hassignificant energy down to a period of about 15 seconds (i.e.,for frequencies below 67 mHz). We also include the effect ofattenuation (i.e., anelastic behavior) in this simulation; thereader is referred to [10] for more details on how attenua-tion is implemented in the SEM (it is not straightforwardto implement attenuation in time-domain methods, becausein principle determining the current state of an anelasticmedium requires a convolution with all the past history ofthat medium, which is of course numerically impossible).

In Figure 5 we represent an image of the seismic waves attwo different times, showing how the pressure, shear and sur-face waves propagate across the Earth. In Figure 6 we com-pare normal-mode and SEM synthetic seismograms (i.e., thedisplacement of the Earth’s surface with time at a given loca-tion) at a distance of 5 degrees south of the epicenter in Bo-livia. We find excellent agreement between the two results.In particular, the pressure (P) wave and shear (S) wave ar-rivals, as well as the strong near-field term linking them,are accurately modeled, and the static offset of 6.6 mmon the vertical component of displacement and 7.3 mm onthe North-South component is also well recovered. This so-called static offset corresponds to a permanent displacementof the surface of the Earth around the epicenter, due to thevery large magnitude of the earthquake (this permanent dis-placement is also very clear on the second image of Figure 5).Note also the distinct arrival called ScS on this componentat 800 seconds and the arrival called sScS at 1080 seconds,which are perfectly reproduced. These so-called ScS phasesare waves that travel down to the core-mantle boundary ata depth of 2891 km, where they are reflected because of thefluid-solid impedance contrast, and then travel back to thesurface.

Next, we check the results of our simulation at a seis-mic station located in Pasadena, California, at an epicentraldistance of 68. In Figure 7 we compare our SEM syn-thetic calculation for the vertical component of velocity tothe real data recorded during the actual event. The signals

Figure 5: Movie of the propagation of seismic wavesin the Earth during the large magnitude 8.2 Bo-livia earthquake of June 9, 1994. The waves travelall across the globe. They can be seen propagatingacross the United States for instance. The earth-quake is so large that it produces a permanent dis-placement of the surface of the Earth of several mil-limeters (1/4th of an inch) around the epicenter inBolivia. Note that this effect extends as far Northas the Amazon river.

-1

-0.5

0

0.5

1

1.5

2

2.5

3

0 200 400 600 800 1000 1200

Dis

plac

emen

t (cm

)

Time (s)

SEM North

P

S

ScSsScS

Modes

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

0 200 400 600 800 1000 1200

Dis

plac

emen

t (cm

)

Time (s)

SEM Vertical

P

S

Modes

Figure 6: SEM (red) and quasi-analytical normal-mode reference (blue) synthetic seismograms for thegreat magnitude 8.2 Bolivia earthquake of June 9,1994 recorded 5 degrees south of the epicenter. Thedepth of the earthquake is 647 km. Anisotropy andattenuation are both included in this simulation.Top: North-South component of displacement of theEarth’s surface, Bottom: vertical component. Notethe strong near-field term linking the pressure (P)and shear (S) waves, the large 6.6 mm and 7.3 mmstatic offsets observed on the two components, aswell as the strong ScS and sScS shear waves reflectedoff the fluid outer core of the Earth.

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

500 1000 1500 2000 2500 3000

Con

volv

ed v

eloc

ity (

mm

/s)

Time (s)

Real data verticalSEM vertical

Figure 7: Comparison between our SEM syntheticseismograms for the anelastic, anisotropic referenceEarth model PREM, and real seismic data recordedin Pasadena, California, during the actual June 9,1994, Bolivia earthquake. The vertical componentof the velocity vector is shown. The agreement ob-tained is very good, considering that our SEM re-sults do not include the full complexity due to 3-Dmodel heterogeneity, in particular in the crust andthe mantle of the Earth.

recorded correspond to the waves that travel across South-ern California on the movie of Figure 5. Both records havebeen lowpass-filtered with the same six-pole two-pass But-terworth filter with a corner period of 40 seconds (i.e., a cor-ner frequency of 25 mHz), and our synthetics have been con-volved with the instrument response of the seismic station.The agreement is quite satisfactory, keeping in mind that oursynthetics are based upon the spherically-symmetric refer-ence model PREM and therefore do not include more com-plex effects due to 3-D model heterogeneity. We are cur-rently in the process of taking such 3-D velocity models intoaccount in our SEM.

To illustrate that our implementation of the inner corewith the central cube of Figure 3 is correct, we show in Fig-ure 8 a close-up of the so-called PKP arrivals on the verticalcomponent of the displacement vector around the antipodeof the epicenter of the earthquake. The PKP phases arewaves that have traveled as pressure waves both in the man-tle and in the outer core of the Earth. These phases havethree branches: PKP(AB) and PKP(BC), which travel inthe fluid outer core but not in the solid inner core, andPKP(DF), which goes into the inner core. The PKP(DF)waveform is very sensitive to the very slow shear-wave ve-locity of about 3.6 km.s−1 in the inner core. If this unusu-ally low velocity is not correctly represented numerically,the PKP(DF) waveform changes considerably. This poses achallenge, because if the mesh is not fine enough the veryslow shear-wave velocity is not sampled by enough pointsper seismic wavelength, and as a result strong numericalnoise is generated.

In our results, the PKP(AB) and PKP(BC) outer corebranches as well as the PKP(DF) inner core branch areall very accurately modeled. The PKP(DF) arrival trav-els through the small cube at the center of the inner corewhich is handled by one processor that needs to interactwith all the other processors in the parallel implementation

130 155 180 205 230Epicentral distance (degrees)

15

17

19

21

23

25

Tim

e (m

inut

es)

AB AB

DF

Figure 8: Record section comparison of outer-core and inner-core PKP pressure waves calculatedfor the anelastic, anisotropic Earth velocity modelPREM based upon the SEM (solid lines) and an in-dependent quasi-analytical normal-mode referencecalculation (dotted lines) between 130 and 230. Ateach epicentral distance we plot both the SEM andthe normal-mode solution. All PKP arrivals, in-cluding PKP(DF), which has traveled through thecentral cube in the mesh (Figure 4), are well re-produced. This validates the master/slave parallelprogramming methodology that is used to imple-ment the inner core, as illustrated in Figure 4. Italso demonstrates that we can correctly handle theunusually low value of shear-wave velocity in theEarth’s inner core.

Figure 9: Projected evolution until 2012 of micro-processor clock speed, memory capacity and numberof transistors per chip, showing that around 2010 thescientific community might have access to Petaflopmachines consisting typically of several tens of thou-sands of chips. Courtesy Dr. Tom Sterling, Caltech.

of the method (Figure 4), therefore this result validates allof our MPI implementation.

6. CONCLUSIONS AND PERSPECTIVESWe have developed and implemented a parallel MPI spec-

tral element method (SEM) for the simulation of seismicwave propagation resulting from large earthquakes at thescale of the full 3-D Earth. The method has been bench-marked against a classical quasi-analytical normal-mode ref-erence solution for the spherically symmetric Earth velocitymodel PREM. Excellent agreement has been obtained.

The calculations presented in this paper required 151 pro-cessors and 50 Gb of memory and used tens of hours ofCPU time. These requirements may seem large, but withinten years, computers reaching 1000 teraflops (1 petaflop)will become available, and the calculations presented in thisstudy will be performed routinely in a matter of seconds orminutes. For instance, Figure 9 (courtesy of Dr. Tom Ster-ling from his Supercomputing’2000 presentation) shows theexpected evolution of microprocessor clock speed, memorycapacity and number of transistors per chip. One can seethat according to these projections, around 2010 the scien-tific community might have access to Petaflop machines con-sisting typically of several tens of thousands of chips. Thisis in agreement with a linear extrapolation (in a semilogscale) until 2010 of the evolution of the speed of the fastest

Figure 10: Linear extrapolation in a semilog scale(yellow curve) of the projected evolution of thespeed of the fastest computer of the TOP500list (www.top500.org) until 2010, showing that thefastest computers might reach the petaflop range10 years from now. Courtesy Dr. Tom Sterling,Caltech.

computer of the TOP500 list (www.top500.org). Figure 10(also courtesy of Dr. Tom Sterling from his Supercomput-ing’2000 presentation) shows that the world’s fastest com-puters might reach the petaflop range 10 years from now.Let us mention that the ASCI and the Blue Gene projectsalready plan to reach 30 to 100 Teraflops around 2003-2005.

We are also currently in the process of adding more com-plexity to our simulations in terms of the Earth model used,i.e., fully 3-D mantle models with lateral variations of den-sity and pressure and shear-wave velocities, ellipticity andsurface topography of the Earth, as well as gravity and ro-tation (Coriolis force) which have an effect on long-periodsurface waves.

7. ACKNOWLEDGMENTSThe authors thank Luis Rivera, Philip and Rachel Aber-

crombie, Roland Martin, Tom Sterling, Emmanuel Chaljub,Yann Capdeville, Jan Lindheim, Ewing Lusk, Hans-PeterBunge and Paul F. Fischer for fruitful discussions and com-ments. This material is based in part upon work supportedby the National Science Foundation under Grant No. 0003716.This is Caltech GPS contribution No. 8823.

8. REFERENCES[1] C. Bernardi, N. Debit, and Y. Maday. Coupling finite

element and spectral methods: first results.Mathematics of Computation, 54:21–39, 1990.

[2] L. Brillouin. Tensors in Mechanics and Elasticity.Academic Press, New York, 3rd edition, 1964.

[3] C. Canuto, M. Y. Hussaini, A. Quarteroni, and T. A.Zang. Spectral methods in fluid dynamics.Springer-Verlag, New York, 1988.

[4] Y. Capdeville. Methode couplee elements spectraux -solution modale pour la propagation d’ondes dans laTerre a l’echelle globale (A coupled method usingspectral elements and normal modes for global wavepropagation in the Earth). PhD thesis, UniversiteParis VII Denis Diderot, Paris, France, 2000.

[5] E. Chaljub. Modelisation numerique de la propagationd’ondes sismiques en geometrie spherique: applicationa la sismologie globale (Numerical modeling of thepropagation of seismic waves in spherical geometry:applications to global seismology). PhD thesis,Universite Paris VII Denis Diderot, Paris, France,2000.

[6] F. A. Dahlen and J. Tromp. Theoretical GlobalSeismology. Princeton University Press, Princeton,1998.

[7] A. M. Dziewonski and D. L. Anderson. Preliminaryreference Earth model. Phys. Earth Planet. Inter.,25:297–356, 1981.

[8] E. Faccioli, F. Maggio, R. Paolucci, andA. Quarteroni. 2D and 3D elastic wave propagation bya pseudo-spectral domain decomposition method. J.Seismol., 1:237–251, 1997.

[9] T. J. R. Hughes. The finite element method, linearstatic and dynamic finite element analysis.Prentice-Hall International, Englewood Cliffs, NJ,1987.

[10] D. Komatitsch and J. Tromp. Introduction to thespectral-element method for 3-D seismic wavepropagation. Geophys. J. Int., 139:806–822, 1999.

[11] D. Komatitsch and J. P. Vilotte. The SpectralElement method: an efficient tool to simulate theseismic response of 2D and 3D geological structures.Bull. Seis. Soc. Am., 88(2):368–392, 1998.

[12] A. T. Patera. A spectral element method for fluiddynamics: laminar flow in a channel expansion. J.Comput. Phys., 54:468–488, 1984.

[13] E. Priolo, J. M. Carcione, and G. Seriani. Numericalsimulation of interface waves by high-order spectralmodeling techniques. J. Acoust. Soc. Am.,95(2):681–693, 1994.

[14] C. Ronchi, R. Ianoco, and P. S. Paolucci. The “CubedSphere”: a new method for the solution of partialdifferential equations in spherical geometry. J.Comput. Phys., 124:93–114, 1996.

[15] R. Sadourny. Conservative finite-differenceapproximations of the primitive equations onquasi-uniform spherical grids. Monthly WeatherReview, 100:136–144, 1972.

[16] G. Seriani and E. Priolo. A spectral element methodfor acoustic wave simulation in heterogeneous media.Finite Elements in Analysis and Design, 16:337–348,1994.

[17] M. Taylor, J. Tribbia, and M. Iskandarani. Thespectral element method for the shallow waterequation on the sphere. J. Comput. Phys., 130:92–108,1997.