HAL Id: hal-01400643https://hal.inria.fr/hal-01400643
Submitted on 24 Nov 2016
HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.
Resolution strategy for the Hybridizable DiscontinuousGalerkin system for solving Helmholtz elastic wave
equationsMarie Bonnasse-Gahot, Henri Calandra, Julien Diaz, Stephane Lanteri
To cite this version:Marie Bonnasse-Gahot, Henri Calandra, Julien Diaz, Stephane Lanteri. Resolution strategy for theHybridizable Discontinuous Galerkin system for solving Helmholtz elastic wave equations. Face toface meeting HPC4E Brazilian-European project, Sep 2016, Gramado, Brazil. hal-01400643
September 14, 2016
Resolution strategy for the HybridizableDiscontinuous Galerkin system for solvingHelmholtz elastic wave equationsM. Bonnasse-Gahot1,2, H. Calandra3, J. Diaz1 and S. Lanteri21 INRIA Bordeaux-Sud-Ouest, team-project Magique 3D2 INRIA Sophia-Antipolis-Méditerranée, team-project Nachos
3 TOTAL Exploration-Production
Motivations
Principles of seismic imaging
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 2/20
Motivations
Examples of seismic imaging campaigns
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 2/20
Motivations
Imaging methodsI Reverse Time Migration (RTM) : based on the reversibility
of wave equationI Full Wave Inversion (FWI) : inversion process requiring to
solve many forward problems
Seismic imaging : time-domain or harmonic-domain ?I Time-domain : imaging condition complicated but quite low
computational costI Harmonic-domain : imaging condition simple but huge
computational cost
Memory usage
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 3/20
Motivations
Imaging methodsI Reverse Time Migration (RTM) : based on the reversibility
of wave equationI Full Wave Inversion (FWI) : inversion process requiring to
solve many forward problems
Seismic imaging : time-domain or harmonic-domain ?I Time-domain : imaging condition complicated but quite low
computational costI Harmonic-domain : imaging condition simple but huge
computational cost
Memory usage
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 3/20
Motivations
Imaging methodsI Reverse Time Migration (RTM) : based on the reversibility
of wave equationI Full Wave Inversion (FWI) : inversion process requiring to
solve many forward problems
Seismic imaging : time-domain or harmonic-domain ?I Time-domain : imaging condition complicated but quite low
computational costI Harmonic-domain : imaging condition simple but huge
computational cost
Memory usage
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 3/20
Motivations
Resolution of the forward problem of the inversion processI Elastic wave propagation in the frequency domain : Helmholtz
equation
First order formulation of Helmholtz wave equationsx = (x , y , z) ∈ Ω ⊂ R3,
iωρ(x)v(x) = ∇·σ(x) + fs(x)
iωσ(x) = C(x) ε(v(x))
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 4/20
MotivationsResolution of the forward problem of the inversion process
I Elastic wave propagation in the frequency domain : Helmholtzequation
First order formulation of Helmholtz wave equationsx = (x , y , z) ∈ Ω ⊂ R3,
iωρ(x)v(x) = ∇·σ(x) + fs(x)
iωσ(x) = C(x) ε(v(x))
I v : velocity vectorI σ : stress tensorI ε : strain tensor
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 4/20
Motivations
Resolution of the forward problem of the inversion processI Elastic wave propagation in the frequency domain : Helmholtz
equation
First order formulation of Helmholtz wave equationsx = (x , y , z) ∈ Ω ⊂ R3,
iωρ(x)v(x) = ∇·σ(x) + fs(x)
iωσ(x) = C(x) ε(v(x))
I ρ : mass densityI C : elasticity tensor
I fs : source term, fs ∈ L2(Ω)
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 4/20
Approximation methods
Discontinuous Galerkin Methods3 unstructured tetrahedral meshes3 combination between FEM and finite volume method (FVM)3 hp-adaptivity3 easily parallelizable method
7 7 large number of DOF as compared to classical FEM
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 5/20
Approximation methods
Discontinuous Galerkin Methods3 unstructured tetrahedral meshes3 combination between FEM and finite volume method (FVM)3 hp-adaptivity3 easily parallelizable method7 7 large number of DOF as compared to classical FEM
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 5/20
Approximation methods
Discontinuous Galerkin Methods3 unstructured tetrahedral meshes3 combination between FEM and finite volume method (FVM)3 hp-adaptivity3 easily parallelizable method7 7 large number of DOF as compared to classical FEM
b b
b
b
b
b b b
b
b b
b
b
b
b b
b
b
b
b
b
b b
b
bb
bb
b
b
b
b
b
bb
bbb b
b b
bb
bb
bb b
bb
bbb
b b
b
b b
b
b
bb
b
bb b
b
b
b
b bbb
b
b b
b b
b b
b
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 5/20
Approximation methods
Hybridizable Discontinuous Galerkin Methods3 same advantages as DG methods : unstructured tetrahedralmeshes, hp-adaptivity, easily parallelizable method, discontinuousbasis functions3 introduction of a new variable defined only on the interfaces3 lower number of coupled DOF than classical DG methods
7 time-domain increases computational costs
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 6/20
Approximation methods
Hybridizable Discontinuous Galerkin Methods3 same advantages as DG methods : unstructured tetrahedralmeshes, hp-adaptivity, easily parallelizable method, discontinuousbasis functions3 introduction of a new variable defined only on the interfaces3 lower number of coupled DOF than classical DG methods
7 time-domain increases computational costs
b b
b
b
b
b b b
b
b b
b
b
b
b b
b
b
b
b
b
b b
b
bb
bb
b
b
b
b
b
bb
bbb b
b b
bb
bb
bb b
bb
bbb
b b
b
b b
b
b
bb
b
bb b
b
b
b
b bbb
b
b b
b b
b b
b
b
b bb b
b b bb
bbb
bb
b bbb
b b
b
b
b b
b
b
b b
b
b
b b
b
b
b
bb
bb
b
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 6/20
Approximation methods
Hybridizable Discontinuous Galerkin Methods3 same advantages as DG methods : unstructured tetrahedralmeshes, hp-adaptivity, easily parallelizable method, discontinuousbasis functions3 introduction of a new variable defined only on the interfaces3 lower number of coupled DOF than classical DG methods7 time-domain increases computational costs
b b
b
b
b
b b b
b
b b
b
b
b
b b
b
b
b
b
b
b b
b
bb
bb
b
b
b
b
b
bb
bbb b
b b
bb
bb
bb b
bb
bbb
b b
b
b b
b
b
bb
b
bb b
b
b
b
b bbb
b
b b
b b
b b
b
b
b bb b
b b bb
bbb
bb
b bbb
b b
b
b
b b
b
b
b b
b
b
b b
b
b
b
bb
bb
b
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 6/20
Hybridizable Discontinuous Galerkin method
B. Cockburn, J. Gopalakrishnan and R. Lazarov. Unifiedhybridization of discontinuous Galerkin, mixed and continuousGalerkin methods for second order elliptic problems. SIAM Journalon Numerical Analysis, Vol. 47 :1319-1365, 2009.
S. Lanteri, L. Li and R. Perrussel. Numerical investigation of a highorder hybridizable discontinuous Galerkin method for 2dtime-harmonic Maxwell’s equations. COMPEL, 32(3)1112-1138,2013.N.C. Nguyen, J. Peraire and B. Cockburn. High-order implicithybridizable discontinuous Galerkin methods for acoustics andelastodynamics. Journal of Computational Physics, 230 :7151-7175,2011N.C. Nguyen and B. Cockburn. Hybridizable discontinuous Galerkinmethods for partial differential equations in continuum mechanics.Journal of Computational Physics 231 :5955–5988, 2012
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 7/20
HDG method
Contents
Hybridizable Discontinuous Galerkin methodFormulationAlgorithm
Numerical results
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 7/20
HDG method Formulation
HDG formulation of the equations
Local HDG formulationiωρv−∇ · σ = 0
iωσ − Cε (v) = 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 8/20
HDG method Formulation
HDG formulation of the equations
Local HDG formulation
∫
KiωρK vK ·w +
∫KσK : ∇w−
∫∂Kσ∂K · n ·w = 0∫
KiωσK : ξ +
∫K
vK · ∇ ·(
CKξ)−∫∂K
v∂K · CKξ · n = 0
σK and vK are numerical traces of σK and vK respectively on ∂K
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 8/20
HDG method Formulation
HDG formulation of the equations
We define :v∂K = λF , ∀F ∈ Fh,
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 9/20
HDG method Formulation
HDG formulation of the equationsWe define :
v∂K = λF , ∀F ∈ Fh,σ∂K · n = σK · n− τ I
(vK − λF ) , on ∂K
where τ is the stabilization parameter (τ > 0)
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 9/20
HDG method Formulation
HDG formulation of the equations
Local HDG formulation∫
KiωρK vK ·w−
∫K
(∇ · σK) ·w +
∫∂Kτ I(vK − λF ) ·w = 0∫
KiωσK : ξ +
∫K
vK · ∇ ·(
CKξ)−∫∂KλF · CKξ · n = 0
We define :W K =
(Vx
K , VyK , Vz
K , σxxK , σyy
K , σzzK , σxy
K , σxzK , σyz
K)T
Λ =(ΛF1 , ΛF2 , ..., ΛFnf
)T, where nf = card(Fh)
Discretization of the local HDG formulation
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 10/20
HDG method Formulation
HDG formulation of the equationsLocal HDG formulation∫
KiωρK vK ·w−
∫K
(∇ · σK) ·w +
∫∂Kτ I(vK − λF ) ·w = 0∫
KiωσK : ξ +
∫K
vK · ∇ ·(
CKξ)−∫∂KλF · CKξ · n = 0
We define :W K =
(Vx
K , VyK , Vz
K , σxxK , σyy
K , σzzK , σxy
K , σxzK , σyz
K)T
Λ =(ΛF1 , ΛF2 , ..., ΛFnf
)T, where nf = card(Fh)
Discretization of the local HDG formulationAK W K +
∑F∈∂K
CK ,F Λ = 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 10/20
HDG method Formulation
HDG formulation of the equationsLocal HDG formulation∫
KiωρK vK ·w−
∫K
(∇ · σK) ·w +
∫∂Kτ I(vK − λF ) ·w = 0∫
KiωσK : ξ +
∫K
vK · ∇ ·(
CKξ)−∫∂KλF · CKξ · n = 0
We define :W K =
(Vx
K , VyK , Vz
K , σxxK , σyy
K , σzzK , σxy
K , σxzK , σyz
K)T
Λ =(ΛF1 , ΛF2 , ..., ΛFnf
)T, where nf = card(Fh)
Discretization of the local HDG formulationAK W K + CK Λ = 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 10/20
HDG method Formulation
HDG formulation of the equations
Transmission conditionIn order to determine λF , the continuity of the normal componentof σ∂K is weakly enforced, rendering this numerical traceconservative : ∫
F[[σ∂K · n]] · η = 0
Discretization of the transmission condition∑K∈Th
[BK W K + LK Λ
]= 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 11/20
HDG method Formulation
HDG formulation of the equations
Transmission conditionIn order to determine λF , the continuity of the normal componentof σ∂K is weakly enforced, rendering this numerical traceconservative : ∫
F[[σ∂K · n]] · η = 0
Discretization of the transmission condition∑K∈Th
[BK W K + LK Λ
]= 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 11/20
HDG method Formulation
HDG formulation of the equations
Global HDG discretizationAK W K + CK Λ = 0∑K∈Th
[BK W K + LK Λ
]= 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 12/20
HDG method Formulation
HDG formulation of the equations
Global HDG discretizationW K = −(AK )−1CK Λ∑K∈Th
[BK W K + LK Λ
]= 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 12/20
HDG method Formulation
HDG formulation of the equations
Global HDG discretization∑K∈Th
[−BK (AK )−1CK + LK ]Λ = 0
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 12/20
HDG method Algorithm
Main steps of the HDG algorithm
1. Construction of the global matrix Mwith M =
∑K∈Th
[−BK (AK )−1CK + LK
]for K = 1 to Nbtri doComputation of matrices BK , (AK )−1,CK and LK
Construction of the corresponding section of Mend for
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 13/20
HDG method Algorithm
Main steps of the HDG algorithm
1. Construction of the global matrix M2. Construction of the right hand side S
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 13/20
HDG method Algorithm
Main steps of the HDG algorithm
1. Construction of the global matrix M2. Construction of the right hand side S3. Resolution MΛ = S, with a direct solver (MUMPS) or hybridsolver (MaPhys)
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 13/20
HDG method Algorithm
Main steps of the HDG algorithm
1. Construction of the global matrix M2. Construction of the right hand side S3. Resolution MΛ = S, with a direct solver (MUMPS) or hybridsolver (MaPhys)4. Computation of the solutions of the initial problem
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 13/20
HDG method Algorithm
Main steps of the HDG algorithm
1. Construction of the global matrix M2. Construction of the right hand side S3. Resolution MΛ = S, with a direct solver (MUMPS) or hybridsolver (MaPhys)4. Computation of the solutions of the initial problem
for K = 1 to Nbtri doCompute W K = −(AK )−1CK Λ
end for
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 13/20
HDG method Algorithm
MaPhys Vs MUMPSPattern of the HDG global matrix for P1 interpolation and for a 3Dmesh composed of 21 000 elements
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 14/20
HDG method Algorithm
MaPhys Vs MUMPS
Software packages for solving systems of linear equations Ax = b,where A is a sparse matrix
I MUMPS (MUltifrontal Massively Parallel sparse directSolver) :
I Direct factorization A = LU or A = LDLT
I Multifrontal approachI MaPhys (Massively Parallel Hybrid Solver) :
I Direct and iterative methodsI non-overlapping algebraic domain decomposition method
(Schur complement method)I resolution of each local problem thanks to direct solver such as
MUMPS or PaStiX.
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 14/20
Numerical results 3D plane wave in an homogeneous medium
3D plane wave in an homogeneous medium
1000 m
1000 m
1000 m
Configuration of thecomputational domainΩ .
I Physical parameters :I ρ = 1 kg.m−3
I λ = 16 GPaI µ = 8 GPa
I Plane wave :
u = ∇ei(kx x+ky y+kz z)
where kx =ω
vpcos θ0 cos θ1,
ky =ω
vpsin θ0 cos θ1, and
kz =ω
vpsin θ1
I ω = 2πf , f = 8 HzI θ0 = 30, θ1 = 0
I Mesh composed of 21 000elements
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 15/20
Numerical results 3D plane wave in an homogeneous medium
Cluster configuration
Features of the nodes :I 2 Dodeca-core Haswell Intel Xeon E5-2680I Frequency : 2,5 GHzI RAM : 128 GoI Storage : 500 GoI Infiniband QDR TrueScale : 40Gb/sI Ethernet : 1Gb/s
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 16/20
Numerical results 3D plane wave in an homogeneous medium
3D Plane wave : Memory consumption
48 96 192 384 576
1
10
# cores
Mem
ory(G
B)
Maximum local memory for HDG-P2 method
MaPhys8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
MUMPS8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
(matrix order = 772 416, # nz=107 495 424)M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 17/20
Numerical results 3D plane wave in an homogeneous medium
3D Plane wave : Memory consumption
48 96 192 384 576
1
10
# cores
Mem
ory(G
B)
Maximum local memory for HDG-P3 method
MaPhys8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
MUMPS8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
(matrix order = 1 287 360, # nz=298 598 400 )M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 17/20
Numerical results 3D plane wave in an homogeneous medium
3D Plane wave : Execution time
48 96 192 384 576
10
100
# cores
Time(s)
Execution time for the resolution of the HDG-P2 system
MaPhys8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
MUMPS8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
(matrix order = 772 416, # nz=107 495 424 )M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 18/20
Numerical results 3D plane wave in an homogeneous medium
3D Plane wave : Execution time
48 96 192 384 576
100
# cores
Time(s)
Execution time for the resolution of the HDG-P3 system
MaPhys8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
MUMPS8 MPI, 3 threads4 MPI, 6 threads2 MPI, 12 threads
(matrix order = 1 287 360, # nz=298 598 400 )M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 18/20
Conclusions-Perspectives
Conclusion-Perspectives
I HDG method implemented in Total program (WP6)I more detailled analysis of the comparison between MUMPS
and MaPhys (WP3)I comparison between to PaStiX solverI extension to elasto-acoustic caseI call for projects PRACE to test bigger test-cases
M. Bonnasse-Gahot - HDG method for Helmholtz wave equations September 14, 2016 - 19/20
Conclusions-Perspectives
Thank you !
Conclusions-Perspectives
Factorization time (s) for the HDG-P2 system(Matrix order = 772 416, # nz = 107 495 424)
2 nodes 4 nodes 8 nodes 16 nodes 24 nodesMaphys Mumps Maphys Mumps Maphys Mumps Maphys Mumps Maphys Mumps
8 MPI/n., 21.77 42.55 7.18 35.06 2.62 37.54 1.32 43.47 0.37 43.473 t./MPI4 MPI/n. 42.37 44.66 14.05 33.69 5.28 26.80 2.48 31.20 1.1 37.276 t./MPI2 MPI/n. 70.20 69.48 29.11 49.69 10.79 33.44 4.22 27.57 2.69 24.5812 t./MPI