CADDCADD::Component-AveragedComponent-Averaged
Domain DecompositionDomain Decomposition
Dan GordonComputer ScienceUniversity of Haifa
Rachel GordonAerospace Engg.
Technion
January 13, 2008January 13, 2008 11Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 22
Outline of TalkOutline of Talk
CADD – Algebraic explanationCADD – DD explanation The Kaczmarz algorithm (KACZ) KACZ CARP (a CADD method) Applications of CARPCARP-CG: CG acceleration of CARP Sample results
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 33
CADD – Algebraic Explanation – 1CADD – Algebraic Explanation – 1 The equations are divided into blocks
(not necessarily disjoint) Initial estimate: vector x=(x1,…,xn) Suppose x1 is a variable (component
of x) that appears in 3 blocks x1 is “cloned” as y1 , z1 , t1 in the
different blocks. Perform one (or more) iterative
operation(s) on each block (independently, in parallel)
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 44
CADD – Algebraic Explanation – 2CADD – Algebraic Explanation – 2
The internal iterations in each block produce 3 new values for the clones of x1 : y1’ , z1’ , t1’
The next iterative value of x1 is
x1’ = (y1’ + z1’ + t1’)/3 The next iterate is
x’ = (x1’ , ... , xn’) Repeat iterations as needed for
convergence
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 55
CADD – non-overlapping domainsCADD – non-overlapping domains
xx xxyy
00 1111
domain Adomain A domain Bdomain B
external gridexternal gridpoint of Apoint of A
clone of clone of xx11
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 66
CADD – overlapping domainsCADD – overlapping domains
xx xxyy
00 1111
domain Adomain A
domain Bdomain B
external gridexternal gridpoint of Apoint of A
clone of clone of xx11
AA∩∩BB
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 77
CADD – Parallel ImplementationCADD – Parallel Implementation
Every processor is in charge of one domain
Mode: BLOCK-PARALLEL– All processors operate in parallel (each
on its domain)– Processors exchange clone values– All values are updated– New values = average of clones
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 88
CADDCADD Vs. Standard DD Methods Vs. Standard DD Methods
Difference is in handling external grid points:
CADD: A clone of the external grid point is modified by the domain's equations and contributes to new value of grid point.
Standard DD: Value of external grid point is fixed (contributes to RHS)
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 99
KACZKACZ: The Kaczmarz algorithm: The Kaczmarz algorithm
Iterative method, due to Kaczmarz (1937). Rediscovered for CT as ART
Basic idea: Consider the hyperplane defined by each equation
Start from an arbitrary initial point Successively project current point
onto the next hyperplane, in cyclic order
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1010
KACZ: Geometric DescriptionKACZ: Geometric Description
eq. 1
eq. 2eq. 3
initialpoint
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1111
KACZKACZ with Relaxation Parameter with Relaxation ParameterKACZ can be used with a relaxation
parameter λ
λ=1: project exactly on the
hyperplaneλ<1: project in front of hyperplane
λ>1: project beyond the hyperplane
Cyclic relaxation: Eq. i is assigned a
relaxation parameter λi
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1212
Convergence Properties of Convergence Properties of KACZKACZ
KACZ with relaxation (0< λ <2) converges for consistent systems:– Herman, Lent & Lutz, 1978
– Trummer, 1981
For inconsistent systems, KACZ converges cyclically:– Tanabe, 1971
– Eggermont, Herman & Lent, 1981 (for cyclic relaxation parameters).
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1313
Another view of Another view of KACZKACZ
Given the system Ax = b
Consider the "normal equations" system AATy = b, x=ATy
Well-known: KACZ is simply SOR applied to the normal equations
The relaxation parameter of KACZ is the usual relax. par. of SOR
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1414
CARPCARP: Component-Averaged Row Projections: Component-Averaged Row Projections
CARP is a CADD method with KACZ iterations in each domain:
The system Ax=b is divided into blocks B1,…,Bn (need not be disjoint)
Each processor is assigned one (or more) block
All the blocks are processed in parallel Results from blocks are “merged” to form
the next iterate Merging is done by CADD averaging
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1515
Overview of Overview of CARPCARP
domain Adomain A domain Bdomain B
KACZKACZiterationsiterations
KACZKACZiterationsiterations
averagingaveraging
cloningcloning
KACZKACZ in superspace in superspace(with cyclic relaxation)(with cyclic relaxation)
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1616
Convergence of CARPConvergence of CARP
Averaging Lemma: the component-
averaging and cloning operations of
CARP are equivalent to KACZ row-
projections in a certain superspace
(with λ = 1) CARP is equivalent to KACZ in the
superspace, with cyclic relaxation parameters – known to converge
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1717
"Historical" Note"Historical" Note
The term "component averaging" was first used by Censor, Gordon & Gordon (2001) for CAV and BICAV, used for image reconstruction in CT
These are Cimmino-type algorithms, with weights related to the sparsity of the system matrix
CADD, CARP use this concept in a different sense
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1818
CARPCARP Application: Solution of stiff Application: Solution of stiff linear systems from PDEslinear systems from PDEs
Elliptic PDEs w/large convection term result in stiff linear systems (large off-diagonal elements)
CARP is very robust on these systems, as compared to leading solver/preconditioner combinations
Downside: Not always efficient
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 1919
CARPCARP Application: Electron Tomography Application: Electron Tomography(joint work with J.-J. Fernández)(joint work with J.-J. Fernández)
3D reconstructions: Each processor is assigned a block of consecutive slices. Data is in overlapping blobs.
The blocks are processed in parallel. The values of shared variables are
transmitted between the processors which share them, averaged, and redestributed.
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2020
CARP-CGCARP-CG: CG acceleration of : CG acceleration of CARPCARP
CARP is KACZ in some superspace (with cyclic relaxation parameter)
Björck & Elfving (BIT 79): developed CGMN, which is a (sequential) CG-acceleration of KACZ (double sweep, fixed relax. parameter)
We extended this result to allow cyclic relaxation parameters
Result: CARP-CG
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2121
CARP-CGCARP-CG: Properties: Properties
Same robustness as CARP Very significant improvement in
performance on stiff linear systems derived from elliptic PDEs
Very competitive runtime compared to leading solver/preconditioner combinations on systems derived from convection-dominated PDEs
Improved performance in ET
CARP-CGCARP-CG on PDEs on PDEs CG acceleration of projection
methods was done before, but with block projections, requiring multi-coloring. CARP-CG avoids this.
Tests were run on 9 convection-dominated PDEs, comparing CARP-CG with restarted GMRES, CGS and Bi-CGSTAB, with and without various preconditioners. Also tested: CGNR.
Domain: unit cube – 80x80x80.
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2222
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2323
512,000 equations
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2424
512,000 equations
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2525
512,000 equations
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2626
512,000 equations
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2727
512,000 equations
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2828
CARP-CGCARP-CG: ET results: ET results
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 2929
Future research on Future research on CADDCADD Different internal solvers:
1. Guaranteed convergence?2. Symmetrizability?
Combine CADD with multigrid CFD applications Additional biomedical applicationsCADD with preconditioners Compare CADD with other DD
methods, such as additive Schwarz
January 13, 2008January 13, 2008 Component-Averaged Domain DecompositionComponent-Averaged Domain Decomposition 3030
CARPCARP Publications PublicationsCARP: SIAM J. Sci. Comput. 27
(2005) 1092-1117.
http://cs.haifa.ac.il/~gordon/carp.pdf
Electron tomography: J. Par. & Dist.
Comput. (2007), in press.
http://cs.haifa.ac.il/~gordon/eltom.pdf
CARP-CG: Submitted for publication.