+ All Categories
Home > Documents > Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone...

Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone...

Date post: 14-Jan-2016
Category:
Upload: edward-bailey
View: 215 times
Download: 0 times
Share this document with a friend
Popular Tags:
38
Parallel Iterative Solvers Parallel Iterative Solvers with the Selective with the Selective Blocking Preconditioning Blocking Preconditioning for Simulations of Fault- for Simulations of Fault- Zone Contact Zone Contact Kengo Nakajima Kengo Nakajima GeoFEM/RIST, Japan. GeoFEM/RIST, Japan. 3rd ACES Workshop, May 5-10, 2002. 3rd ACES Workshop, May 5-10, 2002. Maui, Hawaii, USA. Maui, Hawaii, USA.
Transcript
Page 1: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

Parallel Iterative Solvers with the Parallel Iterative Solvers with the Selective Blocking Selective Blocking

Preconditioning for Simulations Preconditioning for Simulations of Fault-Zone Contactof Fault-Zone Contact

Kengo NakajimaKengo NakajimaGeoFEM/RIST, Japan.GeoFEM/RIST, Japan.

3rd ACES Workshop, May 5-10, 2002.3rd ACES Workshop, May 5-10, 2002.Maui, Hawaii, USA.Maui, Hawaii, USA.

Page 2: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

22

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Solving large-scale linear equations Solving large-scale linear equations Ax=bAx=b is the most important and is the most important and expensiveexpensive part of various types of scientific computing. part of various types of scientific computing. for both linear and nonlinear applicationsfor both linear and nonlinear applications

Various types of methods have been proposed and developed. Various types of methods have been proposed and developed. for dense and sparse matricesfor dense and sparse matrices classified into classified into directdirect and and iterativeiterative methods methods

Dense Matrices : Globally Coupled ProblemsDense Matrices : Globally Coupled Problems BEM, Spectral Methods, MO/MD (gas, liquid)BEM, Spectral Methods, MO/MD (gas, liquid)

Sparse Matrices : Locally Defined ProblemsSparse Matrices : Locally Defined Problems FEMFEM, FDM, DEM, MD (solid), BEM w/FMP, FDM, DEM, MD (solid), BEM w/FMP

I am usually working onI am usually working onsolving Ax=b !!!solving Ax=b !!!

Page 3: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

33

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Gaussian Elimination/LU Factorization.Gaussian Elimination/LU Factorization. compute compute AA-1-1 directly. directly.

Robust for wide range of applications.Robust for wide range of applications. Good for both dense and sparse matricesGood for both dense and sparse matrices

More expensive than iterative methods (memory, CPU)More expensive than iterative methods (memory, CPU) Not suitable for parallel and vector computation due to its global operations.Not suitable for parallel and vector computation due to its global operations.

Direct MethodsDirect Methods

Page 4: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

44

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Stationary Methods (SOR, Gauss-Seidel etc.) and Nonstationary Methods (CG, GMRES, BiCGSTAB etc.)Stationary Methods (SOR, Gauss-Seidel etc.) and Nonstationary Methods (CG, GMRES, BiCGSTAB etc.)

Less expensive than direct methods, especially in memory.Less expensive than direct methods, especially in memory. Suitable for parallel and vector computing.Suitable for parallel and vector computing.

Convergence strongly depends on problems, boundary conditions (condition number etc.)Convergence strongly depends on problems, boundary conditions (condition number etc.) Preconditioning is required.Preconditioning is required.

Iterative MethodsIterative Methods

Page 5: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

55

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Convergence rate of iterative solvers strongly depends on the spectral properties (eigenvalue distribution) of the coefficient matrix Convergence rate of iterative solvers strongly depends on the spectral properties (eigenvalue distribution) of the coefficient matrix AA. .

A preconditioner A preconditioner MM transforms the linear system into one with more favorable spectral properties transforms the linear system into one with more favorable spectral properties In "In "ill-conditionedill-conditioned" problems, "" problems, "condition numbercondition number" (ratio of max/min eigenvalue if " (ratio of max/min eigenvalue if AA is symmetric) is large. is symmetric) is large. MM transforms original equation transforms original equation Ax=bAx=b into into A'x=b'A'x=b' where where A'=MA'=M-1-1AA, , b'=Mb'=M-1-1bb

ILU (Incomplete LU Factorization) or IC (Incomplete Cholesky Factorization) are well-known preconditioners. ILU (Incomplete LU Factorization) or IC (Incomplete Cholesky Factorization) are well-known preconditioners.

Preconditioing for Iterative MethodsPreconditioing for Iterative Methods

Page 6: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

66

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

1.0

10.0

100.0

1000.0

1 10 100 1000

SMP-Node #

GF

LO

PS

3D linear elastic problem for simple cubic geometry on Hitachi SR8000/MPP with 128 SMP nodes (1024 PEs) (not ES40, unfortunately).Block ICCG Solver. The largest problem size so far is 805,306,368 DOF.

Iterative method is the Iterative method is the ONLYONLY choice for large-scale parallel computing. choice for large-scale parallel computing.Problem specific preconditioning method is the most important issue Problem specific preconditioning method is the most important issue although traditional ILU(0)/IC(0) cover wide range of applications. although traditional ILU(0)/IC(0) cover wide range of applications.

Strategy in GeoFEMStrategy in GeoFEM

128 SMP nodes805,306,368 DOF335.2 GFLOPS

16 SMP nodes100,663,296 DOF42.4 GFLOPS

Page 7: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

77

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Contact Problems in Simulations for Earthquake Generation Cycle by GeoFEM.Contact Problems in Simulations for Earthquake Generation Cycle by GeoFEM. Non-linearNon-linear Ill-conditioned problem due to penalty constraint by ALM (Augmented Lagrangean)Ill-conditioned problem due to penalty constraint by ALM (Augmented Lagrangean) AssumptionsAssumptions

Infinitesimal deformation, static contact relationship.Infinitesimal deformation, static contact relationship. Location of nodes is in each "contact pair" is identical.Location of nodes is in each "contact pair" is identical.

No friction : Symmetric coefficient matrixNo friction : Symmetric coefficient matrix

Topics in this PresentationTopics in this Presentation

Special preconditioning : Special preconditioning : Selective Blocking.Selective Blocking. provides robust and smooth convergence in 3D solid mechanics simulations for provides robust and smooth convergence in 3D solid mechanics simulations for

geophysics with contact.geophysics with contact.

Examples on Hitachi SR2201 parallel computer with 128 processing elements. Examples on Hitachi SR2201 parallel computer with 128 processing elements.

Page 8: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

88

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

OVERVIEWOVERVIEWBackgroundBackgroundGeneral Remedy for Ill-Conditioned ProblemsGeneral Remedy for Ill-Conditioned Problems

Deep Fill-inDeep Fill-in BlockingBlocking

Special Method for Fault-Contact ProblemsSpecial Method for Fault-Contact Problems Selective BlockingSelective Blocking Special Repartitioning Special Repartitioning

ExamplesExamples Large Scale Computation on Hitachi SR2201 w/128 PEsLarge Scale Computation on Hitachi SR2201 w/128 PEs

SummarySummary

Page 9: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

99

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Geophysics Application w/ContactGeophysics Application w/ContactAugmented Lagrangean Method with Penalty CoAugmented Lagrangean Method with Penalty Co

nstraint Condition for Contactnstraint Condition for Contact

Eurasia

Philippine

PacificEurasia

Philippine

Pacific

6,156 elements, 7,220 nodes, 21,660 DOF840km1020km600km region

Page 10: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1010

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

1

10

100

1000

10000

1.E+08 1.E+10 1.E+12 1.E+14

Penalty Number

Itera

tion

s

●● Newton Raphson iteration Newton Raphson iteration

▲▲ Solver iteration for entireSolver iteration for entire Newton Raphson iterationNewton Raphson iteration

■■ Solver iteration for Solver iteration for ONEONE Newton Raphson iterationNewton Raphson iteration

Large Penalty providesLarge Penalty provides ・・ Good N-R convergenceGood N-R convergence ・・ Large Condition NumberLarge Condition Number

Optimum ChoiceOptimum Choice

Augmented Lagrangean MethodAugmented Lagrangean MethodPenalty~Iteration Relation for Contact ProblemsPenalty~Iteration Relation for Contact Problems

Newton-Raphson / Iterative Solver Newton-Raphson / Iterative Solver

Page 11: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1111

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Block-type Preconditioning seems to work well for ill-conditioned casesBlock-type Preconditioning seems to work well for ill-conditioned cases

Results in the BenchmarkResults in the Benchmark7,220 nodes, 21,660 DOFs, 7,220 nodes, 21,660 DOFs, =10=10-8-8

GeoFEM's CG solver (scalar version)GeoFEM's CG solver (scalar version)Single PE caseSingle PE case

=1010

IC(0) : 89 iters, 8.9 sec. DIAG : 340 iters, 19.1 sec. Block LU scaling : 165 iters, 11.9 sec.

=1016

IC(0) : >10,000 iters, >1,300.0 sec. DIAG : No Convergence Block LU scaling : 3,727 iters, 268.9 sec.

Page 12: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1212

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

BackgroundGeneral Remedy for Ill-Conditioned ProblemsGeneral Remedy for Ill-Conditioned Problems

Deep Fill-inDeep Fill-in BlockingBlocking

Special Method for Fault-Contact Problems Selective Blocking Special Repartitioning

Examples Large Scale Computation on Hitachi SR2201 w/128 PEs

Summary

Page 13: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1313

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

The world where direct solvers have governed.The world where direct solvers have governed.But iterative methods are the only choice for large-scale massively parallel computation.But iterative methods are the only choice for large-scale massively parallel computation.We need robust preconditioning !!We need robust preconditioning !!

Remedy : Basically Preconditioning like Direct Solver Remedy : Basically Preconditioning like Direct Solver Deep Fill-inDeep Fill-in Blocking and OrderingBlocking and Ordering

Ill-Conditioned ProblemsIll-Conditioned Problems

Page 14: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1414

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Deep Fill-in : LU and ILU(0)/IC(0)Deep Fill-in : LU and ILU(0)/IC(0)Even if A is sparse, AEven if A is sparse, A-1-1 is not necessarily is not necessarily

sparse due to fill-in.sparse due to fill-in.

Gaussian Elimination do i= 2, n do k= 1, i-1 aik := aik/akk do j= k+1, n aij := aij - aik*akj enddo enddo enddo

ILU(0) : keep non-zero pattern of the original coefficient matrix do i= 2, n do k= 1, i-1 if ((i,k)∈ NonZero(A)) thenif ((i,k)∈ NonZero(A)) then aaik ik := a:= aikik/a/akkkk

endifendif do j= k+1, n if ((i,j)∈ NonZero(A)) thenif ((i,j)∈ NonZero(A)) then aaij ij := a:= aijij - a - aikik*a*akjkj

endifendif enddo enddo enddo

DEEP Fill-in

Page 15: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1515

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Deep Fill-in : ILU(p)/IC(p)Deep Fill-in : ILU(p)/IC(p)

LEVLEVijij=0 if ((i,j)∈ NonZero(A)) otherwise LEV=0 if ((i,j)∈ NonZero(A)) otherwise LEVijij= p+1= p+1 do i= 2, ndo i= 2, n do k= 1, i-1do k= 1, i-1 if (LEVif (LEVikik≦p) then≦p) then aaik ik := a:= aikik/a/akkkk

endifendif do j= k+1, ndo j= k+1, n if (LEVif (LEVij ij = min(LEV= min(LEVijij,1+LEV,1+LEVikik+ LEV+ LEVkjkj)≦p) then)≦p) then aaij ij := a:= aijij - a - aikik*a*akjkj

endifendif enddoenddo enddoenddo enddoenddo

DEEP Fill-in

Page 16: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1616

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Close to direct solver if you have DEEPER fill-in.Close to direct solver if you have DEEPER fill-in.

requires additional memory and computation.requires additional memory and computation. x2 for ILU(0) -> ILU(1)x2 for ILU(0) -> ILU(1)

Deep Fill-in : General IssuesDeep Fill-in : General Issues

DEEP Fill-in

Page 17: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1717

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Apply complete/full LU factorization in the certain size block for process DApply complete/full LU factorization in the certain size block for process D-1-1.. Just divided by diagonal component for scalar cases.Just divided by diagonal component for scalar cases.

3x3 block for 3D solid mechanics.3x3 block for 3D solid mechanics. tightly coupled 3-components (u-v-w) on 1-node.tightly coupled 3-components (u-v-w) on 1-node.

Blocking : Forward/Backward Blocking : Forward/Backward Substitution for ILU/IC ProcessSubstitution for ILU/IC Process

M= (L+D)D-1(D+U)

Forward Substitution (L+D)p= q : p= D-1(q-Lp)

Backward Substitution (I+ D-1 U)pnew= pold : p= p - D-1Up

BLOCKING

Page 18: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1818

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

333 Block ILU(0) Preconditioning3 Block ILU(0) PreconditioningForward SubstitutionForward Substitution do i= 1, N SW1= WW(3*i-2,ZP) SW2= WW(3*i-1,ZP) SW3= WW(3*i ,ZP) isL= INL(i-1)+1 ieL= INL(i) do j= isL, ieL k= IAL(j) X1= WW(3*k-2,ZP) X2= WW(3*k-1,ZP) X3= WW(3*k ,ZP) SW1= SW1 - AL(1,1,j)*X1 - AL(1,2,j)*X2 - AL(1,3,j)*X3 SW2= SW2 - AL(2,1,j)*X1 - AL(2,2,j)*X2 - AL(2,3,j)*X3 SW3= SW3 - AL(3,1,j)*X1 - AL(3,2,j)*X2 - AL(3,3,j)*X3 enddo X1= SW1 X2= SW2 X3= SW3 X2= X2 - ALU(2,1,i)*X1 X3= X3 - ALU(3,1,i)*X1 - ALU(3,2,i)*X2 X3= ALU(3,3,i)* X3 X2= ALU(2,2,i)*( X2 - ALU(2,3,i)*X3 ) X1= ALU(1,1,i)*( X1 - ALU(1,3,i)*X3 - ALU(1,2,i)*X2) WW(3*i-2,ZP)= X1 WW(3*i-1,ZP)= X2 WW(3*i ,ZP)= X3 enddo

Full LU FactorizationFull LU Factorizationfor 3x3 Blockfor 3x3 Block

DD-1-1

BLOCKING

Page 19: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

1919

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Iteration number and computation time dramatically decreases by fill-in and blocking.Iteration number and computation time dramatically decreases by fill-in and blocking.

Benchmark : Benchmark : Effect of Fill-in/BlockingEffect of Fill-in/Blocking7,220 nodes, 21,660 DOFs, 7,220 nodes, 21,660 DOFs, =10=10-8-8

CG solver, Single PE caseCG solver, Single PE case

=1016

IC(0) : >10,000 iters, >1,300.0 sec. Block LU scaling : 3,727 iters, 268.9 sec. Block IC(0) : 1,102 iters, 144.3 sec. Block IC(1) : 94 iters, 21.1 sec. Block IC(2) : 33 iters, 15.4 sec.

DEEP Fill-in BLOCKING

Page 20: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2020

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

BackgroundGeneral Remedy for Ill-Conditioned Problems

Deep Fill-in Blocking

Special Method for Fault-Contact ProblemsSpecial Method for Fault-Contact Problems Selective BlockingSelective Blocking Special Repartitioning

Examples Large Scale Computation on Hitachi SR2201 w/128 PEs

Summary

Page 21: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2121

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.Selective BlockingSelective Blocking

Special Method for Contact ProblemSpecial Method for Contact ProblemStrongly coupled nodes are put into the same Strongly coupled nodes are put into the same

diagonal block.diagonal block.

1 2

5 6

9 10

13 14

17 18

25

29 30

26

21 22

1 2 3

10 11 12

19 20 21

28 29 30

37 38 39

46 47 48

55 56 57

64 65 66

73 74 75

82 83 84

91 92 93

ContactContactGroupsGroups

1 2

5 6

9 10

13 14

17 18

25

29 30

26

21 22

1 2 3

10 11 12

19 20 21

28 29 30

37 38 39

46 47 48

55 56 57

64 65 66

73 74 75

82 83 84

91 92 93

1 2

5 6

9 10

13 14

17 18

25

29 30

26

21 22

1 2 3

10 11 12

19 20 21

28 29 30

37 38 39

46 47 48

55 56 57

64 65 66

73 74 75

82 83 84

91 92 93

ContactContactGroupsGroups

Page 22: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2222

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.Selective BlockingSelective Blocking

Special Method for Contact ProblemSpecial Method for Contact ProblemStrongly coupled nodes are put into the same Strongly coupled nodes are put into the same

diagonal block.diagonal block.

Initial Coef. Matrixfind strongly coupled contact groups (each small square:3x3)

Reordered/Blocked Matrixnodes/block

Each block corresponds toa contact group

Page 23: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2323

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Block ILU/IC

Selective Blocking or SupernodeSelective Blocking or Supernode Procedure : Forward Substitution in Lower Tri. PartProcedure : Forward Substitution in Lower Tri. Part

Selective Blocking/Supernode

size of each diagonal block depends on contact group size

Apply full LU factorization for computation of D-1

Page 24: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2424

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Benchmark : Benchmark : SB-BIC(0)SB-BIC(0)Selective Blocking + Block IC(0)Selective Blocking + Block IC(0)

7,220 nodes, 21,660 DOFs, 7,220 nodes, 21,660 DOFs, =10=10-8-8

CG solver, Single PE caseCG solver, Single PE case

=1016

IC(0) : >10,000 iters, >1,300.0 sec. Block LU scaling : 3,727 iters, 268.9 sec. Block IC(0) : 1,102 iters, 144.3 sec. Block IC(1) : 94 iters, 21.1 sec. Block IC(2) : 33 iters, 15.4 sec. SB-Block IC(0) : 82 iters, 11.2 sec.

Page 25: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2525

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Benchmark : Selective BlockingBenchmark : Selective Blocking Selective Blocking converges even if Selective Blocking converges even if =10=102020

1.E-09

1.E-08

1.E-07

1.E-06

1.E-05

1.E-04

1.E-03

1.E-02

1.E-01

1.E+00

1.E+01

0 500 1000 1500 2000Iterations

BIC(1)

BIC(2)SB-BIC(0)

Page 26: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2626

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Benchmark : Benchmark : 4PE cases4PE cases 7,220 nodes, 21,660 DOFs, 7,220 nodes, 21,660 DOFs, =10=10-8-8

=10=101616 ,, CG solverCG solver

Single PE Block IC(0) : 1,102 iters, 144.3 sec. Block IC(1) : 94 iters, 21.1 sec. Block IC(2) : 33 iters, 15.4 sec. SB-BIC(0) : 82 iters, 11.2 sec.

4 PEs Block IC(0) : 2,104 iters, 68.4 sec. Block IC(1) : 1,724 iters, 85.8 sec. Block IC(2) : 962 iters, 69.9 sec. SB-BIC(0) : 1,740 iters, 70.0 sec.

In 4PE case, nodes in tightly connected groups are on different partition and decoupled.In 4PE case, nodes in tightly connected groups are on different partition and decoupled.

Page 27: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2727

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Deep fill-in, blocking and selective-blocking dramatically improve the convergence rate for ill-conditioned problems such as solid mechanics with contact.Deep fill-in, blocking and selective-blocking dramatically improve the convergence rate for ill-conditioned problems such as solid mechanics with contact.

But performance is bad in parallel cases with localized preconditioning when nodes in tightly connected pairs are on different partition and decoupled.But performance is bad in parallel cases with localized preconditioning when nodes in tightly connected pairs are on different partition and decoupled.

Special repartitioning method needed !!Special repartitioning method needed !!

SummarySummary

Page 28: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2828

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

BackgroundGeneral Remedy for Ill-Conditioned Problems

Deep Fill-in Blocking

Special Method for Fault-Contact Problems Selective Blocking Special RepartitioningSpecial Repartitioning

Examples Large Scale Computation on Hitachi SR2201 w/128 PEs

Summary

Page 29: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

2929

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Outline of the RepartitioningOutline of the Repartitioning

BEFOREBEFORErepartitioningrepartitioning

Nodes in contact pairs Nodes in contact pairs are on separated are on separated partition.partition.

AFTER AFTER repartitioningrepartitioning

Nodes in contact pairs Nodes in contact pairs are on same partition, are on same partition, but no load-balancing.but no load-balancing.

AFTERAFTERload-balancingload-balancing

Nodes in contact pairs Nodes in contact pairs are on same partition, are on same partition, and load-balanced.and load-balanced.

Convergence is slow if nodes in each contact group locate on different partition.Repartitioning so that nodes in contact pairs would be in samepartition as INTERIOR nodes will be effective.

Page 30: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3030

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Special RepartitioningSpecial RepartitioningBenchmark: 4PE casesBenchmark: 4PE cases

Precond. Iter # sec. BIC(1) 1010 80 3.8

1016 167 7.4 BIC(2) 1010 71 5.8

1016 74 5.9 1020 No Conv. N/A

SB-BIC(0) 1010 126 2.9 1016 124 2.8 1020 231 5.7

Precond. Iter # sec.BIC(1) 1010 90 4.1

1016 1,724 70.7BIC(2) 1010 86 6.6

1016 962 59.81020 No Conv. N/A

SB-BIC(0) 1010 156 3.51016 1,598 33.91020 2,345 55.5

BEFORE Repartitioning AFTER Repartitioning

Page 31: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3131

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

BackgroundGeneral Remedy for Ill-Conditioned Problems

Deep Fill-in Blocking

Special Method for Fault-Contact Problems Selective Blocking Special Repartitioning

ExamplesExamples Large Scale Computation on Hitachi SR2201 w/128 PEsLarge Scale Computation on Hitachi SR2201 w/128 PEs

SummarySummary

Page 32: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3232

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Large-Scale ComputationLarge-Scale Computation DescriptionDescription

NX1 NX2N

Z1

NZ

2

NZ

1+N

Z2

x= 0

x= N

X1

x= N

X1+

1

x= N

X1+

NX

2+1

z= 0

z= NZ1

z= NZ1+1

z= NZ1+NZ2+1

x

y

z

Page 33: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3333

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Problem Setting & B.C.'sProblem Setting & B.C.'s

MPC at inter-zone boundariesMPC at inter-zone boundaries

Symmetric condition at Symmetric condition at x=0x=0 and and y=0y=0 surfaces surfaces

Dirichlet fixed condition at Dirichlet fixed condition at z=0z=0 surface surface

Uniform distributed load at Uniform distributed load at z= Zmaxz= Zmax surface surface

1 2

5 6

9 10

13 14

17 18

25

29 30

26

21 22

1 2 3

10 11 12

19 20 21

28 29 30

37 38 39

46 47 48

55 56 57

64 65 66

73 74 75

82 83 84

91 92 93

ContactContactGroupsGroups

1 2

5 6

9 10

13 14

17 18

25

29 30

26

21 22

1 2 3

10 11 12

19 20 21

28 29 30

37 38 39

46 47 48

55 56 57

64 65 66

73 74 75

82 83 84

91 92 93

1 2

5 6

9 10

13 14

17 18

25

29 30

26

21 22

1 2 3

10 11 12

19 20 21

28 29 30

37 38 39

46 47 48

55 56 57

64 65 66

73 74 75

82 83 84

91 92 93

ContactContactGroupsGroupsx

y

z

x

y

z

Page 34: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3434

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Sample Mesh Sample Mesh 99 nodes, 80 elements.99 nodes, 80 elements.

1 2

5 6

9 10

13 14

17 18

25

29 30

26

21 22

1 2 3

10 11 12

19 20 21

28 29 30

37 38 39

46 47 48

55 56 57

64 65 66

73 74 75

82 83 84

91 92 93

ContactContactGroupsGroups

Page 35: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3535

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Results on Hitachi SR2201 (128PEs)Results on Hitachi SR2201 (128PEs) NX1=NX2=70, NY=40, NZ1=NZ2=70, NX1=NX2=70, NY=40, NZ1=NZ2=70, Repartitioned.Repartitioned.

2,471,439 DOF, 784,000 Elements2,471,439 DOF, 784,000 ElementsIterations/CPU time until convergence (Iterations/CPU time until convergence (=10=10-8-8))

BIC(0)

/E 102

905 iters 194.5 sec.

104 106

> 8,300 > 1,800.0

108 1010

BIC(1)225 92.5

297 115.2

460165.6

BIC(2)183

139.3201

146.3296

187.7

SB-BIC(0)54269.5

54269.5

54269.5

54369.7

54469.8

Page 36: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3636

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Required MemoryRequired MemoryNX1=NX2=20, NY=20, NZ1=15, NZ2=16NX1=NX2=20, NY=20, NZ1=15, NZ2=16

83,649 DOF, 24,000 Elements83,649 DOF, 24,000 Elements

BIC(0) 105 BIC(0) 105 MBMB

BIC(1) 284 BIC(1) 284 MBMB

BIC(2) 484 BIC(2) 484 MBMB

SB-BIC(0) 128 SB-BIC(0) 128 MBMB

Page 37: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3737

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Concluding RemarksConcluding RemarksRobust Preconditioning Methods for Contact Problem.Robust Preconditioning Methods for Contact Problem. General: Deep Fill-in, Blocking.General: Deep Fill-in, Blocking. Problem Specific: Selective-Blocking using Supernodes.Problem Specific: Selective-Blocking using Supernodes. Large-Scale Problems using 128 PEs of Hitachi SR2201.Large-Scale Problems using 128 PEs of Hitachi SR2201.

Selective-Blocking (SB-BIC(0)) provides robust convergence.Selective-Blocking (SB-BIC(0)) provides robust convergence. More efficient and robust than BIC(0), BIC(1) and BIC(2). More efficient and robust than BIC(0), BIC(1) and BIC(2). Iteration number for convergence remains constant while Iteration number for convergence remains constant while increases. increases.

Page 38: Parallel Iterative Solvers with the Selective Blocking Preconditioning for Simulations of Fault-Zone Contact Kengo Nakajima GeoFEM/RIST, Japan. 3rd ACES.

3838

3rd ACES Workshop, Maui, May5-10, 3rd ACES Workshop, Maui, May5-10, 2002.2002.

Further StudyFurther StudyOptimization for Earth Simulator.Optimization for Earth Simulator.Dynamic Update of Contact Information.Dynamic Update of Contact Information. Large Slip / Large Deformation. Large Slip / Large Deformation. More flexible and robust preconditioner under development such as SPAI (SpaMore flexible and robust preconditioner under development such as SPAI (Spa

rse Approximate Inverse). rse Approximate Inverse).


Recommended