Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration

under contract DE-AC04-94AL85000.

Eric Phippsand

Andy SalingerApplied Computational Methods Department

Sandia National Laboratories

9th Copper Mountain Conference on Iterative MethodsApril 7, 2006

Solving Bordered Systems of Linear Equations for Large-Scale

Continuation and Bifurcation Analysis

Why Do We Need Stability Analysis Algorithms for Large-Scale Applications?

Nonlinear systems exhibit instabilities, e.g.:

• Buckling• Ignition • Onset of Oscillations• Phase Transitions

These phenomena must be understood in order to perform computational design and optimization.

Established stability/bifurcation analysis libraries exist:

• AUTO (Doedel, et al)• CONTENT (Kuznetsov, et al)• MATCONT (Govaerts, et al)• etc…

We need algorithms, software, and experience to impact ASC- and SciDAC-sized applications (millions of unknowns)

Stability/bifurcation analysis provides qualitative information about time evolution of nonlinear systems by computing families of steady-state solutions.

LOCA: Library of Continuation Algorithms

LOCA: Library of Continuation Algorithms

LOCA provides:• Parameter Continuation: Tracks a family of

steady state solutions with parameter

• Linear Stability Analysis: Calculates leading eigenvalues via Anasazi (Thornquist, Lehoucq)

• Bifurcation Tracking: Locates neutral stability point (x,p) and tracks as a function of a second parameter

Application code provides:• Nonlinear steady-state residual and Jacobian fill:

• Newton-like linear solves:

External force

Second parameter

Ext

erna

l for

ce1

1

3

LOCA Designed for Easy Linking to Existing Newton-based Applications

Algorithmic choices for LOCA: • Must work with iterative (approximate) linear solvers on

distributed memory machines• Non-Invasive Implementation (e.g. matrix blind) • Should avoid or limit:

Requiring more derivatives Changing sparsity pattern of matrix Increasing memory requirements

LOCA targets existing codes that are:• Steady-State, Nonlinear• Newton’s Method• Large-Scale, Parallel

Bordering Algorithms Meet These Requirements

… but 4 solves of J per Newton iteration are used to drive J singular!

Turning Point Bifurcation Full Newton Algorithm

Bordering Algorithm

These Techniques Have Been Applied To Many Interesting Problems

-2.6

-2.4

-2.2

-2

-1.8

-1.6

-1.4

0 0.1 0.2 0.3 0.4 0.5

mixedperp

free energy

εwA

Capillary Condensation Flow in CVD Reactor Yeast Cell-Cycle Control

Buckling of Garden Hose Block Copolymer Self-Assembly Propane&Propylene Combustion

LOCA has been rewritten as part of the Trilinos framework

Better algorithms can be implemented with tighter coupling to linear algebra

LOCA can build upon the existing interface between application codes and the NOX nonlinear solver

Sandia’s parallel iterative solvers are made available to LOCA users

• Collection of parallel solver packages, with a common build process, made interoperable where appropriate (Heroux et al)

Trilinos 6.0 Released: September 1, 2005

2004

2004HPC Software Challenge

Bordered Systems of Equations

Only requires solves of J but • Requires m+l linear solves

• Has difficulty when J is nearly singular

Solving bordered systems of equations is a ubiquitous computation:

Pseudo-Arclength Continuation

Constraint Following

Turning Point Identification

Bordering Algorithm

Solving Bordered Systems via QR

1H.F Walker, SIAM J. Sci. Comput., 19992R. Schreiber, SIAM J. Stat. Comput., 1989

Extension of Householder pseudo-arclength technique by Homer Walker1

QR Factorization Compact WY Representation2

where

Rearranged Bordered System

Write then

P is nxn, nonsingular, rank m update to J

Snap-through Buckling of a Shallow Cap(66K tri-shell elements, 200K unknowns, 16 procs)

Salinas (Reese et al., SNL):

• Unstructured finite element, linear elasticity

• Corotational formulation for beams and shells (C. Felippa, CU-Boulder)

• Updated Lagrangian for solid elements

• Analytic, Sparse Jacobian

• Fully Coupled Newton Method (NOX)

• GMRES (Aztec) with RILU(k) Preconditioner (Ifpack)

• Distributed Memory Parallelism

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Turning Point IdentificationMoore-Spence Formulation

… but 4 solves of J per Newton iteration are used to drive J singular!

Turning Point Bifurcation Full Newton Algorithm

Bordering Algorithm

Snap-through Buckling of a Shallow CapTurning Point Bordering Method

• RILU fill factor: 6

• RILU overlap: 6

• Krylov space: 2000

•

Solve 5 bordered systems of equations using QR approach

Then

Modified Turning Point Bordering Algorithm

Snap-through Buckling of a Shallow CapModified Turning Point Bordering Method

• RILU fill factor: 6

• RILU overlap: 6

• Krylov space: 2000

•

Given and , let

then

There are constants such that

Standard formulation:

Note for Newton’s method:

3 linear solves per Newton iteration (5 for modified bordering)!For symmetric problems reduces to 2 solves.

Minimally Augmented Turning Point Formulation

With iterative solvers,

so define

where

Also

Minimally Augmented Turning Point Formulation for Large-Scale Problems

Snap-through Buckling of a Shallow CapMinimally Augmented Formulation

• RILU fill factor: 6

• RILU overlap: 6

• Krylov space: 2000

• Modified bordering

• Minimally augmented

Pseudo-Arclength Continuation of Turning Points

Pseudo-Arclength Equations

Minimally Augmented: 3 total linear solves

Moore-Spence w/Bordering:7 total linear solves

Newton Solve

Snap-through Buckling of a Symmetric Cap Pseudo-arclength Turning Point Continuation

Method Continuation Steps

Failed Steps

Nonlinear Iterations

Linear Solves

Linear Iterations

Total Time (hrs)

Moore-Spence Mod. Bordering

13 1 54 427 117797 4.2

Min. Augmented 12 0 49 122 21888 1.2

Summary

• QR approach provides a convenient way to solve bordered systems– Nonsingular– Only involves one linear solve– Only requires simple vector operations– Doesn’t change dimension of the linear system– Become a workhorse tool in LOCA

• Highly encouraged by minimally augmented turning point formulation– No singular matrix solves– Improves robustness, scalability and accuracy– Requires linear algebra-specific implementation

• Future work– Minimally augmented pitchfork, Hopf bifurcations– Preconditioners for

Points of Contact• LOCA: Trilinos continuation and bifurcation package

– Sub-package of NOX– Andy Salinger (agsalin@sandia.gov)– Eric Phipps (etphipp@sandia.gov)– www.software.sandia.gov/nox

• NOX: Trilinos nonlinear solver package– Roger Pawlowski (rppawlo@sandia.gov)– Tammy Kolda (tgkolda@sandia.gov)– www.software.sandia.gov/nox

• Trilinos: Collection of large-scale linear/nonlinear solvers– Epetra, Ifpack, AztecOO, ML, NOX, LOCA, …– Mike Heroux (maherou@sandia.gov)– www.software.sandia.gov/Trilinos

• Trilinos Release 6.0 currently available, 7.0 this fall– 7.0 will include all LOCA algorithms presented here

• Salinas: Massively Parallel Structural Dynamics Finite Element Code– Garth Reese (gmreese@sandia.gov)

With iterative solvers, solves for u and v are in-exact. Does this impact nonlinear convergence? Instead solve for updates to u and v every nonlinear iteration:

Modified Minimally Augmented Turning Point Formulation for Large-Scale Problems

Snap-through Buckling of a Shallow CapModified Minimally Augmented Formulation

• RILU fill factor: 6

• RILU overlap: 6

• Krylov space: 2000

•

• RILU fill factor: 6

• RILU overlap: 6

• Krylov space: 2000

• Bordering

• Minimally augmented

Snap-through Buckling of a Shallow CapMinimally Augmented Formulation