1

Ake Bjorck Germund Dahlquist

Linkoping University Royal Institute of Technology

Numerical Methods

in

Scientific Computing

Volume II

Working copy, April 10, 2008

siam

cThis material is the property of the authors and is for the sole and exclusive useof the students enrolled in specific courses. It is not to be sold, reproduced, orgenerally distributed.

2

Contents

7 Direct Methods for Linear System 17.1 Linear Algebra and Matrix Computations . . . . . . . . . . . . 1

7.1.1 Matrix Algebra . . . . . . . . . . . . . . . . . . . . . 27.1.2 Submatrices and Block Matrices . . . . . . . . . . . 97.1.3 Permutations and Determinants . . . . . . . . . . . 147.1.4 The Singular Value Decomposition . . . . . . . . . . 197.1.5 Norms of Vectors and Matrices . . . . . . . . . . . . 247.1.6 Matrix Multiplication . . . . . . . . . . . . . . . . . 307.1.7 Floating-Point Arithmetic . . . . . . . . . . . . . . . 317.1.8 Complex Matrix Computations . . . . . . . . . . . . 34

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377.2 Gaussian Elimination . . . . . . . . . . . . . . . . . . . . . . . . 40

7.2.1 Triangular Systems and Gaussian Elimination . . . 417.2.2 LU Factorization . . . . . . . . . . . . . . . . . . . . 487.2.3 Pivoting Strategies . . . . . . . . . . . . . . . . . . . 537.2.4 Computational Variants . . . . . . . . . . . . . . . . 597.2.5 Computing the Inverse . . . . . . . . . . . . . . . . 64

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687.3 Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 69

7.3.1 Symmetric Positive Definite Matrices . . . . . . . . 697.3.2 Cholesky Factorization . . . . . . . . . . . . . . . . 747.3.3 Inertia of Symmetric Matrices . . . . . . . . . . . . 787.3.4 Symmetric Indefinite Matrices . . . . . . . . . . . . 79

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 837.4 Banded Linear Systems . . . . . . . . . . . . . . . . . . . . . . . 84

7.4.1 Multiplication of Banded Matrices . . . . . . . . . . 847.4.2 LU Factorization of Banded Matrices . . . . . . . . 867.4.3 Tridiagonal Linear Systems . . . . . . . . . . . . . . 907.4.4 Inverses of Banded Matrices . . . . . . . . . . . . . 94

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

i

ii Contents

7.5 Block Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 977.5.1 Linear Algebra Software . . . . . . . . . . . . . . . . 977.5.2 Block and Blocked Algorithms . . . . . . . . . . . . 997.5.3 Recursive Fast Matrix Multiply . . . . . . . . . . . . 1057.5.4 Recursive Blocked Matrix Factorizations . . . . . . . 1077.5.5 Kronecker Systems . . . . . . . . . . . . . . . . . . . 110

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1127.6 Perturbation Theory and Condition Estimation . . . . . . . . . 113

7.6.1 Numerical Rank of Matrix . . . . . . . . . . . . . . 1137.6.2 Conditioning of Linear Systems . . . . . . . . . . . . 1147.6.3 Component-Wise Perturbation Analysis . . . . . . . 1187.6.4 Backward Error Bounds . . . . . . . . . . . . . . . . 1217.6.5 Estimating Condition Numbers . . . . . . . . . . . . 123

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1267.7 Rounding Error Analysis . . . . . . . . . . . . . . . . . . . . . . 127

7.7.1 Error Analysis of Gaussian Elimination . . . . . . . 1277.7.2 Scaling of Linear Systems . . . . . . . . . . . . . . . 1327.7.3 Iterative Refinement of Solutions . . . . . . . . . . . 1357.7.4 Interval Matrix Computations . . . . . . . . . . . . 138

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1427.8 Sparse Linear Systems . . . . . . . . . . . . . . . . . . . . . . . 142

7.8.1 Storage Schemes for Sparse Matrices . . . . . . . . . 1447.8.2 Graph Representation of Matrices. . . . . . . . . . . 1467.8.3 Nonzero Diagonal and Block Triangular Form . . . . 1487.8.4 LU Factorization of Sparse Matrices . . . . . . . . . 1507.8.5 Cholesky Factorization of Sparse Matrices . . . . . . 153

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1587.9 Structured Systems . . . . . . . . . . . . . . . . . . . . . . . . . 158

7.9.1 Toeplitz and Hankel Matrices . . . . . . . . . . . . . 1597.9.2 Cauchy-Like Matrices . . . . . . . . . . . . . . . . . 1617.9.3 Vandermonde systems . . . . . . . . . . . . . . . . . 162

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

8 Linear Least Squares Problems 1678.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

8.1.1 The Least Squares Principle . . . . . . . . . . . . . 1678.1.2 The GaussMarkov Theorem . . . . . . . . . . . . . 1718.1.3 Orthogonal and Oblique Projections . . . . . . . . . 1748.1.4 Generalized Inverses and the SVD . . . . . . . . . . 1768.1.5 Matrix Approximation and the SVD . . . . . . . . . 1808.1.6 Elementary Orthogonal Matrices . . . . . . . . . . . 183

Contents iii

8.1.7 Angles Between Subspaces and the CS Decomposition189Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1948.2 The Method of Normal Equations . . . . . . . . . . . . . . . . . 195

8.2.1 Forming and Solving the Normal Equations . . . . . 1958.2.2 Recursive Least Squares. . . . . . . . . . . . . . . . 2008.2.3 Perturbation Bounds for Least Squares Problems . . 2018.2.4 Stability and Accuracy with Normal Equations . . . 2048.2.5 Backward Error Analysis . . . . . . . . . . . . . . . 2078.2.6 The PetersWilkinson method . . . . . . . . . . . . 208

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2108.3 Orthogonal Factorizations . . . . . . . . . . . . . . . . . . . . . 213

8.3.1 Householder QR Factorization . . . . . . . . . . . . 2138.3.2 Least Squares Problems by QR Factorization . . . . 2208.3.3 GramSchmidt QR Factorization . . . . . . . . . . . 2238.3.4 Loss of Orthogonality in GS and MGS . . . . . . . 2278.3.5 Least Squares Problems by GramSchmidt . . . . . 2308.3.6 Condition and Error Estimation . . . . . . . . . . . 2348.3.7 Iterative Refinement of Least Squares Solutions . . . 235

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2398.4 Rank Deficient Problems . . . . . . . . . . . . . . . . . . . . . . 240

8.4.1 Modified Least Squares Problems . . . . . . . . . . . 2408.4.2 Regularized Least Squares Solutions . . . . . . . . . 2448.4.3 QR Factorization of Rank Deficient Matrices . . . . 2468.4.4 Complete QR Factorizations . . . . . . . . . . . . . 2478.4.5 Subset Selection by SVD and RRQR . . . . . . . . . 2508.4.6 Bidiagonalization and Partial Least Squares . . . . . 251

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2588.5 Some Structured Least Squares Problems . . . . . . . . . . . . . 261

8.5.1 Blocked Form of QR Factorization . . . . . . . . . . 2618.5.2 Block Angular Least Squares Problems . . . . . . . 2648.5.3 Banded Least Squares Problems . . . . . . . . . . . 2678.5.4 Block Triangular Form . . . . . . . . . . . . . . . . 2708.5.5 General Sparse QR Factorization . . . . . . . . . . . 2728.5.6 Kronecker and Tensor Product Problems . . . . . . 276

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2788.6 Some Generalized Least Squares Problems . . . . . . . . . . . . 279

8.6.1 Least Squares for General Linear Models . . . . . . 2798.6.2 Indefinite Least Squares . . . . . . . . . . . . . . . . 2838.6.3 Linear Equality Constraints . . . . . . . . . . . . . . 2868.6.4 Quadratic Inequality Constraints . . . . . . . . . . . 2888.6.5 Linear Orthogonal Regression . . . . . . . . . . . . . 294

iv Contents

8.6.6 The Orthogonal Procrustes Problem . . . . . . . . . 297Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2998.7 The Total Least Squares Problem . . . . . . . . . . . . . . . . . 299

8.7.1 Total Least Squares and the SVD . . . . . . . . . . 2998.7.2 Conditioning of the TLS Problem . . . . . . . . . . 3028.7.3 Some Generalized TLS Problems . . . . . . . . . . . 3048.7.4 Bidiagonalization and TLS Problems. . . . . . . . . 3078.7.5 Iteratively Reweighted Least Squares. . . . . . . . . 309

Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310Problems and Computer Exercises . . . . . . . . . . . . . . . . . . . . . 311

9 Matrix Eigenvalue Problems 3159.1 Basic