10/2/2019
1
Computational Science:
Computational Methods in Engineering
Numerical Linear Algebra
Outline
•Direct Solution Methods• Naïve Gauss elimination• Gauss‐Jordan method• LU decomposition
• Iterative Solution Methods• Jacobi method• Gauss‐Seidel method
•Calculating matrix inverse
2
1
2
10/2/2019
2
3
Naïve Gauss Elimination
What is Naïve Gauss Elimination?
4
Naïve Gauss elimination (GE) is the simplest method for solving a system of equations.
A x b
While simple, it can be unstable and “pivoting” is required to stabilize it. The Naïve algorithm ignores this.
In fact, this was sort of already implemented when solving the first system of linear equations. It was just not done in matrix form.
3
4
10/2/2019
3
Step 1
5
Start with a matrix problem…
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3
a a a x b
a a a x b
a a a x b
Step 2
6
Eliminate x1 from rows 2 and 3.
21
11
31
11
New Row 2 Old Row 2 Row 1
New Row 3 Old Row 3 Row 1
a
a
a
a
11 12 13 1 1 11 12 13 1 1
21 22 23 2 2 22 23 2 2
31 32 33 3 3 32 33 3 3
0
0
a a a x b a a a x b
a a a x b a a x b
a a a x b a a x b
5
6
10/2/2019
4
Step 3
7
Eliminate x2 from row 3.
32
22
New Row 3 Old Row 3 Row 2a
a
11 12 13 1 1 11 12 13 1 1
22 23 2 2 22 23 2 2
32 33 3 3 33 3 3
0 0
0 0 0
a a a x b a a a x b
a a x b a a x b
a a x b a x b
Step 4
8
Now x3 is known.
33
33
bx
a
11 12 13 1 1
22 23 2 2
33 3 3
0
0 0
a a a x b
a a x b
a x b
7
8
10/2/2019
5
Step 5
9
Back‐substitute to find x1 and x2.
23 132 1 122 3 1 2 3
22 22 11 11 11
a ab b a
x x x x xa a a a a
11 12 13 1 1
22 23 2 2
33 3 3
0
0 0
a a a x b
a a x b
a x b
Triangular Matrices
10
Notice that an upper triangular matrix is formed from Gauss elimination.
Triangular matrices represent systems of equations that are “almost” solved.
It is usually a very quick and easy procedure to solve triangular matrices.
11 12 13 1 1
22 23 2 2
33 3 3
0
0 0
a a a x b
a a x b
a x b
9
10
10/2/2019
6
Observation
11
Observe that three special factors were calculated.
21
11
31
11
32
2
2
1
2
1
3
32
l
l
l
a
a
a
a
a
a
These will arise again in LU decomposition.
12
Gauss‐Jordan Method
11
12
10/2/2019
7
What is the Gauss‐Jordan Method?
13
The Gauss‐Jordan method is a technique to solve
A x b
or to calculate matrix inverses.
1
A
It is an excellent technique for solving these problems by hand!
Step 1
14
Start with a matrix problem [A][x] = [b] .
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3
a a a x b
a a a x b
a a a x b
13
14
10/2/2019
8
Step 2
15
Construct an augmented matrix from [A] and [b].
11 12 13
21 22 23
1
2
31 32 33 3
a a a
a a a
a
b
b
ba a
A b
11 12 13
21 22 23
31 32 33
1 1
2
3
2
3
a x
x
a a
x
b
ba a a
a a a b
Step 3
16
Normalize the first row by dividing by the diagonal element a11.
11 12 13 1 12 11 13 11 1 11
21 22 23 2 21 22 23 2
31 32 33 3 31 32 33 3
1
a a a b a a a a b a
a a a b a a a b
a a a b a a a b
1312 112 13 1
11 11 11
aa b
a a ba a a
12 13 1
21 22 23 2
31 32 33 3
1
a a b
a a a b
a a a b
15
16
10/2/2019
9
Step 4
17
Eliminate x1 from all other rows.
12 13 1 12 13 1
21 22 23 2 22 23 2
31 32 33 3 32 33 3
1 1
0
0
a a b a a b
a a a b a a b
a a a b a a b
22 22 21 12
23 23 21 13
2 2 21 1
a a a a
a a a a
b b a b
32 32 31 12
33 33 31 13
3 3 31 1
a a a a
a a a a
b b a b
New Row 2 New Row 3
2
31
1
New R
New Row 2
ow 3
Old Row 2
Old Row 3 Row 1
Row 1a
a
Step 5
18
Normalize the second row by dividing by the diagonal element .
12 13 1 12 13 1
22 23 2 23 22 2 22
32 33 3 32 33 3
1 1
0 0 1
0 0
a a b a a b
a a b a a b a
a a b a a b
22a
12 13 1
23 2
32 33 3
1
0 1
0
a a b
a b
a a b
23 23 22 2 2 22 a a a b b a
17
18
10/2/2019
10
Step 6
19
Eliminate x2 from all other rows.
12 13 1 13 1
23 2 23 2
32 33 3 33 3
1 1 0
0 1 0 1
0 0 0
a a b a b
a b a b
a a b a b
13 13 12 23
1 1 12 2
a a a a
b b a b
33 33 32 23
3 3 32 2
a a a a
b b a b
New Row 1 New Row 3
1
32
2
New R
New Row 1
ow 3 Old Row 3 Row
Old Row 1 Row 2
2
a
a
Step 7
20
Normalize the third row by dividing by the diagonal element .
13 1 13 1
23 2 23 2
33 3 3 33
1 0 1 0
0 1 0 1
0 0 0 0 1
a b a b
a b a b
a b b a
33a
13 1
23 2
3
1 0
0 1
0 0 1
a b
a b
b
3 3 33b b a
19
20
10/2/2019
11
Step 8
21
Eliminate x3 from all other rows.
13 1 1
23 2 2
3 3
1 0 1 0 0
0 1 0 1 0
0 0 1 0 0 1
a b b
a b b
b b
1
23
3
New R
New Row 1
ow 2 Old Row 2 Row
Old Row 1 Row
3
3a
a
1 1 13 3b b a b 2 2 23 3b b a b New Row 1 New Row 2
Step 9
22
Extract solution from augmented matrix.
11
3
2 2
3
x b
b
b
x
x
1
2
3
1 0 0
0 1 0
0 0 1
b
b
b
21
22
10/2/2019
12
Example
23
Step 1 – Define problem
Step 2 – Form augmented matrix
Step 3 – Normalize row 1
Step 4 – Subtract row 1 from rows 2 and 3
Step 5 – Normalize row 2
Step 6 – Subtract row 2 from row 1 and 3
Step 7 – Normalize row 3
Step 8 – Subtract row 3 from row 1 and 2
Step 9 – Extract answer from augmented matrix
1 0 0 0
0 1 0 2
0 0 1 4
1 0 2.5 10
0 1 0.25 3
0 0 1 4
1 0 2.5 10
0 1 0.25 3
0 0 2.75 11
1 2 3 16
0 1 0.25 3
0 1 3 14
1 2 3 16
0 4 1 12
0 1 3 14
1 2 3 16
0 4 1 12
1 1 0 2
1 2 3 16
0 4 1 12
1 1 0 2
A b
1 2 3 16
0 4 1 12
1 1 0 2
A b
0
2
4
x
* Be sure to extract the answer from the augmented matrix. The augmented matrix is not the final answer!
Algorithm for Any Size Matrix
24
1. Define [A] and [b]2. Construct augmented matrix
3. Iterate through all rows (m)a) Normalize mth row by dividing by diagonal element.
b) Iterate through all other rows (r), skipping the mth rowi. Subtract row m from row r
4. Extract [x] from augmented matrix
U A b
row row row rmr r mU U a U
U(r,:) = U(r,:) – U(r,m)*U(m,:);
U = [ A b ];
x = U(:,M+1);
column 1Mx U
row row mmm mU U a U(m,:) = U(m,:)/U(m,m);
23
24
10/2/2019
13
How to Find Matrix Inverses
25
1. Define [A]2. Construct augmented matrix
3. Perform Gauss‐Jordan method (iterate through all rows)4. Extract [A]‐1 from augmented matrix
U A I
1U I A
26
LU Decomposition
25
26
10/2/2019
14
Determining the Upper Triangular Matrix [U]
27
Start with
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3
a a a x b
a a a x b A x b
a a a x b
At some point during GE, this system of equations was converted to an upper‐triangular matrix [U].
11 12 13 1 1
22 23 2 2
33 3 3
0
0 0
u u u x d
u u x d U x d
u x d
Determining the Lower Triangular Matrix [L]
28
There exists a lower‐triangular matrix [L] such that
1
0
0
0
A x b
A x b
L U x b
L U x L b
Recall these special terms.
A L U This equation is why the method is called LU decomposition.
Now substitute this decomposition into the original equation.
21
31 32
1 0 0
1 0
1
l
l l
L
10 where L U x d d L b
27
28
10/2/2019
15
Algorithm
29
Step 1 – Decompose [A] into [L] and [U].
a) Use GE to calculate [U]b) Store l terms during GEc) Build [L] from l terms.d) store [L] and [U] together as
Step 2 – Solve [L][d]=[b] using simple forward substitution.
Step 3 – Solve [U][x]=[d] using simple backward substitution.
A x b
U L
L d b
U x d
x
d
LU
Decomposition
[d] by forw
ard
substitution
[x] by backw
ard
substitution
11 12 13
21 22 23
31 32 33
a a a
l a a
l l a
30
Jacobi Method
29
30
10/2/2019
16
What is the Jacobi Method?
31
The Gauss‐Jordan method was a direct solution of [A][x]=[b].
This can be inefficient for large matrices, especially when a good initial guess [x] is known.
We can create an iterative algorithm that improves the initial guess every iteration.
The method only converges for diagonally dominant matrices.The algorithm is very picky about this!
1x
Done?
1i i ix x x
Calculate i
x
no
yesFinished!
Diagonally Dominant Matrices
32
A square matrix is said to be diagonally dominant if for each row, the magnitude of the diagonal element is greater than or equal to the sum of the magnitudes of all other elements in that row.
ii ijj i
a a
Examples
1 2
3 4A
Not diagonally dominant.
3 1 1
0 1 7
2 1 4
A
2 0 1
1 2 2
5 4 8
A
4 2 1
1 3 1
3 2 8
A
Diagonally dominant!
Not diagonally dominant.
Not diagonally dominant.
31
32
10/2/2019
17
Formulation (1 of 2)
33
The matrix problem is
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3
a a a x b
a a a x b
a a a x b
Expand this into its component equations.
11 1 12 2 13 3 1
21 1 22 2 23 3 2
31 1 32 2 33 3 3
a x a x a x b
a x a x a x b
a x a x a x b
Formulation (2 of 2)
34
Solve the first equation for x1, the second equation for x2, and the third equation for x3.
1 12 2 13 31
11
2 21 1 23 32
22
3 31 1 32 23
33
b a x a xx
a
b a x a xx
a
b a x a xx
a
These are the equations that we be iterated.
11 1 12 2 13 3 1
21 1 22 2 23 3 2
31 1 32 2 33 3 3
a x a x a x b
a x a x a x b
a x a x a x b
33
34
10/2/2019
18
Implementation (1 of 2)
35
Step 1 – Define Problem
11 12 13 1
21 22 23 2
31 32 33 3
a a a b
A a a a b b
a a a b
Step 2 – Come up with an initial guess for [x].
1 1 11 2 3, , and x x x
Implementation (2 of 2)
36
Step 3 – Iterate [x] until convergence
a) Calculate new values of [x]
1 1 11 12 2 13 3 2 21 1 23 3 3 31 1 32 21 2 3
11 22 33
i i i i i i
i i ib a x a x b a x a x b a x a xx x x
a a a
b) Check how much the values have changed
11 11 1 1 3 31 1 2 2
1 2 31 1 11 2 3
i ii i i i
i i i
i i i
x xx x x x
x x x
c) Continue to iterate until all values of are sufficiently small.
35
36
10/2/2019
19
Matrix Formulation (1 of 2)
37
Rearrange the update equations this way
1 12 2 13 31 1 1 12 2 13 3
11 11
2 21 1 23 32 2 2 21 1 23 3
22 22
3 31 1 32 23 3 3 31 1 32 2
33 33
1
1
1
b a x a xx x b a x a x
a a
b a x a xx x b a x a x
a a
b a x a xx x b a x a x
a a
By inspecting these equations, the update equation can be written as
1
1diag diag
i ix A b A A x
11
22diag
MM
a
aA
a
Matrix Formulation (2 of 2)
38
For simplicity, let
1
1
1 1 1
1
i i
i i
i i
x D b A D x
D b D A x D D x
x D b A x
Rearrange the “update equation” so that it makes more intuitive sense.
diagD A
Improvement on [x]i
Previous solution
37
38
10/2/2019
20
Matrix Implementation
39
1. Define [A] and [b]2. Make initial guess [x]1
3. Extract diagonal of [A]
4. Iterate until convergencea) Calculate adjustment of [x]i
b) Adjust [x]i
c) Compute relative error
d) Continue iteration until is sufficiently small
diagD A
1
i ix D b A x
1i i ix x x
max i
i
x
x
dx = D\(b – A*x);
x = x + dx;
err = max(abs(dx./x));
D = diag(diag(A));
Block Diagram of Jacobi Method
40
diagD A 1, , and A b x
InputArguments Extract Diagonal of [A] Done?
tol
Calculate Adjustment
1
i ix D b A x
Calculate New [x]
1i i ix x x
Calculate Error
max i
i
x
x
yes
no
Finished!
39
40
10/2/2019
21
41
Gauss‐Seidel Method
Gauss‐Seidel Method
42
The Jacobi method required [A] to be diagonally dominant, which restricts what the method can be used to solve.
The Gauss‐Seidel method is a modification to the Jacobi method to overcome this limitation.
41
42
10/2/2019
22
Formulation (1 of 4)
43
The matrix [A] can be decomposed into the sum of a lower triangular matrix [L’] and an upper triangular matrix [U’].
11 12 1 11 12 1
21 22 2 21 22 2
1 2 1 2
0 0 0
0 0 0
0 0 0
N N
N N
N N NN N N NN
A L U
a a a a a a
a a a a a aA L U
a a a a a a
Note: [L’] and [U’] here are not the same [L] and [U] from LU decomposition.
Formulation (2 of 4)
44
The goal is to solve [A][x]=[b]. Given that 𝐴 𝐿′ 𝑈′ , this becomes
A x b
L U x b
L x U x b
43
44
10/2/2019
23
Formulation (3 of 4)
45
From experience with LU decomposition, it is known that [L’][x]=[b] is very fast and efficient to solve for [x] using forward‐substitution.
Rearrange the matrix equation to take advantage of this.
L x U x b
L x b U x
1x L b U x
This can be solved very fast!
Formulation (4 of 4)
46
The update equation is derived from the last expression.
1
1
1i i
x L b U x
x L b U x
45
46
10/2/2019
24
47
Calculating Matrix Inverse
Using Gauss‐Jordan Method
48
Given matrix [A], form the following augmented matrix
11 12 13
21 22 23
31 32 33
| 1 0 0
| 0 1 0
| 0 0 1
a a a
A I a a a
a a a
Use the Gauss‐Jordan method until the augmented matrix has the form
11 12 13
21 22 23
31 32 33
1 0 0 |
0 1 0 |
0 0 1 |
c c c
c c c
c c c
11 12 13
1
21 22 23
31 32 33
c c c
A c c c
c c c
Here is the matrix inverse.
47
48
10/2/2019
25
Using LU Decomposition
49
Apply the LU decomposition method on a column‐by‐column basis.Each solution is a column in the inverse matrix.
1A
13
23
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3 33
0
0
1
a a c
c
c
a x x
a a a x x
a a a x x
13
23
33
c
c
c
11
21
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3 31
1
0
0
a a c
c
c
a x x
a a a x x
a a a x x
11
21
31
c
c
c
12
22
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3 32
0
1
0
a a c
c
c
a x x
a a a x x
a a a x x
12
22
32
c
c
c
49
![Computer Algebra using Maple Part IV: [Numerical] Linear Algebra](https://static.documents.pub/doc/80x56/6210ccc2e095ed73887ee0c5/computer-algebra-using-maple-part-iv-numerical-linear-algebra.jpg)
