1
Chapters: 3, 4 and 6
Solving Linear Systems
A system of n simultaneous, non-homogenous, algebraic, linear equations is given in
the form:
A : is an nn coefficient matrix. b : is an 1n right-hand side vector.
x : is an 1n vector of unknowns.
nnnn
n
n
AAA
AAA
AAA
21
22221
11211
nx
x
x
2
1
=
nb
b
b
2
1
nnA
1nx
1nb
First: Exact Methods
1) Cramer's Rule: This method is one of the least efficient for solving a large number of non-homogeneous linear equations. In general, given:
ijij
bxA : nji ,,3,2,1
A
A
A
Ax
jj
i
det
det
where,
A : is the determinant of the coefficient matrix A .
jA : is the determinant of the coefficient matrix A with its j th column replaced by
the i
b vector.
bAx
2
Example: Use Cramer's Rule to solve the following system of equations:
53221 xx
521 xx
Solution:
bAx
5
5
11
32
2
1
x
x
4
11
32
15
35
1
1
A
Ax
1
11
32
51
52
2
2
A
Ax
1
4
2
1
x
x
Elementary Row Operations:
(1) Interchange rows k and j .
(2) Multiply row k by 0 . (3) Add times row j to row k .
3
2) Gauss-Jordan Elimination Method:
We can solve bAx using Gauss-Jordan elimination method. Algorithm:
1. Form the augmented matrix: )1(, nnbAC
2. For 1j to n do
{
a) Compute the pivot index "p": jpn such that:
ij
n
jipj
CC
max
b) If 0pj
C , exit (no solution).
c) If jp , interchange rows p and j .
d) Divide row j by the pivot jj
C .
e) For each ji , subtract ij
C times row j from row i . Ri = jiji
RCR
}
3. Partition C as xIC , .
Example: Solve the following system of equations using Gauss-Jordan Elimination method:
6
1
2
210
122
011
3
2
1
x
x
x
Solution:
436210
1122
2011
C
1j , 13 p 2max3
ijji
Cp
0221
p
C
12 jp interchange rows 2 and 1
6210
2011
1122
2
1R
6210
2011
5.05.011
4
6210
5.15.000
5.05.011
2j 23 p , 3p
132C
23 interchange rows 3 and 2
5.15.000
6210
5.05.011
1
2R
5.15.000
6210
5.05.011
5.15.000
6210
5.65.101
3j 33 p , 3p
05.033
C
5.0
3R
3100
6210
5.65.101
3100
6210
5.65.101
3232
3131
RCR
RCR
3100
0010
2001
I x
3
2
1
3
0
2
x
x
x
x
row2 - C21.row1
row3 - C31.row1
row1 - C12.row2 row3 - C32.row2
5
3) Matrix Inverse Method:
bAx bAx 1
- First, find 1
nxnA
- Then, bAx 1 Algorithm:
1. Form the augmented matrix: nnIAC 2,
2. For 1j to n do
{ a) Compute the pivot index jpn such that:
ij
n
jipj
CC
max
b) If 0pj
C , exit (no inverse).
c) If jp , interchange rows p and j .
d) Divide row j by the pivot jj
C .
e) For each ji , subtract ij
C times row j from row i .
}
3. Partition C as 1, AIC .
Example: Consider the following 33 matrix:
120
402
011
A Find 1A
Solution:
63100120
010402
001011
,
IAC
1,
2.02.04.0100
4.01.02.0010
4.01.08.0001
AI
2.02.04.0
4.01.02.0
4.01.08.01A
6
4) Gaussian Elimination Method:
bAx
)1(, nnbAC )1(, nneD
D : is an upper-triangular matrix (matrix with zeros below the main diagonal) Algorithm:
1. Form the augmented matrix: )1(, nnbAC .
2. For 1j to n do
{ a) Compute the pivot index jpn such that:
ij
n
jipj
CC
max
b) If 0pj
C , exit (no solution).
c) If jp , interchange rows p and j .
d) For each ji , subtract
jj
ij
C
Ctimes row j from row i .
Ri
j
jj
ij
iR
c
cR
}
3. Partition C as eDC , , where D is an nn and e is 1n .
4. For nj down to 1, compute:
{
n
jiijij
jj
jxDe
Dx
1
1
}
Example: Solve the following linear algebraic system using Gaussian Elimination method:
814
312
201
2
1
1
3
2
1
x
x
x
7
Solution:
2814
1312
1201
C
1j :
431C , 3p
1201
1312
2814
1
11
21
2R
c
cR
1
11
31
3R
c
cR
5.0025.00
215.10
2814
2j
3j
8333.01667.000
215.10
2814
C
3j
58333.01667.0
1 3
43
i
ijixDx
25.1
35)1(2
5.1
12
5.1
12
5.1
1323
3
32
xDxDxi
iji
9)58()21(24
12
4
12
4
1313212
3
21
xDxDxDxi
iji
5
2
9
3
2
1
x
x
x
0
8
LU Decomposition (Triangular Factorization)
Definition: The nonsingular matrix A has a triangular factorization if it can be expressed as the
product of a lower triangular matrix L and an upper triangular matrix U.
A = LU
L: it has 1’s along the diagonal.
U: it has nonzero diagonal elements.
L =
1
01
001
0001
21
3231
21
nn mm
mm
m U =
nn
n
u
u
uuu
000
00
0 22
11211
e.g
Given,
A =
621
542
134
Convert the matrix A to the form A=LU.
Solution:
621
542
134
=
1
01
001
3231
21
mm
m
33
2322
131211
00
0
u
uu
uuu
4 = u11 , 3 = u12 , -1 = u13
m21 u11 = -2 → m 21 = - 0.5
m21u12 + u22 = - 4 → m21 = - 4 – (- 0.5*3) = -2.5
m21u13 + u23 = 5 → u23 = 5 – (- 0.5*-1) = 4.5
m31u11 = 1 → m31 = 0.25
m31u12 +m32u22 = 2 → m32 = 5.2
3*4
12
= -0.5
m31u13 + m32u23 + u33 = 6
u33 = 5.8)5.4*2
1(1*
4
16
621
542
134
=
5.800
5.45.20
134
15.025.0
015.0
001
A = L U
9
5) Solution of a Linear System: AX = b
Suppose that the coefficient matrix A of the linear system: AX = b has a triangular
factorization A = LU, then the solution to: AX = b → LUX = b can be obtained by
defining:
Y = UX and then solving two systems:
First, solve LY = b for Y using forward substitution.
Then, solve UX = Y for X using back substitution.
e.g.
Solve the following system of equations using the triangular factorization method.
82
79
52
21
610124
88103
4682
1421
4
3
2
1
x
x
x
x
Solution:
1. A = L U
44
3433
242322
14131211
434241
3231
21
000
00
0
1
01
001
0001
610124
88103
4682
1421
u
uu
uuu
uuuu
mmm
mm
m
Find the values of the m’s and u’s. Therefore,
L =
1214
0113
0012
0001
, U =
6000
3200
2240
1421
2. solve LY = b for Y using forward substitution:
1214
0113
0012
0001
4
3
2
1
y
y
y
y
=
82
79
52
21
10
y1 = 21
2y1 + y2 = 52 → y2 = 10
3y1 + y2 + y3 = 79 → y3 = 6
4y1 + y2 + 2y3 + y4 = 82 → y4 = -24
Y =
24
6
10
21
3. solve UX = Y for X using back substitution:
6000
3200
2240
1421
4
3
2
1
x
x
x
x
=
24
6
10
21
x1 + 2x2 + 4x3 + x4 = 21 → x1 = 1
4x2 – 2x3 +2x4 = 10 → x2 = 2
-2x3 +3x4 = 6 → x3 = 3
- 6x4 = -24 → x4 = 4
X =
4
3
2
1
is the solution of the system of the equations.
e.g.
Show that the following matrix can not be factored directly as: A = LU.
A =
532
184
621
11
ILL-Conditional Systems:
Consider the following system:
100
100
401200
200100
2
1
x
x
2011x , 100
2x
Now, consider the system:
100
100
400200
200101
2
1
x
x
501x , 75.24
2x
Clearly the solution is very sensitive to the values of A in this case. When the solution is highly sensitive to the values of the coefficient matrix A or the right-hand side vector b , the equations are said to be ill-conditioned.
Vector and Matrix Norms: A norm of a vector is a generalization of the absolute value of a scalar. That is, a norm is a measure of the length or magnitude of the vector. There are several ways to define a norm of an 1n vector x .
1)
l norm i
n
i
xx max1
x : is called the infinity norm of x .
2) 2
l norm
n
ii
xx1
2
2
2l norm: is called the Euclidean norm (distance from the origin).
In order to develop a measure of the sensitivity of a system, we need to consider the notion of a matrix norm:
n
jij
n
i
AA11
max
12
Example:
13
12
1
.
137
285
964
3
2
1
x
x
x
19}11,15,19max{3
1
3
1max
j
iji
AA
The solution of this system is:
1
0
2
3
2
1
x
x
x
2}1,0,2max{
X
5142
X
Properties of norms:
1. 00 xforx .
2. 00 xforx .
3. xx . : is a scalar.
4. yxyx .
5. xAAx . .
Condition Number:
AX = b
Consider the effect of changes in the right-hand side vector b, say from b to b+∆b.
Let X denotes the solution to the original system, and let X + ∆X denotes the
solution to the perturbed system. Then, A(X + ∆X) = b+∆b which reduces to A ∆X
= ∆b because AX = b
∆X = A-1
∆b
13
Using:
AX ≤ XA .
X ≤ bA .1
Using the properties of norms, the relative change in the solution can be expressed in
terms of the perturbation as follows:
b
bAK
X
X
)(
K(A): is a scalar called the condition number of the matrix A and is defined as follows,
assuming A is nonsingular.
K(A) = 1. AA
Note that K(A) is a measure of the relative sensitivity of the solution to changes in b .
Properties of the condition number:
1. When K(A) becomes large, the system is regarded as being an ill-conditioned.
e.g
A =
401200
200100 determine K(A).
Solution :
A-1
=
12
201.4
valuelarge01.361201.6*601)(
01.6}3,01.6max{
601}601,300max{1
AK
A
A
Our system is ill-conditional system.
2. ∞> K(A) ≥ 1
3. K(I) = 1
4. K(CA) = K(A) C is a scalar.
5. If K(A) is large , then A is close to singular matrix.
14
e.g.
Given, b =
7.9
1.4 , x=
0
1
if b is changed to bnew =
7.9
11.4 and x is changed to xnew =
97.0
34.0, determine K(A).
K(A) ≥
b
b
X
X
∆X =
97.0
66.0 ∆b =
0
01.0
7.9,01.0
1,97.0
bb
XX
K(A) ≥ 9.9407.9/01.0
1
97.0
K(A) ≥940.9
Second: Iterative (Numerical) Methods
1) Jacobi method. 2) Gauss-Seidel method. 3) Relaxation methods:
a. Successive Under-Relaxation.
b. Successive Over-Relaxation.
These are methods by which an approximation to the solution of a system of linear algebraic equations may be obtained. The iterative procedure may not always yield a solution. A sufficient condition for a solution to be found is that the absolute value of the diagonal coefficient in any equation must be greater than the sum of the absolute values of all other coefficients appearing in that equation.
1:
knAAkj
kjkk
15
Example:
3333232131
2323222121
1313212111
bxaxaxa
bxaxaxa
bxaxaxa
Test:
131211aaa
232122aaa
323133aaa
To start the iterative scheme, an initial guess 0x must be chosen. For example, if
nothing is known about the solution, the initial guess 00 x is as good as any. Typically, the iterations are repeated until the norm of the difference between successive estimates is sufficiently small, that is:
)()1( kk xx " : tolerance"
1) Jacobi Method:
Jacobi iteration can be written as a set of scalar equations expressed directly in
terms of the components of A and b ,
1:1 )()1(
inxAbA
xij
k
jiji
ii
k
i
Example: Given the following system of equations:
4
2
7
.
430
153
205
3
2
1
x
x
x
Use Jacobi iterative method to find 21
, xx and 3
x .
Do two iterations, let the initial guess 00 x .
205
135 304
16
)27(5
1 )(
3
)1(
1
kk xx
)32(5
1 )(
3
)(
1
)1(
2
kkk xxx
)34(4
1 )(
2
)1(
3
kk xx
1st iteration (k=0)
4.15
7)27(
5
1 )0(
3
)1(
1 xx : )1(
1x is the value of
1x in the first iteration.
4.05
2)32(
5
1 )0(
3
)0(
1
)1(
2 xxx
0.14
4)34(
4
1 )0(
2
)1(
3 xx
?)0()1( xx
0.1
4.0
4.1
0
0
0
0.1
4.0
4.1)0()1( xx
4.1)0()1( xx
2nd
iteration (k=1)
0.1)]0.12(7[5
1)27(
5
1 )1(
3
)2(
1 xx
24.0)]0.1()4.13(2[5
1)32(
5
1 )1(
3
)1(
1
)2(
2 xxx
7.0)]4.03(4[4
1)34(
4
1 )1(
2
)2(
3 xx
?)1()2( xx
0
0 0
17
3.0
64.0
4.0
0.1
4.0
4.1
7.0
24.0
0.1)1()2( xx
64.0)1()2( xx
Jacobi Estimates:
k 1x 2
x 3x
0 0.0000000 0.0000000 0.0000000
1 1.4000000 0.4000000 - 1.0000000
2 1.0000000 - 0.2400000 - 0.7000000
3 1.1200000 - 0.0600000 - 1.1799999
4 0.9280000 - 0.0360000 - 1.0450000
5 0.9820000 0.0522000 - 1.0270000
6 0.9892000 0.0162000 - 0.9608500
7 1.0156599 - 0.0013500 - 0.9878500
8 1.0048599 - 0.0118259 - 1.0010124
9 0.9995950 - 0.0027135 - 1.0088694
2) Gauss-Seidel Method:
The convergence rate of the Jacobi method can be improved using Gauss-Seidel method. In general, it is better to take advantage of the most recent information when updating an estimate of a solution as follows:
1:1 1
1 1
)()1()1(
inxAxAbA
xi
j
n
ij
k
jij
k
jiji
ii
k
i
18
Example:
Use Gauss-Seidel iterative method to find 21
, xx and 3
x for the same system in the
previous example:
4
2
7
.
430
153
205
3
2
1
x
x
x
Do two iterations, and let the initial guess 00 x .
Solution:
)27(5
1 )(
3
)1(
1
kk xx
)32(5
1 )(
3
)1(
1
)1(
2
kkk xxx
)34(4
1 )1(
2
)1(
3
kk xx
1st iteration (k=0)
4.15
7)27(
5
1 )0(
3
)1(
1 xx
44.0]0)4.13(2[5
1)32(
5
1 )0(
3
)1(
1
)1(
2 xxx
33.1)]44.03(4[4
1)34(
4
1 )1(
2
)1(
3 xx
?)0()1( xx
33.1
44.0
4.1
0
0
0
33.1
44.0
4.1)0()1( xx
4.1)0()1( xx
2nd
iteration (k=1)
868.0)]33.12(7[5
1)27(
5
1 )1(
3
)2(
1 xx
1452.0)]33.1()868.03(2[5
1)32(
5
1 )1(
3
)2(
1
)2(
2 xxx
8911.0)]1452.03(4[4
1)34(
4
1 )2(
2
)2(
3 xx
19
?)1()2( xx
4389.0
5852.0
532.0
33.1
44.0
4.1
8911.0
1452.0
868.0)1()2( xx
5852.0)1()2( xx
Gauss-Seidel Estimates:
k 1x 2
x 3x
0 0.0000000 0.0000000 0.0000000
1 1.4000000 - 0.4400000 - 1.3299999
2 0.8680000 0.1452000 - 0.8911000
3 1.0435599 - 0.0479159 - 1.0359370
4 0.9856252 0.0158123 - 0.9881408
5 1.0047437 - 0.0052180 - 1.0039135
6 0.9984345 0.0017220 - 0.9987085
7 1.0005165 - 0.0005682 - 1.0004262
8 0.9998295 0.0001875 - 0.9998593
9 1.0000563 - 0.0000619 - 1.0000464
20
Jacobi vs Gauss-Seidel Estimates:
k 1
x 2x 3
x
Jacobi Gauss-Seidel Jacobi Gauss-Seidel Jacobi Gauss-Seidel
0 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
1 1.4000000 1.4000000 0.4000000 - 0.4400000 - 1.0000000 - 1.3299999
2 1.0000000 0.8680000 - 0.2400000 0.1452000 - 0.7000000 - 0.8911000
3 1.1200000 1.0435599 - 0.0600000 - 0.0479159 - 1.1799999 - 1.0359370
4 0.9280000 0.9856252 - 0.0360000 0.0158123 - 1.0450000 - 0.9881408
5 0.9820000 1.0047437 0.0522000 - 0.0052180 - 1.0270000 - 1.0039135
6 0.9892000 0.9984345 0.0162000 0.0017220 - 0.9608500 - 0.9987085
7 1.0156599 1.0005165 - 0.0013500 - 0.0005682 - 0.9878500 - 1.0004262
8 1.0048599 0.9998295 - 0.0118259 0.0001875 - 1.0010124 - 0.9998593
9 0.9995950 1.0000563 - 0.0027135 - 0.0000619 - 1.0088694 - 1.0000464
3) Successive Relaxation (SR) Method: The convergence rate can sometimes be increased by introducing a relaxation factor
0 ,
1:)1(1
1 1
)()1()()1(
inxAxAbA
xxi
j
n
ij
k
jij
k
jiji
ii
k
i
k
i
If: 1 , SR method reduces to the Gauss-Seidel method. If: 01 Successive under relaxation (SUR). If: 12 Successive over relaxation (SOR).
Example:
Use Successive Relaxation method to find 21
, xx and 3
x in the previous example
4
2
7
.
430
153
205
3
2
1
x
x
x
Do two iterations, let the initial guess 00 x and 1.1 .
21
Solution:
)1(
1
kx )07(5
1.11.0
3
2
)()(
1
j
k
jij
k xAx
)27(22.01.0 )(
3
)(
1
kk xx
)(
3
)(
144.054.11.0 kk xx
)1(
2
kx )2(5
1.11.0
3
3
)(
2
1
1
)1(
2
)(
2
j
k
jjj
k
jj
k xAxAx
)32(22.01.0 )(
3
)1(
1
)(
2
kkk xxx
)(
3
)1(
1
)(
222.066.044.01.0 kkk xxx
)1(
3
kx )04(4
1.11.0
2
1
)1(
3
)(
3
j
k
jj
k xAx
)304(275.01.0 )1(
2
)(
3
kk xx
)1(
2
)(
3825.01.11.0 kk xx
)1(
1
kx 54.144.01.0 )(
3
)(
1 kk xx
)1(
2
kx 44.022.066.01.0 )(
3
)1(
1
)(
2 kkk xxx
)1(
3
kx 1.1825.01.0 )1(
2
)(
3 kk xx
1st iteration (k=0)
54.154.100)1(
1x
5764.044.0)54.166.0(44.0066.00 )1(
1
)1(
2 xx
5755.11.1)5764.0825.0(1.1825.00 )1(
2
)1(
3 xx
5755.1
5764.0
54.1)0()1( xx
5755.1)0()1( xx
22
2nd
iteration (k=1)
6928.054.144.01.0 )1(
3
)1(
1
)2(
1 xxx
387.044.022.066.01.0 )1(
3
)2(
1
)1(
2
)2(
2 xxxx
6232.01.1825.01.0 )2(
2
)1(
3
)2(
3 xxx
9523.0
9634.0
8472.0
5755.1
5764.0
54.1
6232.0
387.0
6928.0)1()2( xx
9634.0)1()2( xx
Solving a system of m-equations in n-unknowns with m > n
(more equations than unknowns) in the least square sence
(Normal equations)
Solve: Am×n Xn×1 = bm×1
Such systems can be solved using Normal equations as follows:
AT AX = AT b
Note:
A T
mn Am×n → results in a square matrix of size n×n
A T
mn bm×1 → results in a vector of size n×1
Now, ATAX = AT b
Can be solved using the previous exact methods (Gauss-Jordan
elimination, inverse method, Gaussian method, etc…)
23
e.g.
Find the least squares solution to the inconsistent system (more
equations than unknowns):
x1 + 4x2 = -2
x1 + 2x2 = 6
2x1 + 3x2 = 1
Solution:
A =
1
6
2
,
32
21
41
b
ATA =
7
6,
2912
126bAT
7
6
2912
126
2
1
x
x
Solving the system above,
1
3
2
1
X
XX → least squares solution
This solution is NOT an exact solution but rather an approximate
that minimizes the distance between AX and b (i.e. it minimizes
the length of the error vector: e = AX – b).