SOLVING SYSTEMS OF EQUATIONS,
ITERATIVE METHODS
ELM1222 Numerical Analysis
1
Some of the contents are adopted from
Laurene V. Fausett, Applied Numerical Analysis using MATLAB. Prentice Hall Inc., 1999
ELM1222 Numerical Analysis | Dr Muharrem Mercimek
Today’s lecture
• 3 Common classical iterative techniques for linear equation systems
• Jacobi Method
• Gauss- Seidel Method
• Successive Over Relaxation
2 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
Iterative techniques for linear equation systems
• Systems of linear equations for which numerical solutions are needed are
often very large.
• Using general methods such as Gauss elimination is computationally
expensive.
• If the coefficient system is having a specific structure iterative techniques are
preferable.
3 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
Iterative techniques for linear equation systems
4 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
For large, sparse systems (many coefficients whose value is zero)
iterative techniques are preferable.
What is an equation System?
5 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
When you have
derive the equivalent system
and solve it
• Generate a sequence of approximation , where
dCxx kk )1()(
,..., )2()1( xx
3333132131
2323122121
1313212111
bxaxaxa
bxaxaxa
bxaxaxa
33
32
33
321
33
313
22
21
22
231
22
212
11
13
11
132
11
121
a
bx
a
ax
a
ax
a
bx
a
ax
a
ax
a
bx
a
ax
a
ax
BAx
dCxx
• A 2x2 equation system
• 𝐀 =2 11 2
𝐱 =𝑥𝑦 𝐛 =
66
• Start with
• Simultaneous updating
• New values of the variables are not used until a new iteration step is begun
Jacobi Method
6 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
4
113
4
13
2
1
4
113
4
13
2
1
)0()1(
)0()1(
xy
yx
62
62
yx
yx
32
1
32
1
xy
yx
2/1)0()0( yx
𝐀𝐱 = 𝐛 Example 1:
𝐱 = 𝐂𝐱 + 𝐝
𝐂 has zeros in the diagonal
Jacobi Method
7 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
𝐀 =2 11 2
𝐱 =𝑥𝑦 𝐛 =
66
4
113
4
13
2
1
4
113
4
13
2
1
)0()1(
)0()1(
xy
yx
2/1)0()0( yx
Stopping Criteria:
Stop the iterations when
• The function 𝐀𝐱 (𝑘) − 𝐛 𝟐
less than a “𝑡𝑜𝑙” value.( . 𝟐 is 2-Norm or Euclidean Norm)
• The max number of iterations “𝑚𝑎𝑥 _𝑖𝑡𝑒𝑟” has reached
8
133
8
113
2
1
8
133
8
113
2
1
)1()2(
)1()2(
xy
yx
Jacobi Method
8 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
Example 2: Consider the 3x3 system
• Start with
32
62
12
321
321
321
xxx
xxx
xxx
213
12
1
5.05.0
5.0
xxx
xx
x
5.1
0.35.0
5.05.05.0
3
32
x
xx
5.1
0.3
5.0
0.05.05.0
5.00.05.0
5.05.00.0
)0(3
)0(2
)0(1
)1(3
)1(2
)1(1
x
x
x
x
x
x
)0,0,0()0( x
5.1
0.3
5.0
5.1
3
5.0
0.05.05.0
5.00.05.0
5.05.00.0
)2(3
)2(2
)2(1
x
x
x The method converges in 13 iterations
𝐱 (13) = [1.002 2.001 − 0.9997] T
Example 3:
A necessary and sufficient condition for the convergence of the Jacobi method
“the magnitude of the largest eigenvalue of the iteration matrix C be less than 1”
A necessary condition (not sufficient) for the convergence of the Jacobi method
“ A should be diagonally dominant. Magnitude of diagonal element should be greater than
sum f magnitudes of other elements of the row
9 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
𝐀 =1 22 1
𝐱 =𝑥𝑦 𝐛 =
66
62
62
yx
yx
62
62
xy
yx
56162
56162
)0()1(
)0()1(
xy
yx
• When you run the other iterations you will see it diverges
Jacobi Method
• A 2x2 equation system
• 𝐀 =2 11 2
𝐱 =𝑥𝑦 𝐛 =
66
• Start with
• Sequential updating
• New values of the variables are used in the same iteration step
Gauss-Seidel Method
10 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
8
133
8
113
2
1
4
113
4
13
2
1
)1()1(
)0()1(
xy
yx
62
62
yx
yx
32
1
32
1
xy
yx
2/1)0()0( yx
𝐀𝐱 = 𝐛
Example 4:
32
613
32
353
2
1
16
353
16
133
2
1
)2()2(
)1()2(
xy
yx
11
• Example 5: Consider the three-by-three system
• Start with
• After 10 iterations 𝐱=[1.0001 1.9999 -1.0001]
32
62
12
321
321
321
xxx
xxx
xxx
newnewnew
newnew
new
xxx
xx
x
213
12
1
5.05.0
5.0
5.1
0.35.0
5.05.05.0
)(3
)(3
)(2
old
oldold
x
xx
)0,0,0()0( x
Gauss-Seidel Method
11 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
Stopping Criteria:
Stop the iterations when
• The function 𝐀𝐱 (𝑘) − 𝐛 𝟐
less than a “𝑡𝑜𝑙” value.( . 𝟐 is 2-Norm or Euclidean Norm)
• The max number of iterations “𝑚𝑎𝑥 _𝑖𝑡𝑒𝑟” has reached
• Discussion
• The Gauss-Seidel method is sensitive to the form of the coefficient
matrix A
• The Gauss-Seidel method typically converges more rapidly than the
Jacobi method
• The Gauss-Seidel method is more difficult to use for parallel
computation
12
Gauss-Seidel Method
12 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
Similary 𝑥2 and 𝑥3 updates
can be found
Similary 𝑥2 and 𝑥3 updates can be found
13
Successive Over Relaxation
13 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
• Introduce an additional parameter, ω, that may accelerate the convergence of the
iterations. A combination of current update (From Gauss-Seidel) and previous value.
• A proportion from current update, as well as a proportion from previous value is
summed up.
3333132131
2323122121
1313212111
bxaxaxa
bxaxaxa
bxaxaxa
3333132131
2323122121
1313212111
bxaxaxa
bxaxaxa
bxaxaxa
)()1(
)()1(
)()1(
232131333
33
323121222
22
313212111
11
newnewoldnew
oldnewoldnew
oldoldoldnew
xaxaba
xx
xaxaba
xx
xaxaba
xx
𝜔𝑥1 =𝜔(𝑏1−𝑎12𝑥2 −𝑎13 𝑥3)
𝑎11
add 𝑥1 − 𝜔𝑥1 to both sides and use it in
Gauss-Seidel Method updates
multiply each
equation with 𝜔
Similary 𝑥2 and 𝑥3 updates can be found
A proportion from current update
14
• Consider the three-by-three system
1150
562
024
A
43
29
8
b
)
11
43
11
5()1(
)6
29
6
5
3
1()1(
)22
1()1(
133
3122
211
newoldnew
oldnewoldnew
oldoldnew
xxx
xxxx
xxx
Successive Over Relaxation (SOR)
14 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
15
Successive Over Relaxation (SOR)
15 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
• Required number of iterations for different values of the relaxation
parameter
• Start with
• Tolerance = 0.00001
ω 0.8 0.9 1.0 1.2 1.25 1.3 1.4
No. of iterations 44 36 29 18 15 13 16
1150
562
024
A
43
29
8
b
16
• Discussion
• The SOR method can be derived by multiplying the decomposed system
obtained from the Gauss-Seidel method by the relaxation parameter w
• The iterative parameter w should always be chosen such that
0 < w < 2
Successive Over Relaxation (SOR)
16 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
17
Summary • Jacobi method
• SOR method
33
32
33
321
33
313
22
213
22
231
22
212
11
113
11
1312
11
121
a
bx
a
ax
a
ax
a
bx
a
ax
a
ax
a
bx
a
ax
a
ax
kkk
kkk
kkk
)()1(
)()1(
)()1(
232131333
33
323121222
22
313212111
11
newnewoldnew
oldnewoldnew
oldoldoldnew
xaxaba
xx
xaxaba
xx
xaxaba
xx
33
312
33
3211
33
313
22
213
22
2311
22
212
11
113
11
1312
11
121
a
bx
a
ax
a
ax
a
bx
a
ax
a
ax
a
bx
a
ax
a
ax
kkk
kkk
kkk
Gauss-seidel method
17 ELM1222 Numerical Analysis | Dr Muharrem Mercimek
