of 15
7/29/2019 engg-prob-calculus 3
1/15
Engineering Computation ECL3-1
ENGINEERING COMPUTATION Lecture 3
Stephen RobertsMichaelmas Term
Iterative Solution of Simultaneous Equations
Topics covered in this section:
1. Iterative algorithms and the need to test their convergence.2. The Jacobi method of solution to solve Ax=b3. The Gauss-Seidel algorithm.4. Awareness of other numerical approached to solving Ax=b
Engineering Computation ECL3-2
Introduction
So far we have discussed the solution of the simultaneous linear equationset bAx = , and the conditions for ill-conditioning.
The underlying assumption was that we would go ahead and solve the setof equations by a direct method, such as Gaussian elimination.
How many calculations would this take?
Consider Nequations in Nvariables, where A is an NNmatrix. Assumeadditions are quick, and multiplications cost time.
7/29/2019 engg-prob-calculus 3
2/15
Engineering Computation ECL3-3
Example: Consider using the echelon method to solve
=
4
3
1
643
432
321
z
y
x
Process the augmented matrix as follows:
1
1
1
100
210
321
1
1
1
320
210
321
4
3
1
643
432
321
2row2-row33row1-row3
2row1-row2
Then back-substitute: 21321121 ==++=== xzyxyz-y-z .. .
The reduction to echelon form took 42 + 31 = 11 multiplications.The back substitution took a further 1 + 2 + 3 multiplications or divisions,making a total of 17.
Engineering Computation ECL3-4
Introduction
General case:
Consider Nequations in Nvariables, where A is an NNmatrix. Assumeadditions are quick, and multiplications cost time.
Reduction to echelon form will take ~ N(N-1) + (N-1)(N-2) + ..... ~N
3 multiplications. For large systems (say 103 equations) this can bea very large number (~ 109), and solutions may take a long time, evenwith modern fast computers.
Can we use a better approach?
7/29/2019 engg-prob-calculus 3
3/15
7/29/2019 engg-prob-calculus 3
4/15
Engineering Computation ECL3-7
Revision: Iterative solutions of equations in one variable
We attempt to solve the equation ( ) 0=xf . e.g. ( ) 0cos == xxxf .
Rearrange the equation into the form ( )xgx = e.g. xx cos= .
Choose a starting value x0, and repeat the iterative calculation
( ) LL,2,1,0,1 ==+ nxgx nn until xnconverges on a solution.
Engineering Computation ECL3-8
In our example, using nn xx cos1 =+ , we get the following results, starting with x0
= 0.7000:
n xn Error en0 0.7000 -0.03911 0.7648 0.02582 0.7215 -0.01763 0.7508 0.0117... ..........10 0.7383 -0.007
This converges (somewhat slowly! Try it on your calculator) to the solution0.7391 accurate to 4 figures.
But there is no guarantee that the method will converge!
Consider an alternative formulationnn
xx 11 cos
+= .
n xn0 0.70001 0.79542 0.65113 0.8617 This is diverging!
=> need a method to show will converge
7/29/2019 engg-prob-calculus 3
5/15
Engineering Computation ECL3-9
Test for Convergence
Let the true solution of ( )xgx = be sx = , so that ( )sgs = .
The error en+1 after n+ 1 iterations is given by
( ) ( )sgxgsxennn
==++ 11
Expanding the first two terms of a Taylor series about sxn
= ,
( ) ( ) ( ) ( ) ( ) ( ) LL ++=++= sgesgsgsxsgxgnnn
Thus ( )sgee nn +1 . (1)
The iteration scheme will converge if nen
as0 .
Engineering Computation ECL3-10
Thus ( )sgeenn
+1
.
The iteration scheme will converge if nen
as0 .
From ((1), previous slide) The condition for convergence is ( ) 1
7/29/2019 engg-prob-calculus 3
6/15
Engineering Computation ECL3-11
Iterative Methods for solving Ax=b
Engineering Computation ECL3-12
Iterative Methods for solving Ax=b
Lets now consider how a related idea can be applied to develop aniterative technique for solving Ax=b.
As with the case of the one dimensional case, lets translate Ax=binto an equivalent system of the form x=Tx+c.
Then having selected an initial guess x0 we can apply a similar
method to that outlined above.
7/29/2019 engg-prob-calculus 3
7/15
Engineering Computation ECL3-13
Jacobi Algorithm for solving bAx = .
Divide A into a diagonal matrix D and two triangular matrices L and U .
For a 33 matrix,
ULDA ++=
+
+
=
=
000
00
0
0
00
000
00
00
00
23
1312
3231
21
33
22
11
333231
232221
131211
A
AA
AA
A
A
A
A
AAA
AAA
AAA
Rewrite bAx = as ( ) bxULD =++ and re-arrange to give( )xULbDx += .
Now, the inverse D-1 of D is simply the diagonal matrix of the inverse ofthe diagonal elements of A, so write
( ) cBxbDxULDx +=++= 11
This is in the required form Tx+c and suggests the Jacobi iterative
scheme:
( ) cBxbDxULDx +=++= + nnn
11
1
Engineering Computation ECL3-14
Example: Jacobi solution of weighted chain.
Consider a hanging chain of m+ 1 light links with fixedends at height x0 = xm+1 = 0.
It could be the supporting chain for the Clifton
suspension bridge in Bristol, shown .
There is unit tension in the chain, the links are of unit
length.
A downward force bj is applied on the jth hinge.
If we make the approximation that the links make small angles to the horizontal,resolving forces vertically at each hinge gives:
mjbxxx jjjj =+ + 1,2 11
7/29/2019 engg-prob-calculus 3
8/15
Engineering Computation ECL3-15
This is a matrix equation of the form bAx = . In the case m= 5, there are 5links
=
21000
12100
01210
00121
00012
A
Dividing the matrix A into D + L+ U , and taking the case where
[ ]T1,1,1,11,=b gives the Jacobi iteration algorithm in the form
( ) cBxbDxULDx +=++= + nnn11
1 :
=+
21
212
1
21
21
21
21
21
2
1
2
1
21
21
21
1
0000
000000
000
0000
nn xx
Engineering Computation ECL3-16
this is easy to code up in MATLAB.
% Jchain.m
% solves weighted chain with m+1 links with n iterations
function jchain(m,n)
b = ones(m+2,1); % array of 1's - downward forces
x = zeros(m+2,1); % array of 0's - initial guess of position
xnew = x;
fprintf('Iteration 0: ');
fprintf('%.4f ',x);
fprintf('\n');
for i = 1:n % n iterations
for j = 2:m+1 % miss out end points
xnew(j) = 0.5*(x(j-1) + x(j+1) - b(j));
end
x = xnew;
fprintf('Iteration %2d: ', i);
fprintf('%.4f ',x);
fprintf('\n');
end
7/29/2019 engg-prob-calculus 3
9/15
Engineering Computation ECL3-17
Running this, with m = 5 gives, for the first 10 iterations:
jchain(5,10)
Iteration 0: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
Iteration 1: 0.0000 -0.5000 -0.5000 -0.5000 -0.5000 -0.5000 0.0000
Iteration 2: 0.0000 -0.7500 -1.0000 -1.0000 -1.0000 -0.7500 0.0000
Iteration 3: 0.0000 -1.0000 -1.3750 -1.5000 -1.3750 -1.0000 0.0000
Iteration 4: 0.0000 -1.1875 -1.7500 -1.8750 -1.7500 -1.1875 0.0000
Iteration 5: 0.0000 -1.3750 -2.0313 -2.2500 -2.0313 -1.3750 0.0000
Iteration 6: 0.0000 -1.5156 -2.3125 -2.5313 -2.3125 -1.5156 0.0000
Iteration 7: 0.0000 -1.6563 -2.5234 -2.8125 -2.5234 -1.6563 0.0000
Iteration 8: 0.0000 -1.7617 -2.7344 -3.0234 -2.7344 -1.7617 0.0000
Iteration 9: 0.0000 -1.8672 -2.8926 -3.2344 -2.8926 -1.8672 0.0000
Iteration 10: 0.0000 -1.9463 -3.0508 -3.3926 -3.0508 -1.9463 0.0000
It eventually converges to 4 significant figures in 80 iterations:
Iteration 76: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000
Iteration 77: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000
Iteration 78: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000
Iteration 79: 0.0000 -2.5000 -4.0000 -4.4999 -4.0000 -2.5000 0.0000
Iteration 80: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000Iteration 81: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000
You can substitute back into the equation Ax = b to check it is the solution!
Engineering Computation ECL3-18
To really go to town, we ran a 50 link chain in the figure below. As you can see ittook about 4000 iterations to converge. Note similarity of n= 2000 and n=4000 curves. Were getting close!
7/29/2019 engg-prob-calculus 3
10/15
Engineering Computation ECL3-19
Convergence of Matrix Iterations
We need a test for convergence for matrix iteration analogous to the oneused for equations in one variable earlier in this lecture. Since we have astructure Tx+c we can use a similar argument:
For an iteration algorithm of the form cBxx +=+ nn 1
, let the exact solution
be the vector s. Then cBss += .
Define the error vector sxe =nn
.
Then ( ) ( ) ( )nnnnn
BecxBcBscBxsxe ==++==++ 11
Using the compatibility between matrix and vector norms,
nneBe
+1
Thus the iteration cBxx +=+ nn 1
converges if nn
ase 0 if there is
a matrix norm such that1
7/29/2019 engg-prob-calculus 3
11/15
Engineering Computation ECL3-21
An alternative representation of theAn alternative representation of theAn alternative representation of theAn alternative representation of the JacobiJacobiJacobiJacobi AlgorithmAlgorithmAlgorithmAlgorithm
As outlined above we have written the Jacobi algorithm in matrix form. This was
convenient in order to understand the convergence properties of the method.
Alternatively, we could write the technique in its matrix element form as follows:
for j=1,2,.N, and n indicates the iteration index.
(If you have problems in seeing that this is so, compare with the Matlab example
above.)
This is a useful representation of the algorithm, both from the viewpoint of
implementation, but also because it suggests an improvement to the method that
we discuss next.
It also highlights the fact that we cannot have the conditionWhat could we do in this case?
,1
)(
)1(
ii
iN
jii
n
jijn
ia
b
a
xax +
=
=
+
0=ii
a
Engineering Computation ECL3-22
GaussGaussGaussGauss----Seidel AlgorithmSeidel AlgorithmSeidel AlgorithmSeidel Algorithm
the Jacobi algorithm ( ) ccccBxBxBxBxbbbbDDDDxxxxUUUULLLLDDDDxxxx +=++= + nnn11
1 updates the whole of the new
vectorxxxxn+1 from the old vectorxxxxn every iteration. But if we look at the MATLABalgorithm,
for j = 2:m+1 % miss out end points
xnew(j) = 0.5*(x(j-1) + x(j+1) - b(j));
end
we see thatxnew(j) depends onx(j-1).
But xnew(j-1) was calculated last time round the loop, so is now available in shiny newupdated form! so lets use it and modify the code to
. for j = 2:m+1 % miss out end pointsxnew(j) = 0.5*(xnew(j-1) + x(j+1) - b(j));
end
7/29/2019 engg-prob-calculus 3
12/15
Engineering Computation ECL3-23
gschain(5,50)
Iteration 0: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
Iteration 1: 0.0000 -0.5000 -0.7500 -0.8750 -0.9375 -0.9688 0.0000
Iteration 2: 0.0000 -0.8750 -1.3750 -1.6563 -1.8125 -1.4063 0.0000
Iteration 3: 0.0000 -1.1875 -1.9219 -2.3672 -2.3867 -1.6934 0.0000
.................................................
Iteration 40: 0.0000 -2.5000 -3.9999 -4.4999 -4.0000 -2.5000 0.0000
Iteration 41: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000
This method, Known as the Gauss-Seidel method, converges more rapidly than the Jacobimethod.
Engineering Computation ECL3-24
The general formulation of the Gauss-Seidel method is
bDUxDLxDx11
1
1
1
+
+++=
nnn
It looks strange to see the term1
1
+
nLxD on the right hand side, but remember
that, since the matrix multiplications are carried out row by row, and L islower triangular, each line of
1
1
+
nLxD contains only references to elements of
1+nx which are already known
7/29/2019 engg-prob-calculus 3
13/15
Engineering Computation ECL3-25
Elemental FormElemental FormElemental FormElemental Form
The elemental version of the Gauss-Seidel method also clearly shows why the
method is an improvement on the Jacobi method.This takes the form
where j=1,2,..N, and n indicates the iteration index as before.
ii
i
j
i
N
ij
n
jij
n
jij
n
ia
bxaxa
x
= +=
+
+
+
=
1
1 1
)()1(
)1(
)()(
Engineering Computation ECL3-26
GaussGaussGaussGauss----Seidel AlgorithmSeidel AlgorithmSeidel AlgorithmSeidel Algorithm convergence propertiesconvergence propertiesconvergence propertiesconvergence properties
Let us start from the elemental form of the Gauss-Seidel iteration equation where j=1,2,..N,
n indicates the iteration index as before.
Multiply by and collect the terms on the left hand side gives for i=1,2,N
Hence there are N equations
ii
i
ji
N
ij
n
jij
n
jijn
ia
bxaxa
x +
=
= +=
+
+
1
1 1
)()1(
)1(
)()(
iia1+nx
ibnNxNia
Ni
xiianixiia
nxianxia +++
=+++++ ,.........11,1...........1
221
11
Nbn
NxNNansxNa
nxNa
bnN
xN
anxanxanxa
bn
NxNan
xan
xan
xa
+=+
++
++
+=+++
+=+
1...........
12
111
223231
2221
121
113132121
111
M
M
KK
KK
7/29/2019 engg-prob-calculus 3
14/15
Engineering Computation ECL3-27
Hence
or
This is in the standard form Tx+cTx+cTx+cTx+c,,,, hence the convergence condition is
Nbn
NxNNansxNa
nxNa
bn
N
x
N
an
xan
xan
xa
bn
NxNan
xan
xan
xa
+=+
++
++
+=+++
+=+
1...........
12
111
22323
1
222
1
121
113132121
111
M
M
KK
KK
bUxL)xDn1n
+=++
(
bL)(DUxL)(Dx1n11n +
+++=
1