Ch 7. Iterative Techniques in Matrix...

Ch 7. Iterative Techniques in Matrix Algebra

Jeong-hwan Gwak

2

Contents

7.1 Norms of Vectors and Matrices 7.1 Norms of Vectors and Matrices 7.2 7.2 EigenvaluesEigenvalues and Eigenvectorsand Eigenvectors7.3 Iterative Techniques for Solving Linear Systems7.3 Iterative Techniques for Solving Linear Systems7.4 Error Bounds and Iterative Refinement7.4 Error Bounds and Iterative Refinement7.5 The Conjugate Gradient Method7.5 The Conjugate Gradient Method

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

(1)(1)

(2)(2)

19

20

21

Jacobi Method

22

23

24

Gauss-Seidel Method

25

26

27

28

29

Convergence of general iterative technique

30

31

SOR Method

if w=1 : Gauss-Seidel method

32

33

34

35

How the appropriate value of w is chosen?

36

37

38

39

40

41

42

43

44

Department of Mathematics

The conjugate gradient Method

• To solve positive definite linear systemn n×

• More computationally expensive than those in Gaussian elimination

• Very useful when employed as an iterative approximation method for solving large sparse systems with nonzero entries occurring inpredictable patterns

• When the matrix has been preconditioned to make the calculationsmore effective, good results are obtained in only about steps.n


▪ Assumption matrix is positive definite

▪ Notation (inner product)where and are vectors.

▪ Some properties ( inner product )For any vectors , , and , and any real number

A ( , , 0 )tA A unless∀ ⟨ ⟩ = > =x x x x x x 0

, t⟨ ⟩ =x y x y x y dimn −

x y z αi ) , ,⟨ ⟩ = ⟨ ⟩x y y xii ) , , ,α α α⟨ ⟩ = ⟨ ⟩ = ⟨ ⟩x y x y x yiii ) , , ,⟨ + ⟩ = ⟨ ⟩ + ⟨ ⟩x z y x y z yiv) ,⟨ ⟩ ≥x x 0v) ,⟨ ⟩ = ⇔ =x x 0 x 0vi ) , ,A A when A is positive definite⟨ ⟩ = ⟨ ⟩x y x y


The vector is a solution to the positive definite linear system

iff minimizes

Forhas minimum ,

when

(i ) *x =Ax b

( ) , 2 ,g = ⟨ ⟩ − ⟨ ⟩x x Ax x b

(ii ) ,∀ ≠x v 0( )g t+x v

,,

t ⟨ − ⟩=

⟨ ⟩v b Ax

v Av

Theorem

*x

,,

⟨ ⟩=⟨ ⟩

v rv Av


· is an initial approximation to . is an initial search direction.

· For

compute

.

and choose a new search direction .

(0)x *x(1) ( )≠v 0

1 , 2 , 3 ,k =

( ) ( 1)

( ) ( )

,,

k k

k k kt−⟨ − ⟩

=⟨ ⟩

v b Axv Av

( ) ( 1) ( )k k kkt

−= +x x v

( 1)k+v


· Choice of the search directions (method of steepest descent)

1 2( , , , )tnx x x= ⋅ ⋅ ⋅x

1 2( ) ( , , , )ng g x x x=x

, 2 ,A= ⟨ ⟩ − ⟨ ⟩x x x b

1 1 1

2n n n

ij i j i ii j i

a x x x b= = =

= −∑∑ ∑

1( ) 2 2

n

ki i kik

g a x bx =

∂⇒ = −

∂ ∑x

1 2

( ) ( ), ( ), , ( )t

n

g g ggx x x

⎛ ⎞∂ ∂ ∂⇒∇ = ⎜ ⎟∂ ∂ ∂⎝ ⎠

x x x x 2( ) 2A= − = −x b r

( 1) ( ) ( )k k kA+⇒ = = −v r b x

The direction of greatest decrease in the value of is the direction given by (i.e. in the direction of the residual )

( )g x( )g−∇ x r


Alternative approach

( ) ( 1) ( ) ( 1)

( ) ( ) ( ) ( )

, ,

, ,

k k k k

k k k k k

At

A A

− −−⇒ = =

v b x v r

v v v v

Def. A - orthogonality condition

A set of nonzero direction vectors { }(1) (2) ( ), , , nv v vi i i

that satisfy ( ) ( ), 0i jA =v v i j≠if

( ) ( 1) ( )k k kkt

−= +x x v


( ) ( 1)

( ) ( )

,

,

k k

k k k

At

A

−−=

v b x

v v

Theorem

Then, assuming exact arithmetic,

{ }(1) (2) ( ), , , nv v v

: positive definite matrix

( ) ( 1) ( )k k kkt

−= +x x v

A(0)x : arbitrary

Define

for 1 , 2 , 3 ,k =

( )n =Ax b

: A –orthogonal set of nonzero vectors


( ) ( ), 0k j =r v

Theorem

The residual vectors ( )kr

for each

1 , 2 , 3 , ,k n=

1 , 2 , 3 , ,j k=

, for a conjugate direction method, satisfy

, where


1. Initial approximation

Conjugate gradient method(0)x

First search direction (1) (0) (0)= = −v r b Ax

( 1) ( 2) ( 1)1

k k kkt

− − −−= +x x v2.

( ) ( ), 0 ,i j⟨ ⟩ =v Av ( ) ( ), 0i j⟨ ⟩ =r r

and

i j≠for

⇒ (1) ( 1), , k−⋅⋅ ⋅v vfind (1) ( 1), , k−⋅⋅ ⋅x x

3. If is the solution to ,=Ax b done .

Otherwise, ( 1) ( 1)k k− −= − ≠r b Ax 0( 1) ( ), 0k i−⟨ ⟩ =r v for 1, 2, , 1.i k= ⋅⋅⋅ −

⇒ ( ) ( 1) ( 1)1

k k kks− −−= +v r v

( 1)k−x

where


4. choose 1,ks − so that ( 1) ( ), 0k k−⟨ ⟩ =v Av

( ) ( 1) ( 1)1

k k kks− −−= +Av Ar Av

⇒ ( 1) ( ),k k−⟨ ⟩v Av ( 1) ( 1)1 ,k k

ks− −

−+ ⟨ ⟩v Ar( 1) ( 1),k k− −= ⟨ ⟩v Ar

⇒( 1) ( 1)

1 ( 1) ( 1)

,,

k k

k k ks− −

− − −

⟨ ⟩= −

⟨ ⟩v Arv Av

( 1) ( ), 0k k−⟨ ⟩ =v Av⇒

5. ( ) ( ), 0k i⟨ ⟩ =v Av for each 1, 2, , 2 .i k= ⋅⋅⋅ −

⇒ (1) ( ){ , , }k⋅ ⋅ ⋅v v is an A-orthogonal set .


6. Having chosen ( ) ,kv complete

( 1) ( 1) ( 1)( ) ( 1)1

( ) ( ) ( ) ( )

( 1) ( 1) ( 1) ( 1)

1( ) ( ) ( ) ( )

( 1) ( 1)( 1) ( 1)

( ) ( )

( ) ( 1) ( )

,,, ,

, ,, ,, ( , 0)

,

k k kk kk

k k k k k

k k k k

kk k k k

k kk k

k k

k k kk

st

s

t

− − −−−

− − − −

−

− −− −

−

⟨ + ⟩⟨ ⟩= =⟨ ⟩ ⟨ ⟩

⟨ ⟩ ⟨ ⟩= +⟨ ⟩ ⟨ ⟩

⟨ ⟩= ⟨ ⟩ =⟨ ⟩

⇒ = +

r v rv rv Av v Av

r r v rv Av v Avr r v rv Av

x x v

∵


7. Compute

8. ComputeSince ,

`

( )kr( ) ( 1) ( )k k k

kt−= +x x v

( ) ( ) ( 1) ( ) ( ) ( )

( ) ( )

, , ,

,

k k k k k kk

k kk

t

t

−⇒ ⟨ ⟩ = ⟨ ⟩ − ⟨ ⟩

= − ⟨ ⟩

r r r r Av r

r Av

( ) ( 1) ( )k k kkt

−⇒ = −r r Av

( ) ( 1) ( )k k kkt

−⇒ − = − +Ax b Ax b Av

( 1) ( )( , 0 )k k−⟨ ⟩ =r r∵

ks( 1) ( 1)

( ) ( )

,,

k k

k k kt− −⟨ ⟩

=⟨ ⟩

r rv Av

( 1) ( 1) ( ) ( ), ,k k k kkt

− −⟨ ⟩ = ⟨ ⟩r r v Av

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( 1) ( 1) ( 1) ( 1)

, ,, ,

(1 / ) , ,(1 / ) , ,

k k k k

k k k k k

k k k kk

k k k kk

s

tt − − − −

⟨ ⟩ ⟨ ⟩⇒ = − = −

⟨ ⟩ ⟨ ⟩

⟨ ⟩ ⟨ ⟩= =

⟨ ⟩ ⟨ ⟩

v Ar r Avv Av v Av

r r r rr r r r


Summary

for(0 ) (0 ) (1) (0 );= − =r b Ax v r 1 , 2 , 3 , ,k n=

( 1) ( 1)

( ) ( )

,,

k k

k k kt− −⟨ ⟩

=⟨ ⟩

r rv Av

( ) ( 1) ( )k k kkt

−= −x x v

( ) ( 1) ( )k k kkt

−= −r r Av

( ) ( )

( 1) ( 1)

,,

k k

k k ks − −

⟨ ⟩=

⟨ ⟩r r

r r

( 1) ( ) ( )k k kks+ = −v r v


Extend the conjugate gradient Method

• If the matrix A is ill-conditioned,the conjugate gradient method is highly susceptible to rounding errors» the exact answer shoud not be obtained in n steps

n

to include preconditioning

• If the matrix A is well-conditioned,An acceptable approximate solution is often in about steps


-1 -1( )t=A C A C

1. Choose nonsingular conditioning matrix C

2. Consider =Ax b

where t=x C x -1=b C b -1 -( ( ) )t t=C C-1 -1( )( )t t−⇒ = =Ax C AC C Ax C Ax

3. Solve =Ax b xfor-t⇒ =x C x

4. Preconditioning( ) ( )k t k=x C x( ) ( ) 1 1 ( )

1 ( ) 1 ( )

( )( )

k k t t k

k k

− − −

− −

⇒ = − = −

= − =

r b Ax C b C AC C xC b Ax C r


5. Let

6.

( ) ( ) ( ) 1 ( ),k t k k kC−= =v C v w r( ) ( ) 1 ( ) 1 ( )

( 1) ( 1) 1 ( 1) 1 ( 1)

( ) ( )

( 1) ( 1)

, ,, ,,,

k k k k

k k k k k

k k

k k

s− −

− − − − − −

− −

⟨ ⟩ ⟨ ⟩⇒ = =

⟨ ⟩ ⟨ ⟩

⟨ ⟩=

⟨ ⟩

r r C r C rr r C r C r

w ww w

( 1) ( 1) 1 ( 1) 1 ( 1)

( ) 1 ( )( ) ( )

( 1) ( 1) ( 1) ( 1)

( ) 1 ( ) ( ) ( )

, ,,,

, ,, ,

k k k k

k t k t t kk k

k k k k

t k k k k

t− − − − − −

− −

− − − −

−

⟨ ⟩ ⟨ ⟩= =

⟨ ⟩⟨ ⟩

⟨ ⟩ ⟨ ⟩= =

⟨ ⟩ ⟨ ⟩

r r C r C rC v C A C C vv A v

w w w wC v C A v v A v


7.

8.

9.

( ) ( 1) ( )k k kkt

−= +x x v( ) ( 1) ( )t k t k t k

kt−⇒ = +C x C x C v

( ) ( 1) ( )k k kkt

−⇒ = +x x v

( ) ( 1) ( )k k kkt

−= −r r Av1 ( ) 1 ( 1) 1 ( )k k t k

kt− − − − −⇒ = −C r C r C AC v

( ) ( 1) ( )k k t t kkt

− −⇒ = −r r AC C v( ) ( 1) ( )k k k

kt−⇒ = −r r Av

( 1) ( ) ( )k k kks

+ = +v r v( 1) 1 ( ) ( )t k k t k

ks+ −⇒ = +C v C r C v( 1) 1 ( ) ( )

( ) ( )

k t k kk

t k kk

s

s

+ − −

−

⇒ = +

= +

v C C r v

C w v


Summary

( 1) ( 1)

( ) ( )

,,

k k

k k kt− −⟨ ⟩

=⟨ ⟩w wv Av

( ) ( 1) ( )k k kkt

−= +x x v( ) ( 1) ( )k k k

kt−= −r r Av

( ) ( )

( 1) ( 1)

,,

k k

k k ks − −

⟨ ⟩=

⟨ ⟩w w

w w

( 1) ( ) ( )k t k kks

+ −= +v C w v

(0 ) (0 ) (1) (0 );= − =r b Ax v r 1 , 2 , 3 , ,k n=for

Date post:	23-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Ch 7. Iterative Techniques in Matrix...

Documents