Linear algebra

& Numerical Analysis

Direct methods

Marta Jarošová http://homel.vsb.cz/~dom033/

Outline

Gaussian elimination

LU factorization

Pivoting

Cholesky factorization

Solving the systems of linear

equations

System of linear equations

Elementary operations

DEFINITION

1. interchange of 2 equations,

say “interchange the i-th and j-th equations"

2. multiply an equation by a non-zero constant,

say “multiply the k-th equation by c≠0 “

3. add a constant multiple of one equation to

another equation,

say “add c time multiple of the i-th equation to j-th

equation”

Elementary row operations

DEFINITION

1. interchange of 2 rows,

say “interchange the i-th and j-th rows"

2. multiply a row by a non-zero constant,

say “multiply the k-th row by c≠0 “

3. add a constant multiple of one row to another row,

say “add c time multiple of the i-th row to j-th row”

Gaussian elimination

Two steps:

1. Forward elimination – reduces a given system to

either triangular or echelon form

2. Back substitution – finds the solution of the

system above

Echelon matrix

Gaussian elimination – example

Example:

Solution:

augmented matrix

Gaussian elimination – example

Forward elimination

Gaussian elimination – example

Back substitution

LU factorization

Let us consider the factorization

with an upper triangular matrix and a lower triangular

matrix with a unit diagonal.

1 1 1 1

0 1 2 3

0 2 5 9

0 3 9 19

1 1 1 1

0 1 2 3

0 0 1 3

0 0 3 10

k-th column

LU factorization

multipliers

is k-th vector of standard basis in

LU factorization

for

for

pivots

LU factorization

Example:

1 1 1 1

0 1 2 3

0 2 5 9

0 3 9 19

1 1 1 1

0 1 2 3

0 0 1 3

0 0 3 10

k-th column

1 0 0 0

0 1 0 0

0 -2 1 0

0 -3 0 1

=

LU factorization

And what about computing the inverse matrices of ?

Isn’t it expensive?

LU factorization

To obtain it is enough to multiply off-diagonal elements by -1.

So the matrix L can be constructed directly since

Example

Algorithm

function [L,U] = my_lu(A)

n = size(A, 1); I = eye(n); L = I; U = A;

for k=1:n-1

L(k+1:n,k) = U(k+1:n,k)/U(k,k); %multipliers

for j = k+1:n

U(j,k:n) = U(j,k:n)-L(j,k)*U(k,k:n); %rows

end

end

function [L,U] = my_lu2(A)

n = size(A, 1); I = eye(n); L = I; U = A;

for k=1:n-1

L(k+1:n,k) = U(k+1:n,k)/U(k,k); %multipliers

U(k+1:n,k:n) = U(k+1:n,k:n)-L(k+1:n,k)*U(k,k:n); %block

end

Algorithm

function [L,U] = my_lu3(A)

n = size(A, 1); I = eye(n); O = zeros(n);

L = I; U = O;

for k = 1:n

if k == 1

v(k:n) = A(k:n,k);

else

z = L(1:k-1,1:k-1)\A(1:k-1,k);

U(1:k-1,k) = z;

v(k:n) = A(k:n,k)-L(k:n,1:k-1)*z;

end

if k < n, L(k+1:n,k) = v(k+1:n)/v(k); end

U(k,k) = v(k);

end

Which implementation is more efficient?

tic/toc functions work together to measure elapsed time

A = pascal(1000);

tic; lu(sparse(A),0); toc 0.7754 sec

time in sec my_lu(A) my_lu2(A) my_lu3(A)

11.4791

11.1275

11.1100

11.1042

11.1608

5.9897

5.9620

5.9940

5.9959

6.0212

2.8146

2.8878

2.9164

2.9275

2.9194

aver 11.1963 5.9926 2.8931

Alternative implementation

Since L11 = 1 we obtain U1k from (Eq1).

From (Eq2)

Since Lkk = 1 we have

(Eq1)

(Eq2)

and

Computational complexity

System of n equations and n unknowns

1. Forward elimination: in k-th step: (n-k) divisions to evaluate

multipliers and (n-k)(n-k+1) operations (+) and (*) to evaluate

elements aij

2. Back substitution – to evaluate k-th unknown needs one

division and (n-k) operations (+) and (*)

For large n dominates the term

(+) additions and subtractions

(*) multiplications

Pivoting

In the case of a zero pivot element interchanging rows or

columns is necessary.

The strategy which place a "good" element in the diagonal

position (k,k) prior to a particular operation is called

partial pivoting - the interchanging of rows

full pivoting - the interchanging of both rows and columns

Partial pivoting

We consider the LU factorization in the form

PA = LU

where P is permutation matrix.

to place the element (p,k) to the position (k,k) we have to

interchange the rows k and p

p is row index of the element which is maximal in absolute

value of

Algorithm: LU with partial pivoting

function [L,U,P] = my_lu_piv(A)

n = size(A,1); I = eye(n); O = zeros(n); L = I; U = O; P = I;

function change_rows(k,p) …

function change_L(k,p) …

for k = 1:n

if k == 1, v(k:n) = A(k:n,k);

else

z = L(1:k-1,1:k -1)\ A(1:k-1,k);

U(1:k-1,k) = z;

v(k:n) = A(k:n,k)-L(k:n,1:k-1)*z;

end

if k<n

x = v(k:n); p = (k-1)+find(abs(x) == max(abs(x))); % find index p

change_rows(k,p);

L(k+1:n,k) = v(k+1:n)/v(k);

if k > 1, change_L(k,p); end

end

U(k,k) = v(k);

end

end

function change_rows(k,p)

x = P(k,:); P(k,:) = P(p,:); P(p,:) = x;

x = A(k,:); A(k,:) = A(p,:); A(p,:) = x;

x = v(k); v(k) = v(p); v(p) = x;

end

function change_L(k,p)

x = L(k,1:k-1); L(k,1:k-1) = L(p,1:k-1);

L(p,1:k-1) = x;

end

Full pivoting

We consider the LU factorization in the form

PAQ = LU

where P and Q are permutation matrices.

to place the element (p,q) to the position (k,k) we have to

interchange the rows k and p and columns k and q

p/q are row/column indices of the element which is maximal

in absolute value of

MATLAB function: lu

[L,U]=lu(A) stores an upper triangular matrix in U and a

"psychologically lower triangular matrix" (i.e. a product of

lower triangular and permutation matrices) in L, so that A =

L*U. A can be rectangular.

[L,U,P]=lu(A) returns unit lower triangular matrix L,

upper triangular matrix U, and permutation matrix P so that

P*A = L*U.

[L,U]=lu(sparse(A),0) no pivoting

For more details see: help lu

Cholesky factorization

We consider the factorization of a positive definite matrix A

with a lower triangular matrix .

Cholesky factorization can be generalized for positive semi-definite matrices.

Algorithm

function L = my_chol(A)

n = size(A,1); O = zeros(n);

L = O;

for k = 1:n

if k == 1

L(k,k) = sqrt(A(k,k));

L(k+1:n,k) = A(k+1:n,k)/L(k,k);

else

v = L(k,1:k-1)'; %L_k1'

L(k,k) = sqrt(A(k,k)-v'*v);

L(k+1:n,k) = (A(k+1:n,k)-L(k+1:n,1:k-1)*v)/L(k,k);

end

end

Cholesky factorization can be considered also with pivoting: PAPT = LLT.

MATLAB function: chol

chol(A) uses only the diagonal and upper triangle of A. The lower

triangle is assumed to be the (complex conjugate) transpose of the upper triangle. If A is positive definite, then R = chol(A) produces an upper

triangular R so that R'*R = A. If A is not positive definite, an error message

is printed.

L = chol(A,'lower') uses only the diagonal and the lower triangle

of A to produce a lower triangular L so that L*L' = A. If A is not positive

definite, an error message is printed. When A is sparse, this syntax of

CHOL is typically faster.

[R,p] = chol(A) with two output arguments function never produces

an error message.

For more details see: help chol

Solving the systems of linear equations

LU factorization To find the solution of the system Ax = b we will solve two systems with triangular matrices.

Ax = b

L (LT (Ux)) = b

L(Ux) = Pb

d = Pb, Lz = d, Ux = z

In Matlab: x = U\(L\(P*b))

PA = LU

A = PTLU

d z

Solving the systems of linear equations

Cholesky factorization

The situation is similar to previous one.

Ax = b

L (LTx) = b

Lz = b, Ux = z

In Matlab: x = L'\(L\b)

A = LLT

z

References

Gilbert W. Stewart, Matrix Algorithms: Basic

decompositions (available on Google books)

Lloyd Nicholas Trefethen, David Bau, Numerical linear

algebra (available on Google books)

Kozubek, Brzobohatý, Hapla, Jarošová, Markopoulos:

LINEÁRNÍ ALGEBRA S MATLABEM, http://mi21.vsb.cz/

(in Czech)

MATLAB Help