Review of Some Concepts from Linear Algebra Part 1wright/courses/m566/Linear... · 2020-01-12 ·...

Review of Some Concepts from Linear AlgebraPart 1

Department of Mathematics

Boise State University

January 12, 2020

Math 566 Linear Algebra Review: Part 1 January 12, 2020 1 / 36

Numerical Linear Algebra

Numerical linear algebra is one of the pillars of computationalmathematics and scientific computing. Why?

Most problems that arise in the applied sciences, engineering,finance, economics, mathematics, etc. are not scalar.Non-linear problems are solved by approximating them with linearproblems.


Numerical Linear Algebra

There are four main problems in numerical linear algebra:1 Solving large systems of linear systems of equations2 Least squares problems3 Eigenvalue problems4 Singular value decomposition

In these slides we review some concepts from linear algebra that willprepare us to study these problems.


Linear systems of equations

A linear system of m equations and n unknowns can be expressed inthe following general form:

a11x1 + a12x2 + a13x3 + · · · + a1nxn = b1,a21x1 + a22x2 + a23x3 + · · · + a2nxn = b2,a31x1 + a32x2 + a33x3 + · · · + a3nxn = b3,

......

.... . .

......

am1x1 + am2x2 + am3x3 + · · · + amnxn = bm.

(1)

Here aij are the coefficients of the systems, bi are the right hand sides(RHS), and xj are the unknown values that must be determined. aijand bi will be given by the problem.


Linear systems of equations

Linear systems can be classified into the following three types:1 Square linear system: If the number of equations equals the

number of unknowns (i.e. m = n).2 Overdetermined system: If the number of equations is greater

than the number of unknowns (i.e. m > n).3 Underdetermined system: If the number equations is less than

the number of unknowns (i.e. m < n).


Matrices and vectors

A convenient notation to describe a linear system of equations is interms of matrices and vectors.


Matrices

A matrix is just a table of numbers containing m rows and ncolumns and can be expressed as:

A =

a11 a12 a13 · · · a1na21 a22 a23 · · · a2na31 a32 a33 · · · a3n

......

.... . .

...am1 am2 am3 · · · amn

.

We typically use capital letters to denote matrices.A ∈ Rm×n denotes a matrix with m rows and n columns with realnumbers.A ∈ Cm×n denotes a matrix with m rows and n columns withcomplex numbers.


Matrices


A =


......

.... . .

...am1 am2 am3 · · · amn

.



Matrices


A =


......

.... . .

...am1 am2 am3 · · · amn

.



Matrices


A =


......

.... . .

...am1 am2 am3 · · · amn

.



Matrices


A =


......

.... . .

...am1 am2 am3 · · · amn

.



Matrices

A common shorthand notation for a matrix is A ={

aij}

, where thevalues for i and j are understood from the problem.We will also use MATLAB notation A(i : j , k : l) to denote thesubmatrix of A lying in rows i through j and columns k through l .For example, if

A =

1 2 3 4 56 7 8 9 10

11 12 13 14 1516 17 18 19 20

,then

A(2 : 4,3 : 5) =

8 9 1013 14 1518 19 20

.Math 566 Linear Algebra Review: Part 1 January 12, 2020 8 / 36

Vectors

If the matrix only has one row or column then it is called a vector.A column vector with n entries can be expressed as

x =

x1x2x3...

xn

.

A row vector and can be expressed as

x =[x1 x2 x3 · · · xn

].

We typically use bold lower-case letters to denote vectors.A column vector with n real entries is denoted by x ∈ Rn, while arow vector is denoted by x ∈ R1×n.


Vectors


x =

x1x2x3...

xn

.


x =[x1 x2 x3 · · · xn

].



Vectors


x =

x1x2x3...

xn

.


x =[x1 x2 x3 · · · xn

].



Vectors


x =

x1x2x3...

xn

.


x =[x1 x2 x3 · · · xn

].



Vectors


x =

x1x2x3...

xn

.


x =[x1 x2 x3 · · · xn

].



Matrix & vector operations


Matrix & vector operations: Transpose

Let A ∈ Rm×n with entries

A =


......

.... . .

...am1 am2 am3 · · · amn

= {aij},

then the transpose of A switches the columns of A with the rows, i.e.

AT =

a11 a21 a31 · · · am1a12 a22 a32 · · · am2a13 a23 a33 · · · am3

......

.... . .

...a1n a2n a3n · · · amn

= {aji}.

Note that AT ∈ Rn×m and that (AT )T = A.Math 566 Linear Algebra Review: Part 1 January 12, 2020 11 / 36

Matrix & vector operations: Complex transpose

Let A ∈ Cm×n with entries

A =


......

.... . .

...am1 am2 am3 · · · amn

= {aij},

then the complex transpose of A is the transpose of A with theentries complex conjugated, i.e.

A∗ =

a11 a21 a31 · · · am1a12 a22 a32 · · · am2a13 a23 a33 · · · am3

......

.... . .

...a1n a2n a3n · · · amn

= {aji}.

Note that A∗ ∈ Cn×m and that (A∗)∗ = A.Math 566 Linear Algebra Review: Part 1 January 12, 2020 12 / 36

Matrix & vector operations: Transpose

The transpose can also be applied to vectors. In this case if x is a(column) vector then xT is a row vector:

if x =

x1x2x3...

xn

then xT =[x1 x2 x3 · · · xn

].

Similarly if x is row vector then xT is a column vector.

The same is holds for the complex transpose.


Matrix & vector operations: Matrix addition

Let A ∈ Cm×n and B ∈ Cm×n then the sum of A and B is given by

A + B ={

aij + bij

}.

This is just the sum of the corresponding entries of the elements of Aand B.

For this sum to make sense A and B must be the same size.


Matrix & vector operations: scalar multiplication

Let α be a complex number and A ∈ Cm×n then the product of α and Ais given by

αA ={α aij

}.

Note that this is just α times each entry of A.


Matrix & vector operations: Vector-vector products

There are two types of vector-vector products that arise quitefrequently:

Inner productOuter product

These can be derived from the definition for matrix-matrix products(discussed later), but it is worth stating them separately.


Matrix & vector operations: Inner product

Let x,y ∈ Rn then the inner product or dot product of x and y is

xT y =[x1 x2 · · · xn

]

y1y2...

yn

= x1y1 + x2y2 + . . .+ xnyn =n∑

j=1

xjyj .

Note that the inner product is a single number. The inner productis sometimes denoted by x · y.If x and y are complex then their inner product is x∗y.


Matrix & vector operations: Outer product

Let x ∈ Rm and y ∈ Rn then the outer product of x with y is

xyT =

x1x2...

xm

[y1 y2 · · · yn]=

x1y1 x1y2 · · · x1ynx2y1 x2y2 · · · x2yn

......

. . ....

xmy1 xmy2 · · · xmyn

Note that the outer product is a matrix of size m-by-n.If x and y are complex then their inner product is xy∗.


Matrix & vector operations: Matrix-vector products

Let A ∈ Cm×n and x ∈ Cn then the product of A and x is given by

Ax =x1

a11a21a31

...am1

+ x2

a12a22a32

...am2

+ x3

a13a23a33

...am3

+ . . .+ xn

a1na2na3n

...amn

(2)

Thus, the product Ax is a linear combination of the columns of A.

The set of all linear columns of A is called the column space of A.(More on this later...)


Matrix & vector operations: Matrix-vector products

Important observations regarding the matrix-vector product Ax:The only way for this product to make sense is if A has the samenumber of columns as x does rows.Ax ∈ Cm, i.e. the product is a column vector containing m entries.If we let b = Ax then we can alternatively express the i th entry ofb as

bi =n∑

j=1

aijxj , i = 1, . . . ,m.

This illustrates that bi is just the inner product of the i th row of Awith the vector x.In general, computing Ax using the above formulas requires mnmultiplications and m(n − 1) additions.


Matrix & vector operations: Matrix-matrix products

Let A ∈ Cm×n and B ∈ Cn×p, and let B have columns

B =

b1 b2 · · · bp

.The matrix-matrix product C = AB is given as

C =

Ab1 Ab2 · · · Abp

.This shows the k th column of the product AB is a linear combination ofthe columns of A with the coefficients in the linear combinations beingdetermined by entries in the k th column of B.



Important observations regarding the matrix-matrix product AB,A ∈ Cm×n, B ∈ Cn×p:

Number columns of A must equal number rows B.AB ∈ Cm×p, i.e. the product is a matrix containing m rows and pcolumns.In general, AB 6= BA , i.e. the product does not commute.



Important observations regarding the matrix-matrix product AB,A ∈ Cm×n, B ∈ Cn×p:

We can express each entry of C as

cik =n∑

j=1

aijbjk , i = 1, . . . ,m, k = 1, . . . ,p.

So cik is just the inner product of the i th row of A with the k thcolumn of B.Computing AB using the above formulas requires mnpmultiplications and m(n − 1)p additions.The transpose of the product AB satisfies: (AB)T = BT AT .


Linear systems in matrix-vector notation

Recall that we can express a linear system of equations with mequations and n unknowns as

a11x1 + a12x2 + a13x3 + · · · + a1nxn = b1,a21x1 + a22x2 + a23x3 + · · · + a2nxn = b2,a31x1 + a32x2 + a33x3 + · · · + a3nxn = b3,

......

.... . .

......

am1x1 + am2x2 + am3x3 + · · · + amnxn = bm.

(3)

We can express this linear system in matrix-vector notation using theprevious definitions.


Linear systems in matrix-vector notation

Let x ∈ Cn, b ∈ Cm, and A ∈ Cm×n, then the linear system is given asAx = b, or


......

.... . .

...am1 am2 am3 · · · amn

︸︷︷︸

A

x1x2x3...

xn

︸︷︷︸

x

=

b1b2b3...

bm

︸︷︷︸

b

.


Linear systems: solvability

Recall that Ax is a linear combination of the columns of A:

Ax =x1

a11a21a31

...am1

+ x2

a12a22a32

...am2

+ x3

a13a23a33

...am3

+ . . .+ xn

a1na2na3n

...amn

.

Thus, the only way there will be a solution to Ax = b is if b can bewritten as a linear combination of the columns of A.

Or, b is in element in the column space of A.



There are three possibilities for the linear system Ax = b:1 There are an infinite number of solutions that satisfy Ax = b.

An infinite number of ways to linearly combine the columns of A to equal b.

2 There is one unique solution to the linear system.Only one way to linearly combine the columns of A to equal b.

3 There is no solution to the linear system.There is no way to linearly combine the columns of A to equal b.

In the case of (1) and (2) b is an element of the column space of A, butfor (3) b is not.



There are three possibilities for the linear system Ax = b:1 There are an infinite number of solutions that satisfy Ax = b.

An infinite number of ways to linearly combine the columns of A to equal b.

2 There is one unique solution to the linear system.Only one way to linearly combine the columns of A to equal b.

3 There is no solution to the linear system.There is no way to linearly combine the columns of A to equal b.

In the case of (1) and (2) b is an element of the column space of A, butfor (3) b is not.


Special types of matrices: Diagonal matrix

A diagonal matrix is an n-by-n square matrix with zeros in every entryexcept possibly the main diagonal:

D =

d1 0 0 · · · 00 d2 0 · · · 00 0 d3 · · · 0...

......

. . ....

0 0 0 · · · dn

,

where d1,d2, . . . ,dn are some real or complex numbers.


Special types of matrices: Identity matrix

The identity matrix is a diagonal matrix with every diagonal entry equalto 1:

I =

1 0 0 · · · 00 1 0 · · · 00 0 1 · · · 0...

......

. . ....

0 0 0 · · · 1

It has the property that for any matrix A ∈ Rn×n, IA = AI = A.


Special types of matrices: Lower triangular matrix

A matrix L ∈ Cm×n is lower triangular if all the entries above its maindiagonal are zero. Square n-by-n lower triangular matrices take theform

L =

`11 0 0 · · · 0`21 `22 0 · · · 0`31 `32 `33 · · · 0...

......

. . ....

`n1 `n2 `n3 · · · `nn

,

where `i,j , i = 1, . . . ,n, j = i , . . . ,n, are some complex numbers.


Special types of matrices: Upper triangular matrix

A matrix U ∈ Cm×n is upper triangular if all the entries below its maindiagonal are zero. Square n-by-n upper triangular matrices take theform

U =

u11 u12 u13 · · · u1n0 u22 u23 · · · u2n0 0 u33 · · · u3n...

......

. . ....

0 0 0 · · · unn

,

where ui,j , i = 1, . . . ,n, j = i , . . . ,n, are some complex numbers.


Special types of matrices: Lower Hessenbergmatrix

A matrix L ∈ Cm×n is lower Hessenberg if all the entries above its firstsuper diagonal are zero, e.g.

L =

`11 `12 0 0 · · · 0`21 `22 `23 0 · · · 0

`31 `32 `33. . . · · · 0

......

.... . . . . .

......

......

. . . . . . `n−1,n`n1 `n2 `n3 · · · · · · `nn

,

where `i,j , i = 1, . . . ,n, j = i , . . . ,n, are some complex numbers.

An upper Hessenberg matrix is the (complex) transpose of lowerHessenberg matrix.


Special types of matrices: Symmetric matrix

A square matrix A ∈ Rn×n is symmetric if A = AT .

A square matrix A ∈ Cn×n is Hermitian if A = A∗.

A square matrix A ∈ Rn×n is skew-symmetric or anti-symmetric ifA = −AT .

A square matrix A ∈ Cn×n is skew-Hermitian or anti-Hermitian ifA = −A∗.














Inverse of a matrix

Let A ∈ Cn×n and suppose there exists a matrix B ∈ Cn×n such that

BA = AB = I,

where I is the n-by-n identity matrix. Then B is called the inverse of A.

The inverse of A is denoted by A−1.If A−1 exists then A is called nonsingular, otherwise it is singular.


Matrix inverses and linear systems

If A is a square, nonsingular matrix, then the solution to the linearsystem Ax = b is given formally as

x = A−1b.

Important: When solving a linear system, one should never firstcompute A−1 and then compute the product A−1b. There are muchbetter ways to solve the system (for example using Gaussianelimination when n is not too large).


Properties of matrix inverses

Suppose A is nonsingular then the following statements are trueA−1 is uniqueA−1 is nonsingular and its inverse is AAT is nonsingularIf B ∈ Cn×n is nonsingular then AB is nonsingular and(AB)−1 = B−1A−1

The linear system Ax = b has a unique solution.


Date post:	21-May-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

Review of Some Concepts from Linear Algebra Part 1wright/courses/m566/Linear... · 2020-01-12 ·...

Documents