Date post: | 12-Jan-2016 |
Category: |
Documents |
Upload: | edwin-bryan |
View: | 216 times |
Download: | 0 times |
Multivariate Statistics
Matrix Algebra II
W. M. van der VeldUniversity of Amsterdam
Overview
• The determinant of a matrix• The matrix inverse• System of equations
The determinant of a matrix
• The determinant of a matrix is a scalar and is denoted as |A| or det(A). Det(A) only exists when A is a square matrix.
• It has very important mathematical properties, but it is very difficult to provide a substantive definition.
• The determinant is necessary to compute the inverse of a matrix (A-1).– The inverse of a matrix is needed for solving systems of linear
equations; multivariate statistics often comes down to this. – When the determinant is zero, there exists no solution to a
system of linear equations.
• Let’s see how the value of the determinant is found.
The determinant of a matrix
• How to do it? The most simple case, a 2 by 2 matrix .• Det(A)=|A|=?
40
31A
40*34*140
31)( AADet
Cofactors
The determinant of a matrix
• One step further, a 3 by 3 matrix. • Det(A)=|A|=?
)0341(1
40
31*1A
12)160()4840()04(
08
15*2
48
35*2
)8345(2 )8105(2
408
315
221
A
Cofactor
The determinant of a matrix
• You should have noted that for matrices larger than first order, computation of the determinant is a recursive process. This process stops each time a 1 by 1 determinant is encountered, and involves multiplication by the cofactors.
The determinant of a matrix
• Let A be a matrix of order n x n. If we omit one or more rows or columns from A, we obtain a matrix of smaller order, called a minor of the matrix.
• Similarly, we have minors of a determinant, and in particular, if we omit from the determinant the ith row and the jth column, the resulting minor will be square and its determinant will be symbolized |Mij|. This determinant is called a cofactor (cij) if we give it a sign equal to (-1)i+j, so that: cij = (-1)i+j |Mij|. Using this notation we can write a formula for the expansion of a determinant of order n:
In this version the determinant is expanded according to it’s ith row.
n
gigigininiiii cacacaca
12211 A
The determinant of a matrix• The following rules are important for determinants, and
can help you sometimes to simplify calculations:– The determinant of A has the same value as the determinant of
A’.– The value of the determinant changes sign if one row (column)
is interchanged with another row (column).– If a determinant has two equal rows (columns), its value is
zero.– If a determinant has two rows (columns) with proportional
elements, its value is zero.– If all elements in a row (column) are multiplied by a constant,
the value of the determinant is multiplied by that constant.– If a determinant has a row (column) in which all elements are
zero, the value of the determinant is zero.– The value of the determinant remains unchanged if one row
(column) is added to or subtracted form another row (column). Moreover, if a row (column) is multiplied by a constant and then added to or subtracted from another row (column) the value remains unchanged.
The determinant of a matrix
• What is the determinant of:
2412
105D
123
321
842
A
10
01B
123
41C
The matrix inverse
• Let A be a square matrix. If we can find a matrix B of the same order as A such that AB=BA=I, then B is said to be the inverse of A and is symbolized A-1. A-1, if exists, can be found as follows.
• Let C be the matrix of cofactors of A (i.e., cij is the cofactor obtained from the minor |Mij|); then
• Where C’ is the transpose of C (or if one prefers, C’ is the matrix of cofactors of A’). It is immediately seen that the inverse is undefined if A is not square (since then there is no determinant |A|), and also if |A| is equal to zero.
ACA /1
The matrix inverse
• Illustration that AA-1 = A-1A = I.
2 3
4 1A
.1- 3.
.4 2.1
A
1.03.0
4.02.0
2 3
4 11AA
)1.0(2)4.0(3)3.0(2)2.0(3
)1.0(4)4.0(1)3.0(4)2.0(11AA
IAA
10
01
)2.02.1()6.06.0(
)4.04.0()2.12.0(1
The matrix inverse
• How did I get A-1?
2C
14
32C
32
C
4
32C
13
42C
1012223
41A
1.03.0
4.02.0
10
13
42
/1 AA C
2 3
4 1A Compute determinant
Now Compute C
C transpose => C’
Calculate A-1
A
The matrix inverse• Another way to calculate A-1. This way introduces you to
solving systems of equations.
.1- 3.
.4 2.1
A
2 3
4 1A
1 0
0 1
2 3
4 1
2221
12111
xx
xxIAA
10
01
2323
4141
2 3
4 1
22122111
22122111
2221
12111
xxxx
xxxx
xx
xxAA
123
041
023
141
2212
2212
2111
2111
xx
xx
xx
xx
105
;2023
141
2111
2111
2111
xx
xx
xx
205
;2123
041
2212
2212
2212
xx
xx
xx
3100
023
;3141
2111
2111
2111
xx
xx
xx
1100
123
;3041
2212
2212
2212
xx
xx
xx3.010
321
x
2.051
11 x
1.0101
22 x
4.052
12 x
The matrix inverse
• Rules for algebra with inverse matrices:– AA-1 = A-1A = I– (AB)-1 = B-1A-1
– (ABC)-1 = C-1B-1A-1
• Proof that (AB)-1 = B-1A-1.
System of equations
• In the introduction I already mentioned that the basic linear equation y=bx will be very important for multivariate methods.
• Here we will discuss how to solve systems of such linear equations.
System of equations• Illustration. Suppose we have the following set of
equations:-3=1x1+4x2
1=3x1+2x2
• The basic way to think about this problem set is finding the intersection, i.e. for which unknowns are the equations satisfied.
• This can be solved in a simple way (old style).
?.155
,2*231
413
211
21
21
xetcxx
addxx
xx
• The solution is basically the intersection of the lines represented by the equation.
• You won’t be surprised that there is a more general way to solve systems of linear equations, using matrix algebra.
System of equations
• Solution for m equations with n unknowns: m=n.
nmnmm
n
n
n x
x
x
aaa
aaa
aaa
k
k
k
2
1
21
22221
11211
2
1
Axk
Axkif AxAkA 11
xkA 1
IxkA 1
• What to do? Normally you divide by A so that you obtain a solution for x (give example: 15=3x).
• Matrix division is defined as multiplication by the inverse, so:
System of equations• Example. Suppose we have the following set of equations:
-3=1x1+4x2
1=3x1+2x2
• We already solved this one, resulting in x1=1 and x2=-1.
• The set of equations can be written as a matrix operation.
2
1
23
41
1
3
x
xAxk
System of equations
• Thus, we have to find the inverse of: A => A-1 = C’/|A|
23
41A
14
32C
4
32C
32C
2C
• We have to take the transpose of C
13
42C
System of equations
• We have to divide by |A|.
10)34()21(23
41A
13
42
10
11A
• Thus the inverse matrix is.
System of equations
2
1
1
3
13
42
10
1x
x
2
1
)11()33(
)14()32(
10
1x
x
2
1
1
1
10
10
10
1x
x
• Thus a solution for: -3=1x1+4x2 1=3x1+2x2 is found via
2
1
23
41
1
3
x
x
Axk
xkA 1
System of equationsExercise, solve: x1 + 2x2 = 0 3x1 + 7x2 = 1
1
0k
73
21A
xkA
1
2
10
20
1
0
13
271
A-1Ax = Ix = x = A-1k
• So if Ax = k solve via x = A-1k. • .... But it is not always so simple …
System of equations
• Sometimes, the requirement that m=n seems to be fulfilled, so that there should exist a solution.
• But consider the following cases.
(Row 2 = 2 x Row 1)
332
221
111
(Row 3 = Row 1 + Row 2)
1064
541
624
(Column 3 = Column 1 + Column 2), etc.
2
1
24
12
16
8
x
x
System of equations
• These situations are called linear dependence:
– Given vectors: x1, x2,…, xn-1
– Another vector xn is linearly dependent if there exists constants α1, α2,…, αn-1 such that:xn= α1x1+α2x2+ …+αn-1xn-1
• Otherwise the vector xn is linearly independent.
• In case of linear dependence; |A|= 0.• And then the inverse is not defined: A-1=C’/|A|.• And when the inverse is not defined we cannot find a
solution via: A-1k=x.
System of equations
• Generally a unique solution exists only if m=n, and |A|≠0
• When are there ‘problems’?– If m<n there are many solutions, the problem is
underdetermined. 8x1+10x2+14x3=94x1+12x2+16x3=10
– if m>n there are no solutions, the problem is overdetermined.8x1+10x2=94x1+12x2=104x1+10x2=2
System of equations
• Using the idea of linear dependency, the rank of a matrix can be introduced.
• rank(A) = number of linearly independent rows or columns.
• Given an mxn matrix, with m ≥ n, then if – |A| ≠ 0 rank(A) = n full rank, solvable– |A| = 0 rank(A) < n rank deficient
• We will get back to the issue of rank.
Overdetermined Systems
• Minimization problem: • Normal equations: (A’A)x = A’k• Solution: x = (A’A)-1A’k
– A’A must be nonsingular; i.e. |A’A|≠0– (A’A)-1A’ is called the left inverse matrix
)()( kAxkAxΦ
• Find Ax “closest” to k• Least-squares distance measure
Underdetermined Systems• Find “smallest” x that satisfies equations• Minimum norm objective
xxΦ 2
1
• Constrained minimization problem:kAx
Φ
:subject to
minx
• Solution: x = A’(AA’)-1k– AA’ must be nonsingular– A’(AA’)-1 is called the right inverse