Date post: | 22-Apr-2015 |
Category: |
Science |
Upload: | solo-hermelin |
View: | 135 times |
Download: | 6 times |
1
Matrices II
SOLO HERMELIN
Updated: 20.07.07
2
SOLO Matrices
Table of Content
Singular Values
Definitions
Domain and Codomain of a Matrix A
Properties of Square Orthogonal Matrices
Definition of the Singular Values
Geometric Interpretation of Singular Values
Properties of Singular Values
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Householder Transformation
Projection of a vector on a vector . b
a
111 min&min1
nxmxnxmxnxxbxAd
nx
Computation of Moore-Penrose Pseudoinverse Matrix, A †
Properties of Moore-Penrose Pseudoinverse Matrix, A †
Description of Projections Related to Moore-Penrose Pseudoinverse
Particular case (1) r = n ≤ m:
Particular case (2) r = m ≤ n:
3
SOLO Matrices
Table of Content (continue – 1)
General Solution of Amxn Xnxp = Bmxp
Particular case (1) r = m ≤ n
Particular case (2) r = n ≤ n
General Solution of YpxmAmxn = Cpxn
Particular case (1) r = m ≤ n
Particular case (2) r = n ≤ n
Inverse of Partitioned Matrices
Matrix Inverse Lemmas Identities
Matrix Schwarz Inequality
Trace of a Square Matrix
References
4
SOLO Matrices
Singular Values
Definitions:
Any complex matrix A with n rows (r1, r2,…,rn) and m columns (c1,c2,…,cm)
m
n
nxm ccc
r
r
r
A ,,, 21
2
1
can be considered as a linear function (or mapping or transformation) for am-dimensional domain to a n-dimensional codomain.
AcodomyAdomxxAyA nxmxnxm 11;:
In the same way its conjugate transpose:
H
n
HH
H
m
H
H
H
mxn rrr
c
c
c
A ,,, 212
1
is a linear function (or mapping or transformation) for an-dimensional codomain to a m-dimensional domain.
AcdomxAcodomyyAxA mxnx
HH
mxn 111111 ;:Table of Contents
5
SOLO Matrices
Domain and Codomain of a Matrix A
The domain of A can be decomposed into orthogonal subspaces:
ANARAdom H
HAR
AN
HAN
AR
xAy
11 yAx H
Adomxmx 1
11mxx
Acodomy nx 11
1nxyR (AH) – is the row space of AH (dimension r)
N (A) – is the null-space of A (x N (A) A x = 0) or the kernel of A (ker (A)) (dimension m-r)
The codomain of A (domain of AH) can be decomposed into orthogonal subspaces:
HANARAcodom
R (A) – is the column space of A (dimension r)
N (AH) – is the null-space of AH (dimension n-r)
Singular Values
Table of Contents
6
SOLO
Hermitian = Symmetric if A has real components
Hermitian Matrix: AH = A, Symmetric Matrix: AT = A
Matrices
Properties of Square Orthogonal Matrices
Use Pease, “Methods of Matrix Algebra”, Mathematics in Science and Engineering Vol.16, Academic Press 1965
Definitions:
Adjoint Operation (H):
AH = (A*)T (* is complex conjugate and T is transpose of the matrix)
Skew-Hermitian = Anti-Symmetric if A has real components.
Skew-Hermitian: AH = -A, Anti-Symmetric Matrix: AT =-A
Unitary Matrix: UH = U-1, Orthonormal Matix: OT = O-1
Unitary = Orthonormal if A has real components.
Charles Hermite1822 - 1901
7
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 1)
Lemma1: All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal.
Proof of Lemma1:
Pre-multiply by :iii xHx Hix i
Hiii
Hi xxHxx
and take the conjugate transpose: iH
iiiHH
i
H
iH
i xxxHxHxx *
This proves that the eigenvalues of H are real.
Subtract those two equations: 00 ** i
H
iiii
H
iii xxsincexx
From Hij
Hjj
Hj
HHj
Hj xxHxHxHx *
Pre-multiply by and post-multiply by and subtract iii xHx H
jx Hij
Hj xHx ix
0
i
H
jji
i
H
jji
H
j
i
H
jii
H
jxx
xxHxx
xxHxx
0 iH
jji xxIf
If we can use the Gram-Schmidt procedure to obtain an eigenvector orthogonal to .
ji jx~
ix i
iH
i
jH
ijj x
xx
xxxx ~
8
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 2)
Lemma1: All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal.
Proof of Lemma1 (continue – 1):
0
i
H
jji
i
H
jji
H
j
i
H
jii
H
jxx
xxHxx
xxHxx
0 iH
jji xxIf
If we can use the Gram-Schmidt procedure to obtain an eigenvector orthogonal to .
ji jx~
ix i
iH
i
jH
ijj x
xx
xxxx ~
we can see that 0~ i
Hi
iH
i
jH
ij
Hij
Hi xx
xx
xxxxxx
jii
i
H
i
j
H
i
jiii
i
H
i
j
H
i
jji
i
H
i
j
H
i
jj xxxx
xxxx
xx
xxxxH
xx
xxxHxH ~~
q.e.d.
9
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 3)
Lemma2: Any hermitian matrix H can be factored in H = U Λ UH
where Λ=diag (λ1,λ2,…,λn) and U is unitary i.e. U UH = UH U = In.Proof of Lemma2:
Let normalize the orthogonal eigenvectors of H ;i.e. iii xxu /:
or H U = U Λ where U = [u1,u2,…,un]
Because U is a square matrix having orthonormal columns, and is a square matrix,U is also a unitary matrix satisfying UH U=U UH=In. q.e.d.
000
00
00
,,,,,, 2
1
2121
nn uuuuuuH
10
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 4)
Lemma3: A AH and AH A are hermitian matrices that have the same nonzero real positive eigenvalues.
Proof of Lemma3:
q.e.d.
HHHHHH AAAAAA i.e. A AH is hermitian, therefore the eigenvalues λi (A AH) are real and positive according to Lemma 1.
Suppose ui is a normalized eigenvector of λi (A AH) ≠0
i
H
ii
H uAAuAA Pre-multiply by AH and define i
H
H
i
i uAAA
v
1:
i
H
ii
H
H
i
i
HH
iH
i
i
HH
i
HH
ii
HH
vAAvAA
AA
uAAA
AA
uAAAuAAAuAAA
we get
We can see that νi is the eigenvector of AH A and λi (A AH) is the corresponding eigenvalue, meaning that both AH A and A AH have the same nonzero eigenvalues.
From 0
2
2
i
i
i
H
i
i
H
iH
ii
H
i
H
ii
HH
ii
H
ii
H
v
vA
vv
vAvAAAvvAAvAAvvAAvAA
Therefore we can define 0: H
ii AA
11
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 5)
Lemma4: If U is a unitary matrix then all its eigenvalues have unit modulus. .
Proof of Lemma4:
form the inner product
IUUUU HH Consider the set of eigenvalues x1, x2, …, xn which we know to be complete and
iii xUx
nixxIxxUxUxxx iiiiH
iiH
iiHH
iiH
iii ,11**
q.e.d.
12
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 6)
Proof of Lemma5:
Lemma5: Every unitary matrix U can be expressed as an exponential matrix: where H is hermitian (jH is skew-hermitian)
jHeU
Since the eigenvalues of U have unit modulus; i.e. we can writenii ,11
niej HijHiHii ,1sincos
jHjjj eSeeediagSU HnHH 1,,, 21
121 ,,, SdiagSH HnHH where:
q.e.d.
13
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 7)
Table of Contents
Decomposition of Square Matrices:
HHHH AAj
jAAAAAAA22
1
2
1
2
1
HH
H AAAA
2
1
2
1
HHH
H AAAAAA
2
1
2
1
2
1
here: Hermitian
Skew-Hermitian
HHH
H AAj
AAj
AAj
222Hermitian
HH AAj
jAAA22
1 the matrix generalization of the decomposition of a complex number in the real and imaginary part.
14
SOLO Matrices
Lemma6: (6.1) Every complex nxm matrix of rank can be factored into:
Definition of the Singular Values
H
H
rmxrnxrrn
rmrxH
mxmnxmnxnnxm
xmrm
rxmrxr
rnnxnxr V
VUUVUA
2
11
21 00
0
where 0,,, 21211 rrdiagrxr
Unxn and Vmxm are unitary matrices, i.e.:
rnxrn
rxr
H
H
H
H
H
H
I
IUU
U
UUU
U
UUUUU
0
021
2
1
2
121
rmxrm
rxr
H
H
H
H
H
H
I
IVV
V
VVV
V
VVVVV
0
021
2
1
2
121
(6.2) σi i=1,…,r are the positive square roots of the nonzero eigenvalues of AH A or A AH and are called the singular values of A.
(6.3) The dyadic expansion of A is: where ui and vi are the columns of U1 and V1 respectively.
r
i
Hiii vuA
1
Singular Values
15
SOLO Matrices
Lemma6 (continue – 1):
Definition of the Singular Values
(6.5) The columns of V are orthonormal eigenvectors of AH A:
rmxrmxrrm
rmrxH rxrVVVVAA00
02
1
2121
i.e. the columns of V1 are the eigenvectors of the nonzero eigenvalues, and the columns of V2 are the eigenvectors of the zero eigenvalues of AH A.
(6.6) The following relations exist between U1 and V1
1111
1111
rxrnxrmxr
rxrmxrnxr
UAV
VAUH
nxm
nxm
i.e. the columns of U1 are the eigenvectors of the nonzero eigenvalues, and the columns of U2 are the eigenvectors of the zero eigenvalues of A AH.
(6.4) The columns of U are orthonormal eigenvectors of A AH:
rmxrnxrrn
rmrxH rxrUUUUAA00
021
2121
Singular Values
16
SOLO Matrices
Lemma6 (continue – 2):
Definition of the Singular Values
HAR
AN
HAN
AR
xAy
11 yAx H
Adomxmx 1
11mxx
Acodomynx
11
1nxy
(6.7) The columns U1 of form an orthonormal basis for the column space of A: ARUR 1
The columns of U2 form an orthonormal basis for the nullspace of AH: HH AANUR ker2
The columns of V1 form an orthonormal basis for the column space of AH:
HARVR 1
The columns of V2 form an orthonormal basis for the nullspace of A:
AANVR ker2
Singular Values
17
SOLO Matrices
Lemma6 (continue – 3):
Proof of Lemma 6:
Definition of the Singular Values
and
From Lemma 3 we have
H
H
rnxrnxrrn
rnrxH
nxnnxn
H
mxnnxm
xnrn
rxnrxr
rnnxnxrnxn U
UUUUUAA
2
12
1
21100
0
H
H
rmxrmxrrm
rmrxH
mxmmxmnxm
H
mxn
xmrm
rxmrxr
rmmxmxrmxm V
VVVVVAA
2
12
1
21200
0
where 0,,, 21211 rrdiagrxr
and riAAAA H
i
H
ii ,,2,10:
Those equations can be rewritten as: 0
0
1121
1121
VVVAA
UUUAAH
H
or
0
0
2
2
111
111
VAA
UAA
VVAA
UUAA
H
H
H
H
HU 2
HV2
00
00
22222
22222
VAVAVAVAAV
UAUAUAUAAUHHH
HHHHHH
Singular Values
18
SOLO Matrices
Lemma6 (continue – 4):
Proof of Lemma 6 (continue – 1):
Definition of the Singular Values
02 UAH The columns of U2 form an orthonormal basis for the nullspace of AH:
HH AANUR ker2
02 VA The columns of V2 form an orthonormal basis for the nullspace of A:
AANVR ker2
111 UUAA H The columns U1 of form an orthonormal basis for the column space of A:
ARUR 1
111 VVAAH The columns of V1 form an orthonormal basis for the column space of AH:
HARVR 1
HAR
AN
HAN
AR
xAy
11 yAx H
Adomxmx 1
11mxx
Acodomy nx 11
1nxy
Singular Values
19
SOLO Matrices
Lemma6 (continue – 5):
Proof of Lemma 6 (continue – 2):
Definition of the Singular Values HAR
AN
HAN
AR
xAy
11 yAx H
Adomxmx 1
11mxx
Acodomy nx 11
1nxy
From
riuAuAAA
v i
H
i
i
H
H
i
i ,,2,111
:
we have
r
r
H
r uuuAvvv
/100
0/10
00/1
2
1
2121
or 1
111
UAV H
11
1
1
2
11
1
111
2111
UUUAAVAUUAA
H
H
riuuuAAvA iiii
i
i
H
i
i ,,2,111 2
from which 1
111
VAU rivAu i
i
i ,,2,11
111 VAU H
Singular Values
20
SOLO Matrices
Lemma6 (continue – 6):
Proof of Lemma 6 (continue – 3):
Definition of the Singular Values HAR
AN
HAN
AR
xAy
11 yAx H
Adomxmx 1
11mxx
Acodomy nx 11
1nxy
Using and let compute AH A V 111 VAU H02 VA
rmxrnxrrn
rmrx
nxm
H
nxm
H
nxm
H
nxm
H
nxmH
H
mxmnxm
H
nxn
rxr
rmmxxrrnmxrxrrn
rmmxrxnmxrrxn
rmmxmxr
xnrn
rxn
VAUVAU
VAUVAUVVA
U
UVAU
00
01
2212
2111
21
2
1
From this equation we obtain:
r
i
H
iii
H
H
H
rmxrnxrrn
rmrx
nxm
vuVU
V
VUUA
rxmrxrnxr
xmrm
rxmrxr
rnnxnxr
1
111
2
11
21 00
0
Singular Values
Table of Contents
21
SOLO Matrices
Let perform the following transformations in the domain and codomain :
Geometric Interpretation of Singular Values
Suppose, first, that A is square and r = n = m, and consider the spherical hypersurfacein the domain of A for which:
1v 1x
2x
2v
1x
111 vAv
111 xAx
222 xAx
222 vAv
The Indicator Ellipsoid of a 2 x 2 Matrix11
11
nxnxnnx
mxmxmmx
Uy
Vx
Because y = A x: UVVUVAUy H
From which: 11 mxnxmnx
11
22
2
2
2
r
i
i
HHH VVxxx
From we have and the mapping of the sphericalhypersurface, in the codomain of A is the hypersurface of an ellipsoid:
11 mxnxmnx iii /
11
2
r
i i
i
This ellipsoid is called the indicator ellipsoid of A and the singular values arethe lengths of the principal axes of this ellipsoid.
Singular Values
22
SOLO Matrices
If the square matrix A is singular, i.e., r < n = m, the indicator ellipsoid shrinks to zero in the directions of the principal axes vi for which σi = 0. In this case:
Geometric Interpretation of Singular Values (continue – 1)
01 1
1
2
nr
r
i i
i
If the general case of nonsquare matrices with r < n ≠ m, if we choose the cylindrical hypersurface that has a circular hypersurface projection in R (AH):
01 1
1
2 mr
r
i
i
then its mapping will be the surface of the ellipsoid in R (A).
01 1
1
2
nr
r
i i
i
HAR
AN
HAN
AR
Singular Values
Table of Contents
23
SOLO Matrices
Properties of Singular Values
(1) The maximum singular value of Anxm is:
2
2
02121maxmaxmaxmax:
22 x
xAxAxAAA
xxxii
(2) The minimum singular value of Anxm is:
2
2
02121minminminmin:
22 x
xAxAxAAA
xxxii
Proof of (1) and (2)
Using x = V ζ we can write:
m
i
ii
HxV
HHHHH xVVxxAAxxAxAxA1
22222
2
11
22
2
m
i
i
HxV
HHH xVVxxxx
To obtain the maximum/minimum of that satisfy the condition we construct the Hamiltonian by adjoin the constraint to the extremum problem:
2
2xA 12
2x
1:,1
2
1
22m
i
i
m
i
iiH + for maximum- for minimum
Singular Values
24
SOLO Matrices
Properties of Singular ValuesProof of maximum/minimum singular value of Anxm (continue – 1)
The necessary conditions for extremum are:
1:,1
2
1
22m
i
i
m
i
iiH + for maximum- for minimum
miH
ii
i
,,2,102 2
Kuhn-TuckerCondition
miminimumfor
maximumforHi
i
,,2,10
02 2
2
2
Maximization problem solution: AH 22
1,max with
0
10,1
2
1
1
2
21
m
i
im
Kuhn-Tucker Condition
Minimization problem solution: AH m
22,min with
0
10,1
2
1
2
11
m
m
i
imm
Kuhn-Tucker Condition
Singular Values
25
SOLO Matrices
Properties of Singular ValuesProof of maximum/minimum singular value of Anxm (continue – 2)
For any x ≠ 0 we have:
0max
max
2
2
2
2
2
xx
xIAAxAA
xx
xAAxA
x
xAH
HHH
H
HH
The inequality holds because (AHA-I λmax [AHA]) is non-positive definite.
We can see that the equality is satisfied for x = eigenvector (AHA) that correspondsto λmax [AHA], therefore:
0max2
2
0
A
x
xAx
In the same way, or any x ≠ 0 we have:
0min
min
2
2
2
2
2
xx
xIAAxAA
xx
xAAxA
x
xAH
HHH
H
HH
We can see that the equality is satisfied for x = eigenvector (AHA) that correspondsto λmin [AHA], therefore:
0min2
2
0
A
x
xAx
Ax
xAx
2
2
0max
Ax
xAx
2
2
0min
Singular Values
26
SOLO Matrices
Properties of Singular Values
(3) is a norm of Anxm, because it satisfies the norm properties: A
(3.1) is non-negative and if and only if A = 0.
Proof of (3.1):
From Lemma 3:
A 0A 00 AA
(3.2) Multiplication by a complex constant α: AA
(3.3) Triangle Inequalities:
(3.4) Schwarz Inequality:
BABABA
BABA
0A
000 HVUAA
Proof of (3.2):
AAAAxx
2121
22
maxmax
Singular Values
27
SOLO Matrices
Properties of Singular Values
(3) is a norm of Anxm, because it satisfies the norm properties: A
(3.1) is non-negative and if and only if A = 0. A 0A 00 AA
(3.2) Multiplication by a complex constant α: AA
(3.3) Triangle Inequalities:
(3.4) Schwarz Inequality:
BABABA
BABA
Proof of (3.3): BAxBxA
xBxAxBABA
xx
xx
2121
22121
22
22
maxmax
maxmax
BBABBAA From which
BABA
In the same way BAAB BABA
Proof of (3.4):
BAx
xB
y
yA
x
xB
xB
xBA
x
xBABA
xy
xx
2
2
02
2
0
2
2
2
2
02
2
0
maxmax
maxmax
Hermann Amandus Schwarz
1843 - 1921
Singular Values
28
SOLO Matrices
Properties of Singular Values
(4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between the minimum and the maximum singular values:
niAAA i ,,2,1
Proof of (4):
We have: 02
2 xAx
xAA
If xi is any normalized eigenvector: A xi = λi xi, then
nix
x
x
x
x
xAi
i
ii
i
ii
i
i ,,2,12
2
2
2
2
2
Therefore: niAAA i ,,2,1
Singular Values
29
SOLO Matrices
Properties of Singular Values(5) A square matrix Anxn is singular iff its minimal singular value is zero.
0 ASingularA Proof of (5):
AAVdiagUA n
H
n 2121 ,,,
Therefore: 00det AAASingularA n
n
i
i
H
n VdiagUA11
21
1
det,,,detdetdet
(6) For a nonsingular square matrix Anxn we have
11
1&
1
A
AA
ArNonsingulaA
Proof of (6):
H
n
H
n UdiagVAVdiagUA /1,,/1,/1,,, 21
1
21
1
1
1
21 /1/10 AAAA nn
Hence: 11
1&
1
A
AA
A
Singular Values
30
SOLO Matrices
Properties of Singular Values
(7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true.
ABBASingularBA &
Proof of (7):
If (A+B) is singular, there exists a normalized eigenvector u (║u║2=1), s.t.:
22
0 uBuAuBuAuBA From this equation we obtain:
BxBuBuAxAAxx
212221
22
minmax
AxAuAuBxBBxx
212221
22
minmax
To prove that the opposite is not true, consider a counterexample:
15&3430
05
10
04
ABBABA
The right side is satisfied, but is nonsingular.
40
01BA
Singular Values
31
SOLO Matrices
Properties of Singular Values
ABBASingularBA &
Proof of (8):
(8) A sufficient condition that the square matrix (A+B) is nonsingular is:
We just proved:
ABorBAingularonsNBA
The proof follows directly from property (7). If (A+B) is singular then
ABBA & ; hence if
then (A+B) is nonsingular.
ABorBA
(7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true.
Singular Values
32
SOLO
To prove this we will consider the following three cases:- (A+B) singular, - (A+B) nonsingular but A and B are singular, - (A+B) nonsingular but A or B, or both are nonsingular.
Matrices
Properties of Singular Values
(9) The minimum singular value of a square matrix (A+B) satisfies the inequalities:
ABBABAABBA ,min,max
Proof of (9):
(9.1) - (A+B) singular
According to property (5) 0 BASince (A+B) is singular use property (7)
BABAAB 0
BAABBA 0
This completes the proof when (A+B) is singular.
Singular Values
33
SOLO
(9.2) - (A+B) nonsingular but A and B are singular,
Matrices
Properties of Singular Values
(9) The minimum singular value of a square matrix (A+B) satisfies the inequalities:
ABBABAABBA ,min,max
Proof of (9) (continue – 1):
22
0
uBuBAuBuBuAuBA
If A is singular, ,there exists a normalized eigenvector u (║u║2=1), s.t. A u=0: 0A
ABBxBuBuBAxBABAxx
212221
22
maxmin
and
BABABA
In the same way for (A+B) nonsingular and B singular:
ABBAAB
This completes the proof when (A+B) is nonsingular but A and B are singular.
Singular Values
34
SOLO
(9.3) - (A+B) nonsingular but A or B, or both are nonsingular.
Matrices
Properties of Singular Values
(9) The minimum singular value of a square matrix (A+B) satisfies the inequalities:
ABBABAABBA ,min,max
Proof of (9) (continue – 2):
BAC :Suppose that (A+B) and A are nonsingular, and define:
Pre-multiply by C-1 and post-multiply by A-1:1111 ABCCA
Let take any norm of this equation and write triangle an Schwarz inequalities:
BACABC
ABCCAABCC
1111
1111111
BACCABACC 1111111 B
ABAB
A
111
111
Using property (3), we can define , and because property (6) the previousequation is equivalent to:
** BABABA
If B is nonsingular in the same way we can prove that: ABBAAB
This completes the proof when (A+B) is nonsingular but A or B, or both are nonsingular.
Singular Values
35
SOLO
Using this and property (3.3):
Matrices
Properties of Singular Values
(10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B) can be approximated by A:
ABAABABABAIf &
Proof of (10):
We have: BBAA
ABAABABABAA
BABABA
Using: and property (9): BBAA
ABBABAABBA ,min,max
we have:
ABAABABABAA
Singular Values
36
SOLO Matrices
Properties of Singular Values
(11) Multiplicative Inequalities for square matrices:
BABABABABABA &
Proof of (11):
The proof is given in the following steps:
(11.1) is the Schwarz inequality of property (3) BABA
(11.2) prove that : BABA
If A or B is singular ( or is zero) then A B is singular ( det [A B] =
det [A].det [B]=0 and ) and we have equality.
A B 0BA
If A or B is nonsingular then A B is nonsingular ( det [A B] =
det [A].det [B]≠0 ) and: 11111111 ABABBAABBA
We define , and use the property (6): ** 11
1&
1
A
AA
A
to obtain:
BABABA
BA 111
111
This result is opposite to Schwarz inequality, proving that is not a norm. A
Singular Values
37
SOLO Matrices
Properties of Singular Values
(11) Multiplicative Inequalities for square matrices:
BABABABABABA &
Proof of (11) (continue – 1):
(11.3) prove that :
If A is singular then:
BABAorBA
BABA 0
If A is nonsingular then:
BABA
BAA
BAABAAB
1111
If B is singular then: BABA 0
If B is nonsingular then:
BABAB
BABBABBAA
11
Singular Values
38
SOLO Matrices
Properties of Singular Values
(11) Multiplicative Inequalities for square matrices:
BABABABABABA &
Proof of (11) (continue – 2):
(11.4) prove that :
If A or B are singular then A B is singular, and:
BABAorBA
BAorBABA 0If B is nonsingular then:
BABAB
BABBABBAA
11
We also have:
BABA
B
BABBABBABBAA
111
If A is nonsingular then:
BABAA
BABAABAAB
11
We also have:
BABA
A
BABAABAABAAB
111
q.e.d.
Singular Values
39
SOLO Matrices
Properties of Singular Values
(12) Any unitary matrix U (U UH = UH U = I) has all the singular values equal to 1.
Proof of (12):
iIUUU i
H
ii 12/12/1
(13) If U is a unitary matrix (U UH = UH U = I) then:
Proof of (13):
iAUAAU iii
iAAAAUUAAUAUAU i
H
i
HH
i
H
ii 2/12/12/1
iAAAAAUUUAAUUAUAUA i
H
i
HH
i
HH
i
H
ii 2/12/12/12/1
q.e.d.
Singular Values
q.e.d.
40
SOLO Matrices
Properties of Singular Values - Summary
(4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between the minimum and the maximum singular values:
niAAA i ,,2,1
(3) is a norm of Anxm, because it satisfies the norm properties: A
(3.1) is non-negative and if and only if A = 0. A 0A 00 AA
(3.2) Multiplication by a complex constant α: AA
(3.3) Triangle Inequalities:
(3.4) Schwarz Inequality:
BABABA
BABA
(1) The maximum singular value of Anxm is:
2
2
02121maxmaxmaxmax:
22 x
xAxAxAAA
xxxii
(2) The minimum singular value of Anxm is:
2
2
02121minminminmin:
22 x
xAxAxAAA
xxxii
Singular Values
41
SOLO Matrices
Properties of Singular Values – Summary (continue – 1)
(5) A square matrix Anxn is singular iff its minimal singular value is zero.
0 ASingularA
(6) For a nonsingular square matrix Anxn we have
11
1&
1
A
AA
ArNonsingulaA
(9) The minimum singular value of a square matrix (A+B) satisfies the inequalities:
ABBABAABBA ,min,max
(8) A sufficient condition that the square matrix (A+B) is nonsingular is:
ABorBAingularonsNBA
ABBASingularBA &
(7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true.
Singular Values
42
SOLO Matrices
Properties of Singular Values – Summary (continue – 2)
(12) Any unitary matrix U (U UH = UH U = I) has all the singular values equal to 1.
(13) If U is a unitary matrix (U UH = UH U = I) then: iAUAAU iii
(11) Multiplicative Inequalities for square matrices:
BABABABABABA &
(10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B) can be approximated by A:
ABAABABABAIf &
Singular Values
Table of Contents
43
SOLO Matrices
Householder Transformation
n̂
xnn T ˆˆ
xnn T ˆˆ
x
'x
O A
We want to compute the reflection ofover a plane defined by the normal 1ˆˆˆ nnn T
x
From the Figure we can see that:
xHxnnIxnnxx TT ˆˆ2ˆˆ2'
1ˆˆˆˆ2: nnnnIH TT
We can see that H is symmetric:
HnnInnIH TTTT ˆˆ2ˆˆ2
In fact H is also a rotation of around OA so it must be orthogonal, i.e.HTH=H HT=I.
x
InnnnnnInnInnIHHHH TTTTTT ˆˆˆˆ4ˆˆ4ˆˆ2ˆˆ21
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Table of Contents
Alston Scott Householder1904 - 1993
44
SOLO Matrices
The same result is obtained if we compute α that minimizes:
a
b
Pa
ab
p
We want to find such that pba
ap
Projection of a vector on a vector . b
a
or: baaaabapba TTTT 1
0
and: bPbaaaabaaaaap TTTT
11
TT aaaaP 1
: Projection Matrix
aababbabababd TTTT
22
2 2minminminmin
baaa
aad
aabad
TT
T
TT
1
min
2
22
2
0
022
Properties of Projection Matrix
(1) P is idempotent P2 = P
(2) P is symmetric PT = P
cbcPIcPcbP
,
Proof: cbcPIPbcPIbP TT
T ,0
0 PIPTHence: PPP TT PPPPP TTT 2PPPPP TT
b
bP
cP
c
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Table of Contents
45
SOLO
Note: If A and b were real, instead of H (transpose & complex conjugate) wehave only T (transpose).
Matrices
Given: Amxn of rank (Amxn) = r ≤ min (m,n) and1mxb
Find: such that is minimal1nxx
11 mxnxmxn bxAd
If the solution is not unique choose such that is minimal1nxx
1nxx
Solution:
The minimum is obtained when
0
0
2
22
2
11
2
AAx
d
bAxAAbxAAbxAx
d
H
HHH
mxnxmxn
bAAAx HH 1
A unique solution exists if AHA is positive definite, or rank (AHA) = n, or det|AHA| ≠ 0
1111
2
11
2
mxnxmxn
H
mxnxmxnmxnxmxn bxAbxAbxAd
Analytic:
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
111 min&min1
nxmxnxmxnxxbxAd
nx
46
SOLO Matrices
Given: Amxn of rank (Amxn) = r ≤ min (m,n) and1mxb
Find: such that is minimal1nxx
11 mxnxmxn bxAd
If the solution is not unique choose such that is minimal1nxx
1nxx
Solution (continue – 1):
Geometric:
We have A x R (A) for all x domain (A).
We want to find x0 domain (A), such that is normal to A x.
0xAbpb
AdomainxxAbpbxA
0
AdomainxxAAbAxxAbxA HHHH
000
Hence: 00 bAxAA HH
HAR
AN
HAN
ARxA
Adomx 0
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
Rx0
Nx0
0xAp
pb
xA
Adomx
0xA
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
111 min&min1
nxmxnxmxnxxbxAd
nx
47
SOLO Matrices
Let decompose as0x
HAR
AN HAN
ARxA
Adomx 0
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
Rx0
Nx0
0xAp
pb
xA
Adomx
0xA
NR
N
H
R
NR xxAofspaceNullANx
AofspaceRowARxxxx 00
0
0
000
Therefore:
RNR xAxAxAxAp 0
0
000
Hence if:
(a) N (A) = 0 or(b) The rows of A are linearly dependent or
(c) rank (A) = r < m
(d) AHA is singular there are a infinity of solutions NRNR xxxxx 00000
The norm of is:0x
NR
xx
NR xxxxxNR
00000
00
Hence: 0&min 000 NR xxx
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
111 min&min1
nxmxnxmxnxxbxAd
nx
48
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
TAR
AN
HAN
ARxA
Adomx 0
AcodomY
Null Space of AKer (A)
span by VA2T
Row Space of Aspan by VA1
T
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
Rx0
Nx0
0xAp
pb
xA
Adomx
0xA
bAx R
00&min 000 NR xxx
Define the Linear Transformation (Matrix),that gives from , as the Pseudoinverseof A. (A is the direct transformation that gives from :
Rx0
b
p
x
xAp
bAx †
R
0
A† is called Moore-Penrose Pseudoinverse Matrix, because was defined independently by E.H.Moore in 1920 and Roger Penrose in 1955.
Eliakim HastingsMoore
1862 - 1932
Roger Penrose1931 -
111 min&min1
nxmxnxmxnxxbxAd
nx
Table of Contents
49
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . a
HAR
AN HAN
ARxA
Adomx 0
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
Rx0
Nx0
0xAp
pb
xA
Adomx
0xA
bAx †
R
0
bAx †
R
0
Computation of Moore-Penrose Pseudoinverse Matrix, A †
Perform Singular Value Decomposition (S.V.D.) of Amxn:
where 0,,, 21211 rrA diagrxr
UAmxm and VAnxn are unitary matrices, i.e.:
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm V
VUUVUA
2
11
21 00
0
H
AA
H
AA
H
AAH
A
H
A
AA
rmxrm
rxr
AAH
A
H
A
A
H
A UUUUUUU
UUU
I
IUU
U
UUU
2211
2
1
2121
2
1
0
0
H
AA
H
AA
H
AAH
A
H
A
AA
rnxrn
rxr
AAH
A
H
A
A
H
A VVVVVVV
VVV
I
IVV
V
VVV
2211
2
1
2121
2
1
0
0
50
SOLO Matrices
HAR
AN HAN
ARxA
Adomx 0
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
Rx0
Nx0
0xAp
pb
xA
Adomx
0xA
bAx †
R
0
Since the norm is invariant to the product of orthogonal matrices
bUxVbxVUUbxVUbxA HHHHH
Introduce the new unknown: R
H
N
H
R
HH xVxVxVxVy
:
But 00
N
H
R
H
N
H
R
H
NR xVxVUxVxVUxAxAxA
RR
I
HH
RR
HH xxVVxxVxVy
2/1
bUybUybxA H
rnxrmxrrm
r
rnrx
y
H
yx
00
0
0
0
minminmin
1
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Computation of Moore-Penrose Pseudoinverse Matrix, A †
51
SOLO Matrices
HAR
AN HAN
ARxA
Adomx 0
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
Rx0
Nx0
0xAp
pb
xA
Adomx
0xA
bAx †
R
0
Therefore:
R
HH xVxVy
:
RR
I
HT
RR
HH xxVVxxVxVy
2/1
bUybUybxA H
rnxrmxrrm
r
rnrx
y
H
yx
00
0
0
0
minminmin
1
any
xrm
rxH
rmxrnxrrn
r
rmrx
XbUy1
1
1
0
00
/10
0
0/1
Rxyx 0minmin
R
HH
rmxrnxrrn
r
rmrx
xVbUy 0
1
00
/10
0
0/1
bAbUVx †H†
R
0
H†† UVA
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Computation of Moore-Penrose Pseudoinverse Matrix, A †
52
SOLO Matrices
HAR
AN HAN
ARxA
Adomx 0
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
Rx0
Nx0
0xAp
pb
xA
Adomx
0xA
bAx †
R
0
Where:
rmxrnxrrn
r
rmrx
†
nxm
00
0
0
0
: 1
1
1
H
mxm
†
nxmnxn
†
nxm UVA
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Computation of Moore-Penrose Pseudoinverse Matrix, A †
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxrUV
U
UVVA 1
1
11
2
11
1
2100
0:
1
1
1
1
0
0
:
r
1-
A rxr
Table of Contents
53
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Properties of Moore-Penrose Pseudoinverse Matrix, A †
rnxrnxrrn
rnrxrxr
rnxrmxrrm
r
rnrx
rmxrnxrrn
r
rmrx
mxn
†
nxm
I
00
0
00
0
0
0
00
0
0
01
1
1
1
rmxrmxrrm
rmrxrxr
rmxrnxrrn
r
rmrx
rnxrmxrrm
r
rnrx
†
nxmmxn
I
00
0
00
0
0
0
00
0
0
0
1
1
11
†
nxmmxn
rmxrmxrrm
rmrxrxr††
nxmmxn
I
00
0
mxn
†
nxm
rnxrnxrrn
rnrxrxr†
mxn
†
nxm
I
00
0
Using the definition of the Pseudoinverse we can see that
54
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Properties of Moore-Penrose Pseudoinverse Matrix, A †
†
nxm
rmxrnxrrn
r
rmrx
rmxrnxrrn
r
rmrx
rnxrnxrrn
rnrxrxr†
nxmmxn
†
nxm
I
00
0
0
0
00
0
0
0
00
01
1
1
1
1
1
mxn
rnxrmxrrm
r
rnrx
rnxrmxrrm
r
rnrx
rmxrmxrrm
rmrxrxr
mxn
†
nxmmxn
I
00
0
0
0
00
0
0
0
00
0
11
†
nxmmxn
H†H††Def†
†H††H†H††
nxmmxn AAUUUUUUUVVUAA
mxn
†
nxm
H†H††Def†
†H††HH††
mxn
†
nxm AAVVVVVVVUUVAA
Also:
Let check the same operations for Matrix A †
mxn
HH†HH†H
mxn
†
nxmmxn AVUVUVUUVVUAAA
†
nxm
H†H††H†HH††
nxmmxn
†
nxm AUVUVUVVUUVAAA
55
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Properties of Moore-Penrose Pseudoinverse Matrix, A † - Summary)
†††nxmmxnnxmmxn AAAA
mxn
†
nxm
†
mxn
†
nxm AAAA
mxnmxn
†
nxmmxn AAAA
†
nxm
†
nxmmxn
†
nxm AAAA
1
2
3
4
Table of Contents
56
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Description of Projections Related to Moore-Penrose Pseudoinverse
bPbAAxAp †
R
11
HAR
AN
HAN
AR
Rx
Adomx
Acodomb
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
bAx †
R
R
†
xA
bAAp
RxAp
P1 is a projection matrix because
11
1
2
1
PUUUVVUAAP
PAAAAAAPHH†HH†HH†H
†††
P1=A A† projects into column space of A, R (A)b
H†H†H† UUUVVUAAP :1
57
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Description of Projections Related to Moore-Penrose Pseudoinverse
2 HAR
AN
HAN
AR
Rx
Adomx
Acodomb
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
bAx †
R
R
†
xA
bAAp
RxAp
bAAIpb †
0 pbA†
pAx †
R
P2=(I - A A † ) is a projection matrix because
21112
2
2
2
PPIPIPIAAIP
PAAAAAAAAIAAIAAIP
HHH†H
†
A
†††††
Because , is theprojection of into .
HANAR pb
HANb
We can see, also, that:
H†H†
†
UIUUUI
AAIPIP
12 :
00
bbAAAAbAAIApbA
†A
†††††† pAbAx ††
R
bPbAAIpb †
2
58
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Description of Projections Related to Moore-Penrose Pseudoinverse
3 NR
N
H
R
NR xxAofspaceNullANx
AofspaceRowARxxxx
xPxAAxAApAbAx ††††
R
3
P3=A †A is a projection matrix of in R (AH)x
33
3
2
3
PVVVUUVAAP
PAAAAAAAAAAP
HH†HHH†H†H
†
A
††††
†
4 xPxAAIxAAxxxx ††
RN
4
P4=I-A †A is a projection matrix of in N (A)x
43333
4
2
33
2
3
2
4
3
2
PPIPIPIP
PPPIPIP
HHHH
P
HAR
AN
HAN
AR
Adomx
Acodomb
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
bAx †
R
R
†
xA
bAAp
RxAp
bAAIpb †
0 pbA†
pAx †
R
xAAx †
R
HAR
AN
HAN
AR
Adomx
Acodomb
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
bAx †
R
R
†
xA
bAAp
RxAp
bAAIpb †
0 pbA†
pAx †
R
xAAx †
R
xAAIx †
N
0NxA
59
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Description of Projections Related to Moore-Penrose Pseudoinverse (Summary)
3 H††††
R ARxPxAAxAApAbAx
3
4
HAR
AN
HAN
AR
Adomx
Acodomb
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
b
bAx †
R
R
†
xA
bAAp
RxAp
bAAIpb †
0 pbA†
pAx †
R
xAAx †
R
xAAIx †
N
0NxA
ANxPxAAIxAAxxxx ††
RN
4
H† ANbPbAAIpb
22
ARbPbAAxAp †
R
11
H†† UUAAP :1
†AAIPIP 12 :H†† VVAAP :3
AAIPIP † 34 :
Table of Contents
60
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Particular case (1) r = n ≤ m:
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxmVUUVUA
0
1
21
(a) rank (Amxn) = n or
(b) columns of Amxn are linear independent or
(c) N (Amxn) = 0 or
(d) AnxmHAmxn is nonsingular
This is equivalent to:
where 0,,, 21211 nnA diagnxn
H
AA
H
AA
H
AAH
A
H
A
AA
nmxnm
nxn
AAH
A
H
A
A
H
A UUUUUUU
UUU
I
IUU
U
UUU
2211
2
1
2121
2
1
0
0
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxnUV
U
UVA 1
1
1
2
11
1 0:
HAR
0AN HAN
AR
x
b
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
bAAxAp †
pAx †
bAAIpb
xAp
bAx †
0 pbA†
61
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Particular case (1) r = n ≤ n: (continue – 1)
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxnUV
U
UVA 1
1
1
2
11
1 0:
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxmVUUVUA
0
1
21
H
AAA
H
A
A
AAH
A
H
A
AA
H VVVUUU
UVAA 2
1
1
21
2
1
1 00
H
AAA
H VVAA 2
1
1
†H
AAA
H
AAA
H
AAA
HH AUVUVVVAAA
1
1
111
2
1
1
or H
nxmmxn
H
nxm
†
nxm AAAA1
HAR
0AN HAN
AR
x
b
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
bAAxAp †
pAx †
bAAIpb
xAp
bAx †
0 pbA†
We have only one solution that minimize
11 mxnxmxn bxAd
x
and is given by:
1
1
11 mx
H
nxmmxn
H
nxmmx
†
nxmnx bAAAbAx Table of Contents
62
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Particular case (2) r = m ≤ n:
(a) rank (Amxn) = m or
(b) rows of Amxn are linear independent or
(c) N (AnxmH) = 0 or
(d) AmxnAnxmH is nonsingular
This is equivalent to:
where 0,,, 21211 mmA diagmxm
H
AAA
H
A
xmmn
A
AnA
†
nxm mxmmxmnxmmxm
mxm
mnnxmxrUVUVVA 1
11
1
1
210
:
H
AAAH
A
H
A
mnmxAA
H
AAAmxn mxnmxmmxm
xnmn
mxn
mxmmxmnxnmxnmxmVU
V
VUVUA 11
2
1
1 0
H
AA
H
AA
H
AAH
A
H
A
AA
mnxmn
mxm
AAH
A
H
A
A
H
A VVVVVVV
VVV
I
IVV
V
VVV
2211
2
1
2121
2
1
0
0
HAR
AN
ARBAN H &0
AR
xAb
N
†
nxn xxAAI
R
† xxAA
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H Column Space of Aspan by UA1
bAx †
R
RxAb
b
0NxA
63
SOLO Matrices
Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .
Particular case (2) r = m ≤ n: (continue – 1)
H
AAA
H
A
xmmn
A
AnA
†
nxm mxmmxmnxmmxm
mxm
mnnxmxrUVUVVA 1
11
1
1
210
:
H
AAAH
A
H
A
mnmxAA
H
AAAmxn mxnmxmmxm
xnmn
mxn
mxmmxmnxnmxnmxmVU
V
VUVUA 11
2
1
1 0
H
AAA
H
AA
I
A
H
AAA
H UUUVVUAAm
2
11111 H
AAA
H UUAA 2
1
1
†H
AAA
H
AAA
H
AAA
HH AUVUUUVAAA 1
11
2
111
1
or 1 HH† AAAA HAR
AN
ARBAN H &0
AR
xAb
N
†
nxn xxAAI
R
† xxAA
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H Column Space of Aspan by UA1
bAx †
R
RxAb
b
0NxA
We have an infinite number of solutions that minimize
11 mxnxmxn bxAd
bAAAbAx HH†
R
1
The solution that minimizesthe norm is given by:
Rx
Rx
Table of Contents
64
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
X - nxp unknowns with mxp equations
mxpnxpmxn BXA
Perform Singular Value Decomposition (S.V.D.) of Amxn:
where 0,,, 21211 rrA diagrxr
UAmxm and VAnxn are unitary matrices, i.e.:
HAR
AN
HAN
ARBXA
11 yAx H
AdomX
AcodomY
1nxy
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
B
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm V
VUUVUA
2
11
21 00
0
H
AA
H
AA
H
AAH
A
H
A
AA
rmxrm
rxr
AAH
A
H
A
A
H
A UUUUUUU
UUU
I
IUU
U
UUU
2211
2
1
2121
2
1
0
0
H
AA
H
AA
H
AAH
A
H
A
AA
rnxrn
rxr
AAH
A
H
A
A
H
A VVVVVVV
VVV
I
IVV
V
VVV
2211
2
1
2121
2
1
0
0
65
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
Let multiply byand using:
mxpnxpmxn BXA
H
H
U
U
2
1
we obtain:
BU
BUX
V
VUU
U
UH
A
H
A
H
A
H
AA
I
AAH
A
H
A
m
2
1
2
11
21
2
1
00
0
or:
BU
BUX
V
VH
A
H
A
H
A
H
AA
2
1
2
11
00
0
or:
xprmmxp
H
A BUxmrm
02
(m-r)xp - constraints equivalent to condition Bmxp (Amxn)
mxp
H
Anxp
H
AA BUXVrxmrxnrxr 111
rxp - independent equationsnxp – unknownssince r ≤ n → Eq. ≤ Unknown
HAR
AN
HAN
ARBXA
11 yAx H
AdomX
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
B
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm V
VUUVUA
2
11
21 00
0
66
xprmmxp
H
A BUxmrm
02
(m-r)xp - constraints equivalent to condition Bmxp (Amxn)
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
mxp
H
Anxp
H
AA BUXVrxmrxnrxr 111
rxp - independent equationsnxp – unknownssince r ≤ n → Eq. ≤ Unknown
This equation is a Necessary and Sufficient Condition for any solutions of equationAmxn Xnxp = Bmxp. It is equivalent to Bmxp (Amxn) or Bmxp N (AT) = . If this condition is fulfilled, then from we havenxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom.
mxp
H
Anxp
H
AA BUXVrxmrxnrxr 111
mxp
H
AAnxp
H
A BUXVrxmrxrrxn 1
1
11
Since VA1
T VA1=Ir & VA1T VA2 = 0 the
General Solution of Amxn Xnxp = Bmxp is:
AN
xprnA
AR
mxp
H
AAAnxp YVBUVXrnnx
T
rxmrxrnxr
21
1
11
where Y(n-r)xp is any (n-r)xp matrix, i.e. weused all (n-r)xp degrees of freedom.
HAR
AN
HAN
ARBXA AdomX
AcodomY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
BB has to be in thecolumn space of A
ANBorARB
67
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
Check:
mxpmxp
xmrm
H
A
H
A
AAmxpxmrm
H
A
mxp
H
A
AA
xprm
mxp
H
A
AA
xprn
mxp
H
AA
rnxrmxrrm
rnrxA
AA
xprn
I
A
H
Amxp
H
AAA
H
A
xprnA
H
Amxp
H
AA
I
A
H
A
rnxrmxrrm
rnrxA
AA
xprnAmxp
H
AAAH
A
H
A
rnxrmxrrm
rnrxA
AAnxp
H
AAAnxpmxn
BBU
UUUBU
BUUU
BUUU
Y
BUUU
YVVBUVV
YVVBUVVUU
YVBUVV
VUUXVUXA
rxm
rmmxmxr
rxm
rmmxmxr
rxm
rmmxmxr
rxmrxrrxr
rmmxmxr
rnnxxnrnrxmrxrnxrxnrn
rnnxrxnrxmrxrnxrrxnrxr
rmmxmxr
rnnxrxmrxrnxr
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
2
1
21
0
2
1
21
1
21
1
1
11
21
221
1
1
0
12
0
211
1
1111
21
21
1
11
2
11
21
000
0
00
0
00
0
68
SOLO Matrices
where r is such that:
General Solution of Amxn Xnxp = Bmxp
Algorithm to solve Amxn Xnxp = Bmxp:
(1) Compute s.v.d. of Amxn and partition according to:
0,,, 21211 rrA diagrxr
(2) Check if: xprmmxp
H
A BUxmrm
02
(3) If (2) is not true → no solution for (1)
any
xprnAmxp
H
AAAnxp YVBUVXrnnxrxmrxrnxr
21
1
11
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm V
VUUVUA
2
11
21 00
0
If (2) is true → (n-r)xp solutions:
69
SOLO Matrices General Solution of Amxn Xnxp = Bmxp
Moore-Penrose Pseudoinverse of A:
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxrUV
U
UVVA 1
1
11
2
11
1
2100
0:
then
H
AAH
A
H
A
rmxrnxrrn
rmrxA
I
AAH
A
H
A
rnxrmxrrm
rnrxA
AA
†
mxn rxnnxr
xmrm
rxmrxr
nxn
rnnxnxr
xnrn
rxnrxr
rmmxmxrnxmUU
U
UVV
V
VUUAA 11
2
11
1
21
2
11
2100
0
00
0
H
AA
H
AA
H
AA
H
AA
†
nxmmxnmxm xmrmrmmxrxmmxrxmrmrmmxrxmmxrUUUUUUUUAAI
22112211
also
H
AAH
A
H
A
rnxrmxrrm
rnrxA
I
AAH
A
H
A
rmxrnxrrn
rmrxA
AAnxm
†
mxn rxnnxr
xnrn
rxnrxr
mxm
rmmxmxr
xmrm
rxmrxr
rnnxnxrVV
V
VUU
U
UVVAA 11
2
11
21
2
11
1
21 00
0
00
0:
H
AA
H
AA
H
AA
H
AAmxn
†
nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxrVVVVVVVVAAI
22112211
70
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
Moore-Penrose Pseudoinverse of A (continue - ):
Define also Znxp such that: nxp
H
Axprn ZVYxrrn
2:
any
xprnAmxp
H
AAAnxp YVBUVXrnnxrxmrxrnxr
21
1
11
Since, if xprmmxp
H
A BUxmrm
02
The solution of Amxn Xnxp = Bmxp is
H
AA
H
AA
H
AA
H
AAmxn
†
nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxrVVVVVVVVAAI
22112211
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxrUV
U
UVVA 1
1
11
2
11
1
2100
0:
Therefore: any
nxpmxn
†
nxmnxnmxp
†
nxmnxp ZAAIBAX
Note: By writing the solution this way we lose the fact that we have only (n-r)xp different solutions as we have seen.
Check:
xprn
xnrnrnnxrxmrxrnxr
Y
nxp
H
AAmxp
H
AAAnxp ZVVBUVX
221
1
11
71
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
Moore-Penrose Pseudoinverse of A (continue - ):
any
nxpmxn
†
nxmnxnmxp
†
nxm
any
xprnAmxp
H
AAAnxp ZAAIBAYVBUVXrnnxrxmrxrnxr
21
1
11
xprmmxp
H
AA
ANonBofprojection
mxp
H
BUU
BAAI
orANB
orARB
xmrmrmmx
T
0
0
22
Solutions exists iff:
HAR
AN HAN
AR
BXA
YVZAAI A
†
nxn 2
BA†
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
B
BA†Z
i.e. the projection (Imxm – Amxn Anxm †) of B on N (AH) is zero.
mxpmxp
H
AAmxp
†
nxmmxnmxm BUUBAAImnrmrmmx
0
0
22
Table of Contents
72
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
Particular case (1) r = m ≤ n:
solutions always exist
HAR
AN
ARBAN H &0
AR
BXA
YVZAAI A
†
nxn 2
BA†
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H Column Space of Aspan by UA1
B
BA†Z
Since
H
A
H
A
mnmxAA
H
AAAmxn
xnmn
rxn
mxmmxmnxnmxnmxm V
VUVUA
2
1
1 0
xprmmxp
H
ArmmxA BUUxmrmrmmx
00 22
nxp unknowns ≥ mxp equations, meaning (n-m)xp degrees of freedom
any
nxpmxn
†
nxmnxnmxp
†
nxm
any
xprnAmxp
H
AAAnxp
ZAAIBA
YVBUVXrnnxrxmrxrnxr
21
1
11
11
11
1
1
210
:
H
nxmmxn
H
nxm
H
AAA
H
A
xmmn
A
AnA
†
nxm AAAUVUVVAmxmrxrnxrmxm
mxm
mnnxnxm
mxm
H
A
xmmn
A
AAH
A
H
A
mnmxAA
†
nxmmxn IUVVV
VUAA
mxm
mxm
mnnxnxm
xnmn
mxn
mxmmxm
0
01
1
21
2
1
Table of Contents
73
SOLO Matrices
General Solution of Amxn Xnxp = Bmxp
Particular case (2) r = n ≤ n: mxp equations ≥ nxp unknowns, meaning (n-m)xp constraints
Only if solutions exist. xprmmxp
H
A BUxmrm
02
In this case we have nxp unknowns and mxp equations - (m-p)xp constraints = nxp independent equations, i.e. a unique solution:
mxpnxmmxp
H
AAAnxp BABUVXnxmnxnnxn
1
1
11
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxmVUUVUA
0
1
21
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxnUV
U
UVA 1
1
2
11 0:
nxn
H
A
xnnm
A
AAH
A
H
A
nmnxAAmxn
†
nxm IVUUU
UVAA
nxn
nxn
nmmxmxn
xmnm
nxm
nxnnxn
0
01
21
2
11
HAR
0AN HAN
ARBXA
BA†1nxy
Row Space of Aspan by VA1
HColumn Space of A
span by UA1
Left Null Space of Aspan by UA2
BB has to be in thecolumn space of A
ANBorARB
Table of Contents
74
SOLO Matrices
General Solution of YpxmAmxn = Cpxn
Y - pxm unknowns with pxn equations
pxnmxnpxm CAY
Perform Singular Value Decomposition (S.V.D.) of Amxn:
where 0,,, 21211 rrA diagrxr
UAmxm and VAnxn are unitary matrices, i.e.:
H
AA
H
AA
H
AAH
A
H
A
AA
rmxrm
rxr
AAH
A
H
A
A
H
A UUUUUUU
UUU
I
IUU
U
UUU
2211
2
1
2121
2
1
0
0
H
AA
H
AA
H
AAH
A
H
A
AA
rnxrn
rxr
AAH
A
H
A
A
H
A VVVVVVV
VVV
I
IVV
V
VVV
2211
2
1
2121
2
1
0
0
HAR
AN HAN
ARCAY
11 yAx H
CY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm V
VUUVUA
2
11
21 00
0
75
SOLO Matrices
we obtain: 2121
2
11
21 00
0AAAAH
A
H
AA
AA VCVCVVV
VUUY
or:
rnpxApxn rnnxVC
02
px(n-r) - constraints equivalent to condition Cpxn (Amxn
H)
nxrrxrmxr ApxnAApxm VCUY 111 pxr - independent equationspxm – unknownssince r ≤ m → Eq. ≤ Unknown
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm V
VUUVUA
2
11
21 00
0
General Solution of YpxmAmxn = Cpxn HAR
AN HAN
ARCAY
CY
Null Space of AKer (A)
span by VA2T
Row Space of Aspan by VA1
T
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
C has to be in theRow Space of A
HAC
ANC
orARC HLet post-multiply byand using:
21 VVpxnmxnpxm CAY
or: 21
1
21 00
0AA
A
AA VCVCUUY
76
SOLO Matrices
Since UA1T UA1=Ir & UA1
T UA2 = 0 the
General Solution of YpxmAmxn = Cpxn is:
where Xpx(m-r) is any px(m-r) matrix, i.e. weused all px(m-r) degrees of freedom.
General Solution of YpxmAmxn = Cpxn
nxrrxrmxr ApxnAApxm VCUY 111 pxr - independent equationspxm – unknownssince r ≤ m → Eq. ≤ Unknown
This equation is a Necessary and Sufficient Condition for any solutions of equationYpxmAmxn = Cpxn. It is equivalent to Cpxn (Amxn) or Cpxn N (AT) = . If this condition is fulfilled, then from we havenxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom.
H
ApxnAApxm nxrrxrmxrVCUY 111
1
111
rxrnxrmxr AApxnApxm VCUY
px(n-r) - constraints equivalent to condition Cpxn (Amxn
H) rnpxApxn rnnxVC
02
H
A
any
rmpx
H
AAApxnpxm xmrmrxmrxrnxrUXUVCY
21
1
11
HAR
AN HAN
ARCAY
CY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
C has to be in theRow Space of A
HAC
ANC
orARC H
HAC
77
SOLO Matrices
Check:
pxnH
A
H
A
AApxnH
A
H
A
ApxnApxn
H
A
H
A
rnpxApxnH
A
H
A
rnxrmxrrm
rnrxA
rmpxAApxn
H
A
H
A
rnxrmxrrm
rnrxA
I
A
H
ArmpxA
H
AAApxnA
H
Armpx
I
A
H
AAApxn
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
Armpx
H
AAApxnmxnpxm
CV
VVVC
V
VVCVC
V
VVC
V
VXVC
V
V
UUXUUVCUUXUUVC
V
VUUUXUVCAY
xnrn
rxn
rnpxnxr
xnrn
rxn
rnpx
rnpxnxr
xnrn
rxn
nxr
xnrn
rxnrxr
rxrnxr
xnrn
rxnrxr
rmmxxmrmrmmxrxmrxrnxrmxrxmrmmxrrxmrxrnxr
xnrn
rxnrxr
rmmxmxrxmrmrxmrxrnxr
2
1
21
2
1
0
21
2
1
1
2
111
11
2
11
22
0
21
1
11
0
1211
1
11
2
11
2121
1
11
000
0
00
0
00
0
General Solution of YpxmAmxn = Cpxn
78
SOLO Matrices
where r is such that:
Algorithm to solve YpxmAmxn = Cpxn:
(1) Compute s.v.d. of Amxn and partition according to:
0,,, 21211 rrA diagrxr
(2) Check if:
(3) If (2) is not true → no solution for (1)
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm V
VUUVUA
2
11
21 00
0
General Solution of YpxmAmxn = Cpxn
rnpxApxn rnnxVC
02
H
A
any
rmpx
H
AAApxnpxm xmrmrxmrxrnxrUXUVCY
21
1
11
If (2) is true → px(m-r) solutions:
79
SOLO Matrices
Moore-Penrose Pseudoinverse of A (continue - ):
Define also Wpxm such that: rmmxApxmrmpx UWX
2:
H
AA
H
AA
H
AA
H
AAmxn
†
nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxrVVVVVVVVAAI
22112211
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxrUV
U
UVVA 1
1
11
2
11
1
2100
0:
Therefore: †
nxmmxnmxm
any
pxm
†
nxmpxnpxm AAIWACY
Note: By writing the solution this way we lose the fact that we have only px(m-r) different solutions as we have seen.
If rnpxApxn rnnx
VC
02
General Solution of YpxmAmxn = Cpxn
the solution of YpxmAmxn = Cpxn is
H
A
any
rmpx
H
AAApxnpxm xmrmrxmrxrnxrUXUVCY
21
1
11
Check:
H
A
X
Apxm
H
AAApxnpxm xmrm
rmpx
rmmxrxmrxrnxrUUWUVCY
221
1
11
80
SOLO Matrices
Moore-Penrose Pseudoinverse of A (continue - ):
Solutions exists iff:i.e. the projection (Inxn – Anxm
† Amxn) of C on N (A) is zero.
General Solution of YpxmAmxn = Cpxn
†
nxmmxnmxm
any
pxm
†
nxmpxn
H
A
any
rmpx
H
AAApxnpxm
AAIWAC
UXUVCYxmrmrxmrxrnxr
21
1
11
pxn
H
AApxnmxn
†
nxmnxnpxn xnrn
rnpx
rnnxVVCAAIC 02
0
2
HAR
AN HAN
ARCAY
CpxmY
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
C has to be in theRow Space of A
HAC orANC
orARC H
pxn
H
AApxn
ANonprojection
mxnnxmnxnpxn xnrn
rnpx
rnnxVVCAAIC 02
0
2
H
Armpx
ANonWanyofprojection
nxmmxnmxmpxm xrrm
Hpxm
UXAAIW
2
pxmW
HAC
pxn
H
AApxn
ANonprojection
mxn
†
nxmnxnpxn
H
xnrn
rnpx
rnnxVVC
AAIC
orANC
orARC
02
0
2
†††
Table of Contents
81
SOLO Matrices
Particular case (1) r = m ≤ n:
H
A
H
A
mnmxAA
H
AAAmxn
xnmn
rxn
mxmmxmnxnmxnmxm V
VUVUA
2
1
1 0
11
11
1
1
210
:
H
nxmmxn
H
nxm
H
AAA
H
A
xmmn
A
AnA
†
nxm AAAUVUVVAmxmrxrnxrmxm
mxm
mnnxnxm
mxm
H
A
xmmn
A
AAH
A
H
A
mnmxAA
†
nxmmxn IUVVV
VUAA
mxm
mxm
mnnxnxm
xnmn
mxn
mxmmxm
0
01
1
21
2
1
General Solution of YpxmAmxn = Cpxn
Only if solutions exist. rnpxApxn rnnx
VC
02
In this case we have pxm unknowns and pxn equations – px(n-m) constraints = pxm independent equations, i.e. a unique solution:
†
nxmpxn
H
AAApxnpxm
AC
UVCYrxmrxrnxr
1
1
11
HAR
AN 0HAN
ARCAY
C
Null Space of AKer (A)
span by VA2H
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
C has to be in theRow Space of A
HAC
pxn
H
AApxnmxn
†
nxmnxnpxn
H
xnrn
rnpx
rnnxVVCAAIC
orANC
orARC
02
0
2
Table of Contents
82
SOLO Matrices
Particular case (2) r = n ≤ n: pxn equations ≥ pxm unknowns, meaning px(m-n) constraints
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxmVUUVUA
0
1
21
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxnUV
U
UVA 1
1
2
11 0:
nxn
H
A
xnnm
A
AAH
A
H
A
nmnxAAmxn
†
nxm IVUUU
UVAA
nxn
nxn
nmmxmxn
xmnm
nxm
nxnnxn
0
01
21
2
11
Since solutions always exist rnpxApxn rnnx
VC
02
pxm unknowns ≥ pxn equations, meaning px(m-n) degrees of freedom
HAR
0AN HAN
ARCAY
CY
Row Space of Aspan by VA1
H
Column Space of Aspan by UA1
Left Null Space of Aspan by UA2
C has to be in theRow Space of A
HAC
CAAC †
H
AUXAAIW 2
pxn
H
AApxnmxn
†
nxmnxnpxn
H
xnrn
rnpx
rnnxVVCAAIC
orANC
orARC
02
0
2
†
nxmmxnmxm
any
pxm
†
nxmpxn
H
A
any
rmpx
H
AAApxnpxm
AAIWAC
UXUVCYxmrmrxmrxrnxr
21
1
11
General Solution of YpxmAmxn = Cpxn
Table of Contents
83
SOLO Matrices Inverse of Partitioned Matrices
11111
111111
mnnnnmmmnmmmmnnnnmmm
mnnnnmmmmnnnnmmmmnnn
mmnm
mnnn
BADCDCBADC
BADCBADCBA
CD
BA
if and exist. 1nnA 1
nnC
Let find the inverse of such that:
mmnm
mnnn
PN
ML
mmnm
mnnn
CD
BA
mmnm
mnnn
mmnm
mnnn
mmnm
mnnn
I
I
PN
ML
CD
BA
0
0
Proof:
nnnmmnnnnn INBLA 1 nnnnnmmmmnnn ILDCBA
1
2 nmnmmmnnnm NCLD 0nnnmmmnm LDCN
1
3mnmmmnmnnn PBMA 0
mmmnnnmn PBAM
1
4mmmmmmmnnm IPCMD mmmmmnnnnmmm IPBADC
1
84
SOLO Matrices Inverse of Partitioned Matrices
11111
111111
mnnnnmmmnmmmmnnnnmmm
mnnnnmmmmnnnnmmmmnnn
mmnm
mnnn
BADCDCBADC
BADCBADCBA
CD
BA
if and exist. 1nnA 1
nnC
Let find the inverse of such that:
mmnm
mnnn
PN
ML
mmnm
mnnn
CD
BA
mmnm
mnnn
mmnm
mnnn
mmnm
mnnn
I
I
PN
ML
CD
BA
0
0
Proof (continue – 1):
1 nnnnnmmmmnnn ILDCBA
1
2 nnnmmmnm LDCN
1
3
4
mmmnnnmn PBAM
1
mmmmmnnnnmmm IPBADC
1
11
nmmmmnnnnn DCBAL
111
nmmmmnnnnmmmnm DCBADCN
11
mnnnnmmmmm BADCP
111
mnnnnmmmmnnnmn BADCBAM
q.e.d.
85
SOLO Matrices Inverse of Partitioned Matrices
mmnm
mnnn
mmnm
mnnn
mmnm
mnnn
I
I
CD
BA
PN
ML
0
0From:
1
mmmnnmmmmm CBNIPmmmmmmmnnm ICPBN we get:
11
mnnnnmmmmm BADCPSubstitute:
and: 111
nmmmmnnnnmmmnm DCBADCN
1111111
mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC
to obtain:
By inter-changing , in this identity, we obtain:nmmnnnmm DBAC ,
1111111
nnnmmnnnnmmmmnnnnnnmmmmnnn ADBADCBAADCBA
86
SOLO
Pre-multiplying this identity by we obtain
Matrices Inverse of Partitioned Matrices
Let prove the identity:
111111
mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA
1111
mnnnnmmmmnnnnmmmmnnn BADCBADCBA
Proof:
1111
mnnnnmmmmnnnnmmmmnmn BADCBADCBB
11111
mmmnmnnnnmmmmnnnnmmmmmmn CBBADCBADCCB
11
nmmmmnnn DCBA
111
111111
mmmnnmmmmnnn
mnnnnmmmmnnnnmmmmnnnnmmmmnnn
CBDCBA
BADCBADCBADCBA
q.e.d.
111111
nmmmmnnnnmmmnnnmmnnnnmmm DCBADCADBADC
By inter-changing ,in the first identity, we obtain:nmmnnnmm DBAC ,
111111
nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC
q.e.d.
87
SOLO Matrices Inverse of Partitioned Matrices
By using the identities:
111111
nmmmmnnnnmmmnnnmmnnnnmmm DCBADCADBADC
We obtain:
11111
111111
mnnnnmmmnmmmmnnnnmmm
mnnnnmmmmnnnnmmmmnnn
mmnm
mnnn
BADCDCBADC
BADCBADCBA
CD
BA
if and exist. 1nnA 1
nnC
11111
111111111
mnnnnmmmnnnmmnnnnmmm
mnnnnmmmmnnnnnnmmnnnnmmmmnnnnn
mmnm
mnnn
BADCADBADC
BADCBAADBADCBAA
CD
BA
1111111
nnnmmnnnnmmmmnnnnnnmmmmnnn ADBADCBAADCBA
88
SOLO Matrices Inverse of Partitioned Matrices
If and exist, performing the computation M-1M, we can prove: 1nnA 1
nnC
111
1100
mmnnnmmm
mnnn
mmnm
mnnn
CADC
A
CD
A1
2
1
1111
00mmnm
mmmnnnnn
mmnm
mnnn
C
CBAA
C
BA
3
1
11
0
0
0
0
mmnm
mnnn
mmnm
mnnn
C
A
C
A
4 If and : T
nnnn AA T
nnnn CC
11111
111111
mnnnnmT
mmnmT
mmmnnnnmT
mm
mnnnnmT
mmmnnnnmT
mmmnnn
mmnmT
mnnn
BABCBCBABC
BABCBABCBA
CB
BA
Because this is a symmetric matrix
111111
nnnm
Tmnnnnm
Tmmnm
Tmmmnnnnm
Tmm ABBABCBCBABC
Also: 1111111
mmmnnmT
mmmnnnnmT
mmmmmnnnnmT
mm CBBCBABCCBABC
89
SOLO Matrices Inverse of Partitioned Matrices
If m=n and and also exist: 1
nnB1
nnD5
111111
nnnnnnnnnnnnnnnnnnnnnnnn ABBADCBADCBA
111111
nnnnnnnnnnnnnnnnnnnnnnnn CDDCBADCBADC
also
we obtain
1111
11111
nnnnnnnnnnnnnnnn
nnnnnnnnnnnnnnnn
nnnn
nnnn
BADCCDAB
ABCDDCBA
CD
BA
Table of Contents
90
SOLO Matrices
Matrix Inverse Lemmas Identities
1111111
mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC1
Proof:
1111111
mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC
In the identity:
substitute by .1
nnA nnA
Substitute by and by in (1).1
nnAnnA mmC
1
mmC2
mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC
1111
Substitute in (1) nnnn IA 3
111111
mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC
Substitute in (2) nnnn IA 4
mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC
111
91
SOLO Matrices
Matrix Inverse Lemmas Identities
5Substitute in (1), (2), (3), (4) by . (We don’t assume symmetric and )nmD nm
TB nnA mmC
1111111
mmmnnm
Tmmmnnnnm
Tmmmmmnnnnm
Tmm CBBCBABCCBABC
mmmnnmT
mmmnnnnmT
mmmmmnnnnmT
mm CBBCBABCCBABC
1111
111111
mmmnnm
Tmmmnnnnm
Tmmmmmnnm
Tmm CBBCBIBCCBBC
mmmnnmT
mmmnnnnmT
mmmmmnnmT
mm CBBCBIBCCBBC
111
From this we get:
6Substitute in (3) mmmm IC
mnnmmnnnnmmmmnnmmm BDBIDIBDI
11
mnnmmnnmmmmnnmmmmnnmmmmnnmmm
mnnmmmmnnmmnnmmmmnnmmmmnnmmm
mnnmmmmmmnnmmnnnnm
BDBDIBDIBDIBDI
BDIBDBDIBDIBDI
BDIIBDBID
111
111
11
92
SOLO Matrices
Matrix Inverse Lemmas Identities
7Substitute in the identity
111111
mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA
nnnn IA and mmmm IC to obtain:
mnnmmnnnmnnmmmmn BDBIBDIB
11
Pre-multiplying this by we get (6).nmD
By using a similar path with the identity
111111
nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC
nnnn IA and mmmm IC to obtain:with
nmmnnmmmnmmnnnnm DBDIDBID
11
Post-multiplying this by we get (6).mnB
93
SOLO Matrices
By matrix manipulation we obtain:
Matrix Inverse Lemmas Identities
8In the identity
111111
mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBApre-multiplying Anxn by and post-multiplying by Cmxm we get:
mnnmmmmnnnnnmmmnnnnmmmmn BDCBAACBADCB
1111
mnnnnmmmmnnnmnnnnmmmmmmn BADCBIBADCIB
111111
Use now the identity
111111
nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC
Pre-multiplying by Cmxm and post-multiplying by Anxn we get:
nmmnnnnmmmmmnnnmmmmnnnnm DBADCCADCBAD
1111
By matrix manipulation we obtain
nmmmmnnnnmmmnmmmmnnnnnnm DCBADIDCBAID
111111
Table of Contents
94
SOLO Matrices
Matrix Schwarz Inequality
QPPPQPQQ TTTTT 1
Table of Contents
Hermann Amandus Schwarz
1843 - 1921yxyx ,
Let x, y be the elements of an Inner Product space X, than :
This is the Schwarz Inequality.
Let Pmxn and Qmxl be two matrices such that PTP is nonsingular, then:
CxxQPPPQPxxQQx TTTTTTT 1i.e.,:
Furthermore equality holds if and only if exists a matrix Snxl such that Q = P S.
Proof:
Start from the inequality: and choose 0 SPQSPQ T QPPPS TT 1
01
1111
QPPPQPQQ
QPPPPPPPPQQPPPPQQPPPPQQQ
SPPSQPSSPQQQSPQSPQ
TTTTT
TTTTTTTTTTTT
TTTTTTT
The inequality becomes equality if and only if : that is equivalent with
0 SPQSPQ T
SPQ
95
SOLO Matrices
Trace of a Square Matrix The trace of a square matrix is defined as T
nn
n
iiinn AtraceaAtrace
1
:
q.e.d.
ABtraceBAtrace 1
Proof:
n
i
n
jjiij baBAtrace
1 1
BAtracebaabABtracen
i
n
jjiij
n
j
n
iijji
1 11 1
ABtraceBAtraceBAtraceABtraceABtraceBAtrace TTTT111
2
Proof:
ABtraceBAtracebabaBAtracen
i
n
jjiij
n
i
n
jijij
T
1 11 1
Tn
j
n
iijij
T BAtraceabABtrace
1 1q.e.d.
96
SOLO Matrices
Trace of a Square Matrix The trace of a square matrix is defined as T
nn
n
iiinn AtraceaAtrace
1
:
3
Proof:
q.e.d.
n
ii APAPtraceAtrace
1
1
where P is the eigenvector matrix of A related to the eigenvalue matrix Λ of A by
n
PPPA
0
01
AtraceAPPtracePAPtrace 11
1
n
PPPA
0
01
n
PAP
0
01
1
n
iitracePAPtace
1
1
97
SOLO Matrices
Trace of a Square Matrix The trace of a square matrix is defined as T
nn
n
iiinn AtraceaAtrace
1
:
Proof:
q.e.d.
Definition
4 AtraceA ee det
AtraceA eeePeP
PePPePe
n
i
i
1detdetdet
det
1detdetdetdetdet 11
If aij are the coefficients of the matrix Anxn and z is a scalar function of aij, i.e.:
njiazz ij ,,1,
then is the matrix nxn whose coefficients i,j areA
z
njia
z
A
z
ijij
,,1,:
(see Gelb “Applied Optimal Estimation”, pg.23)
98
SOLO Matrices
Trace of a Square Matrix The trace of a square matrix is defined as T
nn
n
iiinn AtraceaAtrace
1
:
Proof:
q.e.d.
5
A
AtraceI
A
Atrace T
n
1
ji
jia
aA
Atraceij
n
iii
ijij1
0
1
6 nmmnTTT RBRCCBBC
A
BCAtrace
A
ABCtrace
1
Proof:
ijTji
m
ppijp
ik
jl
n
l
m
p
n
kklpklp
ijij
BCBCbcabcaA
ABCtrace
11 1 1
q.e.d.
7 If A, B, C Rnxn,i.e. square matrices, then
TTT CBBCA
BCAtrace
A
CABtrace
A
ABCtrace
11
99
SOLO Matrices
Trace of a Square Matrix The trace of a square matrix is defined as T
nn
n
iiinn AtraceaAtrace
1
:
Proof:
q.e.d.
8 nmmn
TTT
RBRCBCA
ABCtrace
A
BCAtrace
A
ABCtrace
721
9
BC
A
BCAtrace
A
CABtrace
A
ABCtrace TTT 811
If A, B, C Rnxn,i.e. square matrices, then
10 TA
A
Atrace2
2
ijTjiji
n
l
n
mmllm
ijijij
Aaaaaaa
Atrace
A
Atrace2
1 1
22
11 1
kT
k
AkA
Atrace
Proof: 1111
kT
k
kTkTkT
k
k
AkAAAA
AAAtrace
A
Atrace
q.e.d.
100
SOLO Matrices
Trace of a Square Matrix The trace of a square matrix is defined as T
nn
n
iiinn AtraceaAtrace
1
:
Proof:
q.e.d.
12 TAA
eA
etrace
TAn
k
n
k
kT
n
kkkT
n
n
k
k
n
n
k
k
n
A
eAk
Ak
k
k
Atrace
Ak
Atrace
AA
etrace
1 0
11
00 !
1lim
!lim
!lim
!lim
13
TT
TTTTTTTTT
TTTTT
TTT
BACBAC
A
ACABtrace
A
BACAtrace
A
ABACtrace
A
CABAtrace
A
BACAtrace
A
CABAtrace
A
ACABtrace
A
BACAtrace
A
ABACtrace
111
21
11
TTTTTTT
BACBACCABBACA
ABACtrace
A
ABACtrace
A
ABACtrace
86
2
2
1
1Proof: q.e.d.
14
AA
AAtrace
A
AAtrace TT
213
101
SOLO Matrices
Trace of a Square Matrix The trace of a square matrix is defined as T
nn
n
iiinn AtraceaAtrace
1
:
Proof:
15 TTTTT ABBAABBAA
ABAtrace
Table of Contents
ijTTTTn
ljlli
n
kkijk
n
l
n
l
n
kklmklm
ijijij
ABBAbababaaaa
ABAtrace
A
ABAtrace
111 1 1
q.e.d.
16 TTTTTT CABBACA
ABACtrace
ijTTTTTTn
l
n
rlirljr
n
k
n
mmikmjk
n
l
n
rrljrli
n
k
n
mmikmjk
n
l
n
k
n
m
n
rrlmrkmlk
ijijij
CABBACcabbac
abcbacabacaa
ABACtrace
A
ABACtrace
1 11 1
1 11 11 1 1 1
Proof:
q.e.d.
102
SOLO
References [1] Pease, “Methods of Matrix Algebra” ,Mathematics in Science and Engineering,
Vol.16, Academic Press, 1965
Matrices
[2] S. Hermelin, “Robustness and Sensitivity Design of Linear Time-Invariant Systems” PhD Thesis, Stanford University, 1986
Table of Contents
April 11, 2023 103
SOLO
TechnionIsraeli Institute of Technology
1964 – 1968 BSc EE1968 – 1971 MSc EE
Israeli Air Force1970 – 1974
RAFAELIsraeli Armament Development Authority
1974 – 2013
Stanford University1983 – 1986 PhD AA
Matrices
104
SOLO
Derivatives of Matrices
Matrices
ljikij
kl
a
a
For vector forms
j
i
ijii
i
iy
x
y
x
y
x
y
x
y
x
y
x
:::
We have the following expressions:
HH
TT
XX
XX
XXtraceX
XXtraceXX
XXXX
YXYXYX
YXYXYX
XtraceXtrace
YXYX
XX
constAifA
1
1
111
detln
detdet
0
105
SOLO
Derivatives of Determinants
Matrices
x
YYtraceY
x
Y 1detdet
x
YY
x
YYtrace
x
YYtrace
x
YYtrace
x
xY
YtraceYx
xY
11
11
1det
det
General Form
106
SOLO
Derivatives of Determinants
Matrices
11
1
detdetdet
detdet
detdet
TT
ijk
jkik
T
XBXAXBXAX
BXA
XXX
X
XXX
X
Linear Form
Square FormsIf X is Square and Invertible, then
TTT
XXAXX
XAX
det2
det
If X is Not Square but A is Symmetric, then
1det2
det
XAXXAXAXX
XAX TTT
If X is Not Square and A is Not Symmetric, then
11det
det
XAXXAXAXXAXAXX
XAX TTTTT
107
SOLO
Derivatives of Determinants
Matrices
Tk
k
T
TT
TT
XXkX
X
XXX
X
XX
XX
XX
XX
detdet
22detln
2detln
2detln
1T1-
†
†
Nonlinear Form
108
SOLO
Derivatives of an Inverse
Matrices
111
Yx
YY
x
Y
From this it follows
T
T
T
TTTT
jlkiij
kl
AXAXX
AXtrace
XABXX
BXAtrace
XXX
X
XbaXX
bXa
XXX
X
111
111
111
1
111
detdet
109
SOLO
Derivatives of Matrices, Vectors and Scalar Forms
Matrices
First Order
ijnm
mjinmn
ijT
ijmn
njimmn
ij
ij
ij
TTTT
TTT
TT
TT
AJAX
AX
AJAX
AX
JX
X
aaX
aXa
X
aXa
abX
bXa
baX
bXa
ax
xa
x
ax
n
mJ mn
000
010
000
110
SOLO
Derivatives of Matrices, Vectors and Scalar Forms
Matrices
Second Order
jlikklijijijT
ij
T
ilkjklT
ljij
klT
TTT
T
TTTT
klkl
klmnmnkl
ij
JXBJJBXX
XBX
XBBXX
XBX
bxBCDdxDCBx
dxDCbxB
bccbXX
cXXb
XXXX
2
n
mJ mn
000
010
000
111
SOLO
Derivatives of Matrices, Vectors and Scalar Forms
Matrices
Second Order (continue)
TTT
TTTTT
TT
bcbXDDcbXDcbXX
bcXDcbXDX
cXDXb
xBBx
xBx
Assume W is symmetric
TT
T
T
T
TT
ssAxWsAxWsAxA
sAxWsAxWsAxx
sxWsxWsxs
sxWsxWsxx
sAxWAsAxWsAxs
2
2
2
2
2
112
SOLO
Derivatives of Matrices, Vectors and Scalar Forms
Matrices
Higher Order and Nonlinear
1
0
1n
rkl
rnijr
ij
kln
XJXX
X
1
0
11
11
0
n
r
TrnTnTrrTnTrnnTnT
Trnn
r
TTrnT
XbaXXXXbaXbXXaX
XbaXbXaX