Numerical Methods Part: Cholesky and Decomposition http://numericalmethods.eng.usf.edu
TLDL
For more details on this topic Go to http://numericalmethods.eng.usf.edu Click on Keyword
http://numericalmethods.eng.usf.edu/�
You are free
to Share – to copy, distribute, display and perform the work
to Remix – to make derivative works
Under the following conditions Attribution — You must attribute the work in
the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
Noncommercial — You may not use this work for commercial purposes.
Share Alike — If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.
Chapter 04.09: Cholesky and Decomposition
Major: All Engineering Majors
Authors: Duc Nguyen
http://numericalmethods.eng.usf.edu Numerical Methods for STEM undergraduates
http://numericalmethods.eng.usf.edu 5 11/14/2011
Lecture # 1 TLDL
http://numericalmethods.eng.usf.edu/�
(1)
1
Introduction
6
http://numericalmethods.eng.usf.edu
][]][[ bxA =
= known coefficient matrix, with dimension
where
][A nn×
][b = known right-hand-side (RHS) 1×n vector
][x = unknown 1×n vector.
http://numericalmethods.eng.usf.edu 7
Symmetrical Positive Definite (SPD) SLE
A matrix can be considered as SPD if either of
(a) If each and every determinant of sub-matrix is positive, or..
the following conditions is satisfied:
for any given vector (b) If
As a quick example, let us make a test a test to see if the given matrix
nnA ×][
),...,2,1( niAii =,0>AyyT 0][ 1
≠×ny
−−−
−=
110121
012][A is SPD?
http://numericalmethods.eng.usf.edu 8
Symmetrical Positive Definite (SPD) SLE
Based on criteria (a):
The given matrix 33× is symmetrical, because
Furthermore,
[ ] 022det 11 >==×A
[ ]
032112
det 22
>=
−−
=×A
jiij aa =
http://numericalmethods.eng.usf.edu 9
[ ]
01110121
012det 33
>=
−−−
−=×A
Hence is SPD. ][A
http://numericalmethods.eng.usf.edu 10
Based on criteria (b): For any given vector
0
3
2
1 ≠
=
yyy
y , one computes
[ ]
( ) { }( ) { }32232221221
3223
2221
21
3
2
1
321
2
2222
110121
012
yyyyyyy
yyyyyyy
yyy
yyy
Ayyscalar T
−+++−=
−++−=
−−−
−=
=
http://numericalmethods.eng.usf.edu 11
( ) ( ) 023221221 >−++−= yyyyyscalar
hence matrix is SPD ][A
12
Step 1: Matrix Factorization phase
][][][ UUA T=
=
33
2322
131211
332313
2212
11
333231
232221
131211
0000
00
uuuuuu
uuuuu
u
aaaaaaaaa
Multiplying two matrices on the right-hand-side (RHS) of Equation (3), one gets the following 6 equations
1111 au =11
1212 u
au =11
1313 u
au =
( )21
2122222 uau −=
22
13122323 u
uuau −= ( )21
223
2133333 uuau −−=
(2)
(3)
(4)
(5)
http://numericalmethods.eng.usf.edu 13
( ) 21
1
1
2
−= ∑
−
=
i
kkiiiii uau
ii
i
kkjkiij
ij u
uuau
∑−
=−
=
1
1
Step 1.1: Compute the numerator of Equation (7), such as
∑−
=−=
1
1
i
kkjkiij uuaSum
Step 1.2 If is an off-diagonal term (say ) iju ji <then (See Equation (7)). Else, if is a
iiij u
Sumu = ijudiagonal term (that is, ), then ji = Sumuii =
(6)
(7)
(See Equation (6))
http://numericalmethods.eng.usf.edu 14
As a quick example, one computes:
55
47453735272517155757 u
uuuuuuuuau −−−−=
Thus, for computing , one only needs )7,5( == jiuto use the (already factorized) data in columns )5(# =iand of , respectively. )7(# =j ][U
(8)
http://numericalmethods.eng.usf.edu 15
k = 1
k = 2
k = 3
k = 4
iiu iju
ju4ju3ju2
ju1
iu4iu3iu2
iiu
i = 5
Col. # i=5 Col. # j=7
Figure 1: Cholesky Factorization for the term iju
http://numericalmethods.eng.usf.edu 16
Step 2: Forward Solution phase Substituting Equation (2) into Equation (1), one gets:
][]][[][ bxUU T =Let us define:
][][][ yxU ≡Then, Equation (9) becomes:
][][][ byU T =
=
3
2
1
3
2
1
332313
2212
11
000
bbb
yyy
uuuuu
u
(9)
(10)
(11)
(12)
http://numericalmethods.eng.usf.edu 17
1111 byu =
11
11 u
by =
From the 2nd row of Equation (12), one gets
2222112 byuyu =+
22
11222 u
yuby −=Similarly
33
22311333 u
yuyuby −−=
(13)
(14)
(15)
http://numericalmethods.eng.usf.edu 18
In general, from the row of Equation (12), one has thj
jj
j
iiijj
j u
yuby
∑−
=−
=
1
1 (16)
http://numericalmethods.eng.usf.edu 19
Step 3: Backward Solution phase
As a quick example, one has (See Equation (10)):
=
3
2
1
3
2
1
33
2322
131211
000
yyy
xxx
uuuuuu
(17)
http://numericalmethods.eng.usf.edu 20
From the last (or ) row of Equation (17), rdthn 3=
one has 3333 yxu =
hence
33
33 u
yx =
Similarly: 22
32322 u
xuyx −=
and
11
31321211 u
xuxuyx −−=
(18)
(19)
(20)
http://numericalmethods.eng.usf.edu 21
In general, one has:
jj
n
jiijij
j u
xuyx
∑+=
−= 1 (21)
http://numericalmethods.eng.usf.edu 22
TLDLA ]][][[][ =For example,
=
10010
1
000000
101001
32
3121
33
22
11
3231
21
333231
232221
131211
lll
dd
d
lll
aaaaaaaaa
Multiplying the three matrices on the RHS of Equation (23), one obtains the following formulas for
the “diagonal” , and “lower-triangular” matrices: ][D ][L
(22)
(23)
http://numericalmethods.eng.usf.edu 23
∑−
=−=
1
1
2j
kkkjkjjjj dlad
×
−= ∑
−
= jj
j
kjkkkikijij d
ldlal 11
1
(24)
(25)
http://numericalmethods.eng.usf.edu 24
Step1: Factorization phase TLDLA ]][][[][ =
Step 2: Forward solution and diagonal scaling phase
Substituting Equation (22) into Equation (1), one gets:
][][]][][[ bxLDL T =Let us define:
][][][ yxL T =
(22, repeated)
(26)
http://numericalmethods.eng.usf.edu 25
Also, define: ][]][[ zyD =
=
3
2
1
3
2
1
33
22
11
000000
zzz
yyy
dd
d
nifordzyii
ii ......,,3,2,1, ==
Then Equation (26) becomes:
][]][[ bzL =
(29)
(30)
http://numericalmethods.eng.usf.edu 26
=
3
2
1
3
2
1
3231
21
101001
bbb
zzz
lll
niforzLbzi
kkikii ......,,3,2,1
1
1=−= ∑
−
=
(31)
(32)
http://numericalmethods.eng.usf.edu 27
Step 3: Backward solution phase
=
3
2
1
3
2
1
32
3121
10010
1
yyy
xxx
lll
1......,,1,;1
−=−= ∑+=
nniforxlyxn
ikkkiii
http://numericalmethods.eng.usf.edu 28
Numerical Example 1 (Cholesky algorithms)
Solve the following SLE system for the unknown vector ? [ ]x ][]][[ bxA =
where [ ]
−−−
−=
110121
012A
=
001
][b
http://numericalmethods.eng.usf.edu 29
Solution:
The factorized, [ ]Uupper triangular matrix can be computed by either referring to Equations (6-7), or looking at Figure 1, as following:
][1
0414.10
7071.0414.1
1
414.12
11
1313
11
1212
1111
Uofrow
uau
uau
au
=
=
=
−=
−=
=
==
=
http://numericalmethods.eng.usf.edu 30
( )
( ){ }( )
( )( )
][2
8165.0225.1
07071.01225.1
1
225.17071.02
2
1312
22
11
123
23
2
21
212
21
11
1
22222
Uofrow
uuU
uuau
u
uau
i
kkjki
i
kki
−=
−−−=
×−−=
−=
=
−−=
−=
−=
∑
∑
=−
=
=−
=
http://numericalmethods.eng.usf.edu 31
( )
{ }( ) ( )
][3
5774.08165.001 22
21
223
21333
21
21
1
23333
Uofrowuua
uaui
kki
=
−−−=
−−=
−= ∑=−
=
Thus, the factorized matrix
[ ]
−
−=
5774.0008165.0225.1007071.0414.1
U
http://numericalmethods.eng.usf.edu 32
The forward solution phase, shown in Equation (11), becomes:
[ ] [ ] [ ]byU T =
=
−−
001
5774.08165.000225.17071.000414.1
3
2
1
yyy
http://numericalmethods.eng.usf.edu 33
( )( )( )
( )( ) ( )( )( )
5774.05774.0
4082.08165.07071.000
4082.0225.1
7071.07071.00
7071.0
414.11
33
223113
21
13
3
22
112
11
12
2
11
11
==
=−=−==−=
−=
==
=−=−=
−=
=
=
=
∑
∑
=−
=
=−
=
uyuyu
u
yuby
uyu
u
yuby
uby
jj
j
iiij
jj
j
iiij
http://numericalmethods.eng.usf.edu 34
The backward solution phase, shown in Equation (10), becomes:
[ ][ ] [ ]yxU =
=
−
−
5774.04082.07071.0
5774.0008165.0225.1007071.0414.1
3
2
1
xxx
http://numericalmethods.eng.usf.edu 35
( )( )
1225.1
18165.04082.0
15774.05774.0
22
3232
3
312
33
3
3
=
−−=
−=
−=
=
=
=
=
∑=
=+=
uxuy
u
xuyx
uy
uy
x
jj
N
jiijij
jj
j
http://numericalmethods.eng.usf.edu 36
( )( ) ( )( )
1414.1
1017071.07071.011
3132121
3
211
=
−−−=
−−=
−=
∑=
=+=
uxuxuy
u
xuyx
jj
N
jiijij
Hence
=
111
][x
http://numericalmethods.eng.usf.edu 37
Numerical Example 2 ( Algorithms) TLDL
Using the same data given in Numerical Example 1, find the unknown vector ][x by algorithms? TLDLSolution:
The factorized matrices and can be computed
from Equation (24), and Equation (25), respectively.
][D ][L
http://numericalmethods.eng.usf.edu 38
[ ] [ ]LandDofmatricesofColumn
dal
da
d
ldlal
alwaysl
a
dlad
jj
j
kjkkkik
j
kkkjk
1
020
5.021
)!(12
11
3131
11
21
01
121
21
11
11
01
1
21111
=
=
=
−=
−=
=
−=
===
−=
∑
∑
=−
=
=−
=
http://numericalmethods.eng.usf.edu 39
( ) ( )
( )( )( )
[ ] [ ]LandDmatricesofColumn
d
ldlal
alwaysl
dl
dlad
j
k
j
kkkjk
2
6667.05.1
5.0201
)!(15.1
25.02
2
22
11
121113132
32
22
211
221
11
1
22222
−=
−−−=
−=
==
−−=
−=
−=
∑
∑
=−
=
=−
=
http://numericalmethods.eng.usf.edu 40
( ) ( ) ( ) ( )[ ] [ ]LandDmatricesofColumndldl
dladj
kkkjk
3
3333.05.16667.0201
122
2223211
231
21
1
23333
=−−−=
−−=
−= ∑=−
=
http://numericalmethods.eng.usf.edu 41
Hence
[ ]
=
3333.00005.10002
D
and
[ ]
−−=
16667.00015.0001
L
http://numericalmethods.eng.usf.edu 42
The forward solution shown in Equation (31), becomes:
[ ][ ] [ ]bzL =
=
−−
001
1667.00015.0001
3
2
1
zzz
or,
∑−
=−=
1
1
i
kkikii zlbz (32, repeated)
http://numericalmethods.eng.usf.edu 43
Hence
( )( )
( )( ) ( )( )3333.0
5.06667.0100
5.015.00
1
23213133
12122
11
=−−−=
−−==
−−=−===
zLzLbz
zLbzbz
http://numericalmethods.eng.usf.edu 44
The diagonal scaling phase, shown in Equation (29), becomes
[ ][ ] [ ]zyD =
=
3333.05.0
1
3333.00005.10002
3
2
1
yyy
http://numericalmethods.eng.usf.edu 45
or
ii
ii d
zy =
Hence
13333.03333.0
3333.05.15.0
5.021
33
33
22
22
11
11
===
===
===
dzy
dzy
dzy
http://numericalmethods.eng.usf.edu 46
The backward solution phase can be found by referring to Equation (27)
[ ] [ ] [ ]yxL T =
=
−
−
1333.05.0
100667.01005.01
3
2
1
xxx
∑+=
−=N
ikkkiii xlyx
1(28, repeated)
http://numericalmethods.eng.usf.edu 47
Hence
( )
( )( ) ( )( )1
1015.05.0
116667.03333.0
1
1
33122111
2
33222
33
=−−−=
−−==
×−−=−=
==
xxlxlyx
x
xlyx
yx
http://numericalmethods.eng.usf.edu 48
Hence
[ ]
=
=
111
3
2
1
xxx
x
http://numericalmethods.eng.usf.edu 49
Re-ordering Algorithms For Minimizing Fill-in Terms [1,2].
During the factorization phase (of Cholesky, or Algorithms ), many “zero” terms in the original/given
matrix will become “non-zero” terms in the factored matrix
TLDL
][A][U . These new non-zero terms are often
called as “fill-in” terms (indicated by the symbol ) F It is, therefore, highly desirable to minimize these fill-in terms , so that both computational time/effort
and computer memory requirements can be substantially reduced.
http://numericalmethods.eng.usf.edu 50
For example, the following matrix and vector are given:
][A ][b
[ ]
=
11001020440030006604010088500345110720007112
A
=
14477094
129121
][b
(33)
(34)
http://numericalmethods.eng.usf.edu 51
The Cholesky factorization matrix , based on the original matrix (see Equation 33) and Equations (6-7), or Figure 1, can be symbolically computed as:
[ ]
××
×××
×××××××
=
000000000
00000
0000
FFF
FFF
U
][U][A
(35)
http://numericalmethods.eng.usf.edu 52
IPERM (new equation #) = {old equation #} such as, for this particular example:
=
123456
654321
IPERM
(36)
(37)
http://numericalmethods.eng.usf.edu 53
Using the above results (see Equation 37), one will be able to construct the following re-arranged matrices:
[ ]
=
11270002711054300588001040660003004402010011
*A
and
=
12112994704714
][ *b
(38)
(39)
http://numericalmethods.eng.usf.edu 54
Remarks: In the original matrix (shown in Equation 33), the nonzero term (old row 1, old column 2) = 7 will move to new location of the new matrix (new row 6, new column 5) = 7, etc.
The non zero term (old row 3, old column 3) = 88 will move to (new row 4, new column 4) = 88, etc.
The value of (old row 4) = 70 will be moved to (or located at) (new row 3) = 70, etc
b*b
A*A
A*A *A
http://numericalmethods.eng.usf.edu 55
Now, one would like to solve the following modified system of linear equations (SLE) for ],[ *x
][]][[ *** bxA =
rather than to solve the original SLE (see Equation1). The original unknown vector can be easily recovered from and ,shown in Equation (37).
}{x][ *x [ ]IPERM
(40)
http://numericalmethods.eng.usf.edu 56
[ ]
×××
××××××
×××
=
000000000
00000000000
000
*
FU
The factorized matrix can be “symbolically” computed from as (by referring to either Figure 1, or Equations 6-7):
][ *U][ *A
(41)
http://numericalmethods.eng.usf.edu 57
4. On-Line Chess-Like Game For Reordering/Factorized Phase [4].
Figure 2 A Chess-Like Game For Learning to Solve SLE.
http://numericalmethods.eng.usf.edu 58
(A)Teaching undergraduate/HS students the process how to use the reordering output IPERM(-), see Equations (36-37) for converting the original/given matrix , see Equation (33), into the new/modified matrix , see Equation (38). This step is reflected in Figure 2, when the “Game Player” decides to swap node (or equation) (say ) with another node (or equation ) , and click the “CONFIRM” icon!
][A][ *A
""i 2=i"" j
http://numericalmethods.eng.usf.edu 59
Since node is currently connected to nodes hence swapping node with the above nodes will “NOT” change the number/pattern of “Fill-in” terms. However, if node is swapped with node then the fill-in terms pattern may change (for better or worse)!
"2" =i;8,7,6,4=j 2=i
"" j2=i
,5,3,1 ororj =
http://numericalmethods.eng.usf.edu 60
(B) Helping undergraduate/HS students to understand the “symbolic” factorization” phase, by symbolically utilizing the Cholesky factorized Equations (6-7). This step is illustrated in Figure 2, for which the “game player” will see (and also hear the computer animated sound, and human voice), the non-zero terms (including fill-in terms) of the original matrix to move to the new locations in the new/modified matrix .
][A][ *A
http://numericalmethods.eng.usf.edu 61
(C) Helping undergraduate/HS students to understand the “numerical factorization” phase, by numerically utilizing the same Cholesky factorized Equations (6-7).
(D) Teaching undergraduate engineering/science students and even high-school (HS) students to “understand existing reordering concepts”, or even to “discover new reordering algorithms”
http://numericalmethods.eng.usf.edu 62
5. Further Explanation On The Developed Game
1. In the above Chess-Like Game, which is available on-line [4], powerful features of FLASH computer environments [3], such as animated sound, human voice, motions, graphical colors etc… have all been incorporated and programmed into the developed game-software for more appealing to game players/learners.
http://numericalmethods.eng.usf.edu 63
2. In the developed “Chess-Like Game”, fictitious monetary (or any kind of ‘scoring system”) is rewarded (and broadcasted by computer animated human voice) to game players, based on how he/she swaps the node (or equation) numbers, and consequently based on how many fill-in terms occurred. In general, less fill-in terms introduced will result in more rewards! ""F
http://numericalmethods.eng.usf.edu 64
3. Based on the original/given matrix , and existing re-ordering algorithms (such as the Reverse Cuthill- Mckee, or RCM algorithms [1-2]) the number of fill-in terms can be computed (using RCM algorithms). This internally generated information will be used to judge how good the players/learners are, and/or broadcast “congratulations message” to a particular player who discovers new “chess-like move” (or, swapping node) strategies which are even better than RCM algorithms!
][A
)"("F
http://numericalmethods.eng.usf.edu 65
4. Initially, the player(s) will select the matrix size ( , or larger is recommended), and the percentage (50%, or larger is suggested) of zero-terms (or sparsity of the matrix). Then, “START Game” icon will be clicked by the player.
88×
http://numericalmethods.eng.usf.edu 66
5. The player will then CLICK one of the selected node (or equation) numbers appearing on the computer screen. The player will see those nodes which are connected to node (based on the given/generated matrix ). The player then has to decide to swap node with one of the possible node
""i"" j
""i][A
""i "" j
http://numericalmethods.eng.usf.edu 67
After confirming the player’s decision, the outcomes/ results will be announced by the computer animated human voice, and the monetary-award will (or will NOT) be given to the players/learners, accordingly. In this software, a maximum of $1,000,000 can be earned by the player, and the “exact dollar amount” will be INVERSELY proportional to the number of fill-in terms occurred (as a consequence of the player’s decision on how to swap node with another node ).
""i"" j
http://numericalmethods.eng.usf.edu 68
6. The next player will continue to play, with his/her move (meaning to swap the node with the node) based on the current best non-zero terms pattern of the matrix.
thi thj
THE END
http://numericalmethods.eng.usf.edu
This instructional power point brought to you by Numerical Methods for STEM undergraduate http://numericalmethods.eng.usf.edu Committed to bringing numerical methods to the undergraduate
Acknowledgement
http://numericalmethods.eng.usf.edu/�
For instructional videos on other topics, go to
http://numericalmethods.eng.usf.edu/videos/ This material is based upon work supported by the National Science Foundation under Grant # 0717624. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
http://numericalmethods.eng.usf.edu/videos/�
The End - Really
�Numerical Methods��� �Part: Cholesky and Decomposition��http://numericalmethods.eng.usf.eduSlide Number 2You are freeUnder the following conditionsChapter 04.09: Cholesky and� Decomposition�Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23Slide Number 24Slide Number 25Slide Number 26Slide Number 27Slide Number 28Slide Number 29Slide Number 30Slide Number 31Slide Number 32Slide Number 33Slide Number 34Slide Number 35Slide Number 36Slide Number 37Slide Number 38Slide Number 39Slide Number 40Slide Number 41Slide Number 42Slide Number 43Slide Number 44Slide Number 45Slide Number 46Slide Number 47Slide Number 48Slide Number 49Slide Number 50Slide Number 51Slide Number 52Slide Number 53Slide Number 54Slide Number 55Slide Number 56Slide Number 57Slide Number 58Slide Number 59Slide Number 60Slide Number 61Slide Number 62Slide Number 63Slide Number 64Slide Number 65Slide Number 66Slide Number 67Slide Number 68The EndAcknowledgementSlide Number 71The End - Really