Solution of Sparse Linear Systems

Numerical Simulation

219A Spring 2008

Thanks to: Kin Sou, Deepak Ramaswamy, Michal Rewienski, Jacob White, Shihhsien Kuo and Karen Veroy and

especially Luca Daniel

2

Outline of today’s lecture

• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– Examples of Problems with Sparse Matrices Examples of Problems with Sparse Matrices

• Struts and joints, resistor grids, 3-D heat flowStruts and joints, resistor grids, 3-D heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization

• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach

– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering

3

X

X

X

X

X

X

X

X

X

1

2

3

4

5

6

7

8

9

2 x 2 block

X =

Sparse MatricesApplications

Space Frame

Space FrameX

X X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

XX

X X

X

X

X X

X

Unknowns : Joint positionsEquations : forces = 0

X

Nodal Matrix

X

X

X

X

X

X

X

XX

X

X

4

1 2 3 4 1m m

2m

1m 2m 3m 2m

( 1) ( 1)m m Unknowns : Node VoltagesEquations : currents = 0

Sparse MatricesApplications

Resistor Grid

Nodal Formulation

Matrix non-zero locations for 100 x 10 Resistor Grid

Sparse MatricesApplications

Resistor Grid

Nodal FormulationSparse MatricesApplications

Temperature in a cube

Temperature known on surface, determine interior temperature

1 2

1m 2m

2 1m 2 2m CircuitModel

7

Outline

• Solution of Dense Linear SystemsSolution of Dense Linear Systems

• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– LU Factorization Reminder.LU Factorization Reminder.– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices

• Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization

• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach

– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering

Nodal FormulationSparse MatricesTridiagonal Example

Matrix Form

1 2 3 4 1m m

X X

X X X

X X X

X X X

X X X

X X X

X X X

X X

m

How many operations to do LU factorization?

9

Sparse MatricesGE Algorithm

Tridiagonal Example

For i = 1 to n-1 { “For each Row”

For j = i+1 to n { “For each target Row below the source”

For k = i+1 to n { “For each Row element beyond Pivot”

} }}

Pivotjiji

ii

MM

M

Multiplier

jk jk ji ikM M M M

Order n Operations!

10

Outline

• Solution of Dense Linear SystemsSolution of Dense Linear Systems

• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– LU Factorization Reminder.LU Factorization Reminder.– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices

• Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization

• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach

– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering

11

1R

5R

3R

Sparse MatricesSparse Matrix Fill-In

4R

2R1V 2V

3V

1Si

1

1

2

3

0

0

S

V

V

iV

1

1

R 2

1

R

2

1

R

2

1

R

2

1

R 3

1

R

4

1

R

4

1

R

4

1

R

4

1

R 5

1

R

SymmetricDiagonally Dominant

Nodal Matrix

Resistor Example

Example

0

12

Sparse MatricesFill-In

Example

X X X

X X 0

X 0 X

X X X

X X 0

X 0 X

X

X

X X

X= Non zero

Matrix Non zero structure Fill Ins

X X

X = fill in where 0 became non-zero

13

Sparse MatricesFill-In

Second Example

X X X X

X X 0 0

0 X X 0

X 0 00

Fill-ins Propagate

XX

X

X

X

X X

X

X X

Fill-ins from Step 1 result in Fill-ins in step 2

14

Sparse MatricesFill-In

Reordering

1V 2V

3V

0

x x x

x x x

x x x

Fill-ins

Node Reordering - Can reduce fill-in - Preserves properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns

No Fill-ins2V 1V

3V

0

x x 0

x x x

0 x x

15

Sparse MatricesFill-In

Reordering

Where can fill-in occur ?

x

x x

x x

x x

x

x

Multipliers

Already Factored

Possible Fill-inLocations

Fill-in Estimate = (Non zeros in unfactored part of Row -i) (Non zeros in unfactored part of Col -i) MarkowitzMarkowitz product

0 0

x 0

0 x

16

Sparse MatricesFill-In

Reordering

Markowitz Reordering

tor oF 1i n with min Mard kowitz Productiagonan lFi d ij

anSw d rows columns a p j ji i

the new row and determine fiFactor ll-insi

End

Greedy Algorithm !

17

Sparse MatricesFill-In

Why only try diagonals ?

• Corresponds to node reordering in Nodal formulation

03

1 2

02

3 1

• Reduces search cost • Preserves Matrix Properties - Diagonal Dominance - Symmetry

Reordering

18

Sparse Matrices Fill-In

Pattern of a Filled-in Matrix

Very Sparse

Very Sparse

Dense

19

Sparse Matrices Fill-In

Unfactored Random Matrix

Symmetric

20

Sparse Matrices Fill-In

Factored Random Matrix

Symmetric

21

Outline

• Solution of Dense Linear SystemsSolution of Dense Linear Systems

• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– LU Factorization Reminder.LU Factorization Reminder.– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices

• Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization

• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach

– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering

22

X

X

X

X

X

X

X X

X

XX

X

X

X X 1

2

4

3

5

• One Node Per Matrix Row

• One Edge Per Off-diagonal Pair

X

X

Sparse MatricesMatrix Graphs

Construction

Structurally Symmetric Matrices and Graphs

23

Sparse MatricesMatrix Graphs

Markowitz Products

X

X

X

X

X

X

X X

X

XX

X

X

X X

X

X 1

2

4

3

5

Markowitz Products = (Node Degree)2

11 3 3 9M 2(degree 1) 933

22 2 2 4M 2(deg ree 2) 422

33 3 3 9M 2(deg ree 3) 933

44 2 2 4M 2(degree 4) 422

55 2 2 4M 2(degree 5) 422

24

X

X

X

X

X

X

X X

X

XX

X

X

X X 1

2

4

3

5

• Delete the node associated with pivot row

• “Tie together” the graph edges

X

X

X

X

X

Sparse MatricesMatrix Graphs

Factorization

X

One Step of LU Factorization

25

Sparse MatricesMatrix Graphs

Example

x x x x

x x x

x x x x

x x x x

x x x x

1 2 3 4 5

Graph

Markowitz products( = Node degree)

Col Row

3 3 9

2 2 4

3 3 9

3 3 9

3 3 9

1

2

3

4

5

26

Sparse MatricesMatrix Graphs

Example

x x x

x x 0 x x

x 0 x x x

x x x x

x x x x

Swap 2 with 1

X

X

2 3 4 5

Graph

2 3 4 5

Eliminate 1

112

27

1 2 3 4 1m m

2m

1m 2m 3m 2m

( 1) ( 1)m m Unknowns : Node VoltagesEquations : currents = 0

Sparse MatricesGraphs

Resistor Grid Example

28

Sparse MatricesMatrix Graphs

Grid Example

29

Sparse Matrices For growth control?Or to avoid fill-ins?

What should you pivot for?

A) LU factorization applied to a strictly diagonally dominant matrix will never produce a zero pivot

B) The matrix entries produced by LU factorization applied to a strictly diagonally dominant matrix will never increase by more than a factor 2(n-1)

[which is anyway the best you can do by pivoting for growth control]

Bottom line:if your matrix is strictly diagonally dominantno need for numerical pivot for growth control!

Sparse Factorization Approach

1)1) Assume matrix requires NO numerical pivoting.Assume matrix requires NO numerical pivoting.Diagonally dominant or symmetric positive definite.Diagonally dominant or symmetric positive definite.

2)2) Pre-Process:Pre-Process:• Use Graphs to Determine Matrix OrderingUse Graphs to Determine Matrix Ordering

• Many graph manipulation tricks used.Many graph manipulation tricks used.

• Form Data Structure for Storing Filled-in MatrixForm Data Structure for Storing Filled-in Matrix• Lots of additional non-zero addedLots of additional non-zero added

3)3) Put numerical values in Data Structure and factorPut numerical values in Data Structure and factorComputation must be organized carefully!Computation must be organized carefully!

Summary of Sparse Systems

• LU Factorization and Diagonal Dominance.LU Factorization and Diagonal Dominance.– Factor without numerical pivotingFactor without numerical pivoting

• Sparse Matrices Sparse Matrices – Struts, resistor grids, 3-d heat flow -> O(N) non-zerosStruts, resistor grids, 3-d heat flow -> O(N) non-zeros

• Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– Factor in O(N) operationsFactor in O(N) operations

• General Sparse FactorizationGeneral Sparse Factorization– Markowitz Reordering to minimize fillMarkowitz Reordering to minimize fill

• Graph Based ApproachGraph Based Approach– Factorization and Fill-inFactorization and Fill-in– Useful for estimating Sparse GE complexityUseful for estimating Sparse GE complexity

32

Outline

• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices– Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization

• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach• Summary of the AlgorithmSummary of the Algorithm

– Sparse Matrix Data StructuresSparse Matrix Data Structures• Scattering and symbolic fill inScattering and symbolic fill in

33

Sparse Matrices Sparse Data Structure

Val 11

Col 11

Val 12

Col 12

Val 1K

Col 1K

Val 21

Col 21

Val 22

Col 22

Val 2L

Col 2L

Val N1

Col N1

Val N2

Col N2

Val Nj

Col Nj

1

N

Arrays of Data in a RowVector of row pointers

Matrix entriesColumn index

Row pointersColumn Indices

Goal:Never store a 0Never multiply by 0

34

Sparse Matrices Sparse Data Structure

Problem of Misses

, 1 , 4 , 5 , 7 , 9 , 12 , 15

1 4 5 7 9 12 15j k j k j k j k j k j k j kM M M M M M M

k k k k k k k

, 1 , 7 , 15

1 7 15i k i k i kM M M

k k k

Eliminating Source Row i from Target row j

Row i

Row j

Must read all the row j entries to find the 3 that match row i

35

Sparse Matrices Sparse Data Structure

Data on Misses

Rows Ops Misses

Res 300 904387 248967

RAM 2806 1017289 3817587

Grid 4356 3180726 3597746

Every Miss is an unneeded Memory Reference!

More missesthanops!

36

Sparse Matrices Sparse Data Structure

Scattering for Miss Avoidance

Row j

, 1j kM , 4j kM , 5j kM , 7j kM , 9j kM , 12j kM , 15j kM

Use target row approach – Row j is the target.1) Read all the elements in Row j, and scatter them in an n-length vector2) Access only the needed elements using array indexing!

, 1 , 4 , 5 , 7 , 9 , 12 , 15

1 4 5 7 9 12 15j k j k j k j k j k j k j kM M M M M M M

k k k k k k k

37

Sparse Matrices – Another Data StructureOrthogonal linked list

But if fill in occurs, pointer structures change. What do we do?

Pre-compute pivoting order and sparse fill in structure symbolically

38

Summary of Sparse Systems

• LU Factorization and Diagonal Dominance.LU Factorization and Diagonal Dominance.– Factor without numerical pivotingFactor without numerical pivoting– hard problems (ill-conditioned)hard problems (ill-conditioned)

• Sparse Matrices Sparse Matrices – Struts, resistor grids, 3-d heat flow -> O(N) nonzerosStruts, resistor grids, 3-d heat flow -> O(N) nonzeros

• Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– Factor in O(N) operationsFactor in O(N) operations

• General Sparse FactorizationGeneral Sparse Factorization– Markowitz Reordering to minimize fillMarkowitz Reordering to minimize fill

• Graph Based ApproachGraph Based Approach– Factorization and Fill-inFactorization and Fill-in– Useful for estimating Sparse GE complexityUseful for estimating Sparse GE complexity

• Sparse Data StructuresSparse Data Structures– Scattering and symbolic fill inScattering and symbolic fill in