+ All Categories
Home > Documents > Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum...

Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum...

Date post: 22-May-2020
Category:
Upload: others
View: 13 times
Download: 0 times
Share this document with a friend
36
Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center and Uncertainty Quantification Center, KAUST (joint work with M. Genton, Y. Sun and D. Keyes) Center for Uncertainty Quantification http://sri-uq.kaust.edu.sa/
Transcript
Page 1: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

Hierarchical matrix techniques for maximumlikelihood covariance estimation

Alexander Litvinenko,Extreme Computing Research Center and Uncertainty

Quantification Center, KAUST(joint work with M. Genton, Y. Sun and D. Keyes)

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

http://sri-uq.kaust.edu.sa/

Page 2: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

The structure of the talk

1. Motivation

2. Hierarchical matrices [Hackbusch 1999]:

3. Matern covariance function

4. Uncertain parameters of the covariance function:

4.1 Uncertain covariance length4.2 Uncertain smoothness parameter

5. Identification of these parameters via maximizing thelog-likelihood.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

2 / 36

Page 3: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Motivation, problem 1

Task: to predict temperature, velocity, salinity, estimate parameters ofcovariance

Grid: 50Mi locations on 50 levels, 4*(X*Y*Z) + X*Y= 4*500*500*50 +

500*500 = 50Mi.

High-resolution time-dependent data about Red Sea: zonal velocity and

temperatureCenter for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

3 / 36

Page 4: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Motivation, problem 2

Task: to predict moisture, compute covariance, estimate its parameters

Grid: 1830× 1329 = 2, 432, 070 locations with 2,153,888 observations

and 278,182 missing values.

−120 −110 −100 −90 −80 −70

25

30

35

40

45

50

Soil moisture

longitude

latit

ude

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

High-resolution daily soil moisture data at the top layer of the Mississippibasin, U.S.A., 01.01.2014 (Chaney et al., in review).

Important for agriculture, defense. Moisture is very heterogeneous.Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

4 / 36

Page 5: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Motivation, estimation of uncertain parameters

H-matrix rank

3 7 9

cov. le

ngth

0.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055

0.06

Box-plots for ` = 0.0334 (domain [0, 1]2) vs different H-matrixranks k = {3, 7, 9}.Which H-matrix rank is sufficient for identification of parametersof a particular type of cov. matrix?

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

5 / 36

Page 6: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Motivation for H-matrices

General dense matrix requires O(n3) storage and time. Is veryexpensive.

If covariance matrix is structured (diagonal, Toeplitz, circulant)then we can apply e.g. FFT with O(nlogn), but if not ?

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

6 / 36

Page 7: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

H-matrix storage and complexity (p-proc. on shared mem.)

Operation Sequential Compl. Parallel Complexity(R.Kriemann 2005)

building(M) N = O(nlogn) Np +O(|V (T )\L(T )|)

storage(M) N = O(knlogn) N

Mx N = O(knlogn) Np + n√

p

αM ′ ⊕ βM ′′ N = O(k2nlogn) Np

αM ′ �M ′′ ⊕ βM N = O(k2nlog2n) Np +O(Csp(T )|V (T )|)

M−1 N = O(k2nlog2n) Np +O(nn2

min)

LU N = O(k2nlog2n) N

H-LU N = O(k2nlog2n) Np +O(k

2nlog2n

n1/d )

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

7 / 36

Page 8: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Hierarchical (H)-matrices

Introduction into Hierarchical (H)-matrix technique

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

8 / 36

Page 9: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Examples of H-matrix approximations

25 20

20 20

20 16

20 16

20 20

16 16

20 16

16 16

4 4

20 4 32

4 4

16 4 32

4 20

4 4

4 16

4 4

32 32

20 20

20 20 32

32 32

4 3

4 4 32

20 4

16 4 32

32 4

32 32

4 32

32 32

32 4

32 32

4 4

4 4

20 16

4 4

32 32

4 32

32 32

32 32

4 32

32 32

4 32

20 20

20 20 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

4 44 4

20 4 32

32 32 4

4 4

32 4

32 32 4

4 4

32 32

4 32 4

4 4

32 32

32 32 4

4

4 20

4 4 32

32 32

4 4

432 4

32 32

4 4

432 32

4 32

4 4

432 32

32 32

4 4

20 20

20 20 32

32 32

4 4

20 4 32

32 32

4 20

4 4 32

32 32

20 20

20 20 32

32 32

32 4

32 32

32 4

32 32

32 4

32 32

32 4

32 32

32 32

4 32

32 32

4 32

32 32

4 32

32 32

4 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

4 4

4 4 44 4

20 4 32

32 32

32 4

32 32

4 32

32 32

32 4

32 32

4 4

4 4

4 4

4 4 4

4 4

32 4

32 32 4

4 4

4 4

4 4

4 4 4

432 4

32 32

4 4

4 4

4 4

4 4

4 4 432 4

32 32

32 4

32 32

32 4

32 32

32 4

32 32

4 4

4 4

4 4

4 4

4 20

4 4 32

32 32

4 32

32 32

32 32

4 32

32 32

4 32

44 4

4 4

4 4

4 4

4 4

32 32

4 32 4

44 3

4 4

4 4

4 4

432 32

4 32

4 4

44 4

4 4

4 4

4 4

32 32

4 32

32 32

4 32

32 32

4 32

32 32

4 32

44 4

4 4

20 20

20 20 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

4 4

20 4 32

32 32

32 4

32 32

4 32

32 32

32 4

32 32

4 20

4 4 32

32 32

4 32

32 32

32 32

4 32

32 32

4 32

20 20

20 20 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

4 4

32 32

32 32 4

4 4

32 4

32 32 4

4 4

32 32

4 32 4

4 4

32 32

32 32 4

432 32

32 32

4 4

432 4

32 32

4 4

432 32

4 32

4 4

432 32

32 32

4 4

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 4

32 32

32 4

32 4

32 4

32 32

32 4

32 4

32 32

4 32

32 32

4 32

32 32

4 4

32 32

4 4

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

25 11

11 20 12

1320 11

9 1613

13

20 11

11 20 13

13 3213

13

20 8

10 20 13

13 32 13

1332 13

13 32

13

13

20 11

11 20 13

13 32 13

13

20 10

10 20 12

12 3213

13

32 13

13 32 13

1332 13

13 32

13

13

20 11

11 20 13

13 32 13

1332 13

13 3213

13

20 9

9 20 13

13 32 13

1332 13

13 32

13

13

32 13

13 32 13

1332 13

13 3213

13

32 13

13 32 13

1332 13

13 32

25 25

25

13 13

13 22

8 25

32

13 7

22 8

8 32

25

13 22

9 9

32 32

32

24 21

21 21

23 25

23

6 7

8 14

7 6

27 27

23 9

23

7 7

8 7

27 9

27 27

23 23

25

9 9

9 14

23 23

99 9

9 9

7 27

6 27

27 27

9 27

23 23

23

19 14

14 14

23 23

27 27

23 27

23 27

27 27

27 27

9 6

30 8

30 20 9

8 2

32 8

7 5

20 7 27 6

7 5

27 6

9 6 8

8 3

27 27

27 27 9

930 30

9 20

6 9

927 9

9 9

9 9

832

9 20

9 9

9 27

2 8

927 27

27 27

9 9

30 20

20 20

9 30

20 20

9 20

30 20

30 30

30

15 15

15 19

30 9

20 8

5 6

20 6

30 8

6 6

5 7 27

21 8

21

5 7

17 16

30 20

9 9

30

9 9

9 9

9 27

9 20

9 9

21 21

9

9 17

9 16

20 20

20 22 27

27 27

21

6 5

17 7

21 27

21 21

9 17

9 9 27

21 21

21

17 16

16 16

25 25

9 8

6 5

8 8

8 9

8 8

6 9

23 23

328 17

7 14

23 23

7 6

14 6 32

7 8

27 27

27 8

27 279

7 1

9 7

3

9 4

9 3

7 7

27 8 8 8

6 9

2 6

1 6

2

8

9 7

6 5 9

27 27

1 7 9

8 9

7 6

5 5

5 7 24

9 9

32 25

9 8

5 6

28

3

5 4

6 2

3 3

18 19 30

8 9

27 8

8 7

27 8

8 9

7 17

6 7 9

8 8

32 99

25 9

25 9 9

6 9

5 9 9

23 32

23

9 9

17 14

9 27

9 27

23

9 14

9 9

23 32

27 27

9 27

8 9

8 9 9

9 9

9 9

9 9

9 9

9 9 27

9 27 9

9 9

9

9 9

9

9 6

9 3

9

7 9

1 8

99 27

9 9

9 9

3 9

4 9

99

9 9

9 9

9 24

99 32

9 25

9 27

9 9

9

9 9

17 9

9 9

9 27

9 9

9 32

9 9

9

9 28

9

9 9

9 9

9

9

9 18

9 19

9 309

32 32

32

20 17

17 17

32 32

7 5

17 6 32

329 17

9 9

32 32

23 14

14 14 32

32 32

9 8

20 19

17 17 24

8 6

25

6 16

7 6

9 7

32 8

9 8

25 32

9

20 17

19 17

9 24

9 32

9 9

9 25

99 9

16 9

9 25

9 32

21 19

19 19 24

24 24

25

7 7

17 16

24 24

25 24

9 17

9 16 24

25 25

25

17 16

16 16

8 3

7 6

14 7 24

32 24 9

8

3 1

6 30

8 7

28 8

6 5

5 30

6 2

9 24

8 8 8

95 30

3 24

25 25

28

5 5

16 16

6 25

6 30

9

9 14

9 9 32

24 24

9 9

99 9

24 9

9 9

9

9 28

9 9

9 9

9 30

9 9

9 30

9

25 28

25

9 16

9 16

9 9

30 24

9 9

25 30

24 18

18 18 24

24 24

32 25

24 24

32 24

25 24

32 25

25 25

8 9

24 24

6 6

6 8 8

24 6

7 8

8 25

32 8

25 25

9 24

9 24

9 9

9 25

9 9

9 9 24

9 9

32 25

9 25

28 28

28

18 16

16 16

9 28

18 7

16 16 30

9

18 16

9 16

28 30

18 18

18 22 30

30 30

Figure : Three examples of H-matrix approximations, (left-middle)∈ Rn×n, n = 210, of the discretised covariance function cov(x , y) = e−r ,`1 = 0.15, `2 = 0.2, x , y ∈ [0, 1]2; (right) Soil moisture from exampleabove with 999 dofs. The biggest dense (dark) blocks ∈ R32×32, max.rank k = 4 on the left, k = 13 in the middle, and k = 9 on the right.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

9 / 36

Page 10: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Low-rank (rank-k) matrices

How do we compute green blocks?M ∈ Rn×m, U ≈ U ∈ Rn×k , V ≈ V ∈ Rm×k , k � min(n,m).The storage M = UΣV T is k(n + m) instead of n ·m for Mrepresented in the full matrix format.

VU ΣT=M

UVΣ∼

∼ ∼ T

=M∼

Figure : Reduced SVD, only k biggest singular values are taken.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

10 / 36

Page 11: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

H-matrices (Hackbusch ’99), main steps

1. Build cluster tree TI and block cluster tree TI×I .

I

I

I I

I

I

I I I I1

1

2

2

11 12 21 22

I11

I12

I21

I22

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

11 / 36

Page 12: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

H - Matrices

Let h =√∑2

i=1 h2i /`

2i , where hi := xi − yi , `i are cov. lengths and

d = 1.

exponential cov(h) = σ2 · exp(−h),The cov. matrix C ∈ Rn×n, n = 652.

`1 `2‖C−CH‖2

‖C‖2

0.01 0.02 3e − 20.1 0.2 8e − 31 2 2.8e − 6

10 20 3.7e − 9

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

12 / 36

Page 13: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Matern covariance functions

Matern covariance functions

Cθ =2σ2

Γ(ν)

( r

2`

)νKν

( r

`

), θ = (σ2, ν, `).

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

13 / 36

Page 14: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Examples of Matern covariance matrices

Cν=3/2(r) =

(1 +

√3r

`

)exp

(−√

3r

`

)(1)

Cν=5/2(r) =

(1 +

√5r

`+

5r 2

3`2

)exp

(−√

5r

`

)(2)

ν = 1/2 exponential covariance function Cν=1/2(r) = exp(−r),ν →∞ Gaussian covariance function Cν=∞(r) = exp(−r 2),.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

14 / 36

Page 15: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Numerical experiments with H-matrices

H-matrix approximations of covariance matrices,computing time and storage

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

15 / 36

Page 16: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Memory and computational times

n rank k size, MB t, sec. ε ε2

for C C C C C

4.0 · 103 10 48 3 0.8 0.08 7 · 10−3 2.0 · 10−4

1.05 · 104 18 439 19 7.0 0.4 7 · 10−4 1.0 · 10−4

2.1 · 104 25 2054 64 45.0 1.4 1 · 10−5 4.4 · 10−6

Table : The accuracy of the H-matrix approximation (weak admissibility)of the exp. covariance function, `1 = `3 = 0.1, `2 = 0.5, L-shape 3Ddomain (right) [Khoromskij et al’ 09]

.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

16 / 36

Page 17: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Identifying uncertain parameters

Identifying uncertain parameters

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

17 / 36

Page 18: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Identifying uncertain parameters

Given: a vector of measurements z = (z1, ..., zn)T with acovariance matrix C (θ∗) = C (σ2, ν, `).

Cθ =2σ2

Γ(ν)

( r

2`

)νKν

( r

`

), θ = (σ2, ν, `).

To identify: uncertain parameters (σ2, ν, `).Plan: Maximize the log-likelihood function

L(θ) = −1

2

(N log2π + log det{C (θ)}+ zTC (θ)−1z

),

On each iteration i we have a new matrix C (θi ).

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

18 / 36

Page 19: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Other works

1. S. AMBIKASARAN, et al., Fast direct methods for gaussian processes and the analysis of NASA Keplermission, arXiv:1403.6015, (2014).

2. S. AMBIKASARAN, J. Y. LI, P. K. KITANIDIS, AND E. DARVE, Large-scale stochastic linear inversionusing hierarchical matrices, Computational Geosciences, (2013)

3. J. BALLANI AND D. KRESSNER, Sparse inverse covariance estimation with hierarchical matrices, (2015).

4. M. BEBENDORF, Why approximate LU decompositions of finite element discretizations of ellipticoperators can be computed with almost linear complexity, (2007).

5. S. BOERM AND J. GARCKE, Approximating gaussian processes with H2-matrices, 2007.

6. J. E. CASTRILLON, M. G. GENTON, AND R. YOKOTA, Multi-Level Restricted Maximum LikelihoodCovariance Estimation and Kriging for Large Non-Gridded Spatial Datasets, (2015).

7. J. DOELZ, H. HARBRECHT, AND C. SCHWAB, Covariance regularity and H-matrix approximation forrough random fields, ETH-Zuerich, 2014.

8. H. HARBRECHT et al, Efficient approximation of random fields for numerical applications, NumericalLinear Algebra with Applications, (2015).

9. C.-J. HSIEH, et al, Big QUIC: Sparse inverse covariance estimation for a million variables, 2013

10. J. QUINONERO-CANDELA, et al, A unifying view of sparse approximate gaussian process regression,(2005).

11. A. SAIBABA, S. AMBIKASARAN, J. YUE LI, P. KITANIDIS, AND E. DARVE, Application of hierarchicalmatrices to linear inverse problems in geostatistics, Oil & Gas Science (2012).

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

19 / 36

Page 20: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Convergence of the optimization method

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

20 / 36

Page 21: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Details of the identification

To maximize the log-likelihood function we use the Brent’s method[Brent’73] (combining bisection method, secant method andinverse quadratic interpolation).

1. C (θ) ≈ CH(θ, k).

2. H-Cholesky: CH(θ, k) = LLT

3. zTC−1z = zT (LLT )−1z = vT · v , where v is a solution ofL(θ, k)v(θ) := z(θ∗).

4. Let λi be diagonal elements of H-Cholesky factor L, then

log det{C} = log det{LLT} = log det{n∏

i=1

λ2i } = 2

n∑i=1

logλi ,

L(θ, k) = −N

2log(2π)−

N∑i=1

log{Lii (θ, k)} − 1

2(v(θ)T · v(θ)). (3)

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

21 / 36

Page 22: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

0 10 20 30 40−4000

−3000

−2000

−1000

0

1000

2000

parameter θ, truth θ*=12

Log−

likelih

ood(θ

)

Shape of Log−likelihood(θ)

log(det(C))

zTC

−1z

Log−likelihood

Figure : Minimum of negative log-likelihood (black) is atθ = (·, ·, `) ≈ 12 (σ2 and ν are fixed)

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

22 / 36

Page 23: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

What will change?

Approximate C by CH

1. How the eigenvalues of C and CH differ ? [See Diss. ThomasMach, Magdeburg 2013]

2. How det(C ) differs from det(CH) ? [Below]

3. How L differs from LH ? [Mario Bebendorf et al]

4. How C−1 differs from (CH)−1 ? [Mario Bebendorf et al]

5. How L(θ, k) differs from L(θ)? [Below]

6. What is optimal H-matrix rank? [Below]

7. How θH differs from θ? [Below]

For theory, estimates for the rank and accuracy see works ofBebendorf, Grasedyck, Le Borne, Hackbusch,...

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

23 / 36

Page 24: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Remark

For a small H-matrix rank k the H-matrix Cholesky of CH can benot so stable (talk of Ralf Zimmermann) when eigenvalues of Ccome very close to zero. A remedy is to increase the rank k .In our example for n = 652 we increased k from 7 to 9.

To avoid this instability, we can modify CHm = CH + δ2I . Assumeλi are eigenvalues of CH. Then eigenvalues of CHm will be λi + δ2.

log det(CHm ) = logn∏

i=1

(λi + δ2) =n∑

i=1

log(λi + δ2). (4)

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

24 / 36

Page 25: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Error analysis

Theorem (Existence of H-matrix inverse in [Bebendorf’11,Ballani, Kressner’14)

Under certain conditions an H-matrix inverse exist

‖C−1H − C−1‖ ≤ ε‖C−1‖, (5)

theoretical estimations for rank kinv of C−1H are given.

Theorem (Error in log det)

Let E := C − CH, (CH)−1E := (CH)−1C − I and for the spectralradius

ρ((CH)−1E ) = ρ((CH)−1C − I) ≤ ε < ε. (6)

Then |log det(C )− log det(CH)| ≤ −plog(1− ε).

Proof: See [Ballani, Kressner 14], [Ipsen’05].

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

25 / 36

Page 26: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

How sensitive is Log-Likelihood to the H-matrix rank ?

It is not at all sensible.H-matrix approximation changes function L(θ, k) and estimationof θ very-very small.

θ 0.05 1.05 2.04 3.04 4.03 5.03 6.02 7.02 8.01 9 10L(exact) 1628 -2354 -1450 27 1744 3594 5529 7522 9559 11628 13727L(7) 1625 -2354 -1450 27 1745 3595 5530 7524 9560 11630 13726L(20) 1625 -2354 -1450 27 1745 3595 5530 7524 9561 11630 13725

Comparison of three likelihood functions, computed with differentH-matrix ranks: exact, H-rank 7, H-rank 20. Exponentialcovariance function, with covariance length ` = 0.9, domainG = [0, 1]2.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

26 / 36

Page 27: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

How sensitive is Log-Likelihood to the H-matrix rank ?

0 5 10−5000

0

5000

10000

15000

θ

−lo

glik

elih

ood

Figure : Three negative log-likelihood functions: exact, commuted withH-matrix rank 7 and 17. One can see that even with rank 7 one canachieve very accurate results.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

27 / 36

Page 28: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Decreasing of error bars with increasing number of measurements

Error bars (mean +/- st. dev.) computed for different n.

Decreasing of error bars with increasing of number ofmeasurements/dimension, n = {172, 332, 652}. The mean andmedian are obtained after averaging 200 simulations.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

28 / 36

Page 29: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Convergence of H-matrix approximations

0 10 20 30 40 50 60 70 80 90 100−25

−20

−15

−10

−5

0

rank k

log(r

el.

error)

Spectral norm, L=0.1, nu=1

Frob. norm, L=0.1

Spectral norm, L=0.2

Frob. norm, L=0.2

Spectral norm, L=0.5

Frob. norm, L=0.5

0 10 20 30 40 50 60 70 80 90 100−16

−14

−12

−10

−8

−6

−4

−2

0

rank k

log(r

el.

error)

Spectral norm, L=0.1, nu=0.5

Frob. norm, L=0.1

Spectral norm, L=0.2

Frob. norm, L=0.2

Spectral norm, L=0.5

Frob. norm, L=0.5

ν = 1(left) and ν = 0.5 (right) for different cov. lengths` = {0.1, 02, 0.5}

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

29 / 36

Page 30: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Convergence of H-matrix approximations

0 10 20 30 40 50 60 70 80 90 100−16

−14

−12

−10

−8

−6

−4

−2

0

rank k

log(r

el.

error)

Spectral norm, nu=1.5, L=0.1

Spectral norm, nu=1

Spectral norm, nu=0.5

0 10 20 30 40 50 60 70 80 90 100−22

−20

−18

−16

−14

−12

−10

−8

−6

−4

−2

rank k

log(r

el.

error)

Spectral norm, nu=1.5, L=0.5

Spectral norm, nu=1

Spectral norm, nu=0.5

ν = {1.5, 1, 0.5}, ` = 0.1 (left) and ν = {1.5, 1, 0.5}, ` = 0.5(right)

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

30 / 36

Page 31: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Take into account the gradient

∂L(θi )

∂θi=

1

2tr

(C−1 ∂C

∂θi

)− 1

2zTC−1 ∂C

∂θiC−1z . (7)

For an exponential random field, have

∂C (θi )

∂θi=

∂`exp

(−√|x − y |2`

)=−√|x − y |2`2

exp

(−√|x − y |2`

)(8)

∂C (θi )

∂θi=: C2

∂L(θi )

∂θi=

1

2tr(C−1C2

)− 1

2zTC−1C2C−1 · z .

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

31 / 36

Page 32: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

H-matrix approximation is robust w.r.t. parameter ν

Figure : Dependence of H-matrix approximation error on parameter ν.

Relative error ‖C−CH‖2

‖CH‖2via smoothness parameter ν. H-matrix rank

k = 8, n = 16641, Matern covariance matrix.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

32 / 36

Page 33: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

H-matrix approximation is robust w.r.t. cov. length `

Figure : Dependence of H-matrix approximation error on cov. length `.

Relative error ‖C−CH‖2

‖CH‖2via covariance length `. H-matrix rank k = 8,

n = 16641, Matern covariance matrix.

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

33 / 36

Page 34: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Conclusion

I Covariance matrices can be approximated in H-matrix format.

I Hypotes: H-matrix approximation accuracy is robust w.r.t. νand `.

I Influence of H-matrix approximation error on the estimatedparameters is small.

I With application of H-matricesI we extend the class of covariance functions to work with,I allows non-regular discretization of the covariance function on

large spatial grids.

I With the maximizing algorithm we are able to identify bothparameters: covariance lengths ` and the smoothness ν

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

34 / 36

Page 35: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Literature

1. PCE of random coefficients and the solution of stochastic partialdifferential equations in the Tensor Train format, S. Dolgov, B. N.Khoromskij, A. Litvinenko, H. G. Matthies, 2015/3/11, arXiv:1503.032102. Efficient analysis of high dimensional data in tensor formats, M. Espig,W. Hackbusch, A. Litvinenko, H.G. Matthies, E. Zander Sparse Grids andApplications, 31-56, 40, 20133. Application of hierarchical matrices for computing the Karhunen-Loeveexpansion, B.N. Khoromskij, A. Litvinenko, H.G. Matthies, Computing84 (1-2), 49-67, 31, 20094. Efficient low-rank approximation of the stochastic Galerkin matrix intensor formats, M. Espig, W. Hackbusch, A. Litvinenko, H.G. Matthies,P. Waehnert, Comp. & Math. with Appl. 67 (4), 818-829, 2012

5. Numerical Methods for Uncertainty Quantification and Bayesian

Update in Aerodynamics, A. Litvinenko, H. G. Matthies, Book

”Management and Minimisation of Uncertainties and Errors in Numerical

Aerodynamics” pp 265-282, 2013

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

35 / 36

Page 36: Alexander Litvinenko, Quanti cation Center, KAUST · Hierarchical matrix techniques for maximum likelihood covariance estimation Alexander Litvinenko, Extreme Computing Research Center

4*

Acknowledgement

1. Lars Grasedyck (RWTH Aachen) and Steffen Boerm (UniKiel) for HLIB (www.hlib.org)

2. KAUST Research Computing group, KAUST SupercomputingLab (KSL)

3. Stochastic Galerkin library (sglib from E. Zander). Type inyour terminalgit clone git://github.com/ezander/sglib.git

To initialize all variables, run startup.m You will find:generalised PCE, sparse grids, (Q)MC, stochastic Galerkin, linearsolvers, KLE, covariance matrices, statistics, quadratures(multivariate Chebyshev, Laguerre, Lagrange, Hermite ) etc

There are: many examples, many test, rich demos

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

36 / 36


Recommended