Low-rank tensor methods for stochastic forward and inverse problems

transcript

Low-rank tensor methods for PDEs withuncertain coefficients and

Bayesian Update surrogate

Alexander Litvinenko

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

http://sri-uq.kaust.edu.sa/

Extreme Computing Research Center, KAUST

Alexander Litvinenko Low-rank tensor methods for PDEs with uncertain coefficients and Bayesian Update surrogate

The structure of the talk

Part I (Stochastic forward problem):1. Motivation2. Elliptic PDE with uncertain coefficients3. Discretization and low-rank tensor approximations4. Tensor calculus to compute QoI

Part II (Bayesian update):1. Bayesian update surrogate2. Examples

61 117

83 132

97 132

82 128

2917 18

121 121

115 121

101 101

I received very rich collaboration experience as a co-organizator of:I 3 UQ workshops,I 2 Scalable Hierarchical Algorithms for eXtreme Computing

(SHAXC) workshopsI 1 HPC Conference (www.hpcsaudi.org, 2017)

My interests and collaborations

Motivation to do Uncertainty Quantification (UQ)

Motivation: there is an urgent need to quantify and reduce theuncertainty in output quantities of computer simulations withincomplex (multiscale-multiphysics) applications.

Typical challenges: classical sampling methods are often veryinefficient, whereas straightforward functional representationsare subject to the well-known Curse of Dimensionality.

My goal is systematic, mathematically founded, development ofUQ methods and low-rank algorithms relevant for applications.

-1 / 39

UQ and its relevance

Nowadays computational predictions are used in criticalengineering decisions and thanks to modern computers we areable to simulate very complex phenomena. But, how reliableare these predictions? Can they be trusted?

Example: Saudi Aramco currently has a simulator,GigaPOWERS, which runs with 9 billion cells. How sensitiveare the simulation results with respect to the unknown reservoirproperties?

0 / 39

Part I: Stochastic forward problem

Part I: Stochastic Galerkin method to solveelliptic PDE with uncertain coefficients

PDE with uncertain coefficient and RHS

Consider− div(κ(x , ω)∇u(x , ω)) = f (x , ω) in G × Ω, G ⊂ R2,u = 0 on ∂G, (1)

where κ(x , ω) - uncertain diffusion coefficient. Since κ positive,usually κ(x , ω) = eγ(x ,ω).For well-posedness see [Sarkis 09, Gittelson 10, H.J.Starkloff11, Ullmann 10].Further we will assume that covκ(x , y) is given.

1 / 39

My previous work

After applying the stochastic Galerkin method, obtain:Ku = f, where all ingredients are represented in a tensor format

Compute maxu, var(u), level sets of u, sign(u)[1] Efficient Analysis of High Dimensional Data in Tensor Formats,

Espig, Hackbusch, A.L., Matthies and Zander, 2012.

Research which ingredients influence on the tensor rank of K[2] Efficient low-rank approximation of the stochastic Galerkin matrix in tensor formats,

Wahnert, Espig, Hackbusch, A.L., Matthies, 2013.

Approximate κ(x , ω), stochastic Galerkin operator K in TensorTrain (TT) format, solve for u, postprocessing[3] Polynomial Chaos Expansion of random coefficients and the solution of stochastic

partial differential equations in the Tensor Train format, Dolgov, Litvinenko, Khoromskij, Matthies, 2016.

2 / 39

Typical quantities of interest

Keeping all input and intermediate data in a tensorrepresentation one wants to perform different tasks:

I evaluation for specific parameters (ω1, . . . , ωM),I finding maxima and minima,I finding ‘level sets’ (needed for histogram and probability

density).Example of level set: all elements of a high dimensional tensorfrom the interval [0.7,0.8].

3 / 39

Canonical and Tucker tensor formats

Definition and Examples of tensors

4 / 39

Canonical and Tucker tensor formats

[Pictures are taken from B. Khoromskij and A. Auer lecture course]

Storage: O(nd )→ O(dRn) and O(Rd + dRn).

5 / 39

Definition of tensor of order d

Tensor of order d is a multidimensional array over a d-tupleindex set I = I1 × · · · × Id ,

A = [ai1...id : i` ∈ I`] ∈ RI , I` = 1, ...,n`, ` = 1, ..,d .

A is an element of the linear space

Vn =d⊗`=1

V`, V` = RI`

equipped with the Euclidean scalar product 〈·, ·〉 : Vn ×Vn → R,defined as

〈A,B〉 :=∑

(i1...id )∈I

ai1...id bi1...id , for A, B ∈ Vn.

6 / 39

Examples of rank-1 and rank-2 tensors

Rank-1:f (x1, ..., xd ) = exp(f1(x1) + ...+ fd (xd )) =

∏dj=1 exp(fj(xj))

Rank-2: f (x1, ..., xd ) = sin(∑d

j=1 xj), since

2i · sin(∑d

j=1 xj) = ei∑d

j=1 xj − e−i∑d

j=1 xj

Rank-d function f (x1, ..., xd ) = x1 + x2 + ...+ xd can beapproximated by rank-2: with any prescribed accuracy:

f ≈∏d

j=1(1 + εxj)

ε−∏d

j=1 1ε

+O(ε), as ε→ 0

7 / 39

Tensor and Matrices

Rank-1 tensor

A = u1 ⊗ u2 ⊗ ...⊗ ud =:d⊗µ=1

Ai1,...,id = (u1)i1 · ... · (ud )id

Rank-1 tensor A = u ⊗ v , matrix A = uvT , A = vuT , u ∈ Rn,v ∈ Rm,Rank-k tensor A =

∑ki=1 ui ⊗ vi , matrix A =

∑ki=1 uivT

i .Kronecker product of n × n and m ×m matrices is a new blockmatrix A⊗ B ∈ Rnm×nm, whose ij-th block is [AijB].

8 / 39

Discretization of elliptic PDE

Now let us discretize our diffusion equation withuncertain coefficients

9 / 39

Karhunen Loeve and Polynomial Chaos Expansions

Apply bothKarhunen Loeve Expansion (KLE):κ(x , ω) = κ0(x) +

∑∞j=1 κjgj(x)ξj(θ(ω)), where

θ = θ(ω) = (θ1(ω), θ2(ω), ..., ),ξj(θ) = 1

∫G (κ(x , ω)− κ0(x)) gj(x)dx .

Polynomial Chaos Expansion (PCE)κ(x , ω) =

∑α κ

(α)(x)Hα(θ), compute ξj(θ) =∑

α∈J ξ(α)j Hα(θ),

where ξ(α)j = 1κj

∫G κ

(α)(x)gj(x)dx .

Further compute ξ(α)j ≈∑s

`=1(ξ`)j∏∞

k=1(ξ`, k )αk .

10 / 39

Final discretized stochastic PDE

Ku = f, where

K:=∑s

`=1K` ⊗⊗M

µ=1∆`µ, K` ∈ RN×N , ∆`µ ∈ RRµ×Rµ ,u:=

∑rj=1 uj ⊗

⊗Mµ=1 ujµ, uj ∈ RN , ujµ ∈ RRµ ,

f:=∑R

k=1 f k ⊗⊗M

µ=1 gkµ, f k ∈ RN and gkµ ∈ RRµ .(Wahnert, Espig, Hackbusch, Litvinenko, Matthies, 2011)

Examples of stochastic Galerkin matrices:

11 / 39

Computing QoI in low-rank tensor format

Now, we consider how tofind maxima in a high-dimensional tensor

Maximum norm and corresponding index

Let u =∑r

j=1⊗d

µ=1 ujµ ∈ Rr , compute

‖u‖∞ := maxi:=(i1,...,id )∈I

|ui | = maxi:=(i1,...,id )∈I

∣∣∣∣∣∣r∑

d∏µ=1

(ujµ)

∣∣∣∣∣∣ .Computing ‖u‖∞ is equivalent to the following e.v. problem.

Let i∗ := (i∗1 , . . . , i∗d ) ∈ I, #I =

∏dµ=1 nµ.

‖u‖∞ = |ui∗ | =

∣∣∣∣∣∣r∑

d∏µ=1

(ujµ)

i∗µ

∣∣∣∣∣∣ and e(i∗) :=d⊗µ=1

ei∗µ ,

where ei∗µ ∈ Rnµ the i∗µ-th canonical vector in Rnµ (µ ∈ N≤d ).

12 / 39

u e(i∗) =

r∑j=1

d⊗µ=1

ei∗µ

d⊗µ=1

ujµ ei∗µ

d⊗µ=1

[(ujµ)i∗µei∗µ

r∑j=1

d∏µ=1

(ujµ)i∗µ

︸︷︷︸

ui∗=

d⊗µ=1

e(i∗µ) = ui∗e(i∗).

Thus, we obtained an “eigenvalue problem”:

u e(i∗) = ui∗e(i∗).

13 / 39

Computing ‖u‖∞, u ∈ Rr by vector iteration

By defining the following diagonal matrix

D(u) :=r∑

d⊗µ=1

diag((ujµ)`µ

)`µ∈N≤nµ

with representation rank r , obtain D(u)v = u v .Now apply the well-known vector iteration method (with ranktruncation) to

D(u)e(i∗) = ui∗e(i∗),

obtain ‖u‖∞.[Approximate iteration, Khoromskij, Hackbusch, Tyrtyshnikov 05],

and [Espig, Hackbusch 2010]

14 / 39

How to compute the mean value in CP format

Let u =∑r

j=1⊗d

µ=1 ujµ ∈ Rr , then the mean value u can becomputed as a scalar product

⟨ r∑j=1

d⊗µ=1

⟩ =r∑

d⊗µ=1

⟨ujµ, 1µ

⟩nµ

d∏µ=1

( nµ∑k=1

(ujµ)k

), (4)

where 1µ := (1, . . . ,1)T ∈ Rnµ .Numerical cost is O

(r ·∑d

µ=1 nµ)

15 / 39

Numerical Experiments

2D L-shape domain, N = 557 dofs.Total stochastic dimension is Mu = Mk + Mf = 20, there are|J | = 231 PCE coefficients

u =231∑j=1

uj,0 ⊗20⊗µ=1

ujµ ∈ R557 ⊗20⊗µ=1

16 / 39

Level sets

Now we compute level sets

sign(b‖u‖∞1− u)for b ∈ 0.2, 0.4, 0.6, 0.8.

I Tensor u has 320 ∗ 557 ≈ 2 · 1012 entries ≈ 16 TB ofmemory.

I The computing time of one level set was 10 minutes.I Intermediate ranks of sign(b‖u‖∞1− u) and of rank(uk )

were less than 24.

17 / 39

Part II

Part II: Bayesian update

We will speak about Gauss-Markov-Kalman filter for theBayesian updating of parameters in comput. model.

Mathematical setup

Consider

K (u; q) = f ⇒ u = S(f ; q),

where S is solution operator.Operator depends on parameters q ∈ Q,hence state u ∈ U is also function of q:

Measurement operator Y with values in Y:

y = Y (q; u) = Y (q,S(f ; q)).

Examples of measurements:y(ω) =

u(ω, x)dx , or u in few points

18 / 39

Random QoI

With state u a RV, the quantity to be measured

y(ω) = Y (q(ω),u(ω)))

is also uncertain, a random variable.Noisy data: y + ε(ω),

where y is the “true” value and a random error ε.

Forecast of the measurement: z(ω) = y(ω) + ε(ω).

19 / 39

Conditional probability and expectation

Classically, Bayes’s theorem gives conditional probability

P(Iq|Mz) =P(Mz |Iq)

P(Mz)P(Iq) (orπq(q|z) =

p(z|q)

Zspq(q));

Expectation with this posterior measure is conditionalexpectation.

Kolmogorov starts from conditional expectation E (·|Mz),from this conditional probability via P(Iq|Mz) = E

(χIq |Mz

20 / 39

Conditional expectation

The conditional expectation is defined asorthogonal projection onto the closed subspace L2(Ω,P, σ(z)):

E(q|σ(z)) := PQ∞q = argminq∈L2(Ω,P,σ(z)) ‖q − q‖2L2

The subspace Q∞ := L2(Ω,P, σ(z)) represents the availableinformation.

The update, also called the assimilated valueqa(ω) := PQ∞q = E(q|σ(z)), is a Q-valued RV

and represents new state of knowledge after the measurement.Doob-Dynkin: Q∞ = ϕ ∈ Q : ϕ = φ z, φmeasurable.

21 / 39

Numerical computation of NLBU

Look for ϕ such that q(ξ) = ϕ(z(ξ)), z(ξ) = y(ξ) + ε(ω):

ϕ ≈ ϕ =∑α∈Jp

ϕαΦα(z(ξ))

and minimize ‖q(ξ)− ϕ(z(ξ))‖2L2, where Φα are polynomials

(e.g. Hermite, Laguerre, Chebyshev or something else).Taking derivatives with respect to ϕα:

∂ϕα〈q(ξ)− ϕ(z(ξ)),q(ξ)− ϕ(z(ξ))〉 = 0 ∀α ∈ Jp

Inserting representation for ϕ, obtain:

22 / 39

∂ϕαE

q2(ξ)− 2∑β∈J

qϕβΦβ(z) +∑β,γ∈J

ϕβϕγΦβ(z)Φγ(z)

−qΦα(z) +∑β∈J

ϕβΦβ(z)Φα(z)

∑β∈J

E [Φβ(z)Φα(z)]ϕβ − E [qΦα(z)]

= 0 ∀α ∈ J .

23 / 39

Now, rewriting the last sum in a matrix form, obtain the linearsystem of equations (=: A) to compute coefficients ϕβ: ... ... ...

... E [Φα(z(ξ))Φβ(z(ξ))]...

... ... ...

...ϕβ

E [q(ξ)Φα(z(ξ))]...

where α, β ∈ J , A is of size |J | × |J |.

24 / 39

We can rewrite the system above in the compact form:

[Φ] [diag(...wi ...)] [Φ]T

...ϕβ...

= [Φ]

w0q(ξ0)...

wNq(ξN)

[Φ] ∈ RJα×N , [diag(...wi ...)] ∈ RN×N , [Φ] ∈ RJα×N .Solving this system, obtain vector of coefficients (...ϕβ...)

T forall β.Finally, the assimilated parameter qa will be

qa = qf + ϕ(y)− ϕ(z), (5)

z(ξ) = y(ξ) + ε(ω), ϕ =∑

β∈JpϕβΦβ(z(ξ))

25 / 39

Example: Lorenz 1963 problem (chaotic system of ODEs)

x = σ(ω)(y − x)

y = x(ρ(ω)− z)− yz = xy − β(ω)z

Initial state q0(ω) = (x0(ω), y0(ω), z0(ω)) are uncertain.

Solving in t0, t1, ..., t10, Noisy Measur. → UPDATE, solving int11, t12, ..., t20, Noisy Measur. → UPDATE,...

IDEA of the Bayesian Update (BU):Take qf (ω) = q0(ω).Linear BU: qa = qf + K · (z − y)Non-Linear BU: qa = qf + H1 · (z − y) + (z − y)T · H2 · (z − y).

26 / 39

Trajectories of x,y and z in time. After each update (newinformation coming) the uncertainty drops. [O. Pajonk, B. V. Rosic, A.

Litvinenko, and H. G. Matthies, 2012]

27 / 39

Example: Lorenz problem

10 0 100

0.10.20.30.40.50.60.70.8

20 0 200

0.050.1

0.150.2

0.250.3

0.350.4

0 10 200

0.10.20.30.40.50.60.70.80.9

Figure: quadratic BU surrogate, measure the state (x(t), y(t), z(t)).Prior and posterior after one update.

28 / 39

Example: Lorenz Problem

10 5 00

0.10.20.30.40.50.60.70.80.9

15 10 50

0.050.1

0.150.2

0.250.3

0.350.4

0.450.5

5 10 150

0.10.20.30.40.50.6

Figure: Comparison of the posterior functions computed by linear andquadratic BU after second update.

29 / 39

Example: Lorenz Problem

20 0 200

0.020.040.060.080.1

0.120.140.16

50 0 500

0.010.020.030.040.050.060.070.080.09

0 10 200

0.020.040.060.080.1

0.120.140.160.18

Figure: Quadratic measurement (x(t)2, y(t)2, z(t)2): Comparison of apriori and a posterior for NLBU

30 / 39

Example: 1D elliptic PDE with uncertain coeffs

−∇ · (κ(x , ξ)∇u(x , ξ)) = f (x , ξ), x ∈ [0,1]

+ Dirichlet random b.c. g(0, ξ) and g(1, ξ).3 measurements: u(0.3) = 22, s.d. 0.2, x(0.5) = 28, s.d. 0.3,x(0.8) = 18, s.d. 0.3.

I κ(x, ξ): N = 100 dofs, M = 5, number of KLE terms 35, beta distribution for κ, Gaussian covκ, cov.length 0.1, multi-variate Hermite polynomial of order pκ = 2;

I RHS f (x, ξ): Mf = 5, number of KLE terms 40, beta distribution for κ, exponential covf , cov. length 0.03,multi-variate Hermite polynomial of order pf = 2;

I b.c. g(x, ξ): Mg = 2, number of KLE terms 2, normal distribution for g, Gaussian covg , cov. length 10,multi-variate Hermite polynomial of order pg = 1;

I pφ = 3 and pu = 3

31 / 39

Example: updating of the solution u

0 0.5 1-20

Figure: Original and updated solutions, mean value plus/minus 1,2,3standard deviations

[graphics are built in the stochastic Galerkin library sglib, written by E. Zander in TU Braunschweig]

32 / 39

Example: Updating of the parameter

0 0.5 10

Figure: Original and updated parameter κ.

33 / 39

Future plans and possible collaboration

Future plans and possible collaboration ideas

Future plans, Idea N1

Possible collaboration work with Troy Butler: To develop alow-rank adaptive goal-oriented Bayesian update technique. Thesolution of the forward and inverse problems will be considered as awhole adaptive process, controlled by error/uncertainty estimators.

(y - z) q

forward update

low-rank and adaptive

(y - z)

forwardy q.....

low-rank and adaptive

... q update

Stochastic forward spatial discret.

stochastic discret.

low-rank approx.

Inverse problem

Errors

inverse operator approx.

Edge between Green functions in PDEs and covariancematrices.Possible collaboration with statistical group, Doug Nychka(NCAR), Havard Rue

34 / 39

Data assimilation techniques, Bayesian update surrogare.Develop non-linear, non-Gaussian Bayesian updateapproximation for gPCE coefficients.Possible collaboration with Jan Mandel, Troy Butler, Kody Law,Y. Marzouk, H. Najm, TU Braunschweig and KAUST

Collaborators

1. Uncertainty quantification and Bayesian Update: Prof. H.Matthies, Bojana V. Rosic, Elmar Zander, Oliver Pajonkfrom TU Braunschweig, Germany,

2. Low-rank tensor calculus: Mike Espig from RWTH Aachen,Boris and Venera Khoromskij from MPI Leipzig

3. Spatial and environmental statistics: Marc Genton, YingSun, Raphael Huser, Brian Reich, Ben Shaby and DavidBolin.

4. Some others: UQ, data assimilation, high-dimensionalproblems/statistics

Conclusion

I Introduced low-rank tensor methods to solve elliptic PDEswith uncertain coefficients,

I Explained how to compute the maximum, the mean, levelsets,... in low-rank tensor format,

I Derived Bayesian update surrogate ϕ (as a linear,quadratic, cubic etc approximation), i.e. computeconditional expectation of q, given measurement y .

34 / 39

Example: Canonical rank d , whereas TT rank 2

d-Laplacian over uniform tensor grid. It is known to have theKronecker rank-d representation,

∆d = A⊗IN⊗...⊗IN +IN⊗A⊗...⊗IN +...+IN⊗IN⊗...⊗A ∈ RI⊗d⊗I⊗d

(6)with A = ∆1 = tridiag−1,2,−1 ∈ RN×N , and IN being theN × N identity. Notice that for the canonical rank we have rankkC(∆d ) = d , while TT-rank of ∆d is equal to 2 for anydimension due to the explicit representation

∆d = (∆1 I)×(

I 0∆1 I

)× ...×

∆1 I

where the rank product operation ”×” is defined as a regularmatrix product of the two corresponding core matrices, theirblocks being multiplied by means of tensor product. The similarbound is true for the Tucker rank rankTuck (∆d ) = 2.

Advantages and disadvantages

Denote k - rank, d-dimension, n = # dofs in 1D:

1. CP: ill-posed approx. alg-m, O(dnk), hard to computeapprox.

2. Tucker: reliable arithmetic based on SVD, O(dnk + kd )

3. Hierarchical Tucker: based on SVD, storage O(dnk + dk3),truncation O(dnk2 + dk4)

4. TT: based on SVD, O(dnk2) or O(dnk3), stable5. Quantics-TT: O(nd )→ O(d logqn)

How to compute the variance in CP format

Let u ∈ Rr and

u := u − ud⊗µ=1

1 =r+1∑j=1

d⊗µ=1

ujµ ∈ Rr+1, (8)

then the variance var(u) of u can be computed as follows

var(u) =〈u, u〉∏dµ=1 nµ

=1∏d

µ=1 nµ

⟨r+1∑i=1

d⊗µ=1

r+1∑j=1

d⊗ν=1

=r+1∑i=1

r+1∑j=1

d∏µ=1

⟨uiµ, ujµ

Numerical cost is O(

(r + 1)2 ·∑d

µ=1 nµ)

Computing QoI in low-rank tensor format

Now, we consider how tofind ‘level sets’,

for instance, all entries of tensor u from interval [a,b].

Definitions of characteristic and sign functions

1. To compute level sets and frequencies we needcharacteristic function.2. To compute characteristic function we need sign function.

The characteristic χI(u) ∈ T of u ∈ T in I ⊂ R is for every multi-index i ∈ I pointwise defined as

(χI(u))i :=

1, ui ∈ I,0, ui /∈ I.

Furthermore, the sign(u) ∈ T is for all i ∈ I pointwise definedby

(sign(u))i :=

1, ui > 0;−1, ui < 0;0, ui = 0.

36 / 39

sign(u) is needed for computing χI(u)

LemmaLet u ∈ T , a,b ∈ R, and 1 =

⊗dµ=1 1µ, where

1µ := (1, . . . ,1)t ∈ Rnµ .(i) If I = R<b, then we have χI(u) = 1

2(1+ sign(b1− u)).

(ii) If I = R>a, then we have χI(u) = 12(1− sign(a1− u)).

(iii) If I = (a,b), then we haveχI(u) = 1

2(sign(b1− u)− sign(a1− u)).

Computing sign(u), u ∈ Rr , via hybrid Newton-Schulz iterationwith rank truncation after each iteration.

37 / 39

Level Set, Frequency

Definition (Level Set, Frequency)Let I ⊂ R and u ∈ T . The level set LI(u) ∈ T of u respect to I ispointwise defined by

(LI(u))i :=

ui ,ui ∈ I ;0,ui /∈ I ,

for all i ∈ I.The frequency FI(u) ∈ N of u respect to I is defined as

FI(u) := # suppχI(u).

38 / 39

Computation of level sets and frequency

PropositionLet I ⊂ R, u ∈ T , and χI(u) its characteristic. We have

LI(u) = χI(u) u

and rank(LI(u)) ≤ rank(χI(u)) rank(u).The frequency FI(u) ∈ N of u respect to I is

FI(u) = 〈χI(u),1〉 ,

where 1 =⊗d

µ=1 1µ, 1µ := (1, . . . ,1)T ∈ Rnµ .

39 / 39

Low-rank tensor methods for stochastic forward and inverse problems

Education