Introduction to EnsembleKalman Filter

E. Kalnay with Hong Li and T. MiyoshiUniversity of MarylandChaos-Weather Group

B. Hunt, J. Yorke, E.OttK. Ide, T. Miyoshi, E. Kalnay,

I. Szunyogh (TAMU), E. Kostelich (ASU) and 16 currentstudents and Ph.D.s

JCSDA Summer Colloquium 2009

Recall the basic formulation of OIRecall the basic formulation of OI

Ta= T

b+ w(T

o! T

b) 0 " w " 1

Optimal weight to minimize the analysis errors: w =!b

2

!b

2+!

o

2

OI for a scalar (analysis):

Recall the basic formulation of OIRecall the basic formulation of OI

OI for a scalar (analysis):Ta= T

b+ w(T

o! T

b) 0 " w " 1

Optimal weight to minimize the analysis errors: w =!b

2

!b

2+!

o

2

OI for a vector: xib= Mx

i!1a

xi

a= x

i

b+K(y

i

o ! Hxi

b)

K = PbHT HPbHT+ R"# $%

!1

Pa = I !KH[ ]Pb

Pb= !x

b•!x

b

T!x

b= x

b" x

t

In OI is statistically pre-estimated and constant in time.Is this a good approximation?

Pb

OI and Kalman Filter for a scalarOI and Kalman Filter for a scalar

OI for a scalar: Tb(ti) = MT

a(ti!1); "

b

2= (1+ a)"

a

2=

1

1! w"

a

2= const

w ="b

2

"b

2+"

o

2; "

a

2= (1! w)"

b

2

Ta(ti) = T

b(ti) + w(T

o(ti) ! T

b(ti)) 0 # w # 1

OI and Kalman Filter for a scalarOI and Kalman Filter for a scalar

OI for a scalar: Tb(ti) = MT

a(ti!1); "

b

2= (1+ a)"

a

2=

1

1! w"

a

2= const

w ="b

2

"b

2+"

o

2; "

a

2= (1! w)"

b

2

Ta(ti) = T

b(ti) + w(T

o(ti) ! T

b(ti)) 0 # w # 1

Kalman Filterfor a scalar:

Now the background error variance is forecasted using thelinear tangent model and its “adjoint”.

Tb(ti) = MT

a(ti!1); "

b

2= (L"

a)(L"

a)T; L = dM / dT

w ="b

2

"b

2+"

o

2; "

a

2= (1! w)"

b

2

Ta(ti) = T

b(ti) + w(T

o(ti) ! T

b(ti)) 0 # w # 1

Ensemble Kalman Filter (EnKF)

Pib=

1

m !1(xk

f! x

t)i (xk

f! x

t)iT

k=1

m

" , m#$

We use the ensemble to estimate *

Pib!

1

K "1(xk

f" x

f)i (xk

f" x

f)iT

k=1

m

#

=1

K "1Xi

bXi

bT

Problem left: how to estimate theanalysis error covariance and theanalysis perturbations?

Pi

a=

1

K !1X

i

aX

i

aT

Local Ensemble Transform Kalman Filter(Ott et al, 2004, Hunt et al, 2004, 2007)

• Model independent(black box)• Obs. assimilatedsimultaneously at eachgrid point• 100% parallel: fast• No adjoint needed• 4D LETKF extension

(Start with initial ensemble)

LETKFObservationoperator

Model

ensemble analyses

ensemble forecasts

ensemble“observations”

Observations

Perform data assimilation in a local volume, choosing observations

The state estimate is updated at thecentral grid red dot

Localization based on observations

Perform data assimilation in a local volume, choosing observations

The state estimate is updated at thecentral grid red dot

All observations (purple diamonds)within the local region are assimilated

Localization based on observations

The LETKF algorithm can be described in a single slide

Local Ensemble Transform Kalman Filter (Local Ensemble Transform Kalman Filter (LETKFLETKF))

Forecast step:Analysis step: construct

Locally: Choose for each grid point the observations to be used, andcompute the local analysis error covariance and perturbations inensemble space:

Analysis mean in ensemble space:and add to to get the analysis ensemble in ensemble space

The new ensemble analyses in model space are the columns of . Gathering the grid point analyses forms the new

global analyses. Note that the the output of the LETKF are analysisweights and perturbation analysis matrices of weights . Theseweights multiply the ensemble forecasts.

x

n,k

b= M

nx

n!1,k

a( )Xb

= x1

b ! xb | ... | xK

b ! xb"# $%;

yi

b= H (x

i

b); Y

n

b= y

1

b ! yb | ... | yK

b ! yb"# $%

!Pa= K !1( )I + YbT

R!1Y

b"# $%!1;W

a= [(K !1) !Pa ]1/2

Xn

a= X

n

bW

a+ x

b

wa= !P

aY

bTR

!1(yo! y

b )

Wa

Globally:

wa

Wa

Repeat: Repeat: LETKFLETKF, globally, with notes (4D-LETKF), globally, with notes (4D-LETKF)

Forecast step:

Analysis step: construct

x

n,k

b= M

nx

n!1,k

a( )

Xb= x

1

b ! xb | ... | xK

b ! xb"# $%;

yi

b= H (x

i

b); Y

n

b= y

1

b ! yb | ... | yK

b ! yb"# $%

Globally:

Notes: Pb= X

bX

bT

Yb! HX

b , but it is computed nonlinearly

4D-LETKF is done by simply concatenating in the vertical thevectors and matrix valid at different times in theassimilation window. The rest is the same.

yo, y

b

Yb

Repeat: LETKFRepeat: LETKF, locally, with notes, locally, with notesLocally: Choose for each grid point the observations to be used, andcompute the local analysis error covariance and perturbations inensemble space:

Analysis mean in ensemble space:and add to to get the analysis ensemble in ensemble space

The new ensemble analyses in model space are the columns of . Gathering the grid point analyses forms the new

global analyses. Note that the the output of the LETKF are analysisweights and perturbation analysis matrices of weights . Theseweights multiply the ensemble forecasts.

!Pa= K !1( )I + YbT

R!1Y

b"# $%!1; W

a= [(K !1) !Pa ]1/2

Xn

a= X

n

bW

a+ x

b

wa= !P

aY

bTR

!1(yo! y

b )

Wa

wa

Wa

!Pa= K !1( )I + YbT

R!1Y

b"# $%!1

Notes: is Pa!1

= Pb!1

+HTR

!1H"# $%

in ensemble space, where is the unit matrix/(K-1)! !Pb!1

= I / (K !1)

Note that here, the columns of are the full analysisensemble members (not the perturbations).

Xa

The 4D-LETKF produces an analysis in terms ofweights of the ensemble forecast members at theanalysis time tn, giving the trajectory that best fits allthe observations in the assimilation window.

analysis time weightsImplication of getting analysis weights

The 4D-LETKF produces an analysis in terms ofweights of the ensemble forecast members at theanalysis time tn, giving the trajectory that best fits allthe observations in the assimilation window.

analysis time weightsA linear comb. of trajectories is ~a trajectory. If it is closeto the truth at the end it should be close to the truththroughout the trajectory (neglecting model errors).

No-cost LETKF smoother ( ): apply at tn-1 the sameweights found optimal at tn. It works for 3D- or 4D-LETKF

The no-cost smoother makes possible: Outer loop (like in 4D-Var) “Running in place” (faster spin-up) Use of future data in reanalysis Ability to use longer windows

No-cost LETKF smoothertested on a QG model: It works!

“Smoother” reanalysis

LETKF Analysisxna= xn

f+ Xn

fwn

aLETKF analysis

at time n

Smoother analysis at time n-1

!xn!1a

= xn!1f

+ Xn!1

fwn

a

This very simple smoother allows us to go backand forth in time within an assimilation window:it allows assimilation of future data in reanalysis

Example: forecast sensitivity to observationsLiu, Hong and Kalnay, in preparation

The only difference between and is the assimilation of observations at 00hr.

Observation impact on the reduction of forecast error:

(Adapted from Langlandand Baker, 2004)

et |0 = xt |0f! xt

a

et |0

et |!6

!e2= (e

t |0

Tet |0" e

t |"6

Tet |"6)

analysis t

et |!6

et |0

-6hr 00hr

OBS.

Example: forecast sensitivity to observations

!e2= (et |0

Tet |0 " et |"6

Tet |"6

) = (et |0T " et |"6

T)(et |0 + et |"6

)

= (xt |0f " xt |"6

f)T

(et |0 + et |"6)

= M(x0

a " x0|"6

b)#$ %&

T

(et |0 + et |"6), so that

!e2= MK(y " H (x

0|"6

b)#$ %&

T

(et |0 + et |"6)

Langland and Baker (2004) solve this with the adjoint:!e2 = MK(y " H (x

0|"6b)#$ %&

T

(et |0+ e

t |"6 )

= (y " H (x0|"6b)#$ %&

T

KTM

T(e

t |0+ e

t |"6 )

This requires the adjoint of the model and of the dataassimilation system (Langland and Baker, 2004)

Example: forecast sensitivity to observations

Langland and Baker (2004):

!e2 = MK(y " H (x0|"6b)#$ %&

T

(et |0+ e

t |"6 )

= (y " H (x0|"6b)#$ %&

T

KTM

T(e

t |0+ e

t |"6 )

With EnKF we can use the original equation without “adjointing”:

!e2 = MK(y " H (x0|"6b)#$ %&

T

(et |0 + et |"6 )

= Xt |0

f Y0

aTR"1(y " H (x

0|"6b)#$ %&

T

(et |0 + et |"6 )

K = PaH

TR

!1= X

aX

aTH

TR

!1 so that we can write

MK =MXa(X

aTH

T)R

!1= Xt |0

fY

aTR

!1

This is a simple product using the available nonlinear forecastensemble and

so thatRecall that

Xt |0

fY0

aT= (HP

a)T

Test ability to detect the poor quality observation onthe Lorenz 40 variable model

Like adjoint method, ensemble sensitivity method can detect the observationpoor quality (11th observation location)

The ensemble sensitivity method has a stronger signal when the observation hasnegative impact on the forecast.

Observation impact from LB (red) and from ensemble sensitivity method (green)

Larger random error Biased observation case

Test ability to detect poor quality observation fordifferent forecast lengths

After 2-days theadjoint has the wrongsensitivity sign!

The ensemblesensitivity method hasa strong signal evenafter forecast error hassaturated!

Larger random error Biased observation case

2 days

5 days

20 days

How can we possibly detect bad observations evenafter all skill is lost???

After 20-days there is noforecast skill but theensemble sensitivity stilldetects the wrongobservation.

The ensemble sensitivityis based on the assumptionthat the analysis weightscan be used in theforecasts. This is accurateeven after forecast errorhas saturated (triangles).

As a result we canidentify a bad observationeven after forecast skill islost.

20 days

Mean Square Error of the -6hr weighted forecasts (diamonds),MSE of the 0hr ensemble mean (circles) and MS Differencebetween ensemble mean and weighted forecasts (triangles).

Error made by using the -6hrweights in the forecasts

Summary

• EnKF is simple, does not require linear tangentmodel or adjoint of either the model M or the forwardoperator H

• There are two types of EnKF: perturbed observations(e.g., Evensen, Houtekamer), and Square Root Filter(e.g., Whitaker and Hamill, Anderson, Bishop). Theseassimilate one obs at a time.

• The LETKF (Hunt et al.) is a SQRF done locally inspace: the analysis and analysis perturbations areweighted averages of the ensemble forecasts.

• This makes possible a “no cost smoother”.• EnKF calculates exactly Pa (the Hessian!), and the

Kalman gain matrix K. This makes possible tocompute longer forecasts sensitivity to observations.