The Variational EM Algorithm for On-line Identification of Extended

The Variational EM Algorithm for On-lineIdentification of Extended AR Models

Václav Šmídl Anthony Quinn

UTIA, Czech Academy of Science Trinity College Dublin, Ireland

March 22, ICASSP 05

Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 1 / 1

Outline


Univariate AR model

AR model:

dt = −a1dt−1 − a2dt−2 − . . .− aqdt−q + σet , et ∼ N (0, 1)

with

AR parameters a = [a1, . . . , aq]′, and σ ,

regressor x t = [dt−1, . . . , dt−q]′,

history Dt = [d1, d2, . . . , dt ] ,

Observation model:

f (dt |a, σ, Dt−1) = f (dt |a, σ, x t) = N(−a′x t , σ

2) .


Recursive Bayesian Inference

Conjugacy:

f (a, σ|Dt) ∝ f (dt |a, σ, x t) f (a, σ|Dt−1) ,

same functional form

f (a, σ|Dt) = f (a, σ|s (Dt))

st = s (Dt) is time-invariant function generating sufficient statistics.

BTf (a, σ|Dt−1) f (a, σ|Dt )

f (dt |a, σ, Dt−1)

st−1 st

dt

update



Conjugacy:



f (a, σ|Dt) = f (a, σ|s (Dt))



f (dt |a, σ, Dt−1)

st−1 st

dt

update



Conjugacy:



f (a, σ|Dt) = f (a, σ|s (Dt))



f (dt |a, σ, Dt−1)

st−1 st

dt

update


Recursive Bayesian Inference of AR parameters

Observation model for AR:

f (dt |a, σ, x t) = N(−a′x t , σ

2) ,

Conjugate distribution is Normal-inverse-Gamma, N iG:

f (a, σ|Dt) = N iG (Vt , νt)

sufficient statistics, st = {Vt , νt}:

Vt = Vt−1 +

[dt

x t

] [dt x t

]=

[vd,d ;t v ′

x ,d ;tv x ,d ;t Vx ,x ;t

],

νt = νt−1 + 1,

Moments:Ef (a|Dt ) [a] = V−1

x ,x ;tv x ,d ;t .


Extending AR model

AR Extended AR (EAR)

etσ

a2

a1

aq

z−1

z−2

z−q

x2;t

x1;t

xq;t

dtdtet

σg−1

0

g1

g2

gq

a1

a2

aq

yt

x1;t

x2;t

xq;t

z−1

a g dtetσ yt

x t

Posterior:

f (a, σ|Dt) = N iG (Vt , νt) ,Vt = Vt−1 + z tz ′

tνt = νt−1 + 1

z t = [dt , dt−1, dt−2, . . . , dt−q]′ z t = g ′ (Dt) =[g0 (Dt) , g1 (Dt−1) , . . . , gq (Dt−1)]

′


Mixture-based Extension of AR (MEAR)

Relaxing the assumption of known time-variant transformation.Treating g as time-variant discrete random variable:

g → g t ∈ {g1, g2, . . . , g c} ⇔ l t ∈ {l 1, l 2, . . . , l c}

The observation model is

f (dt |a, σ, l t , Dt−1) =∏c

i=1 f (dt |a, σ, x i;t , g i)li;t ,

f (li;t) = αi ,∑c

i=1αi = 1, i = 1, . . . , c,

f (dt |a, σ, Dt−1) =∑c

i=1 αi f (dt |a, σ, x i;t , g i),

EAR MEAR

a g dtetσ yt

x t

g2AR

g c

etσ

dt

α

g1


Conjugacy Lost and Conjugacy Regained

Bayes’ rule:

f (a, σ, l t |Dt) ∝ f (dt |a, σ, l t , Dt−1) f (l t) f (a, σ|Dt−1) ,

Conditional independence approximation:

f (a, σ, l t |Dt) ≈ f (a, σ, l t |Dt) = f (a, σ|Dt) f (l t |Dt) ,

Then:

f (a, σ|Dt) f (l t |Dt) ∝ f (dt |a, σ, l t , Dt−1) f (l t) f (a, σ|Dt−1) .

BTf (a, σ|Dt−1)

Pf (dt |a, σ, l t , Dt−1)

f (a, σ|Dt )Pf (a, σ, l t |Dt )

approx.


Restoring conjugacy in mixture-type models

Traditionally, conditional independence was achieved heuristically:

f (a, σ, l t |Dt) ≈ f (a, σ|Dt) f (l t |Dt) ,

f (l t |Dt) =∫

a,σf (a, σ, l t |Dt) dadσ,

f (a, σ|Dt) = f(

a, σ|Dt , l t

).

Quasi-Bayes: l t = Ef (l t |Dt )[l t ] ,

Viterbi-like: l t = arg maxl t f (l t |Dt) .

We favour free-form functional optimization.


The Variational Bayes (VB) method

Two ingredients:

1 Conditional independence assumption:

f (a, σ, l t |Dt) = f1 (a, σ|Dt) f2 (l t |Dt) ,

2 Minimization of the KL divergence,

f1 (·) , f2 (·) = arg minf1 ,f2

KL(

f (a, σ|Dt) f (l t |Dt) ||f (a, σ, l t |Dt))

,

Necessary condition for minimum:

f1 (a, σ|Dt) ∝ exp(

Ef2(l t |Dt )[ln (f (a, σ, l t , Dt))]

),

f2 (l t |Dt) ∝ exp(

Ef1(a,σ|Dt )[ln (f (a, σ, l t , Dt))]

).

Solved iteratively via the Variational EM (VEM) algorithm.



Two ingredients:




f1 (·) , f2 (·) = arg minf1 ,f2

KL(


,




),



).




Two ingredients:




f1 (·) , f2 (·) = arg minf1 ,f2

KL(


,




),



).



VB-observation model


Ef (l t |Dt )

[ln

(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)

)]),

f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)

VB the generates a partial VB-observation model for each variable.




Ef (l t |Dt )

[ln


)]),






Ef (l t |Dt )

[ln


)]),




f1(dt |a, σ, Dt−1)

BTf (l t |Dt−1) f (l t |Dt )

f2(dt |l t , Dt−1)




Ef (l t |Dt )

[ln


)]),




f1(dt |a, σ, Dt−1)

BTf (l t |Dt−1) f (l t |Dt )

f2(dt |l t , Dt−1)

Observation models interact with posteriors via moments.


VB for MEAR

VB-posterior distributions:

f (a, σ|Dt) = N iG (Vt , νt) ,

f (lt |Dt) = Mu (w t) ,

with statistics expressed implicitly:

Vt = Vt−1 +c∑

i=1

wi;tz i;tz ′i;t

νt = νt−1 + 1,

wi;t ∝ |Ji;t |exp[− 1

2σ z ′j;t

[−1, a′

]′ [−1, a′

]z j;t (1)

− 12 z ′

j;tV−1x ,x ;tz j;t

],

Solved iteratively.Note: Statistics Vt is updated by rank-c structure.


VB for MEAR



f (lt |Dt) = Mu (w t) ,


Vt = Vt−1 +c∑

i=1

wi;tz i;tz ′i;t

νt = νt−1 + 1,


2σ z ′j;t

[−1, a′

]′ [−1, a′

]z j;t (1)

− 12 z ′


],



VB for MEAR



f (lt |Dt) = Mu (w t) ,


Vt = Vt−1 +c∑

i=1

wi;tz i;tz ′i;t

νt = νt−1 + 1,


2σ z ′j;t

[−1, a′

]′ [−1, a′

]z j;t (1)

− 12 z ′


],



MEAR with time-variant weights

Consider Markov model of label evolution:

f (lt |lt−1, Q) =∏c

i,j=1 q li;t lj;t−1

i,j ,

with unknown transition matrix Q.Conditional independence is imposed between the weights:

f (a, σ, lt , lt−1|Dt) ≈ f (a, σ|Dt) f (lt |Dt) f (lt−1|Dt) ,

Posterior distributions are again Multinomial on both labels, lt , lt−1,and Dirichlet on Q.


MEAR with time-variant parameters

Since conjugacy was restored using VB, the technique of forgettingcan be used for dealing with time-variant parameters.

f (at , σt |Dt−1) ∝[f (at−1, σt−1|Dt−1)

]φ

at−1 → at

σt−1 → σt

[f (at , σt)

]1−φ

In combination with time-variant weights we use two approximationsof time-update step of stochastic filtering: (i) VB for weights, (ii)forgetting for a and σ.

BT f (at , σt |Dt )

f1(dt |a, σ, Dt−1)

φ

f (at , σt )

f (at−1, σt−1|Dt−1)

f (at , σt |Dt−1)


Outlier corrupted AR

Outlier:

yt = −ax t + σet ,

dt = yt + ξt , ξt =

{het

0

Transformations:g1 : z t = [dt , . . . , dt−q]′,g2 :z t = 1

h [dt , . . . , dt−q]′,g i : i = 3 . . . qz t = [dt , . . . , yt−i , . . . dt−q]

′,

VB result

55 60 65 700

1

time (t )

w4,

t

0

1

w3,

t

0

1

w2,

t

0

1

w1,

t

component weights (first outlier)

60 70 80 90− 2

− 1

0

1

time (t )

Signals and reconstructions

QB result

55 60 65 700

1

time (t )

w4,

t

0

1

w3,

t

0

1

w2,

t

0

1

w1,

t

60 70 80 90− 2

− 1

0

1

time (t )


Burst noise corrupted AR

Burst noise:

yt = −ax t + σet ,

dt = yt + ξt , ξt ∼ N (0, h) ,

with unknown h.Transformations:g1 : z t = [dt , . . . , dt−q]′,g i : z t = 1

κi[dt , x i;t ],

where κi and x i;t are from Kalmanfilter.Experiment was performed withforgetting factor φ = 0.95.

1100 1200 1300 1400 1500

−0.4

−0.2

0

0.2

Unc

orru

pted

Speach segment 1

1100 1200 1300 1400 1500

−0.4

−0.2

0

0.2

Cor

rupt

ed

1100 1200 1300 1400 1500

−0.4

−0.2

0

0.2

VB

reco

nstru

ctio

n1100 1200 1300 1400 1500

−0.4

−0.2

0

0.2

QB

reco

nstru

ctio

n

1100 1200 1300 1400 1500

−0.4

−0.2

0

0.2

time (t)

VL

reco

nstru

ctio

n


Conclusion

Mixture-based extension of the AR for dealing with unknowntransformations was presented.

The Variational Bayes technique was used to achieve conjugateupdate of parameter statistics

Applied to reconstruction of speech corrupted by burst noise.


Spanning

The chosen transformation (filter-bank) forms nodes of a simplex ofpossible dyads. Update by a linear combination of these dyads allowsus to span interior of the simplex.

desired behaviour undesired behaviourg2g1

g3

g2g1

g3

This should be remembered when designing a filter-bank.


Computational flow

w tO.P.

w t−1

11−z−1

11−z−1

z−1

g2

z2;tO.P.

z2;t z′2;t

g c

zc;tO.P.

zc;t z′c;t

......

g1

z1;tO.P.

z1;t z′1;t

dt

w2;tEq. (1)

um

x

...

wc;t

...

w1;tEq. (1)

Eq. (1)

Φt

Vt

Reminiscent of multiple model approach. In our approach, however,we propagate statistics.


Date post:	12-Sep-2021
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

The Variational EM Algorithm for On-line Identification of Extended

Documents