Signal Reconstruction from its Spectrogram

Radu Balan

IMAHA 2010, Northern Illinois University, April 24, 2010

2/23

Overview

1. Problem formulation

2. Reconstruction from absolute value of frame coefficients

3. Our approach– Embedding into the Hilbert-Schmidt space– Discrete Gabor multipliers– Quadratic reconstruction

4. Numerical example

3/23

1. Problem formulation

• Typical signal processing “pipeline”:

Analysis Processing SynthesisIn Out

Features:Relative low complexity O(Nlog(N))On-line version if possible

4/23

<·,gi>x

Analysis Synthesis

Ii

iigc ˆc y

Hx

ighc

Ilc

,

)(

i

2

Iiiigcy

Hy

The Analysis/Synthesis Components:

Example: Short-Time Fourier Transform

fkfk gxc ,, , )()( /2, kbtgetg Fiftfk

)(

10,);,(2 ZlH

ZZFfZkfkI F

)()( 2, kbtgetg kbtiffk

5/23

*

=

fft fft

=

*

Data frame index (k)

f

ck,0

ck,F-1ck+1,F-1

ck+1,0

g(t)

x(t+kb:t+kb+M-1) x(t+kb+M:t+kb+2M-1)

x(t+kb)g(t) x(t+(k+1)b)g(t)

6/23

*

=

ifft ifft

=*

ĝ(t)

ck,0

ck,F-1ck+1,F-1

ck+1,0

+

7/23

Problem: Given the Short-Time Fourier Amplitudes (STFA):

we want an efficient reconstruction algorithm:Reduced computational complexityOn-line (“on-the-fly”) processing

1

/2,, )()(,

Mkb

kbt

Fiftfkfk ekbtgtxgxd

|.| Reconstructionck,f dk,f x

8/23

• Where is this problem important:– Speech enhancement– Speech separation– Old recording processing

9/23

• Setup: – H=En , where E=R or E=C

– F={f1,f2,...,fm} a spanning set of m>n vectors

• Consider the map:

• Problem 1: When is N injective?• Problem 2: Assume N is injective, Given c=N(x)

construct a vector y equivalent to x (that is, invert N up to a constant phase factor)

2. Reconstruction from absolute value of frame coefficients

mkk

mn fxxNREN

1

,)( , ~/:

1||scalar somefor ,~ zzyxyx

10/23

Theorem [R.B.,Casazza, Edidin, ACHA(2006)]

For E = R :• if m 2n-1, and a generic frame set F, then N is

injective;• if m2n-2 then for any set F, N cannot be injective;• N is injective iff for any subset JF either J or F\J

spans Rn.• if any n-element subset of F is linearly independent,

then N is injective; for m=2n-1 this is a necessary and sufficient condition.

mkk

mn fxxNRRN

1

,)( , ~/:

1scalar somefor ,~ zzyxyx

11/23

Theorem [R.B.,Casazza, Edidin, ACHA(2006)]

For E = C :• if m 4n-2, and a generic frame set F, then N is

injective.• if m2n and a generic frame set F, then the set of

points in Cn where N fails to be injective is thin (its complement has dense interior).

mkk

mn fxxNRCN

1

,)( , ~/:

1||scalar somefor ,~ zzyxyx

12/23

3. Our approach

• First observation:

fkfkgx

HSgxgxfkfk

ggyyKxxyyK

KKKKtrgxd

fk

fkfk

,,

*2

,2,

,)( , ,)(

,,

,

,,

Hilbert-Schmidt

Signal space: l2(Z)

x KxK

nonlinearembedding

Kgk,f

E=span{Kgk,f}

Hilbert-Schmidt: HS(l2(Z))

FZZFfZkfkIZlH 10,);,( , )(2

Ifkkbtgetg kbtiffk ),( , )()( 2

,Recall:

13/23

• Assume {Kgk,f} form a frame for its span, E. Then the projection PE can be written as:

where {Qk,f} is the canonical dual of {Kgk,f} .

fk

HSfkgE QKP

fk ,,

,,

)(

,)(

,

,,

1,

,

fk

fkfk

gfk

fkg

HSg

KSQ

KKXXSX

Frame operator

14/23

• Second observation:since:

it follows:

)()(: , )()(: where /2

,

bthtThhTthetMhhM

gTMgFit

kffk

** : , : where

,

TXTXXMXMXX

KK gkf

g fk

15/23

• However:

• Explicitely:

0,0,

S and

QQ

SSSfk

fk

kbtkbt

Fttif

ttfk QeQ

21

21

21 ,0,0/)(2

,,

16/23

Short digression: Gabor Multipliers

• Goes back to Weyl, Klauder, Daubechies• More recently: Feichtinger (2000), Benedetto-

Pfander (2006), Dörfler-Toressani (2008)

LatticeggmmG

dggm

,)( :Multiplierabor

,m : Multiplier STFT 2

Theorem [F’00] Assume {g , Lattice} is a frame for L2(R).Then the following are equivalent:1. {<.,g>g,Lattice} is a frame for its span, in HS(L2(R));2. {<.,g>g,Lattice} is a Riesz basis for its span, in HS(L2(R));3. The function H does not vanish,

)( , ,)()(2

LatticeDualGroupeggeeHLattice

17/23

• Return to our setting. Let

Theorem Assume {gk,f}(k,f)ZxZF is a frame for l2(Z).

Then

1. is a frame for its span in HS(l2(Z)) iff for each mZF, H(,m) either vanishes identically in , or it is never zero;

2. is a Riesz basis for its span in HS(l2(Z)) iff for each mZF and , H(,m) is never zero.

Zk

fkZf

F

mfki

ggemHF

2

,

2

,),(

Fg ZZfkKfk

),(;,

18/23

• Third observation. Under the following settings:– For translation step b=1;– For window support supp(g)={0,1,2,...,L-1}– For F2L

• The span of is the set

of 2L-1 diagonal band matrices.

Fg ZZfkKfk

),(;,

g

LgLgg

ggg

Lggggg

K g

000000

0)1()1()0(0

00

0)1()1()0(0

0)1()0()1()0()0(0

000000

2

2

2

19/23

• The reproducing condition (i.e. of the projection onto E) implies that Q must satisfy:

bandttXQgXgfk

ttttfkfkfk 21,

,,,,, , and X allfor , ,2121

By working out this condition we obtain:

1

02

2

,0,0)()(

1

d

epgpg

e

FQ

p

ip

it

tt

20/23

• The fourth observation:We are able now to reconstruct up to L-1 diagonals of Kx.

This means we can estimate

11

2,,, Lttttt xxxxx

Assuming we already estimated xs for s<t,we estimate xt by a minimization problem:

2

,

2

1,11,

2ˆˆmin JttxJtJttxtttxx KxxwKxxwKx

for some JL-1 and weights w1,...,wJ.

Remark: This algorithm is similar to Nawab, Quatieri, Lim [’83]IEEE paper.

21/23

Reconstruction Scheme

• Putting all blocks together we get:

IFFT

|ck,0|2

|ck,F-1|2

W0

WL-1

0ˆtz

1ˆ Ltz

LeastSquareSolver

ttx̂

Stage 1 Stage 2

t

t

ttzQzW

,0,01)(

22/23

3. Numerical Example

23/23

Conclusions

All is well but ...

• For nice analysis windows (Hamming, Hanning, gaussian) the set {Kgk,f} DOES NOT form a frame for its span! The lower frame bound is 0. This is the (main) reason for the observed numerical instability!

• Solution: Regularization.