Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Formant Based Synthesizer

Qin Yan

Communication & Multimedia Signal Processing Group

Dept of Electronic & Computer Engineering, Brunel University

28 July, 2004

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Main Progress

• Kalman filter based formant tracking system in clean speech

• Speech Synthesis via formant tracks

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Formant Candidate Estimation

LP PoleAnalysis

Kalman Filter

NoisySpeech

Restored Formant & Bandwidth

tracks

Formant Candidate

Estimation

Kalman Filter

Vowel/Consonant

Classification

Voiced?Yes

No

Noise Model

LP-basedSpectral

Subtraction

VAD

Pos.& neg. PolesReconstruction

LP Spectrum Reconstruction

Residual

Real Pole

Speech Reconstruction

Enhanced Speech

Formant Track Restoration Module

Formant based Speech Enhancement System

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

ConfidenceScore

Calculation

LP Pole Analysis

Kalman Filter

Clean Speech

Formant & Bandwidth

tracks

Real Poles

Speech Reconstruction

Output Speech

Residual

Confidence Score

Calculation

Kalman Filter

Positive Poles

Vowel/Consonant

ClassificationVowel?

Yes

No

Formant Candidate

Interpolation

Formant Candidate

Interpolation

Speech Synthesis System

Kalman Filter based Formant Tracker for Clean Speech

Speech Synthesizer via Formant Tracks

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Vowel/Consonant Classification

• Discriminant feature used is the slope coefficient of a 1st order polynomial of LP spectrum;• Positive slope: Consonant; Negative slope: Vowel

Confidence Scores of Formant Candidates

• The score quantifies how significant a pole is • Score for Vowels: Mag(m) /BW(m)• Score for Consonant: m*Mag(m) / BW(m)• The candidate with highest score is interpolated with the closest formant candidate. The rest of formant candidates are sorted in ascending order.• Interpolation function:

Where W(m) is the weights

Parallel Kalman Filters

• Two kalman filters: One for vowel segments, the other for consonant segments.

Kalman Filter based Formant Track in Clean Speech

22 )),1(),((*),(

1)(

ktFkmPkmPmW

FBW

Nk

m

N

mF

mW

mWkmPk

1

1

)(

)(*),(

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Performance

Red : Formant tracks from 2D-HMM; Green : Formant tracks from Kalman filter

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Speech Synthesis via Formant tracks

Pos.& neg. PolesReconstruction

Noisy Speech

Real Pole

Speech Reconstruction

Enhanced Speech

Residual

RestoredFormant track

LP Pole Analysis

Real poles are included to adjust the slope of LP spectrum

LP order = Number of formant tracks + 1

HMM based Formant tracks

Kalman Filter based Formant Tracks

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

The End

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Performance Evaluation

-1

4

9

14

19

24

0 5 10 15 20Global SNR(dB)

snris1kl/mean(kl)IS2

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

ConfidenceScore

Calculation

LP Pole Analysis

Kalman Filter

Clean Speech

Formant & Bandwidth

tracks

Real Poles

Speech Reconstruction

Output Speech

Residual

Confidence Score

Calculation

Kalman Filter

Positive Poles

Vowel/Consonant

ClassificationVowel?

Yes

No

Formant Candidate

Interpolation

Formant Candidate

Interpolation

Kalman Filter based Formant Tracker for Clean Speech

Speech Synthesizer via Formant Tracks

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

SignificanceScore

Calculation

LP PoleAnalysis

Kalman Filter

NoisySpeech

Formant & Bandwidth

tracks

Significance Score

Calculation

Kalman Filter

Vowel/Consonant

Classification

Voiced? Yes

No

Formant Candidate

Interpolation

Formant Candidate

Interpolation

Noise Model

LP-basedSpectral

Subtraction

VAD

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing Source

Speech

Cepstral Feature Analysis

LP Pole Analysis

Speech HMMs

Training

FormantFeatures

Extraction

Speech Labelling &Segmentation

Formant HMMsTraining

Formant candidates classification

Formant Candidates Interpolation

Formant Tracks

State-dependentKalman Filter

R

Fi , BWi

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

LP PoleAnalysis

NoisySpeech

Restored Formant & Bandwidth

tracks

Formant Candidate

Estimation

Kalman Filter

Vowel/Consonant

ClassificationLP ModelOf Noise

LP-Analysisand

LP-Spectral Subtraction

VAD

Pos.& neg. PolesReconstruction

LP Spectrum Reconstruction

ResidualSpeech

ReconstructionEnhanced

Speech

Formant Track Restoration Module

Co

mm

un

icat

ion

s &

Mu

ltim

edia

Sig

nal

Pro

cess

ing

Formant Candidate Estimation

LP PoleAnalysis

Kalman Filter

NoisySpeech

Restored Formant & Bandwidth tracks

Formant Candidate

Estimation

Kalman Filter

Vowel/Consonant

Classification

Voiced?Yes

No

Noise Model

LP-basedSpectral

Subtraction

VAD