Part 3 Vector Quantization and Mixture Density Model

CSE717, SPRING 2008

CUBS, Univ at Buffalo

Vector Quantization

Quantization Represents continuous range of values by a

set of discrete values Example: floating-point representation of real

numbers in computer Vector Quantization

Represent a data space (vector space) by discrete set of vectors

Vector Quantizer

A mapping from vector space onto a finite subset of the vector space

Y = y1,y2,…,yN finite subset of IRk, referred to as the codebook of Q

Q is usually determined by training data

kk YQ IRIR:

Q

1y2y

3y

Partition of Vector Quantizer

The vector space is partitioned into N cells by the vector quantizer

})(|IR{)(1i

kii yxQxyQR

Q

1y2y

3y

1R 2R

3R

Properties of Partition

Vector quantizer Q defines a complete and disjoint partition of IRk into R1,R2,…,RN

)(, and IR1

jijiRRR jik

N

ii

Quantization Error

Quantization Error for single vector x

is a suitable distance measure Overall Quantization Error

))(,()|( xQxdQx

)|( d

k

dxxpxQxd

XQXdEQXEQ

X

IR

)())(,(

))}(,({)}|({)(

Nearest-Neighbor Condition

The minimum quantization error of a given codebook Y is given by partition

}),(min),(|{Yy

ii yxdyxdxR

x

y1 y2

y3

1

321 ),(),(),(

Rx

yxdyxdyxd

Centroid Condition

Centroid of a cell Ri

is minimized by choosing

as the codebook For Euclidean distance d

}|),({minarg)(cent iRy

i RXyXdERi

i

i

RRXXii dxxxpRXXER )(}|{)(cent |

)}(cent),...,({cent 1 NRRY

)(Q

Centroid

Vector Quantizer Design – General Steps

1. Determine initial codebook Y0

2. Adjust partition of sample data for the current codebook Ym using nearest-neighbor condition

3. Update the codebook Ym →Ym+1 using centroid condition

4. Check a certain condition of convergence. If it converges, return the current codebook Ym+1; otherwise go to step 2

my1

my2

my3

mR1mR2

mR3

},,{ 03

02

01 yyyY

111

mm yy1

22 mm yy

133

mm yy

mR1mR2

mR3

Converge?Y

N

Lloyd’s Algorithm

Lloyd’s Algorithm (Cont)

LBG Algorithm

LBG Algorithm (Cont)

k-Means Algorithm

k-Means Algorithm (Cont)

Mixture Density Model

A mixture model of N random variables X1,…,XN is defined as follows:

is a random variables defined on N labels

},...,1{ NC

N

iiXCiIX

1

),(

function Impulse otherwise ,0

if ,1),(

ji

jiI

,...2,1 21 XXCXXC

Mixture Density Model

Suppose the p.d.f.’s of X1,…,XN are

and

then

)()(

1

xpxpiX

N

iiX

)(),...,(1

xpxpNXX

iiC )Pr(

Example: Gaussian Mixture Model of Two Components

)1Pr(},1,0{),1( CI

),(~),,(~ 2222

2111 NXNX

21 )1( XXX

Histogram of samples Mixture Density

Estimation of Gaussian Mixture Model ML Estimation (Value X and label are given)

Samples in the format of

(-0.39, 0), (0.12, 0), (0.94, 1), (1.67, 0), (1.76, 1), …

S1 (Subset of ): (0.94, 1), (1.76, 1), …

S2 (Subset of ): (-0.39, 0), (0.12, 0), (1.67, 0), …

211

Estimation ML

1 ˆ,ˆ S 222

Estimation ML

2 ˆ,ˆ S

|)||/(|||ˆ 211 SSS

21 )1( XXX

),( kkx

1k 0k

]ˆ,ˆ,ˆ,ˆ,ˆ[ˆ 222

211 θ

Estimation of Gaussian Mixture Model EM Algorithm (Value X is given, label is unknown)

1. Choose initial values of

2. E-Step:

For each sample xk, label is missing. But we can estimate

using its expected value

Samples in the format of ; is missing

(-0.39), (0.12), (0.94), (1.67), (1.76), …

)ˆ,ˆ;()ˆ1()ˆ,ˆ;(ˆ

)ˆ,ˆ;(ˆ

)ˆ,|1Pr(

}ˆ,|{

222

211

211

21

1

kXkX

kX

kk

kk

xpxp

xp

x

xE

θ

θ

k

k

]ˆ,ˆ,ˆ,ˆ,ˆ[ˆ 222

211 θ

x1

x2

)ˆ,|1Pr()ˆ,|1Pr( 2211 θθ xx

)( kx k

Estimation of Gaussian Mixture Model EM Algorithm (Value X is given, label is unknown)

3. M-Step:

We can estimate again using the labels

estimated in the E-Step: }ˆ,|{ˆ θkkk xE

kk

kkk

kk

kkk xx

ˆ

)ˆ(ˆ

ˆ,ˆ

ˆ

ˆ

21

11

]ˆ,ˆ,ˆ,ˆ,ˆ[ˆ 222

211 θ

kk

kkk

kk

kkk xx

)ˆ1(

)ˆ)(ˆ1(

ˆ,)ˆ1(

)ˆ1(

ˆ

22

22

))ˆ1(ˆ/(ˆˆ k

kk

kk

k

Estimation of Gaussian Mixture Model EM Algorithm (Value X is given, label is unknown)

4. Termination:

The log likelihood of n samples

At the end of m-th iteration, if

terminate; otherwise go to step 2 (the E-Step).

n

kkXkXn xpxpxxl

1

222

2111 )]ˆ,ˆ;()ˆ1()ˆ,ˆ;(ˆlog[),...,(

22

thrxxlxxlxxl nm

nm

nm ),...,(/|),...,(),...,(| 111

1