Lecture 6 Scalar and Vector Quantization
Wen-Hsiao Peng, Ph.D
Multimedia Architecture and Processing Laboratory (MAPL)Department of Computer Science, National Chiao Tung University
May 2013
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 1 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantization
Lossy compression method
Reduce distinct output values to a smaller setMap an input value/vector to an approximated value/vector
Approaches
Scalar quant. �quantize each sample separately
Uniform vs. Non-uniformMSE vs. MAE vs. ....
Vector quant. �quantize a group of samples jointly
Uniform (lattice quantizer) vs. Non-uniformMSE vs. MAE vs. ....
Objectives1 Minimize distortion s.t. number of reconstruction levels2 Minimize distortion s.t. entropy rate constraint (or vice versa)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 2 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Example
Scalar quantization
Quantization levels L = 2R
Reconstruction values gl , l = 1, 2, ..., LBoundary values bl , l = 0, 1, ..., LQuantizer mapping function
Q(f ) = gl if f 2 Bl = [bl�1, bl )
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 3 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantizer Mapping Function
Q(f ) = gl if f 2 Bl Q(f ) =jf �fminq
k�q+ q
2+f min
Non-uniform Quantizer Uniform Quantizer
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 4 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Uniform vs. Non-uniform
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 5 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Midrise vs. Midtread Quantizers
Midrise uniform quantizer (even L)"0" is a decision boundary
Midtread uniform quantizer (odd L)"0" is a reconstruction levelBetter for small L in video/image coding
Midrise vs. Midtread (for symmetric distributions)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 6 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantizer Distortion
Dq = σ2q(granular )| {z }our focus
+ σ2q(overload )| {z }0 for bounded inputs
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 7 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantizer Distortion
Quantizer distortion Dq1
Dq = E (d(F ,Q(F )))
= ∑l2L
�Zf 2Bl
d(f ,Q(f ))p(f )df�
= ∑l2LP(Bl )
ZBld(f ,Q(f ))p(f jF 2 Bl )df| {z }
Dq,l
Example: d(F ,Q(F )) = (F �Q(F ))2
Dq = E (d(F ,Q(F )) = E�(F �Q(F ))2
�| {z }
MSE
= σ2q
where quant. error F �Q(F ) is usually of zero mean (as we shallshow later)
1Depend on distortion measure d(�) and source distribution p(f )Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 8 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Scalar Quantizer
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 9 / 61
Lecture 6 Scalar and Vector Quantization Uniform Scalar Quantizer
Uniform Scalar Quantizer
Equal distances betweenadjacent boundaries andreconstruction values
bl � bl�1 = gl � gl�1 = q
where
bl = fmin + l � q
gl =bl + bl�1
2 Q(f ) =�f � fminq
��q+q
2+f min| {z }
Closed�form
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 10 / 61
Lecture 6 Scalar and Vector Quantization Uniform Scalar Quantizer
Uniform Quantizer Optimized for Uniform Distribution
Uniform distribution
p(f ) =�1/B f 2 (fmin, fmax)0 otherwise
where B = fmax � fmin
Distortion measure
d(F ,Q(F )) � (F �Q(F ))2
Quantizer distortion
Dq = ∑l2LP(Bl )Dq,l = Dq,l =
q2
12= σ2f 2
�2R
SNR = 10 logσ2fDq
= (20 log 2)R = 6.02R
where
Dq,l =q2
12, q =
BL, L = 2R , signal variance σ2f = B
2/12
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 11 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
Minimum Mean Square Error Scalar Quantizer
Distortion measure
d(F ,Q(F )) � (F �Q(F ))2
Objective
(b�, g�) = arg minfb,gg
∑l2L
Z bl
bl�1(f � gl )2p(f )df
!| {z }
J (b,g)
whereb = (b0, b1, ..., bL)T , g = (g1, g2, ..., gL)T
Necessary conditions
rJ(b�, g�) = 0) (1) ∂J(b�, g�)/∂bl = 0(2) ∂J(b�, g�)/∂gl = 0
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 12 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
MMSE Scalar Quantizer
Solution
(1) ∂J(b�, g�)/∂bl = (b�l � g �l )2 p(b�l )� (b�l � g �l+1)2p(b�l ) = 0
(2) ∂J(b�, g�)/∂gl = �Z b�l
b�l�12(f � g �l )p(f )df = 0
Nearest-neighbor condition
(b�l � g �l )2 = (b�l � g �l+1)2 (1)
) b�l =g �l + g
�l+1
2) Bl = ff : d(f , gl ) � d(f , gl 0), 8l 0 6= lg
Centroid condition
g �l =1
P(Bl )
Z b�l
b�l�1f p(f )df = E (FjF 2 Bl )| {z }
Conditional Mean
(2)
Eqs. (1) & (2) generally DO NOT have a closed-form solutionWen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 13 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
Properties of MMSE Scalar Quantizer
SymbolsF : quantizer inputG: quantizer outputQ = F � G: quantization error
G is an unbiased estimator of FE (G) = E (F ),E (Q) = 0
G is orthogonal to (and uncorrelated with) Q, whereas F and Q arecorrelated
E (GQ) = 0,E (FQ) 6= 0Reduced signal variance
σ2G = σ2F � σ2Q
Equalized error contribution
P(Bl )Dq,l =DqL, 8l 2 L
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 14 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
MMSE Scalar Quantizers of Various Sources
All sources are with zero mean and unit variance2
Quantizer Design (y j= g j , xj = bj ) SNRU: Uniform, G: Gaussian, L: Laplacian, Γ: Gamma
2See Appendix for arbitrary PDFWen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 15 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer: High Rate Approximation
High rate approx.: p(f ) is approximately �at in all intervals Bl
Dq = ∑l2L
Z bl
bl�1(f � gl )2 p(f )|{z} df
' ∑l2Lp(gl )| {z }
Z bl
bl�1(f � gl )2df
= ∑l2L
P(Bl )∆l| {z }
Z bl
bl�1(f � gl )2df , where ∆l = bl � bl�1
Centroid condition
∂Dq(b�, g�)∂gl
= 0)Z b�l
b�l�1(f � g �l )df = 0) g �l =
b�l + b�l�1
2| {z }High Rate
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 16 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer: High Rate Approximation
Quantizer distortion
Dq ' ∑l2L
P(Bl )∆l
Z b�l
b�l�1(f � g �l )2df = ∑
l2LP(Bl )
∆2l12|{z}Dq,l
where ∆l = bl � bl�1Equalized error contribution
P(Bl )Dq,l =112P(Bl )"
∆2l#= constant
Special case: uniform quantizer and uniform source
P(Bl ) = P(Bl 0),∆l = ∆l 0 = ∆,Dq =∆2
12
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 17 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer: High Rate Approximation
Alternative expression (more useful)
Dq ' ∑l2LP(Bl )
∆2l12' ∑
l2L
pF (g �l )∆3l
12' σ2f 2
�2R ε2 (See Appendix)
where
ε2 =112
�Z fmax/σf
fmin/σf
3qpF 0(f )df
�3pF 0(f ) = σf pF (σf f ) is PDF of an unit variance source
Uniform quantizer/source vs. non-uniform quantizer/source
Uniform : SNR ' 10 log σ2fDq
= 10 log 22R = 6.02R
Non-uniform : SNR ' 10 log σ2fDq
= 6.02R � 20 log ε
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 18 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer of Various Sources
∆SNR = 6.02R (Uniform Source and Quantizer)�maxfSNRgSolid (PDF-opt. non-uniform quantizer) vs. Dashed (PDF-opt.uniform quantizer)
Uniform quant. becomes increasingly ine¢ cient with increasing RNon-uniform quant. attains a PDF-speci�c asymptote with increasingR
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 19 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
Companding: Compressing and Expanding
A technique for analyzingquantizers at high rate
Companding law c(x) de�nes Bl a
Interval intensity
dc(gl )dx
� 2xmaxL∆l
where
c(xmax) = xmax,c(0) = 0,c(xmin) = xmin
aQuantizers designed by c(x) is nolonger common
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 20 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
Companding: Compressing and Expanding
Distortion measure
d(F ,Q(F )) � (F �Q(F ))2
Quantizer distortion (in terms of c(x))
Dq ' ∑l2LP(Bl )
∆2l12
' x2max3L2 ∑
l2Lp(gl )∆l
�dc(gl )dx
��2' x2max
3L2
Z xmax
xminp(x)
�dc(x)dx
��2dx
Constraint Z xmax
xmin
dc(x)dx
dx = c(xmax)� c(xmin) = 2xmax
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 21 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer Design Using Companding
MMSE scalar quantizer
minx2max3L2
Z xmax
xminp(x)
�dc(x)dx
��2dx| {z }
Dq
s.t.Z xmax
xmin
dc(x)dx
dx = 2xmax
Lagrange multiplier (�nd dc�(x)/dx = g �)
g � = argmin�Z xmax
xminp(x) (g)�2 dx + λ
�Z xmax
xmingdx � 2xmax
��Optimal companding law c�(x)
g � =dc�(x)dx
=xmaxR xmax
03pp(x)dx
3qp(x)
D�q '23L2
�Z xmax
0
3qp(x)dx
�3= σ2f 2
�2R ε2
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 22 / 61
Lecture 6 Scalar and Vector Quantization MMAE Scalar Quantizer
Minimum Mean Absolute Error Quantizer
Distortion measure
d(F ,Q(F )) � jF �Q(F )j
Objective
(b�, g�) = arg minfb,gg
∑l2L
Z bl
bl�1jf � gl j p(f )df
!| {z }
J (b,g)
whereb = (b0, b1, ..., bL)T , g = (g1, g2, ..., gL)T
Necessary conditions
rJ(b�, g�) = 0) (1) ∂J(b�, g�)/∂bl = 0(2) ∂J(b�, g�)/∂gl = 0
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 23 / 61
Lecture 6 Scalar and Vector Quantization MMAE Scalar Quantizer
Minimum Mean Absolute Error Quantizer
Solution
(1) ∂J(b�, g�)/∂bl = jb�l � g �l j p(b�l )� jb�l � g �l+1j p(b�l ) = 0
(2) ∂J(b�, g�)/∂gl =Z g �l
b�l�1p(f )df �
Z b�l
g �lp(f )df = 0 (Appendix)
Nearest-neighbor condition (same as MMSE Quantizer)
jb�l � g �l j p(b�l )� jb�l � g �l+1j p(b�l ) = 0) b�l =g �l + g
�l+1
2
Generalized centroid conditionZ g �l
b�l�1p(f )df =
Z b�l
g �lp(f )df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 24 / 61
Lecture 6 Scalar and Vector Quantization Optimal Scalar Quantizer
Optimal Scalar Quantizer
Nearest-neighbor condition (same as MMSE case and obvious)
B�l = ff : d(f , gl ) � d(f , gl 0), 8l 0 6= lg
Generalized centroid condition3
g �l = argminglE (d(F , g)jF 2 Bl )
Remarks
Nearest-neighbor (distortion measure dependent) is NOT a¤ectedCentroid SHALL be adapted to distortion measureSource distribution MUST be knownReconstruction level is indexed by a �xed-length code
3Solution depends on distortion measureWen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 25 / 61
Lecture 6 Scalar and Vector Quantization Optimal Scalar Quantizer
Lloyd-Max Algorithm
Optimal quantizer design based on training data
Applicable when source distribution is unknownUse sample averages to replace expectations
Update reconstruction and boundary values iteratively1 Initialization
Choose initial reconstruction valuesCalculate initial boundary values (nearest neighbor)Calculate initial distortion
2 Iterations
Find new reconstructions (centroid, sample mean)Find new boundaries (nearest neighbor)Calculate new distortionRepeat till distortion converges
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 26 / 61
Lecture 6 Scalar and Vector Quantization Optimal Scalar Quantizer
Lloyd-Max Algorithm
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 27 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Entropy Constrained Optimal Scalar Quantizer
Minimize quantizer output entropy s.t. quantizer distortion in MSE
minHQ = �∑l2LP(Bl ) log2 P(Bl )| {z }
Quantizer output entropy
s.t. Dq =x2max3L2
Z xmax
xminp(x)
�dc(x)dx
��2dx| {z } = D
Quantizer distortion with L Levels (high rate approx.)
Parameters to be determined: L and dc (x )dx
Assumptions � (1) high rate, (2) the value of L is arbitrary
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 28 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Entropy Constrained Optimal Scalar Quantizer
Quantizer output entropy (in terms of c(x))
HQ = �∑l2LP(Bl ) log2 P(Bl ) (3)
' �∑l2Lp(gl )∆l log2 (p(gl )∆l )
' �Z xmax
xminp(x) log2 p(x)dx| {z }
h(X )=E (� log2 p(x )) di¤erential entropy
+ log2L
2xmax+Z xmax
xminp(x) log2
dc(x)dx
dx
Optimal quantizer is equivalent to �nding g � = dc�(x)/dx such that
minZ xmax
xminp(x) log2
dc(x)dx
dx + λ
Z xmax
xminp(x)
�dc(x)dx
��2dx �D 0
!Here we have assumed that L takes on its best value
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 29 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Entropy Constrained Optimal Scalar Quantizer
Uniform quantizer attains minimum output entropy regardless ofPDF.
dc�(x)dx
=p2λ ln 2 = constant
c�(x) = x given c(xmax) = xmax, c(0) = 0 (4)
Optimal output entropy (from Eqs. (3) & (4))
H�Q (x) = h(X )� log∆ = h(X )� 12log2 (12Dq)
Information-theoretical interpretation:
h(X ) : average information amount at quantizer inputlog2 ∆ = h(Q) : average information loss due to quantization (oraverage information conveyed by quantization error)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 30 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with Rate-Distortion Bound
Memoryless Non-Gaussian Sources (D(R) has NO closed-form)
LD(R)| {z }Lower Bound
� D(R) � D(R)G| {z }Upper Bound
where distortion D is in MSE and
LD(R) = (2πe)�12�2[R�h(X )],
LR(D) = h(X )� 12log2 2πeD
For a given distortion
H�Q (x)�L R(D) = 0.255bits
Lower e¢ ciency is caused by scalar quantization ) vectorquantization can be bene�cial even for memoryless sources
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 31 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with MMSE Scalar Quantizer
For a given distortion, maximum bit rate reduction
maxf∆Rg = 12log2
σ2fDq
ε2 �H�Q (x) =12log2
�12σ2f ε2
�� h(X )
For a given rate, maximum SNR gain
maxf∆SNRg = SNREC � SNRMMSE = 6.02maxf∆Rg dB
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 32 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with MMSE Scalar Quantizer
maxf∆RgRate reduction with entropy constrained quantizerA higher number of quantization levels
∆Rpdf �optRate reduction by entropy encoding MMSE quantizer outputsA �xed number of quantization levels
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 33 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with MMSE Scalar Quantizer
Gaussian Source(MMSE Scalar Quantizer L=8)
For a given L, HQ (entropy of quantizer output symbols) can bereduced at the cost of σ2q . How?For a given σ2q , the entropy can be reduced by increasing L; likewise,for a given output entropy HQ , σ2q can be reduced by increasing L.
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 34 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Summary
∆Rpdf �optNOT the best value achievable by entropy coding
maxf∆Rg = (3/2)∆Rpdf �optAchieved with uniform quantizer and more quantization levelsAdditional quantization levels are used for outer part of PDFFall short of R-D bound by 0.255bits (1.53dB) at high ratesApproach R-D bound at low rates
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 35 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Vector Quantizer
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 36 / 61
Lecture 6 Scalar and Vector Quantization Vector Quantization
Vector Quantization
Why Vector Quantization?
Samples in a block are correlatedSome block patterns are more likely to occur than others
Parameters
Quantization levels LReconstruction vectors (or Codewords) gl , l = 1, 2, ..., LPartition regions Bl , l = 1, 2, ..., LQuantizer mapping function
gl = Q(f) if f 2 Bl
wheref = [f1, f2, ..., fN ]
T , gl = [gl ;1, gl ;2, ..., gl ;N ]T
Bits/sample R = 1N dlog2 Le
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 37 / 61
Lecture 6 Scalar and Vector Quantization Nearest-Neighbor Vector Quantizer
Nearest-Neighbor Vector Quantizer
Quantized vector gl determined by Nearest-Neighbor criterionComplexity increases exponentially with vector size N
Operations NL (or N2NR ), storage NL (or N2NR )
Bl = ff 2 RN : dN (f, gl ) � dN (f, g0l ), 8l 0 6= lg
dN (f, gl ) =1N ∑N
n=1(fn � gl ;n)2
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 38 / 61
Lecture 6 Scalar and Vector Quantization Lattice Vector Quantizer
Lattice Vector Quantizer
Realization of Uniform Vector Quantizer
All partitions have same shape and size (good for uniform source)Closed-form quantizer mapping function (low complexity)
Reconstruction vectors gl are Lattice PointsLattice Λ (de�ned by a set of basis vectors fvng)
gl =N
∑n=1
ml ;nvn = [V]ml ,
whereml 2 ZN
[V] = [v1, v2, ..., vN ] is Lattice Generating Matrix
Determination of quantized vector
1 m =[V]�1f, where m 2 RN is a real index vector2 Evaluate distortions of bm = dme or bmc, where bm 2 ZN
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 39 / 61
Lecture 6 Scalar and Vector Quantization Lattice Vector Quantizer
Quantizer Distortion
Vector quantizer distortion
Dq = E (dN (�!F ,Q(�!F )))
=ZpN (f)dN (f,Q(f))df
=L
∑l=1
P(Bl )Dq,l
whereDq,l =
ZBlpN (f j
�!F 2 Bl )dN (f, gl )df
Example: lattice quantizer for uniform distribution
Dq =L
∑l=1
P(Bl )Dq,l = Dq,l and Dq,l =1
jdet[V]j
ZBldN (f, 0)df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 40 / 61
Lecture 6 Scalar and Vector Quantization Lattice Vector Quantizer
Lattice Vector Quantizer
Rectangular V1 =�1 00 1
�vs. Hexagonal V2 =
� p3/2 01/2 1
�
Better space packing makes VQ outperform SQ even with i.i.d source.
dN (f, gl ) = maxf2Bl
dN (f, gl )
Rectangular lattice: 2 uniform scalar quantizers
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 41 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Optimal Vector Quantizer
Objective
(B�1 ,B�2 , ...,B�L; g�1 , g�2 , ..., g�L)= argminE (d(
�!F ,Q(�!F )))
= argmin�Z
dN (f,Q(f))p(f)df�
= argmin�
∑P(Bl )ZBldN (f,Q(f))p(f j
�!F 2 Bl )df�
Optimal Solution1 Given
�g�1, g
�2, ..., g
�L
�, what would be optimal (B1,B2, ...,BL)?
2 Given�B�1 ,B�2 , ...,B�L
�, what would be optimal (g1, g2, ..., gL)?
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 42 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Optimal Vector Quantizer
Given (g�1 , g�2 , ..., g
�L), what would be optimal (B1,B2, ...,BL)?
Q�(f) = arg minQ (f)=fg�1 ,g�2 ,...,g�Lg
�ZdN (f,Q(f))p(f)df
�= arg min
Q (f)=fg�1 ,g�2 ,...,g�LgdN (f,Q(f))
B�l = ff : dN (f, g�l ) � dN (f, g�l 0), 8l 0 6= lg (Nearest Neighbor)
Given (B�1 ,B�2 , ...,B�L), what would be optimal (g1, g2, ..., gL)?
g�l = argming
ZB�ldN (f, g)p(f j
�!F 2 B�l )df
= argmingE (dN (
�!F , g)j�!F 2 B�l ) (Centroid)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 43 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Optimal Vector Quantizer
Nearest-Neighbor and Centroid are Necessary Conditions (Candidates)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 44 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Lloyd-Max Algorithm
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 45 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Lloyd-Max Algorithm
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 46 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Vector Quantizer
Entropy Constrained Vector Quantizer
ObjectiveminDq s.t. �∑
l2LP(Bl ) log2 P(Bl )| {z }Entropy Rate
= RN
Solution (Necessary Conditions)Lagrangian
(B�1 ,B�2 , ...,B�L; g�1, g�2, ..., g�L)
= argmin
Dq + λ�
�∑l2LP(Bl ) log2 P(Bl )� RN
!!| {z }
J (B1,B2,...,BL ;g1,g2,...,gL)4
λ� must be chosen such that
�∑l2LP(B�l ) log2 P(B�l ) = RN
4Lloyd-Max Algorithm is still applicable by replacing Dq with J(�)Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 47 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Vector Quantizer
Gaussian-Markov Source + 8-D VQ
http://www.stanford.edu/class/ee368b/Handouts/06-Quantization.pdf
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 48 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Vector Quantizer
Memoryless Laplacian + 8-D VQ
http://www.stanford.edu/class/ee368b/Handouts/06-Quantization.pdf
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 49 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Appendix
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 50 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Properties of MMSE Quantizer
G is an unbiased estimator of F .
E (G) = ∑l
P(Bl )gl
= ∑l
P(Bl )�Z
Blf p(f jF 2 Bl )df
�= ∑
l
ZBlf p(f )df
=ZBf p(f )df = E (F )
Q = F � G is of zero mean.
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 51 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Properties of MMSE Quantizer
G is orthogonal to (and uncorrelated with) Q.
E (GQ) = E (G(F � G))= EGEF jG((G(F � G)))= EG(GEF jG(F )� G2)= EG(G2 � G2)= 0
Note that E (FQ) = E (F (F � G)) = E (F 2)� E (FG) =E (F 2)� E (G2) 6= 0
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 52 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Properties of MMSE Quantizer
Reduced signal variance
σ2G = σ2F � σ2Q
By de�nition
σ2Q = E ((F � G)2)= E (
�(F�µF )�
�G�µG
��2)
= σ2F + σ2G � 2E ((F�µF )�G�µG
�)
= σ2F + σ2G � 2(E (FG)�µFµG)
= σ2F + σ2G � 2(E (G2)�µ2G)
= σ2F � σ2G
where µF = µG = E (G) and E (FG) =E (G2)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 53 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer of Various Sources
Given a source F 0= (F � µF ) /σF of zero mean and unit variance
P(F 0 � f 0)| {z }CF0 (f
0)
= P(F � σF f0 + µF )| {z }
CF (σF f 0+µF )
) ddf 0CF 0(f
0) =ddf 0CF (σF f
0 + µF )
) pF 0(f0) = σFpF (σF f
0 + µF ) or pF (f ) =1
σFpF 0(
f � µFσF
)
where
pF 0(f0) and pF (f ) are PDF of F 0 and F , respectively
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 54 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer of Various Sources
Optimal Scalar Quantizer for F 0
(1) ebl = egl+egl+12
(2) egl = E (F 0jF 0 2 eBl ) = R eblebl�1 f 0 pF0 (f 0)df 0R eblebl�1 pF0 (f 0)df 0Optimal Scalar Quantizer for F
(1) bl =gl+gl+1
2 = σFebl + µF
(2) gl = E (FjF 2 Bl ) =R blbl�1
f pF (f )dfR blbl�1
pF (f )df= σFegl + µF
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 55 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer of Various Sources
Let
f 0 =f � µF
σFAssume ebl = bl � µF
σFThen optimal gl for F can be obtained as follows:
gl = E (FjF 2 Bl ) =R blbl�1
f pF (f )dfR blbl�1
pF (f )df=
R blbl�1
f 1σFpF 0(
f �µFσF
)dfR blbl�1
1σFpF 0(
f �µFσF
)df
=
R eblebl�1 (σF f 0 + µF ) pF 0(f0)df 0R eblebl�1 pF 0(f 0)df 0 = σFegl + µF (5)
From Eq. (5), the assumption ebl = (bl � µF ) /σF is justi�ed
bl = (gl + gl+1) /2 = σFebl + µFWen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 56 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer: High Rate Approximation
Quantizer Distortion
Dq ' ∑l2L
P(Bl )∆2l12
= ∑l2L
p(g �l )∆3l
12= ∑
l2L
α3l12
whereαl =
3qp(g �l )∆l
Observe that
∑l2L
αl = ∑l2L
3qp(g �l )∆l '
Z fmax
fmin
3qp(f )df = constant
MMSE Scalar Quantizer
min ∑l2L
α3l12s.t. ∑
l2Lαl = C
) α�l = constant =1L
Z fmax
fmin
3qp(f )df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 57 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer High Rate Approximation
Given
α�l =1L
Z fmax
fmin
3qp(f )df
Quantizer distortion
Dq ' ∑l2L
(α�l )3
12= L
(α�l )3
12
=1L2112
�Z fmax
fmin
3qp(f )df
�3= 2�2R
112
�Z fmax
fmin
3qp(f )df
�3where L = 2R
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 58 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer High Rate Approximation
Given F 0 = F/σf is an unit variance source
P(F 0 � f 0)| {z }CF0 (f
0)
= P(F � σf f0)| {z }
CF (σf f 0)
) ddf 0CF 0(f
0) =ddf 0CF (σF f
0)
) pF 0(f0) = σf pF (σf f
0) or pF (f ) =1
σfpF 0(
fσf)
Quantizer distortion
Dq ' 2�2R
12
�Z fmax
fmin
3qpF (f )df
�3=2�2R
12
Z fmax
fmin
3
s1
σfpF 0(
fσf)df
!3= σ2f 2
�2R ε2
where ε2 = 112
�R fmax/σffmin/σf
3ppF 0(f )df
�3Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 59 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMAE Scalar Quantizer
Observe thatZ bl
bl�1jf � gl j p(f )df
=Z gl
bl�1(gl � f ) p(f )df +
Z bl
gl(f � gl ) p(f )df
= glZ gl
bl�1p(f )df �
Z gl
bl�1f p(f )df +
Z bl
glf p(f )df � gl
Z bl
glp(f )df
Then
∂
∂glJ(b�, g�) =
∂
∂gl
�Z bl
bl�1jf � gl j p(f )df
�=
Z gl
bl�1p(f )df �
Z bl
glp(f )df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 60 / 61
Lecture 6 Scalar and Vector Quantization Appendix
References
1 Y. Wang, et. al - Video Processing and Communications2 Jayant, N. S., et. al - Digital Coding of Waveforms3 B. Girod, http://www.stanford.edu/class/ee368b/Handouts/
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2013 61 / 61