Lecture 20: Quantization and Rate-Distortion

• Quantization

• Introduction to rate-distortion theorem

Dr. Yao Xie, ECE587, Information Theory, Duke University

Approximating continuous signals...

Dr. Yao Xie, ECE587, Information Theory, Duke University 1

Dr. Yao Xie, ECE587, Information Theory, Duke University 2

Lossy source coding

• we have seen an information source cannot be losslessly compressedbeyond its entropy

• in speech, image and video compression, we may tolerate a certaindistortion to achieve better compression

• if source is continuous, any compression scheme which translates it intobits will involve distortion

• consider lossy compression framework

Dr. Yao Xie, ECE587, Information Theory, Duke University 3

Quantization

• let X be a continuous random variable

• we approximate X by X̂(X)

• using R bits to represent X, then X̂(X) has 2nR possible values

• find the optimal set of values for X̂ and associated regions of each value

Dr. Yao Xie, ECE587, Information Theory, Duke University 4

linear scalar quantizer

Dr. Yao Xie, ECE587, Information Theory, Duke University 5

Example: quantizing Gaussian random variable

• let X ∼ N (0, σ2)

• minimize mean square error E(X − X̂(X))2

• if we use 1 bit to represent X, we should let the bit to distinguish thesign of X

• the estimated X̂ = {E(X|X ≥ 0), E(X|X < 0)}

X̂ =

√

2πσ; x ≥ 0

−√

2πσ; x ≤ 0

Dr. Yao Xie, ECE587, Information Theory, Duke University 6

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

−2.5 −2 −1.5

−.79 +.79

−1 −0.5 0 0.5 1 1.5 2 2.5x

f(x)

Dr. Yao Xie, ECE587, Information Theory, Duke University 7

• if we are given 2 bits to represent

• we want to divide the real line into 4 regions and use points within eachregion to represent the sample

• a more complicated optimization problem: boundaries, reconstructionpoints

• two properties of optimal boundaries and reconstruction points

1) given reconstruction points {X̂}, distortion is minimized by assignedvalues to its closest point – Voronoi or Dirichlet partition

2) given partition: reconstruction point should be conditional mean

• iterate these two steps is Lloyd algorithm

Dr. Yao Xie, ECE587, Information Theory, Duke University 8

Voronoi partitions

Dr. Yao Xie, ECE587, Information Theory, Duke University 9

Lloyd algorithmSimple example:

✲

✻

1-1

1

-1

x

x

x

x

x

x

x

x

x

x

x

x

S1S4

S2S3

✲

✻

1-1

1

-1

①D(1)

①D(2)

①D(4)

①

D(3)

✡✡

✡✡

✭✭✭✭✭✭✭✭✭✭✭✭✭✭❆❆❆❆❆❆❆❆❆❆❆❆❆❆

✭✭✭✭✭✭✭✭✭✭✭✭✭✭

▲▲

▲▲

▲▲

▲▲

▲▲

▲

x

x

x

x

x

x

x

x

x

x

x

x

S1S4

S2S3

Dr. Yao Xie, ECE587, Information Theory, Duke University 10

Vector quantization

• given a set of n samples are i.i.d. from Gaussian

• we want to jointly quantize the vector [X1, . . . , Xn]

• represent these vectors using nR bits

• represent entire sequence by a single index taking 2nR values

• vector quantization achieve lower distortion than linear quantization

Dr. Yao Xie, ECE587, Information Theory, Duke University 11

vector quantizer

Dr. Yao Xie, ECE587, Information Theory, Duke University 12

Rate-distortion tradeoff

• intuition: more bits used, lower quantization error

• can we quantize this tradeoff

• what is the fundamental lower-bound on distortion for a given rate R

Dr. Yao Xie, ECE587, Information Theory, Duke University 13

Rate-distortion code

• assume a source produces a i.i.d. sequences: X1, . . . , Xn, Xn ∼ p(x)

• encoder describes the source sequence Xn by encoding function

• encoding function: fn : X̂n → {1, . . . , 2nR} maps a sequence to anindex

• decoder: represent Xn by an estimate X̂n

• decoding function: gn : {1, . . . , 2nR} → X̂n maps an index toreconstructed sequence

• define this (2nR, n)-rate distortion code

Dr. Yao Xie, ECE587, Information Theory, Duke University 14

Distortion function

• distortion function: cost of representing symbol by its quantized version

d : X × X̂ → R+

• assume maxx∈X ,x̂∈X d(x, x̂) < ∞

• example: Hamming distortion d(x, x̂) =

{0, x = x̂1, x ̸= x̂

Ed(X, X̂) = P (X ̸= X̂)

• example: squared-error distortion d(x, x̂) = (x− x̂)2

• example: Itakura-Saito distance: relative entropy between multivariablenormal processes

Dr. Yao Xie, ECE587, Information Theory, Duke University 15

• for a sequence

d(xn, x̂n) =1

n

n∑i=1

d(xi, x̂i)

• distortion for a (2nR, n) code:

D = Ed(Xn, gn(fn(Xn))) =

∑xn

p(xn)d(xn, gn(fn(xn)))

• a rate distortion pair (R,D) is said to be achievable if there exists asequence of (2nR, n)-rate distortion codes (fn, gn) with

limn→

Ed(Xn, gn(fn(Xn))) ≤ D

Dr. Yao Xie, ECE587, Information Theory, Duke University 16

Rate-distortion theorem

• the rate distortion region for a source is the closure of the set ofachievable rate distortion pairs (R,D)

• rate-distortion function: R(D), is the infimum of rates R such that(R,D) is in the rate distortion region of the source for a givendistortion D

Theorem. The rate distortion function for an i.i.d. source X withdistribution p(x) and bounded d(x, x̂) is equal to

R(D) = minp(x̂|x):

∑(x,x̂) p(x)p(x̂|x)d(x,x̂)≤D

I(X; X̂)

Dr. Yao Xie, ECE587, Information Theory, Duke University 17

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D

R(D

)

FIGURE 10.4. Rate distortion function for a Bernoulli ( 12) source.

Dr. Yao Xie, ECE587, Information Theory, Duke University 18