§3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2...

§3 Discrete memoryless sources and their rate-distortion function

§3.1 Source coding

§3.2 Distortionless source coding theorem

§3.3 The rate-distortion function

§3.4 Distortion source coding theorem

§3.1 Source coding




1. Source coder

§3.1 Source coding

Source coder}1...,1,0{ rA },...,,{ 110 rC

|| iin

}1,...,1,0{ sS

)...,,( 110 rpppp

1 2( ), , ( 1, ..., )ii i i in ij is s s s S j n

Source alphabet

Channel input alphabet

Code

mnm

1 2( ), , ( 1, ..., )ii i i in ij is s s c S j n

Extendedsource coder

mA

}1,...,1,0{ sS

1 2( )mU U U U

iU A

Example 3.1

§3.1 Source coding

2. Examples

1) ASCII source coder

ASCII coder

{0,1}

{English symbol , command} {binary code, 7 bits}

2) Morse source coder

Source coder(1)

Source coder(2)

{0,1}{. , —}

{A,B,…,Z} Binary code

2. Examples

§3.1 Source coding

3) Chinese telegraph coder

“中”

“0022”

“01101 01101 11001 11001”

2. Examples

§3.1 Source coding

Constant-length codes

Variable-length codes

Distortionless codes

Distortion codes

2. Classification of the source coding

Uniquely decodable (UD) codes

Non-UD codes

§3.1 Source coding

in n

in

( ; ) ( )I A C H A

( ; ) ( )I A C H A

The code C is called uniquely decodable (UD) if each string in each Ck arises in only one way as a concatenation of codewords. This means that if say

and each of the τ’s and σ’s is a codeword, then

Thus every string in Ck can be uniquely decoded into a concatenation of codewords.

1 2 1 2* * * * * *k k

1 1 2 2, , , .k k

2. Classification of the source coding

Example 3.2

§3.1 Source coding

Source symbol

si

Symbol probability

P(si) Code

1

Code

2

Code

3

Code

4

Code

5

S1 1／2 0 0 1 1 00 S2 1／4 11 10 10 01 01 S3 1／8 00 00 100 001 10 S4 1／8 11 01 1000 0001 11

3. Parameters about source coding

1) Average length of coding

1

0

1

0

||)(r

iii

r

iii nppn

mr

i

mi

mim npn

1

For extended source coding:

(code/sig)

code/m-sigs

Length of codeword

§3.1 Source coding

mnn

m (code/sig)

2) Information rate of coding

( ) /R H p n

( )

/mm

H pR

n m

(bit/code)

(bit/code)


§3.1 Source coding

3) Coding efficiency

smn

pH

m log

)(

Actual rate

Maximum rate


§3.1 Source coding

log

R

s

For extended source coding:

s

npH

log

/)(



§3.1 Source coding




Example 3.3

The binary DMS has the probability space:

4

1

4

3)(

21 aa

aP

A

i


1) “0” a1, “1” a2

2) a1a1: 0 a1a2: 10 a2a1: 110 a2a2: 111

)/(811.03

4log

4

34log

4

1)( signbitAH

Average length of coding: )/(11 sigcoden

Code efficiency: 811.01


“0” a1, “1” a2

Rate:1

1

( )0.811 ( / )

H pR bit code

n

Example 3.3

4

1

4

3)(

21 aa

aP

A

i

Extended source coding

i )( iP code Length of codeword

a1a1 16

90 1

a1a2 16

310 2

a2a1 16

3110 3

a2a2 16

1111 3

Average length of coding :3

16

13

16

32

16

31

16

92 n

Code efficiency:

)/(961.0844.0

811.02 codebitR

Rate:

961.02

)2/(688.1 sigscode

)/(844.02

2 sigcoden


Example 3.3

m times extended source coding

m = 3: 985.03

R3 = 0.985 (bit/code)

m = 4: 991.04

R4 = 0.991 (bit/code)

m 1m


Example 3.3


Distortionless source coding theorem

Theorem 3.1 If the code C is UD, its average length mustexceed the s-ary entropy of the source , that is,

1

0

log)(r

iisis pppHn

(Theorem 11.3 in textbook)



Theorem 3.2

1)()()( pHpnpH sss



Theorem 3.3 )()(

1lim pHpn

m sm

sm



The source can indeed be represented faithfully using s-ary symbols per source symbol.

p

( )sH p



corollary

The efficient UD codes are achievable if rate R ≤ C.

(C is the capacity of s-ary lossless channel )

Review

• KeyWords:

Source coder

Variable-length codes

distortionless codes

Uniquely decodable codes

Average length of coding

Information rate of coding

Coding efficiency

Shannon’s TH1

Homework

1. p344: 11.12

2. p345:11.20


§3.1 Source coding





1. IntroductionReview

Distortionless source coding theorem (corollary) The efficient UD codes are achievable if rate R ≤ C.

(C is the capacity of s-ary lossless channel )

Conversely, any sequence of (2nR, n) codes with must have R ≤ C.

0EP

The channel coding theorem (Statement 2 ):

All rates below capacity C are achievable. Specifically,

for every rate R ≤ C, there exists a sequence of (2nR,n) codes

with maximum probability of error .0EP


1. Introduction

Review

For distortionless coding: R≤C - (PE→0, R→C - )

But actually……

Given a source distribution and a distortion measure, what is the minimum expected distortion achievable at aparticular rate?what is the minimum rate description required to achieve a particular distortion?


2. Distortion measure

coding channelui

vj

AU={u1,u2,…,ur} AV={v1,v2,…,vs}

k

iii vudvud

1

),(),(

kV

kUkk AAvvvuuuvuif ),...,,;,...,,(),( 2121

( , )i jd u v

Source symbol Destination symbol


Average distortion measure:

, ,

( ) [ ( , )] ( ) ( , ) ( ) ( | ) ( , )u v u v

D k E d U V p uv d u v p u p v u d u v

Let the input and output of the channel be U=(U1,U2,…,Uk)and V=(V1,V2,…,Vk) respectively

kV

kUkk AAvvvuuuvu ),...,,;,...,,(),( 2121

where,


,

,

( ) [ ( , )] ( ) ( , )

( ) ( | ) ( , )

i j i j i jU V

i j i i jU V

D E d E d u v p u v d u v

p u p v u d u v

Example 3.3.1

AU = AV = {0,1};source statistics p(0) = p, p(1) = q = 1-p,where p ½; and distortion matrix

01

10D



Example 3.3.2

AU = {-1,0,+1}, AV = {-1/2, +1/2};source statistics(1/3,1/3,1/3)and distortion matrix

12

11

21

D




Fidelity criterion:


( )D or D k k ,

Test channel:

Let the source statistics p(u) and distortion measure d(u,v) are fixed.

( | ) :j iB P v u D

( | ) : ( )or B P V U D k k

3. Rate-distortion function

1) Definition

The function is a function of the source statistics(p(u)) ,the distortion matrix D, and the real number .

)(kR


The information rate distortion function Rk(δ) for asource U with distortion measure d(U, V) is defined as

1 1, ( ,

( )

) (( , ..., ), ( ,

min{ ( ; )

..., ))

: ( ) }k

k kwhere U V

R I U V D k k

U U V V

The information rate distortion function Rk(δ) for asource U with distortion measure d(U, V) is defined as


UAu

vvudup ),(min)(min

21 )()( 21 kk RR ③If , then


②The minimum possible value of is ,wheremink( )D k

R(δ) and C

①The function I(U;V) actually achieves its minimum value on the region of ;D


2) Properties

Theorem 3.4 is a convex function of .)(kR min (Theorem 3.1 in textbook)


R(0)=H(U)

maxmin

( )kR

max

( )kR

maxmin},:);(min{)( DVUIR

max( ) 0, iff.R


2) Properties

Theorem 3.4 is a convex function of .)(kR min

Theorem 3.5 For a DMS, for all k and .min )()( 1 kRRk





Example 3.3.1 (continued)

AU = AV = {0,1};source statistics p(0) = p, p(1) = q = 1-p,where p ½; and distortion matrix

01

10D


min

max

( ) (0) ( ) ( ),

( ) 0

R R H U H p

R

D

2) Properties

with different

(bit/sig)

0.0

1.0

0.8

0.6

0.4

0.2

0.50.40.30.20.1

0.5p

0.2p

0.1p

( )R

( )R p

0.3p


AU = AV = {0,1,…,r-1},

P{U=u}=1/r

Distortions are given by:

vuif

vuifvud

,1

,0),(

0.6 1.0

3.0

2.0

1.0

with different r

(bit/sig)

0.0 0.80.40.2

( )R

( )R

8r4r2r

2) Properties

Example 3.3.3



§3.1 Source coding




1. Distortion source coding theorem

(modified on Theorem 3.4 in textbook)


Theorem 3.6 (Shannon’s source coding theorem with a fidelity criterion) If , there exists a source code C of length k with M codewords, where:

)(RR

Db

Ma kR

)(

2)(

If ,no such codes exist.)(RR A source symbol can be compressed into R(δ) bits

if a distortion δ is allowable.

2. Relation of shannon’s theorems


Source Distortion

source coder

Distortionless source coder

Sink Distortion

source decoder

Distortionless source decoder

channel

Channel coder

Channel decoder

A general communication system

Review

• KeyWords:

Distortion measure

Average distortion measure

Fidelity criterion

Test channel

Rate-distortion function

Shannon’s TH3

thinking

Source X has the alphabet set {a1,a2,…,a2n},P{X = ai}=1/2n,i = 1,2,…,2n. The distortion measure is Hamming distortionmeasure ,that is

ji

jid ij ,0

,1

Design a source coding method with δ=1/2.


Date post:	02-Jan-2016
Category:	Documents
Upload:	arlene-bishop
View:	219 times
Download:	1 times

§3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2...

Documents