§4 Continuous source and Gaussian channel

§4.1 Continuous source

§4.2 Gaussian channel

§4.1 Continuous source

§4.2 Gaussian channel

§4.1 Continuous source

1. Differential entropy

Definition:

Let X be a random variable with cumulative distribution function F(x) = Pr(X≤x). If F(x) is continuous, the random variable is said to be continuous. Let when the derivative is defined. If , then p(x) is called the probability density function for X. The set where p(x) > 0 is called the support set of X.

( ) '( )p x F x( ) 1p x dx

§4.1 Continuous source

1. Differential entropy

Definition:

The differential entropy h(X) of a continuous random variable X with a density p(x) is defined as

1( ) ( ) log

( )Sh X p x dx

p x

where S is the support set of the random variable.

§4.1 Continuous source

1. Differential entropy

21( ) log(2 ) ( )

2h X e bits

Example 4.1.1

(X~N (m,σ2), Normal distribution)please calculate the differential entropy.

),2

)(exp(

2

1)(~Let 2

2

mx

xpX

§4.1 Continuous source

1. Differential entropyDefinition:

The differential entropy of a set X,Y of random variables with density p(xy) is defined as

1( ) ( ) log

( )XY

S Sh XY p xy dxdy

p xy

If X,Y have a joint density p(xy), we can define the conditional differential entropy h(X|Y) as

1( | ) ( ) log

( | )XY

S Sh X Y p xy dxdy

p x y

§4.1 Continuous source

2. Properties of differential entropy

1) h (XY) = h(X) + h(Y|X) = h(Y) + h(X|Y)

)()|(),()|( YhXYhXhYXh

)()()( YhXhXYh

§4.1 Continuous source

2. Properties of differential entropy

2) h(X) can be negative.

Example 4.1.2

else

bxaabxp

,0

,1

)(If (b-a)<1, h(X) < 0.

Consider a random variable distributed uniformly from a to b.

1( ) log( ) log( )

b

ah X b a dx b a

b a

§4.1 Continuous source

2. Properties of differential entropy

3) h(X) is a convex function of the input probabilities p(x),it has the maximum.

Theorem 4.1

If the peak power of the random variable X is restricted, the maximizing distribution is the uniform distribution.

If the average power of the random variable X is restricted, the maximizing distribution is the normal distribution.

§4.1 Continuous source

2. Properties of differential entropy4) let Y=g(X), the differential entropy of Y may be different with h(X).

Example 4.1.3Let X is a random variable distributed uniformly from -1 to 1, and Y=2X. h(X)=? h(Y)=?

Theorem 4.2 aXhaXh log)()(

Theorem 4.3 )()( XhXah

Review

• KeyWords:

Differential entropy 1( ) ( ) log

( )Sh X p x dx

p x

Chain rule of differential entropy

Conditioning reduces entropy

Independent bound of differential entropy

may be negative convex functiontransformative

Homework

1. Prove the following conclusions:

) ( ) ( )b h a X h X

a) h (XY) = h(X) + h(Y|X) = h(Y) + h(X|Y)

§4 Continuous source and Gaussian channel

§4.1 Continuous source

§4.2 Gaussian channel

§4.2 Gaussian channel

1.The model of Gaussian channel

X

Z

Y

Normal, mean 0,variance σz

2

Y=X+Z

X and Z are independent

2. Average mutual information

§4.2 Gaussian channel

I(X;Y) = h(Y) – h(Y|X)

= h(Y) – h(Z|X) = h(Y) – h(Z)

(Y=X+Z)

Let X~N(0,σx2),

Example 4.2.1

Y~N(0,σx2+σz

2),

2 21( ) log(2 ( ))

2 x zh Y e

2 2

2

1( ; ) log

2x z

z

I X Y

1log( )

2

P N

N

§4.2 Gaussian channel

3. The channel capacity

Definition: The information capacity of the Gaussian channel

with power constraint P is

);(max][:)( 2

YXICPXExp

I(X;Y) = h(Y) – h(Z) 1( ) log 2

2h Z eN

( )

1max[ ( ) log 2 ]

2p xC h Y eN

1 1log 2 ( ) log 2

2 2e P N eN

1log( )

2

P N

N

1log(1 )

2

P

N

3. The channel capacity

§4.2 Gaussian channel

2( [ ] )E X P

1log(1 )

2

PC

N

3. The channel capacity

§4.2 Gaussian channel

Thinking about the band-limited channels, transmission bandwidth is W,

20( )zN N W

0

1log(1 )

2

PC

N W (bits/sample )

There are 2W samples per second,

0

log(1 )

tC NC

PW

N W

4. Shannon’s formula

§4.2 Gaussian channel

0

log(1 )t

PC W

N W

(bits/sec)

Shannon’s famous expression for the capacity of a band-limited, power-limited Gaussian channel.

§4.2 Gaussian channel

4. Shannon’s formula

Remarks:

1 ） Ct 、 W 、 SNR can be interchanged.

2 ） 1, 0tSNR C

0

log(1 )t

PC W

N W

§4.2 Gaussian channel

4. Shannon’s formula

For infinite bandwidth channels

0

0 0

lim lim log(1 )tW W

WNP PC

N P WN

0 0

lim log 1.4427tW

P PC e

N N

0

1.4427P

N

0/ NP

0/ NP

Ct (bps)

W

3 ） shannon limit

0

1lim ln(1 ) 1x

xx

§4.2 Gaussian channel

4. Shannon’s formula

Let Eb is the energy per bit, then 0

log(1 )t

PC W

N W

)1log(0N

E

W

C

W

C btt

WCN

E

t

WCb

t

/

12 /

0

W As

dBWCN

E

t

WC

W

bt

6.12ln/

12lim

/

min0

Review

• KeyWords:

Capacity of Gaussian channel

(Band limited, power limited)Shannon’s fomula

Shannon limit

Information rate of Gaussian channel

Homework

1. In image transmission, there are 2.25*106 pixels per frame.Reproducing image needs 4 bits per pixel (assume that eachbit has equal probability to choose ‘0’ and ‘1’). Compute the channel bandwidth needed for transmitting 30 frames image per second . (P/N = 30dB)

2. Consider a power-limited Gaussian channel , bandwidth is3kHz, and (P + N)/N = 10dB. (1) Compute the maximum rate of this channel. (bps) (2) If SNR decreases to 5 dB, give the channel bandwidth with the same maximum rate.