BySaima Israil

Feb 20th, 2014

Introduction General Model of Communication Shannon’s Information Theory Source Coding theorem Channel Coding Theory Information Capacity Theorem Conclusion

In defining information, Shannon identified the critical relationships among theelements of a communication system the power at the source of a signal the bandwidth or frequency range of an information channel through which the signal travelsthe noise of the channel, such as unpredictable static on a radio, which will alter the signal by the time it reaches the last element of the Systemthe receiver, which must decode the signal.

all communication involves three steps

Coding a message at its source Transmitting the message through a communications channel

Decoding the message at its destination.

Noise can be considered data without meaning; that is, data that is not being used to transmit a signal, but is simply produced as an unwanted by-product of other activities. Noise is still considered information, in the sense of Information Theory.

A quantitative measure of the disorder of a system and inversely related to the

amount of energy available to do work in an isolated system. The more energy has become dispersed, the less work it can perform and the greater the entropy.

The theorem can be stated as follows:Given a discrete source of entropy H(S) , the average code-word length L for any distortionless source coding is bounded as

L > H(S)This theorem provides the mathematical tool for assessing data compaction

The source coding theorem is also known as the "noiseless coding theorem" in the sense that it establishes the condition for error-free encoding to be possible

For reliable communication , needs channel encoding & decoding.“any coding scheme which gives the error as small as possible, and which is efficient enough that code rate is not too small?”

Let Source with alphabet X have entropy H(X) and produce symbols once every Ts, and Channel have capacity C and be used once every Tc . Then,

i) if , there exists a coding scheme.

ⅱ) if , it is not possible to transmit with arbitrary small error.

cs TC

TXH

)(

cs TC

TXH

)(

This Theorem defines an upper limit to the capacity of a link, in bits per second (bps), as a function of the available bandwidth and the signal-to-noise ratio of the link.

The Theorem can be stated as:C = B * log2(1+ S/N)

where C is the achievable channel capacity, B is the bandwidth of the line, S is the average signal power and N is the average noise power.

The phrase signal-to-noise ratio, often abbreviated SNR or S/N, is the ratio between the magnitude of a signal (meaningful information) and the magnitude of background noise.

The signal-to-noise ratio (S/N) is usually expressed in decibels (dB) given by the formula:

10 * log10(S/N)so for example a signal-to-noise ratio of 1000 is commonly expressed as

10 * log10(1000) = 30 dB.

2

/

Define ideal system as , where is the Tx energy per bit

Then, log (1 )

2 1 /

For infinite bandwidth channel

ln 2 0.693 1.6

b

b b

b

oC B

b

o

b

o B

R CP E C E

E CCB N B

EN C B

EdB

N

2 lim logB o

PC C eN

-10 0 10 20 30 40 500.1

1

10

20

Eb/No(dB)

C/B

-1.6

Rb < C

Rb > C

Rb = C

Shannon's limit

If the SNR is 20 dB, and the bandwidth available is 4 kHz, which is appropriate for telephone

communications, then C = 4 log2(1 + 100) = 4 log2 (101) = 26.63 kbit/s. Note that the value of

100 is appropriate for an SNR of 20 dB.

If it is required to transmit at 50 kbit/s, and a bandwidth of 1 MHz is used, then the minimum SNR required is given by 50 = 1000 log2(1+S/N) so S/N = 2C/W -1 = 0.035 corresponding to an SNR of -14.5 dB. This shows that it is possible to transmit using signals which are actually much weaker than the background noise level.

o Shannon’s Information Theory provide us the basis for the field of Information Theory

o Source coding is bounded as L > H(S)

o if , there exists a coding scheme.

o Channel capacity C = B * log2(1+ S/N)

cs TC

TXH

)(