Information Theory: the Day after Yesterdayeceweb1.rutgers.edu/~csi/Guo.pdf · 2017. 9. 19. · 2...

Outline Yesterday Today: applications Today: theory Outlook

Information Theory: the Day after Yesterday

Dongning Guo

Department of Electrical Engineering and Computer ScienceNorthwestern University

Chicago’s Shannon Centennial EventSeptember 23, 2016

Dongning Guo Northwestern University

Information Theory


Information Theory: the Day after Yesterday

↓

↓

IT today


Information Theory


Outline

The birth of information theory;Applications;The theory today;An outlook.


Information Theory


Outline



Information Theory


Outline



Information Theory


Outline



Information Theory


Communication systems existing in mid 20th century

telegraph (1830s)Morse code: A . _ B _ . . . C _ . _telephone (Bell 1876)wireless telegraph (Marconi, 1887)AM radio (early 1900s)television (1925–1927)frequency modulation (FM) (Armstrong, 1936)pulse-coded modulation (PCM) (Reeves, 1937–1939)vocoder (Dudley, 1939)spread spectrum (1940s)

Known techniques: efficient encoding of text, understanding ofbandwidth, digital vs. continuous-time signaling, tradeoff betweenfidelity and bandwidth.


Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory


Fundamental questions

Nyquist, “Certain factors affecting telegraph speed,” 1924.W ∝ log(the number of signal levels)How much improvement in telegraphy trasmission rate couldbe achieved by replacing the Morse code by an “optimum”code?Hartley, “Transmission of information,” 1928.The capacity of a channel is proportional to its bandwith.What is the maximum telegraph signaling speed sustainable bybandlimited linear systems? Answered by the samplingtheorem (Küpfmüller 1924, Nyquist 1928, Kotelnikov 1933, J.Whittaker 1915)


Information Theory





Information Theory





Information Theory


Inception of a unifying theory

Excerpt of a letter from Claude Shannon to Vannevar Bush onFeb. 16, 1939 [Library of Congress]:


Information Theory


Inception of a unifying theory

Excerpt of a letter from Claude Shannon to Vannevar Bush onFeb. 16, 1939 [Library of Congress]:


Information Theory


The Bell System Technical Journal

Vol. XXVII July, 194S No. 3

A Mathematical Theory of Communication

By C. E. SHANNON

Introduction

THE recent development of various methods of modulation such as PCMand PPM which exchange bandwidth for signal-to-noise ratio has in-

tensified the interest in a general theory of communication. A basis for

such a theory is contained in the important papers of Nyquist 1 and Hartley2

on this subject. In the present paper we will extend the theory to include a

number of new factors, in particular the effect of noise in the channel, and

the savings possible due to the statistical structure of the original message

and due to the nature of the final destination of the information.

The fundamental problem of communication is that of reproducing at

one point either exactly or approximately a message selected at another

point. Frequently the messages have meaning; that is they refer to or are

correlated according to some system with certain physical or conceptual

entities. These semantic aspects of communication are irrelevant to the

engineering problem. The significant aspect is that the actual message is

one selected from a set of possible messages. The system must be designed

to operate for each possible selection, not just the one which will actually

be chosen since this is unknown at the time of design.

If the number of messages in the set is finite then this number or any

monotonic function of this number can be regarded as a measure of the in-

formation produced when one message is chosen from the set, all choices

being equally likely. As was pointed out by Hartley the most natural

choice is the logarithmic function. Although this definition must be gen-

eralized considerably when we consider the influence of the statistics of the

message and when we have a continuous range of messages, we will in all

cases use an essentially logarithmic measure.

The logarithmic measure is more convenient for various reasons:

1. It is practically more useful. Parameters of engineering importance

1 Nyquist, H., "Certain Factors Affecting Telegraph Speed," Bell System Technical Jour-

nal, April 1924, p. 324; "Certain Topics in Telegraph Transmission Theory," .4. I.E. E.

Tians., v. 47, April 1928, p. 617.2 Hartley, R. V. L., "Transmission of Information," Bell Svsteni Technical Journal, Julv

1928, p. 535.

379


Information Theory


Shannon’s abstraction

MATHEMATICAL THEORY OE COMMUNICATION 381

several variables—in color television the message consists of three functions

/(•v> 3*> Oj S(xt Ji 0> h(x, y, I) defined in a three-dimensional continuum

—

we may also think of these three functions as components of a vector field

defined in the region—similarly, several black and white television sources

would produce "messages" consisting of a number of functions of three

variables; (f) Various combinations also occur, for example in television

with an associated audio channel.

2. A transmitter which operates on the message in some way to produce a

signal suitable for transmission over the channel. In telephony this opera-

tion consists merely of changing sound pressure into a proportional electrical

current. In telegraphy we have an encoding operation which produces a

sequence of dots, dashes and spaces on the channel corresponding to the

message. In a multiplex PCM system the different speech functions must

be sampled, compressed, quantized and encoded, and finally interleaved

INFORMATIONSOURCE TRANSMITTER RECEIVER DESTINATION

SIGNAL OMESSAGE

RECEIVEDSIGNAL

NOISESOURCE

Fig. 1—Schematic diagram of a general communication system.

properly to construct the signal. Vocoder systems, television, and fre-

quency modulation are other examples of complex operations applied to the

message to obtain the signal.

3. The channel is merely the medium used to transmit the signal from

transmitter to receiver. It may be a pair of wires, a coaxial cable, a band of

radio frequencies, a beam of light, etc.

4. The receiver ordinarily performs the inverse operation of that done by

the transmitter, reconstructing the message from the signal.

5. The destination is the person (or thing) for whom the message is in-

tended.

We wish to consider certain general problems involving communication

systems. To do this it is first necessary to represent the various elements

involved as mathematical entities, suitably idealized from their physical

counterparts. We may roughly classify communication systems into three

main categories: discrete, continuous and mixed. By a discrete system we

will mean one in which both the message and the signal are a sequence of

Source as random process (Shannon worked on cryptography);Channel modeled as a random transformation;Related work:

N. Wiener, “Extrapolation, interpolation, and smoothing ofstationary time series,” 1949;S. O. Rice, “Mathematical analysis of random noise,” 1952.


Information Theory






—














SIGNAL OMESSAGE

RECEIVEDSIGNAL

NOISESOURCE











tended.










Information Theory






—














SIGNAL OMESSAGE

RECEIVEDSIGNAL

NOISESOURCE











tended.










Information Theory






—














SIGNAL OMESSAGE

RECEIVEDSIGNAL

NOISESOURCE











tended.










Information Theory






—














SIGNAL OMESSAGE

RECEIVEDSIGNAL

NOISESOURCE











tended.










Information Theory


Shannon’s theorems

Theorem (Lossless source coding)

H < R

Theorem (Channel coding)

R < C

Theorem (Source–channel separation)

H < C


Information Theory




H < R


R < C


H < C


Information Theory




H < R


R < C


H < C


Information Theory


Roles played by IT since 1948

IT predicts the fundamental limitslossless data compressionlossy data compressionchannel codingnetwork codingsignal processingstatistical inferencecomplexity theoryportfolio theory

IT as a design driverIT as a foundation of sciences and engineering


Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory






Information Theory


Lossless data compression

Huffman coding (a component in JPEG, MP3, MPEG-4)


Information Theory


Universal lossless data compression

Softwares based on the Lempel-Ziv algorithm:


Information Theory


Lossy data compression

Examples: mp3, JPEG, MPEG-4.


Information Theory


Channel coding: deep-space communication

Over 100 million km away.


Information Theory


Channel coding: modem

Example: trellis codesAlso, CRC for the internet


Information Theory


Channel coding: Compact Disk (CD)

Reed-Solomon codes


Information Theory


Channel capacity: WiFi and multiple antennas

Also, space-time codes.


Information Theory


Source and channel coding: cellular networks

Vocoder (speech ↔ bit stream); modem (bit stream ↔ waveform);Also, MIMO, OFDM, CDMA, multiuser detection.


Information Theory


What is IT?

theorems about the fundamental limits

algorithms for achieving/approachingthose limits

peoplewho call themselves informationtheorists and develop thosetheorems and algorithms


Information Theory


Editorial areas:coding techniquescoding theorycommunicationscommunication networkscomplexitycryptographydetection and estimationmachine learningprobability and statisticsquantum information theorysequencesShannon theorysignal processingsource codingstatistical learning


Information Theory




Information Theory




Information Theory


Single-user information theory

Efficient lossless/lossy codes for ergodic sources;Efficient capacity-achieving codes for ergodic channels;Capacity of Gaussian MIMO channels generally known;Capacity of certain quantum channels known;...


Information Theory





Information Theory





Information Theory





Information Theory





Information Theory


Finite-blocklength channel coding rate2328 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 5, MAY 2010

Fig. 7. Bounds for the AWGN channel, �� , � � �� .

asymptotic expansion of . In Figs. 6 and 7 we cansee that the bound is also quite competitive for finite .

Comparing the bound and the classical bounds of Fein-stein and Gallager, we see that, as expected, the bound is uni-formly better than Feinstein’s bound. In the setup of Fig. 6, the

bound is a significant improvement over Gallager’s bound,coming very close to the Shannon bound as well as the con-verse. In Fig. 7, both the and Gallager bounds are again veryclose to the Shannon bound but this time Gallager’s bound isbetter for small . There are two reasons for this. First, recallthat we have analyzed a suboptimal decoder based on hypoth-esis testing, whereas Gallager used the maximum likelihood de-coder. It seems that for small it is important to use optimal de-coding. Moreover, Gallager’s analysis is targeted at very small .Indeed, as we go from to , the tightness of Gallager’sbound improves significantly. In general we observe that Gal-lager’s bound improves as the channel becomes better and as

gets smaller. On the other hand, the bound is much moreuniform over both SNR and . In Section IV, the bound, incontrast to Gallager’s bound, yields the correct term in theasymptotic expansion of .

Comparing the RCU bound and the DT bound (and its rel-ative, the bound), the DT bound is very handy theoreti-cally and does not lose much nonasymptotically compared tothe RCU bound. In fact, for the BEC the DT bound is tighterthan the RCU bound. Also, the DT bound (in the form of Theo-rems 22 and 25) and the bound are directly applicable to the

maximal probability of error, whereas the RCU bound requiresfurther manipulation (e.g., Appendix A).

IV. NORMAL APPROXIMATION

We turn to the asymptotic analysis of the maximum achiev-able rate for a given blocklength. In this section, our goal isto show a normal-approximation refinement of the coding the-orem. To that end, we introduce the following definition.

Definition 1: The channel dispersion (measured in squaredinformation units per channel use) of a channel with capacityis equal to

(221)

(222)

In fact, we show that for both discrete memoryless channelsand Gaussian channels,

(223)

The asymptotic behavior in (223) is particularly useful in con-junction with the nonasymptotic upper and lower bounds devel-oped in Section III, as (223) turns out to be an accurate and suc-cinct approximation to the fundamental finite blocklength limitfor even rather short blocklengths and rates well below capacity.

R(n, ε) = C −√n−1V Q−1(ε) +O(n−1 log n).


Information Theory


Almost lossless analog compression

I.i.d. (analog) source X1, . . . , Xn, with Renyi informationdimension d(X).

Encoder Decoder Minimum ε-achievable ratelinear Borel R∗(ε) = d(X)

continuous continuous R0(ε) = 0

Borel Lipschitz R(ε) = d(X)

Compressed sensing exploits the sparsity of natural signals.


Information Theory


Almost lossless analog compression

I.i.d. (analog) source X1, . . . , Xn, with Renyi informationdimension d(X).

Encoder Decoder Minimum ε-achievable ratelinear Borel R∗(ε) = d(X)

continuous continuous R0(ε) = 0

Borel Lipschitz R(ε) = d(X)

Compressed sensing exploits the sparsity of natural signals.


Information Theory


Multi-user information theory

extension of Shannon’s basic theorems to networks withmultiple sources and receivers;new ingredients: interference, cooperation, side information;multiaccess channel capacity known;degraded broadcast channel capacity known;capacity of Gaussian MIMO broadcast channel known;capacity of Gaussian interference channel well approximated;lossy compression of correlated sources partially solved;degrees of freedom of MIMO interference channels known;fundamental limits of caching;...


Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory


The new many-user regime

k = 1,n → ∞

classical single-user IT

k fixed,n → ∞ multiuser IT

k → ∞after

n → ∞Large-system analysis

k, n→∞ many-user IT(with application to the IoT)


Information Theory


Foundation for thermal physics?


Information Theory


Foundation for evolution, neuroscience?

Examples:evolution and information acquisition;understanding neural spikes.


Information Theory


Foundation for evolution, neuroscience?

Examples:evolution and information acquisition;understanding neural spikes.


Information Theory


Foundation for economics?


Information Theory


Use of entropy, mutual information, relative entropy

probability and statisticscomplexity theorycomputational theorybiostatisticsmachine learningphysicschemistryeconomicsneuroscience...


Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory


Open problems: single-user

exact throughput-delay-reliability trade-off (non-asymptotics);the channel reliability function;capacity of channels with memory;deletions, insertions, synchronization;joint source-channel coding;...


Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory


Open problems: multiuser problems

multiple description of sources;broadcast channel capacity;interference channel capacity;two-way channel capacity;relay channel capacity;capacity of multiuser channels with feedback;...


Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory





Information Theory


Open problem: mobile ad hoc networks

DARPA Spectrum Collaboration Challenge (2016–2019)

Email questions to [email protected] A. Approved for public release

Match Overview

27

Incumbent

Team 1 Team 2 Team 3 Team 4 Team 5

Ensemble of up to 5 teams placed in arena

One node per network serves as a gateway

Collaboration takes place over internet-like infrastructure connected to the gateway (models realistic internets)

Arena may also contain other Non-Collaborative Radios (NCR):• Incumbents• “Jammers”

IPTraffic

Each node is given IP traffic

Sources and destinations are contained in the same network

Traffic will emulate multiple canonical types

Radio environment emulated in real-time:• Large-scale path loss• Multipath & Doppler• Channel correlation• Motion


Information Theory


IT is a way of thinking

Like Shannon, we should always question what are thefundamental limits;We then invent schemes to achieve those limits;New progress is often made when we challenge existingconstraints and assumptions (network coding, full duplex, ...).


Information Theory





Information Theory





Information Theory


What about Shannon’s premises?

The fundamental problem of communication is that ofreproducing at one point either exactly or approximately amessage selected at another point. Frequently themessages have meaning; that is they refer to or arecorrelated according to some system with certain physicalor conceptual entities. These semantic aspects ofcommunication are irrelevant to the engineering problem.The significant aspect is that the actual message is oneselected from a set of possible messages. The systemmust be designed to operate for each possible selection,not just the one which will actually be chosen since this isunknown at the time of design.


Information Theory



Information Theory

Date post:	12-Mar-2021
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Information Theory: the Day after Yesterdayeceweb1.rutgers.edu/~csi/Guo.pdf · 2017. 9. 19. · 2...

Documents