Universal Schemes in Information Theory Introductory lecture … · 2011. 9. 27. · Universal...

Universal Schemes in Information Theory

Introductory lecture

immediate

September 16, 2011

Abstract

bla

1 Examples

1.1 Lossless compression

Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:

1. x � P is stochastic

2. x is an individual sequence

Answers:

1. entropy rate

2. finite-state compressibiity

Yeah, but can they be attained by us mortals, i.e., without

1. knowledge of P

2. knowledge of x

Ziv-Lempel CompressionAchieves:

• The fundamental limits from the non-universal settings:

– The entropy rate of any stationary source

– The finite-state compressibility of any individual sequence

• Linear complexity

• Cuteness

• State of the art performance on real data



immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness


EE477

Tuesday, September 27, 11



immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness





immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness




immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness




immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness





immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness


(cont.)



immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness





immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness


(cont.)



immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness




immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate


Yeah, but can they be attained without

1. knowledge of P

2. knowledge of x






• Cuteness





immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness


• LZ77 (sliding window) • LZ78 (incremental parsing)




immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness


(cont.)



immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness





immediate

September 16, 2011

Abstract

bla

1 Examples





Answers:

1. entropy rate



1. knowledge of P

2. knowledge of x






• Cuteness


(cont.)

Of wide current use:

Gif, Zip, Gzip, PNG, ...


1.2 Prediction

Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `

Questions: what are the fundamental limits assuming:

1. x ⇠ P is stochastic


Answers:

1. Performance of the “Bayes-optimal” predictor

2. Finite-state predictability


1. knowledge of P

2. knowledge of x

Feder-Merhav-Gutman PredictorAchieves:


– Performance of the “Bayes-optimal” predictor

– Finite-state predictability


• Cuteness

• Will beat you in a game of rock-paper-scissors any day, see

http://www.mit.edu/~emin/writings/lz_rps/

Note: universal predictor induced by universal compressor

2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2


1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

(cont.)

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2


1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2


1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

(cont.)

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2


http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

(cont.)

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2

1.2 Prediction





Answers:




1. knowledge of P

2. knowledge of x






• Cuteness




2


http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/

1.3 Discrete Denoising

Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn), so as to minimize theper-symbol prediction loss, as measured by a given loss function `




Answers:

1. Performance of the “Bayes-optimal” denoiser

2. Sliding-window “denoisability”


1. knowledge of P

2. knowledge of x

DUDE: Discrete Universal DEnoiserAchieves:


– Performance of the “Bayes-optimal” denoiser

– Sliding-window “denoisability”


• Cuteness?

• In use @ HP and maybe Google...

3






Answers:




1. knowledge of P

2. knowledge of x






• Cuteness?


3


Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given `Questions: what are the fundamental limits assuming:



Answers:




1. knowledge of P

2. knowledge of x






• Cuteness?


3

Goal:







Answers:




1. knowledge of P

2. knowledge of x






• Cuteness?


3

(cont.)1.3 Discrete Denoising

Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:

1. x ⇥ P is stochastic


Answers:




1. knowledge of P

2. knowledge of x



– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”


• Cuteness?


3





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?


3





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?


3






Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?


3






Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?


3

(cont.)





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?

• In use @ HP, cf.

http://www.hpl.hp.com/research/info_theory/dude/index.htm

and maybe Google...

3





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?



and maybe Google...

3


http://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htm





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?


3

(cont.)





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?



and maybe Google...

3





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?



and maybe Google...

3





Answers:




1. knowledge of P

2. knowledge of x





• Cuteness?



and maybe Google...

3


http://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htm

1.4 Filtering

Problem: Like the discrete denoising problem, but with causality, i.e., the estimate of xi may depend only on(z1, z2, . . . zi)




Answers:

1. Performance of the “Bayes-optimal” filter

2. Finite-state “filterability”


1. knowledge of P

2. knowledge of x

LZ + DUDE - based filterAchieves:


– Performance of the “Bayes-optimal” filter– Finite-state “filterability”


• Cuteness

Note: universal filter induced by universal predictor

4

1.4 Filtering

Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:



Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4

1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4

1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4


1.4 Filtering

Problem: Like the discrete denoising problem, but with causality, i.e., the estimate of xi may depend only on(z1, z2, . . . zi)




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4

(cont.)

1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4

1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4


1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4


1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4

(cont.)

1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x



– Performance of the “Bayes-optimal” filter

– Finite-state “filterability”


• Cuteness


4


1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x





• Cuteness


4

(cont.)

1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x






• Cuteness


4

1.4 Filtering




Answers:




1. knowledge of P

2. knowledge of x






• Cuteness


4


1.5 Lossy compression

Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:



Answers:

1. rate distortion curve

2. finite-state compressibiity with distortion


1. knowledge of P

2. knowledge of x

Lossy compression via MCMCShow movieAchieves:

• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source

• O(1) complexity per iteration...

• Cuteness?

• “State of the art” performance on discrete (small alphabet) data

References

[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.

[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.

[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.

[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.

[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.

[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.

[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.

[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751

[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141

[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).

5





Answers:




1. knowledge of P

2. knowledge of x




• Cuteness?


References











5





Answers:




1. knowledge of P

2. knowledge of x




• Cuteness?


References











5





Answers:


2. finite-state compressibility with distortion


1. knowledge of P

2. knowledge of x




• Cuteness?


References











5





Answers:



Yeah, but can they be attained by us mortals?

1. knowledge of P

2. knowledge of x




• Cuteness?


References











5






Answers:




1. knowledge of P

2. knowledge of x




• Cuteness?


References











5






Answers:




1. knowledge of P

2. knowledge of x




• Cuteness?


References











5

introduce noise such that resulting signal is• corrupted• more compressible






Answers:




1. knowledge of P

2. knowledge of x




• Cuteness?


References











5


Idea:






Answers:




1. knowledge of P

2. knowledge of x




• Cuteness?


References











5





Answers:




1. knowledge of P

2. knowledge of x




• Cuteness?


References











5


Idea:


http://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://ww

Date post:	25-Jan-2021
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Universal Schemes in Information Theory Introductory lecture … · 2011. 9. 27. · Universal...

Documents