Dirty Paper Coding

Gwanmo Ku

Adaptive Signal Processing and Information Theory Research Group

Nov. 7, 2012

Outline

Introduction

Writing on Dirty Paper

System Model & Channel Capacity

Achievable Rate Region

Encoding & Decoding

Achievability Proof

Example of Dirty Paper Coding

2/22

Outline

Introduction

Writing on Dirty Paper

System Model & Channel Capacity

Achievable Rate Region

Encoding & Decoding

Achievability Proof

Example of Dirty Paper Coding

Writing on Dirty Paper (Costa, 83)

4/22

Costa’s Idea

⊕ ⊕ Encoder Decoder

𝐒 ∼ 𝓝(𝟎,𝑸) 𝒁 ∼ 𝓝(𝟎,𝑵)

𝑊 𝒀 𝑊

𝒀 = 𝑿 + 𝑺 + 𝒁

State noise source (AWGN)

Known at Encoder Channel noise channel (AWGN)

𝑿

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵)

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵 + 𝑸)

State 𝑆 does not effect the capacity

Average power constraint

𝑷

Capacity

Writing on Dirty Paper (Costa, 83)

5/22

S : a piece of paper covered with normally distributed dots spot

W : a message on it wit a limited amount of ink (𝑷)

How much information can we reliably send?

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵)

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵 + 𝑸)

Dirt spots do not effect the capacity

⊕ ⊕ Encoder Decoder

𝐒 ∼ 𝓝(𝟎,𝑸) 𝒁 ∼ 𝓝(𝟎,𝑵)

𝑊 𝒀 𝑊

𝒀 = 𝑿 + 𝑺 + 𝒁

First noise source (AWGN)

Known at Encoder Second noise channel (AWGN)

𝑿 Average power constraint

𝑷

Writing on Dirty Paper (Costa, 83)

6/22

Practical Importance

Diversity without capacity loss

Layering by dirt spots (Watermarking)

Extension to MIMO case for Diversity

Centered at on 𝐒𝟏 Centered at on 𝐒𝟐

Outline

Introduction

Writing on Dirty Paper

System Model & Channel Capacity

Achievable Rate Region

Encoding & Decoding

Achievability Proof

Example of Dirty Paper Coding

Writing on Dirty Paper

8/22

System Model

Capacity

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵)

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵 + 𝑸)

State 𝑆 does not effect the capacity

⊕ ⊕ Encoder Decoder

𝐒 𝐙

𝑊

𝑆𝑛

𝑿𝒏 𝒀𝒏 𝑊

𝒀 = 𝑿 + 𝑺 + 𝒁

State

S ∼ 𝒩(0, 𝑄)

Channel

Z ∼ 𝒩(0, 𝑁)

1

𝑛 𝑋𝑖

2 ≤ 𝑃

𝑛

𝑖=1

𝑊 ∈ {1,… , 𝑒𝑛𝑅}

Independent

Known at

Encoder

Capacity Region

9/22

Refer to Gelfand Pinsker Theorem

Capacity of the with DM state (S) known only at Encoder

Extension to DMC with continuous alphabet

Proof : Refer to Gelfand Pinsker Paper or El Gamal’ book (Ch 7.6)

Capacity Region of DPC

Normally Distributed State 𝑺

Input power constraint 𝑷

AWGN channel with variance of 𝑸

𝑪𝑺𝑰−𝑬 = max𝒑(𝒖,𝒙|𝒔)

[𝑰 𝑼; 𝒀 − 𝑰 𝑼; 𝑺 ]

sup𝒑(𝒖,𝒙|𝒔)

[𝑰 𝑼; 𝒀 − 𝑰 𝑼; 𝑺 ]

𝟏

𝟐𝐥𝐨𝐠 (𝟏 +

𝑷

𝑵)

𝑪𝑺𝑰−𝑬 = sup𝒑(𝒖,𝒙|𝒔)

[𝑰 𝑼𝒅; 𝒀𝒑 − 𝑰 𝑼𝒅; 𝑺𝒒 ]

Encoding

10/22

Encoding Scheme

Generate 𝒆𝒏[𝑰 𝑼;𝒀 −𝝐] iid sequences 𝑼

According to Uniform Distribution over the set of typical 𝑈

Distribute these sequences uniformly over 𝒆𝒏𝑹 bins

Each sequence 𝑢 with bin index 𝑖(𝑢)

Given state vector 𝑺 and the message 𝑾

Look in bin 𝑊 for a sequence 𝑈 s.t. joint typical (𝑈, 𝑆)

Choose 𝑿 s.t. jointly typical (𝑿,𝑼, 𝑺), send it

Decoding

11/22

Decoding Scheme

Look for unique sequence 𝑼 s.t. jointly typical (𝑼, 𝒀)

Declare Error

More than one or no such sequence exist

Set the estimate 𝑾

Equal to the index of bin containing the obtained sequence 𝑈

If 𝑹 < 𝑰 𝑼; 𝒀 − 𝑰 𝑼; 𝑺 − 𝝐 − 𝜹

The probability of error averaged over all codes

decrease exponentially to zero as 𝑛 → ∞

Achievability Proof

12/22

𝑹 ≤𝟏

𝟐𝐥𝐨𝐠 (𝟏 +

𝑷

𝑵)

Step 1 : Set 𝑈 = 𝑋 + 𝛼𝑆

𝐼(𝑈; 𝑌) = ℎ 𝑌 − ℎ(𝑌|𝑈)

= ℎ 𝑋 + 𝑆 + 𝑍 − ℎ(𝑋 + 𝑆 + 𝑍|𝑋 + 𝛼𝑆)

= ℎ 𝑋 + 𝑆 + 𝑍 + ℎ 𝑋 + 𝛼𝑆 − ℎ(𝑋 + 𝛼𝑆, 𝑋 + 𝑆 + 𝑍)

=1

2ln[ 2 𝜋𝑒 𝑛(𝑃 + 𝑄 + 𝑁)(𝑃 + 𝛼2𝑄)

− 1

2ln{ 2 𝜋𝑒 𝑛[ 𝑃 + 𝑄 + 𝑁 𝑃 + 𝛼2𝑄 − 𝑃 + 𝛼𝑄 2]}

=1

2ln(𝑃 + 𝑄 + 𝑁)(𝑃 + 𝛼2𝑄)

𝑃𝑄 1 − 𝛼 2 + 𝑁(𝑃 + 𝛼2𝑄)

Achievability Proof

13/22

𝐼(𝑈; 𝑆) = ℎ 𝑈 − ℎ(𝑈|𝑆)

= ℎ 𝑋 + 𝛼𝑆 − ℎ(𝑋 + 𝛼𝑆|𝑆)

= ℎ 𝑋 + 𝛼𝑆 + ℎ 𝑋

=1

2ln[ 2 𝜋𝑒 𝑛(𝑃 + 𝛼2𝑄)] −

1

2ln[ 2 𝜋𝑒 𝑛𝑃]

=1

2ln(𝑃 + 𝛼2𝑄)

𝑃

Let 𝑅 𝛼 = 𝐼 𝑈; 𝑌 − 𝐼(𝑈; 𝑆),

𝑅(𝛼) =1

2ln

𝑃(𝑃 + 𝑄 + 𝑁)

𝑃𝑄 1 − 𝛼 2 + 𝑁(𝑃 + 𝛼2𝑄)

𝐼(𝑈∗, 𝑆) =1

2ln[1 +

𝑃𝑄

𝑃 + 𝑁]

Achievability Proof

14/22

Find maximum 𝑅(𝛼)

Fixed 𝑃(𝑃 + 𝑄 + 𝑁)

minimize 𝑃𝑄 1 − 𝛼 2 + 𝑁(𝑃 + 𝛼2𝑄)

𝛼∗ = argmin𝛼[𝑃𝑄 1 − 𝛼 2 + 𝑁 𝑃 + 𝛼2𝑄 ] =

−2𝑃𝑄

−2𝑄(𝑁+𝑃)=

𝑃

𝑃+𝑁

Also,

𝑅(𝛼) =1

2ln

𝑃(𝑃 + 𝑄 + 𝑁)

𝑃𝑄 1 − 𝛼 2 + 𝑁(𝑃 + 𝛼2𝑄)

𝑹 𝜶 ≤ 𝑹 𝜶∗ |𝜶∗=

𝑷𝑷+𝑵

𝐼(𝑈∗, 𝑌) =1

2ln[1 +

𝑃(𝑃 + 𝑄 + 𝑁)

𝑁(𝑃 + 𝑁)] 𝐼(𝑈∗, 𝑆) =

1

2ln[1 +

𝑃𝑄

𝑃 + 𝑁]

=1

2ln[1 +

𝑃

𝑁]

Achievability Proof

15/22

Rate Regions and Capacity

𝑹(𝜶∗)

Encoding & Decoding with 𝜶∗

16/22

Encoding Scheme

Generate 𝒆𝒏[𝑰 𝑼∗;𝒀 −𝝐] sequences 𝑼

With components independently drawn according to 𝓝(0, 𝑃 + 𝛼∗2𝑄)

Place these sequences into 𝒆𝒏𝑹 = 𝒆𝒏(𝑪∗−𝟐 𝝐) bins

Each bin contain same number of sequences

Share Codebook between Encoder and Decoder

Given a State Vector 𝑺 = 𝑺𝟎 and a message 𝑾 = 𝒌

Look for jointly typical pair (𝑈, 𝑆0) among the 𝑈 in bin 𝑘

𝑈 − 𝛼∗𝑆0 𝑆′0 ≤ 𝛿

Encoder calculate 𝐗𝟎 = 𝐔𝟎 − 𝛂∗𝐒𝟎 , send 𝐗𝟎

Encoding & Decoding

17/22

Decoding Scheme

Receive 𝒀 = 𝒀𝟎

Look for a sequence 𝑈 s.t jointly typical (𝑈, 𝑌0)

Declare Error

More than one or no such sequence

Find 𝑈0

Set the estimate 𝑾

Equal to the index of bin containing this sequence

The probability of error averaged over random choice of code

decrease exponentially to zero as 𝑛 → ∞

Conclusion of DPC

18/22

System Model

Capacity

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵)

𝟏

𝟐 𝐥𝐧 (𝟏 +

𝑷

𝑵 + 𝑸)

State 𝑆 does not effect the capacity

⊕ ⊕ Encoder Decoder

𝐒 𝐙

𝑊

𝑆𝑛

𝑿𝒏 𝒀𝒏 𝑊

𝒀 = 𝑿 + 𝑺 + 𝒁

State

S ∼ 𝒩(0, 𝑄)

Channel

Z ∼ 𝒩(0, 𝑁)

1

𝑛 𝑋𝑖

2 ≤ 𝑃

𝑛

𝑖=1

𝑊 ∈ {1,… , 𝑒𝑛𝑅}

Independent

Known at

Encoder

Outline

Introduction

Writing on Dirty Paper

System Model & Channel Capacity

Achievable Rate Region

Encoding & Decoding

Achievability Proof

Example of Dirty Paper Coding

Example of DPC

20/22

DPC for Gaussian BC

Assumptions

𝒀𝟏 = 𝑿 + 𝑺𝟏 + 𝒁𝟏

𝒀𝟐 = 𝑿 + 𝑺𝟐 + 𝒁𝟐

𝑆1 ∼ 𝒩(0, 𝑄1) 𝑆2 ∼ 𝒩(0, 𝑄2)

𝑍1 ∼ 𝒩(0,𝑁1) 𝑍2 ∼ 𝒩(0,𝑁2) 𝑁2 ≥ 𝑁1

Average Power Constraint 𝑃 on 𝑋

Two States 𝑆1 and 𝑆2 are only known at Encoder

𝑹𝟏 ≤ 𝑪(𝜶𝑷

𝑵𝟏)

𝑹𝟐 ≤ 𝑪((𝟏 − 𝜶)𝑷

𝜶𝑷 + 𝑵𝟐)

𝜶 ∈ [𝟎, 𝟏]

Example of DPC

21/22

Encoding & Decoding

Split 𝑋 into two independent parts 𝑋1 and 𝑋2

𝑋1 ∼ 𝓝(0, 𝛼𝑃) and 𝑋2 ∼ 𝓝(0, (1 − 𝛼)𝑃)

For weaker receiver 𝑌2,

𝑌2 = 𝑋2 + 𝑆2 + (𝑋1 + 𝑍2) with known state 𝑆2

Taking 𝑈2 = 𝑋2 + 𝛽2𝑆2 with 𝛽2 =1−𝛼 𝑃

𝑃+𝑁2

Then, achieve

𝑹𝟐 ≤ 𝑰 𝑼𝟐; 𝒀𝟐 − 𝑰 𝑼𝟐; 𝑺𝟐 ≤ 𝑪((𝟏 − 𝜶)𝑷

𝜶𝑷 + 𝑵𝟐)

Example of DPC

22/22

For stronger receiver 𝑌1,

𝑌1 = 𝑋1 + (𝑋2 + 𝑆1) + 𝑍1 with known state 𝑆1

Taking 𝑈1 = 𝑋1 + 𝛽1(𝑋2 + 𝑆1) with 𝛽1 =𝛼𝑃

𝛼𝑃+𝑁1

Then, achieve

𝑹𝟏 ≤ 𝑪(𝜶𝑷

𝑵𝟏)

Converse Proof of

Capacity Region of DPC

Gwanmo Ku

Adaptive Signal Processing and Information Theory Research Group

Nov. 9, 2012

Outline

Converse Proof System Structure and Gel’fand Pinsker Theorem

Csiszar Sum Equality

Applying Non-causality in Capacity

2/11

References 1. S. I. Gelf’and and M. S. Pinsker, “Coding for channel with random parameters”,

Problems of Control and Information Theory, Vol 9, No 1, pp 19-31, 1980

2. Abbas El Gamal and Young-han Kim, “Network Information Theory”

3. Thomas Cover, “Elements of Information Theory”

Non-causal State Information at Encoder

3/11

DMC with DM State

Applying Gel’fand Pinsker Theorem

Encoder Decoder

𝑺𝐧, iid ~ 𝒑(𝒔)

𝑀 𝒀𝒏 𝑀�

Non-causally Known Random State Information

at Encoder

Channel

𝑚 ∈ {1, … , 2𝑛𝑛}

𝑪𝑺𝑺−𝑬 = max𝑝 𝑢 𝑠 ,𝑥=𝑓(𝑢,𝑠)

[𝑺 𝑼;𝒀 − 𝑺 𝑼;𝑺 ]

𝑿𝒏

𝑪𝑮𝑮𝒍′𝒇𝒇𝒏𝒇 = max𝑝(𝑢,𝑠)

[𝑺 𝑼;𝒀 − 𝑺 𝑼;𝑺 ]

𝑓(𝑢, 𝑠)

Converse Proof

4/11

⋅ Let 𝑅 be an achievable rate, ∃ (2𝑛𝑛 ,𝑛) code with 𝑃𝑒 → 0 as 𝑛 → ∞.

⋅ Find the auxiliary random variable 𝑈𝑖 that forms

Markov Chain 𝑼𝒊 → 𝑿𝒊,𝑺𝒊 → 𝒀𝒊,

𝑛𝑅 = 𝐻(𝑀)

= 𝐻 𝑀 −𝐻 𝑀 𝑌𝑛 + 𝐻(𝑀|𝑌𝑛)

= 𝐼(𝑀;𝑌𝑛) + 𝐻(𝑀|𝑌𝑛) ← By Fano’s Iemma 𝐻 𝑀 𝑌𝑛 ≤ 𝑛 𝜖𝑛

≤ 𝐼 𝑀;𝑌𝑛 + 𝑛𝜖𝑛

= 𝐻 𝑌𝑛 − 𝐻 𝑌𝑛 𝑀 + 𝑛 𝜖𝑛

← +0

Converse Proof

5/11

← By Chain Rule

= 𝐻 𝑌𝑛 − 𝐻 𝑌𝑛 𝑀 + 𝑛 𝜖𝑛

= �𝐻 𝑌𝑖|𝑌𝑖−1𝑛

𝑖=1

−�𝐻(𝑌𝑖|𝑌𝑖−1,𝑀)𝑛

𝑖=1

+ 𝑛 𝜖𝑛

≤�𝐻 𝑌𝑖

𝑛

𝑖=1

−�𝐻(𝑌𝑖|𝑌𝑖−1,𝑀)𝑛

𝑖=1

+ 𝑛 𝜖𝑛

← conditioning reduces entropy

= �𝐼(𝑌𝑖;𝑌𝑖−1,𝑀) 𝑛

𝑖=1

+ 𝑛 𝜖𝑛

Converse Proof

6/11

= �𝐼(𝑌𝑖;𝑌𝑖−1,𝑀) 𝑛

𝑖=1

+ 𝑛 𝜖𝑛

Define 𝑛 random variables 𝑈1, …, 𝑈𝑛 as follow

𝑈𝑖 = (𝑀,𝑌1, … ,𝑌𝑖−1, 𝑆𝑖+1, … , 𝑆𝑛) = (𝑀,𝑌𝑖−1, 𝑆𝑖+1𝑛 ) ≅ (𝑄,𝑈𝑄)

Where 𝑄 = (𝑀,𝑌𝑖−1) and 𝑈𝑄 = 𝑆𝑖+1𝑛

𝐼 𝑌𝑄;𝑄 = 𝐼 𝑌𝑄;𝑈𝑄,𝑄 − 𝐼(𝑌𝑄;𝑈𝑄|𝑄)

�𝐼(𝑌𝑖;𝑌𝑖−1,𝑀) 𝑛

𝑖=1

= �[𝑺 𝑴,𝒀𝒊−𝟏,𝑺𝒊+𝟏𝒏 ;𝒀𝒊 − 𝑺 𝒀𝒊;𝑺𝒊+𝟏𝒏 𝑴,𝒀𝒊−𝟏 ] 𝒏

𝒊=𝟏

When 𝑄 is independent 𝑌𝑄 = 𝑌𝑖, then

By Eq. 15.281 in Elements of Information Theory ,Thomas Cover

𝐼 𝑌𝑄;𝑈𝑄 𝑄 = 𝐼 𝑌𝑄;𝑈𝑄,𝑄

Converse Proof

7/11

← By Csiszar Sum Equality

≤�[𝐼 𝑀,𝑌𝑖−1, 𝑆𝑖+1𝑛 ;𝑌𝑖 − 𝑺 𝒀𝒊;𝑺𝒊+𝟏𝒏 𝑴,𝒀𝒊−𝟏 ] 𝑛

𝑖=1

+ 𝑛 𝜖𝑛

= �[𝐼 𝑀,𝑌𝑖−1, 𝑆𝑖+1𝑛 ;𝑌𝑖 − 𝑺 𝒀𝒊−𝟏;𝑺𝒊 𝑺𝒊+𝟏𝒏 ,𝑴 ] 𝑛

𝑖=1

+ 𝑛 𝜖𝑛

�𝑺 𝒀𝒊;𝑿𝒊+𝟏𝒏 𝒀𝒊−𝟏𝒏

𝒊=𝟏

= �𝑺(𝒀𝒊−𝟏;𝑿𝒊|𝑿𝒊+𝟏𝒏 ) 𝒏

𝒊=𝟏

Csiszar Sum Equality For Two random vectors 𝑋𝑛 and 𝑌𝑛 with joint probability distribution, then

Csiszar Sum Equality Proof

8/11

�𝐼 𝑌𝑖;𝑋𝑖+1𝑛 𝑌𝑖−1𝑛

𝑖=1

= � � 𝐼(𝑋𝑗;𝑌𝑖|𝑌𝑖−1,𝑋𝑗+1𝑛 ) 𝑛

𝑗=𝑖+1

𝑛

𝑖=1

= ��𝐼(𝑋𝑗;𝑌𝑖|𝑌𝑖−1,𝑋𝑗+1𝑛 ) 𝑗−1

𝑖=1

𝑛

𝑗=2

= �𝐼(𝑋𝑗;𝑌𝑗−1|𝑋𝑗+1𝑛 )𝑛

𝑗=2

= �𝐼(𝑋𝑗;𝑌𝑗−1|𝑋𝑗+1𝑛 )𝑛

𝑗=1

= �𝐼(𝑋𝑖;𝑌𝑖−1|𝑋𝑖+1𝑛 )𝑛

𝑖=1

Chain rule

Switching of summation

Chain rule

Also define 𝐼 𝑋1;𝑌0 𝑋2𝑛 = 0

Switching of Index

Converse Proof

9/11

= �[𝐼 𝑀,𝑌𝑖−1, 𝑆𝑖+1𝑛 ;𝑌𝑖 − 𝐼 𝑌𝑖−1; 𝑆𝑖 𝑆𝑖+1𝑛 ,𝑀 ] 𝑛

𝑖=1

+ 𝑛 𝜖𝑛

= �[𝐼 𝑀,𝑌𝑖−1, 𝑆𝑖+1𝑛 ;𝑌𝑖 − 𝐼(𝑆𝑖+1𝑛 ,𝑀,𝑌𝑖−1; 𝑆𝑖)] 𝑛

𝑖=1

+ 𝑛 𝜖𝑛

Independency between (𝑀, 𝑆𝑖+1𝑛 ) and 𝑆𝑖

Use 𝑈𝑖 = (𝑀,𝑌𝑖−1, 𝑆𝑖+1𝑛 )

𝑛𝑅 ≤�[𝐼 𝑈𝑖;𝑌𝑖 − 𝐼(𝑈𝑖; 𝑆𝑖)] 𝑛

𝑖=1

+ 𝑛 𝜖𝑛

≤ 𝑛 max𝑝(𝑢,𝑥|𝑠)

[𝐼 𝑈;𝑌 − 𝐼 𝑈; 𝑆 ] + 𝑛 𝜖𝑛

Converse Proof

10/11

𝑅 ≤ max𝑝(𝑢,𝑥|𝑠)

[𝐼 𝑈;𝑌 − 𝐼 𝑈; 𝑆 ] + 𝜖𝑛

Also show that it suffices to maximize over 𝑝(𝑢|𝑠) and function 𝑥 = 𝑓(𝑢, 𝑠)

𝑝 𝑢, 𝑥 𝑠 = 𝑝 𝑢 𝑠 𝑝(𝑥|𝑢, 𝑠)

𝑝 𝑥 𝑢, 𝑠 = 0,1

Since 𝑥 = 𝑓(𝑢, 𝑠) is deterministic function of 𝑢 and 𝑠,

For fixed 𝑝(𝑢|𝑠),

𝑅 ≤ max𝑝 𝑢 𝑠 ,𝑥=𝑓(𝑢,𝑠)

[𝐼 𝑈;𝑌 − 𝐼 𝑈; 𝑆 ] + 𝜖𝑛 Fixed

Only over 𝐼(𝑈;𝑌) Convex of 𝑝(𝑦|𝑢) fixing 𝑝(𝑢|𝑠)

Converse Proof

11/11

Since mutual information 𝐼(𝑈;𝑌) is a convex function of 𝑝(𝑦|𝑢) for a fixed 𝑝(𝑢|𝑠)

𝑝 𝑦 𝑢 = �𝑝 𝑠 𝑢 𝑝 𝑥 𝑠,𝑢 𝑝(𝑦|𝑥, 𝑠)𝑥,𝑠

linear in 𝑝(𝑥|𝑢, 𝑠)

𝐼 𝑈;𝑌 is also convex of 𝑝(𝑥|𝑢, 𝑠)

Maximum is attained at an extreme points of the set of pmf 𝑝(𝑥|𝑢, 𝑠)