Fundamentals of Bit-Interleaved Coded Modulation and Reliable Source ... - Home |...

Fundamentals of Bit-Interleaved

Coded Modulation and Reliable

Source Transmission

Li Peng

Department of Engineering

University of Cambridge

Supervisor

Dr. Albert Guillen i Fabregas

This dissertation was submitted for the degree of Doctor of

Philosophy

Dec 2012

ThesisFigs/UnivShield.eps

To Grandma, Mom, Dad and Jing

Declaration

I hereby declare that this dissertation is the result of my own work andincludes nothing which is the outcome of work done in collaboration

except where specifically indicated in the text. I also declare thatthe length of this dissertation is less than 65,000 words and that the

number of figures is less than 150.

Fundamentals of Bit-Interleaved Coded

Modulation and Reliable Source TransmissionLi Peng

Department of Engineering

University of Cambridge

This dissertation was submitted for the degree of Doctor of

Philosophy

Dec 2012

In the first part of this dissertation, we study the achievable rates, error expo-

nents and error probability of Bit-Interleaved Coded Modulation (BICM) usingthe mismatched decoder model. We consider BICM with probabilistic shaping,

where instead of the true bit or symbol probabilities and constellation used atthe transmitter, the decoder can use arbitrary probabilities or reference constel-

lations. We first study both the generalized mutual information (GMI) and theLM rate and we investigate the effect of shaping on the achievable rates. In

addition, we study the corresponding low and high SNR regimes and show that

even in the presence of extra sources of mismatch, shaping can bridge the gapbetween BICM and coded modulation (CM) in terms of rate. We then analyze

the error exponents and expurgated error exponents of mismatched decoder fordifferent code ensembles, namely, i.i.d and cost-constrained i.i.d. ensembles. We

demonstrate the benefits of using cost-constrained i.i.d. ensemble and give theoptimal system parameters. We also generalize the expurgated error exponent for

mismatched decoder and show the collective effect of the codewords expurgationand codewords cost constraint on the error exponents. Finally, our information-

theoretic results are verified with error probability analysis, and some numericalsimulations are presented.

In the second part, we study analogue source transmission over multiple-inputand multiple-output (MIMO) block-fading channels, which is relevant for outage-

limited, non-ergodic applications. Unlike previous work which considers the end-to-end expected distortion or distortion vs. outage as the figure of merit, we

study the excess distortion probability, i.e., the probability that the instantaneous

distortion exceeds a certain desired target. We establish the achievability andconverse theorems associated to excess distortion probability by studying the

upper and lower bounds. A simple consequence of this analysis is that source-channel separation, not only achieves the optimal SNR exponents, but it can

also achieve the fundamental limit. This is in contrast to the works that studythe expected distortion, and shows that the expected distortion approach lacks a

coding theorem and a converse in the outage-limited block-fading channels.

Acknowledgements

This dissertation is the destination of my PhD journey for the last

four years. It is an adventurous trip with both joyful and painful mo-ments. The first time I realize how much I have grown, both mentally

and physically, over the years was when I looked at my enrollment

photo at the entrance of my research group one day last year. ThenI realized how lucky I am to be able to spend four years exploring

the unknown universe in a place like Cambridge. I feel grateful and Iwould like to acknowledge a number people who have mentored and

accompanied me every step of the journey, because without you mytime in Cambridge could not have been complete.

First, I am deeply thankful to my supervisor Prof. Albert Guillen i

Fabregas for giving me the opportunity to study in one of the bestuniversities in the world. Albert has been very patient and supportive

in all circumstances. He encouraged me to define my own researchproblem and work independently. I admire his strict attitude towards

research and in turn it has shaped my understanding of how researchshould be done. Our discussions over all these years have taught me

to uncover the principles of the universe with an inquiring mind, andI hope that we can keep working together in the future.

I would like to express my sincere gratitude to Dr. Alfonso Mar-

tinez. Alfonso is a very creative thinker and gave me many valuablesuggestions on my research when I was stuck. Every time after our

discussion, be it academic or non-academic, I am able to walk awaywith a clearer mind. He also gave many helpful suggestions on my

writing and presentation skills. In particular, some parts of this dis-sertation have been improved following his comments. I hope that we

can continue to work together.

The work included in this dissertation has mostly benefited from theinteraction with the members of our research group. Thanks to Dr.

Alex Alvarado for giving me some stimulating comments on someparts of my research. Thanks to Dr. Jossy Sayir and Dr. Tobias

Koch, who have shown me the other area of research on informationtheory. It has broadened by vision and strengthened my interest in in-

formation theory. Thanks to Jonathan Scarlett, who has proofread mydissertation. His feedback has improved the presentation of this dis-

sertation. Thanks to Dr. Taufiq Asyhari for being my research matein the last 3.5 years. We started our PhD together and shared many

interesting academic and non-academic discussions. I also would like

to thank Dr. Adria Tauste, Dr. Gonzalo Vazquez-Vilar, Jing Guo foryour help and support on my research.

This PhD would not have been possible without the support fromthe China Scholarship Council, Cambridge Overseas Trust and Henry

Lester Trust.

I would like to thank the Division F staff in Cambridge University En-

gineering Department, in particular Rachel Fogg, Janet Milne, LauraReed and Roger Wareham for providing administrative and comput-

ing supports.

I spent seven months in UPF, Barcelona, Spain. Muchas Gracias toall those at UPF who made my stay in Barcelona so pleasant: Joana

Clotet, Vanessa Jimenez and Beatriz Abad.

Doubtless, this work could not have been possible without the mentalsupport of my lovely friends. Thanks to Xiaochuan Yuan for being

my best buddy and tennis partner for the past three years! Thanks toJianqiang Wang, Xiaowen Zeng and Tuo Shi for all the unforgettable

moments we share in Cambridge. Thanks to Yingsong Zhang forbeing an elder sister to me in our research group. Thanks to Jun

Kong, Yi Sun, Chunjing Gu for taking me to travel around the UK.Thanks to Ni Yang for the mental support during the final stages of

my PhD. Thanks to Peng Chen, Hao Hu, Gengyang Luo, Hsuan-Yin

Lin, Baihe Xu and Wei Qin for making my stay in Barcelona such awonderful time. I am also indebted to Xi Chen, Ligong Yang, Ioannis

Mitsos, Aran Rachmin, Huang Tan, Meidai Sun, Rui Hao, ChenweiWang, among others.

Finally, I would like to thank my family for their unconditional sup-

port and love. I am who I am today because of everything my parentshave given and taught me. Their constant care and concern warmed

my heart so that I do not feel lonely being so far away from home. Iam deeply thankful to Jing, who has stayed by my side every step of

the journey. I am so lucky to have you. I understand that you havesacrificed a lot for me and it has been hard to have a long distance

relationship across two continents from last year. I am sorry for thetime that I have missed and I will make it up for you. I am also

thankful to my grandma, who have loved me so much ever since I was

a child. May you rest in peace in heaven.

Contents

Contents xi

List of Figures xv

List of Tables xxi

Nomenclature xxiii

1 Preliminaries 1

1.1 Dissertation Outline . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

I Mismatched Shaping Schemes for Bit-Interleaved CodedModulation 5

2 Introduction 7

3 System Setup and Code Ensemble 9

3.1 Channel Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2 Mismatched Decoder . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2.1 Generalized Mutual Information (GMI) . . . . . . . . . . . 10

3.2.2 The LM rate . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3 Coded Modulation . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.4 Bit-Interleaved Coded Modulation . . . . . . . . . . . . . . . . . . 12

4 BICM Achievable Rates with Probabilistic Shaping 15

4.1 Discrete Memoryless Channels (DMC) . . . . . . . . . . . . . . . 17

4.1.1 Numerical Results . . . . . . . . . . . . . . . . . . . . . . 18

4.1.2 When Is LM Rate Significantly Larger Than the GMI? . . 19

xi

CONTENTS

4.1.3 Non-product Distribution . . . . . . . . . . . . . . . . . . 21

4.2 The AWGN Channel . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.2.1 Channel Model and Achievable Rates . . . . . . . . . . . . 24

4.2.2 Low SNR Regime for GMI . . . . . . . . . . . . . . . . . . 30

4.2.3 High SNR Regime for GMI . . . . . . . . . . . . . . . . . 34

4.2.4 Non-product Distribution . . . . . . . . . . . . . . . . . . 36

4.3 Chapter Review and Conclusion . . . . . . . . . . . . . . . . . . . 37

5 Random Coding Error Exponent of BICM 41

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.2 Maximum Likelihood Decoder . . . . . . . . . . . . . . . . . . . . 45

5.3 Mismatched BICM Decoder . . . . . . . . . . . . . . . . . . . . . 53

5.4 Multiple Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61


6 Error Probability Analysis 65

6.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

6.1.1 BICM Schemes and Its Realizations . . . . . . . . . . . . . 67

6.2 Error Probability of Maximal Metric Decoding . . . . . . . . . . . 70

6.3 Cumulant Transform . . . . . . . . . . . . . . . . . . . . . . . . . 72

6.4 Saddlepoint Approximation (SPA) . . . . . . . . . . . . . . . . . . 75

6.5 Results: BICM Schemes with Convolutional Codes and 8PAM

Modulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

6.5.1 BICM2(2PAM-8PAM) . . . . . . . . . . . . . . . . . . . . 75

6.5.2 BICM3(2PAM-8PAM) . . . . . . . . . . . . . . . . . . . . 81


II Reliable Source Transmission over MIMOBlock-FadingChannels 85

7 Introduction to the Source Transmission Problem 87

8 System Model and Basic Principles 91

8.1 End-to-End Systems: The Rate Distortion Theory Paradigm . . . 92

8.1.1 The Source Coding Problem . . . . . . . . . . . . . . . . . 94

8.1.2 Excess Distortion Probability and its Error Exponent . . . 95

8.2 Separation and Joint Source-Channel Coding . . . . . . . . . . . . 96

8.2.1 The Separation Scheme . . . . . . . . . . . . . . . . . . . . 96

8.2.2 Joint Source-Channel Coding . . . . . . . . . . . . . . . . 98

xii

CONTENTS

9 Excess-Distortion Probability in Block-Fading Channels 101

9.1 Previous Work on Joint Source-Channel Coding over Non-ErgodicChannels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

9.1.1 MIMO Block-Fading Channel . . . . . . . . . . . . . . . . 1019.1.2 Fundamental Principles . . . . . . . . . . . . . . . . . . . . 103

9.1.3 Expected Distortion Perspective . . . . . . . . . . . . . . . 105

9.1.4 Distortion vs. Outage . . . . . . . . . . . . . . . . . . . . 1089.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . 109

9.3 Lower Bound to Excess Distortion Probability . . . . . . . . . . . 110

9.3.1 SNR Exponent . . . . . . . . . . . . . . . . . . . . . . . . 1139.4 Upper bound to the Excess Distortion Probability for Separation . 115

9.4.1 Separation Optimality . . . . . . . . . . . . . . . . . . . . 117

9.4.2 SNR Exponent . . . . . . . . . . . . . . . . . . . . . . . . 1189.4.3 Connection with the Work of Liang et al. [59, 60] . . . . . 120

9.5 Layered Source Coding . . . . . . . . . . . . . . . . . . . . . . . . 122


10 Conclusion and Future Work 12710.1 Main Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . 127

10.1.1 Part I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

10.1.2 Part II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12810.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

A 131A.1 Ibicm0,s (s, P bicm

X ) and Ibicm1,s

(s, a(·), P bicm

X

)Properties . . . . . . . . . 131

A.1.1 Concavity of Ibicm0,s (s, P bicmX ) . . . . . . . . . . . . . . . . . 131

A.2 Concavity of Ibicm1,s

(s, a(·), P bicm

X

). . . . . . . . . . . . . . . . . . . 132

A.3 Shaping Algorithm for BICM1 GMI over DMC . . . . . . . . . . . 132

A.3.1 Bit-by-bit Algorithm . . . . . . . . . . . . . . . . . . . . . 134A.3.2 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . 135

A.4 Proof of Expression for c2 . . . . . . . . . . . . . . . . . . . . . . 136

A.4.1 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 146A.4.1.1 E[T 1

j (x, Z, b)] . . . . . . . . . . . . . . . . . . . . 147

A.4.1.2 E[T 2j (x, Z, b)] . . . . . . . . . . . . . . . . . . . . 148

A.4.1.3 E[T 3j (x, Z, b)] . . . . . . . . . . . . . . . . . . . . 149

A.4.1.4 E[T 4j (x, Z, b)] . . . . . . . . . . . . . . . . . . . . 150

A.4.1.5 E[T 5j (x, Z, b)] . . . . . . . . . . . . . . . . . . . . 151

A.4.1.6 E[T 6j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 152

A.4.1.7 E[T 7j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 153

A.4.1.8 E[T 8j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 155

xiii

CONTENTS

A.4.1.9 E[T 9j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 157

A.4.1.10 E[T 10j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 158

A.4.1.11 E[T 11j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 159

A.4.1.12 E[T 12j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 161

A.4.1.13 E[T 13j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 162

A.4.1.14 E[T 14j (x, Z)] . . . . . . . . . . . . . . . . . . . . . 164

A.5 Proof of Proposition 4.2.1 and 4.2.2 . . . . . . . . . . . . . . . . . 165

A.5.1 Proof of Proposition 4.2.1 . . . . . . . . . . . . . . . . . . 165A.5.2 Proof of Proposition 4.2.2 . . . . . . . . . . . . . . . . . . 165

B 167

B.1 Proof of Eqs. (5.8) to (5.12) . . . . . . . . . . . . . . . . . . . . . 167B.2 Concavity of E1′ and E1 . . . . . . . . . . . . . . . . . . . . . . . 169

B.2.1 Concavity of E1′(ρ, s, a(·)

). . . . . . . . . . . . . . . . . . 169

B.2.2 Concavity of E1

(ρ, s, r, r, a(·)

). . . . . . . . . . . . . . . . 170

B.3 Proof of Proposition 5.2.1 . . . . . . . . . . . . . . . . . . . . . . 170B.4 Proof of Lemma 5.2.1 . . . . . . . . . . . . . . . . . . . . . . . . . 172

B.5 Proof of Theorem 5.2.1 . . . . . . . . . . . . . . . . . . . . . . . . 173B.6 Proof of Theorem 5.2.3 . . . . . . . . . . . . . . . . . . . . . . . . 176

B.7 Proof of Proposition 5.3.1 . . . . . . . . . . . . . . . . . . . . . . 179

References 181

xiv

List of Figures

3.1 BICM random generation of codebook. The length of a BICM

code is mN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.1 Example discrete memoryless 4-ary channel. We index the sym-

bol from top to bottom as x1, x2, x3, x4. The probability that a

transmission error occurs for each symbol is p = 0.1748, hence the

capacity of this channel with coded modulation is 1 bit. . . . . . . 17

4.2 The solid line represents the value of Ibicm0 (PB) throughout the

iterations using the proposed algorithm with uniform initial dis-

tribution. All the dashed lines represent the value of Ibicm0 (PB)

throughout the iterations using the proposed algorithm with a ran-

domly generated initial distribution. The dash-dotted line is the

GMI of uniform input distribution. The dotted line is the Cbicm0

found by exhaustive search. . . . . . . . . . . . . . . . . . . . . . 20

4.3Ibicm1 (P bicm

X )−Ibicm0 (P bicmX )

Ibicm0 (P bicmX )

for BICM schemes with decoding metric q(x, y)

in Eq. (4.10) over all input distributions. . . . . . . . . . . . . . . 22

4.4Ibicm1 (P bicm


Ibicm0 (P bicmX )


in Eq. (4.11) over all input distributions. . . . . . . . . . . . . . . 23

4.5 Comparison of Cbicm0 among different BICM schemes with 8PAM

modulation and Gray labeling [000, 001, 011, 010, 110, 111, 101, 100].

Schemes are shown as in Table 4.4. . . . . . . . . . . . . . . . . . 26

4.6 Optimal constellation X for different snr for BICM1 with shaping

and 8PAM modulation with Gray labeling. A symbol is inactive

means it is used with zero probability. . . . . . . . . . . . . . . . 27

4.7 Optimal constellation X and input distribution for different snr for

the GMI of BICM1 with 8PAM modulation with Gray labeling. . 28

4.8 Example of 8PAM constellation with Gray labeling. We have la-

beled the pairs with numbers. . . . . . . . . . . . . . . . . . . . . 29

xv

LIST OF FIGURES

4.9 Comparison of Cbicm1 among different BICM schemes with 8PAM

modulation with Gray labeling [000, 001, 011, 010, 110, 111, 101, 100].

Schemes are shown as in Table 4.4. . . . . . . . . . . . . . . . . . 31

4.10 BICM performance in the wideband regime. Four different BICM3

configurations with Gray labeling are considered. P ∗B and s∗ de-

note the optimal bit probabilities and s. For 64QAM, we assume

symmetry between in-phase and quadrature, i.e. PB1(0) = PB4(0),

PB2(0) = PB5(0), PB3(0) = PB6(0). . . . . . . . . . . . . . . . . . . 35

4.11 High SNR analysis of different (over different subsets of the prob-

ability space) BICM2 8PAM shaping schemes with Gray labeling.

The decoding reference constellation X = X satisfies (4.21) for all

schemes. The solid line corresponds to BICM2 shaping scheme

introduced in Section 4.2.1. The thick solid line corresponds to

fixing PB1(0) = 0.5, PB2(0) = 1, PB3(0) = 0.5 and maximizing

s. The dotted line corresponds to fixing PB2(0) = 1, PB3(0) =

1 and maximizing PB1(0) and s. The dashed line corresponds

to fixing PB2(0) = 1 and maximizing PB1(0), PB3(0) and s. The

dash-dotted line corresponds to fixing PB2(0) = 0 and maximizing

PB1(0), PB3(0) and s. The horizontal lines plot sups,PBlimsnr→∞

(H(X)−

Hsnr(X|Y )), and indicates the saturating value of the GMI for

snr → ∞. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.12 Cbicm0 for BICM1 and BICM2 transmissions over AWGN channel

using 8PAM as channel inputs. The solid line plots Cbicm0 under

product distribution, and the dotted line plots Cbicm0 under non-

product distribution. Gray labeling is used, and the schemes are

described in Table 4.4. . . . . . . . . . . . . . . . . . . . . . . . . 38

5.1 Error exponents Er,0(R), Er,1(R) for 8PAM Coded Modulation

over the AWGN channel. The uniform input distribution is used.

The exponents are calculated for snr = 0, 5, 10 dB, and the corre-

sponding I(PX) are marked with dots on the R axis. . . . . . . . 45

5.2 The value of a(x)−E[a(X)]i(x)−I(PX)

for 8PAM CM transmission over AWGN

channel for snr = −5dB. The markers ’·’ denote the 8PAM constel-

lation. The values of a(x) are obtained using the gradient descent

type convex optimization method. . . . . . . . . . . . . . . . . . . 47

xvi

LIST OF FIGURES

5.3 The ratio and difference between the LM exponent Er,1(R) and

GMI exponent Er,0(R) for 8PAM CM transmission over the AWGN

channel. The solid line is the ratio of the two exponents and the

dash-dotted line is the difference of the exponents. The dotted

line indicates the largest achievable rate at this SNR. The marker

is the analytical limit of the ratioEr,1(R)

Er,0(R)as R → I(PX) given by

Theorem 5.2.2. We use snr = 5 dB. . . . . . . . . . . . . . . . . . 48

5.4 The limiting value of Er,0(R)

Er,1(R)as R → I for 4PAM, 8PAM and

16PAM CM transmission over AWGN channel for different snr.

The case M → ∞ is also shown by using a continuous uniform

distribution between [−√3,√3] . . . . . . . . . . . . . . . . . . . 52

5.5 GMI and LM rate error exponents for BICM0 (as in Tab. 4.4)

8PAM transmission over AWGN channel at snr = 5 dB. The expo-

nents of CM transmission are also shown as reference. The corre-

sponding I(PX), Ibicm0 (P bicm

X ) and Ibicm1 (P bicmX ) are marked on the

R axis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.6 The GMI and the LM error exponents and expurgated error expo-

nents for BICM1 with equiprobable input distribution (as in Tab.

4.4), using 8PAM transmission over the AWGN channel at snr = 5

dB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

6.1 System diagram of BICM2(2PAM-MPAM) and BICM3(2PAM-MPAM)

transmission scheme with shaping code, whose bit probability is

PB1(0) = 0.5, PB2(0) = 1, . . . , PBm(0) = 1. This shaping code is

obtained by using an Appender that append m − 1 ‘0’ after each

bit at the output of the interleaver. And at the receiver, the LLRs

calculated for each 2nd until m-th bit positions are discarded as

we are only interested in the ones for the 1st bit position . . . . . 69

6.2 Simulated density of Ξb for BICM2(2PAM-MPAM) with the setup

in Figure 6.1. Gray mapping and the AWGN channel are used. We

set snr = 0 dB. The solid line shows the simulated density and

the dashed line plots the density approximated using the Gaussian

approximation method. . . . . . . . . . . . . . . . . . . . . . . . . 72

6.3 Simulated density of Ξb for BICM2(2PAM-MPAM) with the setup

in Figure 6.1. Gray mapping and the AWGN channel are used. We

set snr = 5 dB. The solid line shows the simulated density and

the dashed line plots the density approximated using the Gaussian

approximation method. . . . . . . . . . . . . . . . . . . . . . . . . 73

xvii

LIST OF FIGURES

6.4 Comparison of BER simulation results between BICM2(2PAM-

8PAM) and BICM1(8PAM). Chernoff union bound, saddlepoint

approximation and Gaussian approximation of the BER are also

shown. All results are with Gray mapping and AWGN channel

with rate 18convolutional codes. . . . . . . . . . . . . . . . . . . . 76


8PAM) and BICM1(2PAM). 2PAM closed-form approximation [65],

Chernoff union bound, saddlepoint approximation and Gaussian

approximation of the BER are also shown. All results are with

Gray mapping and AWGN channel with rate 18convolutional codes. 77


8PAM) (rate 12convolutional code), BICM1(8PAM) (rate 1

6con-

volutional code) and BICM1(2PAM) (rate 12convolutional code).

2PAM closed-form approximation [65], Chernoff union bound, sad-

dlepoint approximation and Gaussian approximation of the BER

are also shown. All results are with Gray mapping and AWGN

channel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

6.7 Saddlepoint approximation of the error probability for BICM2(2PAM-

8PAM) (rate 12convolutional code) and BICM1(8PAM) (rate 1

6

convolutional code) with Gray mapping for the AWGN channel. . 79

6.8 Comparison between the Chernoff union bound (CUB) and Gaus-

sian approximation (GA) for BICM3(2PAM-8PAM). The AWGN

channel and Gray mapping are used. The Thick-solid line repre-

sents bounds for BICM3(2PAM-8PAM) BER with input (X1, P 1B).

The Dashed line represents the bounds for BICM3(2PAM-8PAM)

BER with input (X2, P 2B). The Dash-dotted line represents the

bounds for BICM3(2PAM-8PAM) BER with input (X3, P 3B). The

Dotted line represents the bounds for BICM3(2PAM-8PAM) BER

with input (X4, P 4B). The Solid line with marker ’·’ represents the

BER of BICM2(2PAM-8PAM). . . . . . . . . . . . . . . . . . . . 80

6.9 Simulation results and saddlepoint approximation of BER for BICM3(2PAM-

8PAM) transmission over the AWGN channel with Gray mapping.

A rate 18convolutional codes is used. To avoid overcrowding the

figure, we only show two cases here (when the outermost pair is

chosen, and when the innermost pair is chosen.) . . . . . . . . . . 82

8.1 Diagram of source coding. . . . . . . . . . . . . . . . . . . . . . . 93

8.2 Source-channel separate coding system diagram. . . . . . . . . . . 97

8.3 Block diagram of joint source-channel coding scheme. . . . . . . . 99

xviii

LIST OF FIGURES

9.1 Transmission and reception with multiple antennas. The channelcoefficients are described by the nr × nt matrix H . . . . . . . . . 102

9.2 Space-time coding channel model. For convenience of presentationwe have omitted the dependence of x and x on u and u. . . . . . 103

9.3 Distortion exponent as a function of bandwidth ratio for 4 × 1

MIMO, N = 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1069.4 One block of the channel codewords for layered source coding scheme.107

9.5 Excess distortion probability SNR exponents upper bound δe in a4×4 MIMO block-fading channel with B = 2. Different channel

inputs and target distortion D are considered. . . . . . . . . . . . 1149.6 Excess distortion probability SNR exponents upper bound δe in a

4×4 MIMO block-fading channel with B = 2. Target distortionD = 0.05 are considered. . . . . . . . . . . . . . . . . . . . . . . . 115

9.7 SNR exponents of the excess distortion probability and expecteddistortion in a 4 × 4 MIMO block-fading channel with B = 2,

Gaussian inputs and D = 0.05. The thick solid line corresponds tothe excess distortion exponent, while the dashed and dash-dotted

lines correspond to the expected distortion exponent. . . . . . . . 1199.8 Lower bound to excess distortion probability and separation achiev-

ability upper bound with zero mean unit variance Gaussian source

and Gaussian channel inputs, bandwidth ratio b = 2, target distor-tion D = 0.05 in a 2× 2 MIMO system. In this case Rs(D)

b≈ 1.080. 120

9.9 Lower bound to excess distortion probability and separation achiev-ability upper bound with zero mean unit variance Gaussian source

and BPSK channel inputs, bandwidth ratio b = 1.5, number ofblocks B = 2, target distortion D = 0.06 in a 2×2 MIMO system.

In this case Rs(D)b

≈ 1.353. . . . . . . . . . . . . . . . . . . . . . . 1219.10 Example of layered source coding, with pre-assigned channel cod-

ing rate Rc,k, k = 1, 2, . . . , l. The horizontal axis is the rate axisand the vertical axis denotes the distortion level. The dotted hor-

izontal line denotes the target distortion, which lies in the interval[DLS

k∗+1,DLSk∗ ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

xix

LIST OF FIGURES

xx

List of Tables

4.1 BICM Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2 The GMI and LM rate Simulation Results . . . . . . . . . . . . . 19

4.3 Cbicm0 and other optimal parameters for BICM1 . . . . . . . . . . . 22

4.4 BICM Schemes of Interest . . . . . . . . . . . . . . . . . . . . . . 24

4.5 Optimal s for BICM2 and BICM3 with different inputs at snr → 0 34

6.1 PrΞb > 0 with computer simulated density and Gaussian ap-

proximation density. . . . . . . . . . . . . . . . . . . . . . . . . . 74

xxi

LIST OF TABLES

xxii

Nomenclature

Roman Symbols

X Random scalar

x Realizations of random scalar X

X Random vector

x Realizations of random vector X

X Random matrix

X Realizations of random matrix X

X Channel input signal set

M Cardinality of the set X

m Number of bits per symbol

u Index of the message

CM Mismatched Capacity

R Set of real numbers

C Set of complex numbers

b Bandwidth ratio

nt Number of transmit antenna

nr Number of receive antenna

Other Symbols

xxiii

LIST OF TABLES

11 · Indicator function

Acronyms

BICM Bit Interleaved Coded Modulation

GMI Generalized Mutual Infomation

AWGN Additive White Gaussian noise

CM Coded Modulation

MLC Multi-Level Code

SNR Signal-to-Noise Ratio

pmf Probability Mass Function

pdf Probability Density Function

MAP Maximum A Posteriori

DMC Discrete Memoryless Channels

LLR Log Likelihood Ratio

PEP Pairewise Error Probability

GA Gaussian Approximation

SPA Saddlepoint Approximation

JSCC Joint Source Channel Coding

DMS Discrete Memoryless Source

MIMO Multiple-input Multiple-output

HDA Hybrid Digital-Analog Transmission

LS Layered Source Transmission

HLS Hybrid Digital-Analog Transmission with Layered Source

BS Broadcase Strategy with Layered Source

xxiv

Chapter 1

Preliminaries

In this preliminary chapter, we provide an outline of each chapter in the disser-

tation and summarize the work that has been published and in preparation for

publication.

1.1 Dissertation Outline

The dissertation is structured into two main parts.

Part I

Part I studies the fundamental limits of BICM and includes the following five

chapters.

Chapter 2: Introduction

In this chapter, we introduce the background and motivation of the Part I of the

thesis. We also give a brief summary on the previous work and establish links

between classical results and our results.

Chapter 3: System Setup and Code Ensemble

In this chapter, we first describe the memoryless channel model. We then intro-

duce the mismatched decoder model as our tool for studying the BICM system,

and review some of the information-theoretic results on mismatched decoder. In

this section, we revisit the Generalized Mutual Information (GMI) and the LM

rate as achievable rates with a fixed decoding measure, investigate the ensembles

used to achieve these rates. At the end of the chapter, we introduce the Coded

Modulation (CM) and Bit-Interleaved Coded Modulation (BICM) systems setups

and discuss their differences.

1

1. PRELIMINARIES

Chapter 4: BICM Achievable Rates with Probabilistic ShapingThis chapter studies the achievable rates of some BICM schemes with probabilis-

tic shaping. These schemes decode using arbitrary bit or symbol probabilities andreference constellations instead of the true ones that are used at the transmitter.

We first study the GMI and the LM rate of the schemes over the DMCs, and

then for the AWGN channels. In particular, we obtain an efficient algorithm forfinding the maximizing input distributions for the GMI of BICM. We compare

the BICM achievable rates of different schemes and also study the behavior of thisdoubly-mismatched decoder in the low and high SNR regime analytically. The

results suggest that shaping can help us bridge the gap between CM and BICMin terms of achievable rates, and provide meaningful system design guidance for

BICM, especially at the low SNR regime.

Chapter 5: Random Coding Error Exponent of BICMSeveral different error exponents and expurgated error exponents with the i.i.d.

ensemble and cost-constrained i.i.d. ensemble are studied for mismatched de-coders in this chapter. In particular, we revisit the links between error exponents

and achievable rates and investigate some important properties of the exponents.We start with some analytical and numerical results on the error exponents of

the maximum likelihood (ML) decoder, and then extended the analysis for mis-matched BICM decoder. Finally, taking into consideration the cost constraint on

the codeword, we generalize Gallager’s expurgated error exponent for the mis-

matched decoder, and the results are applied to BICM.

Chapter 6: Error Probability AnalysisUsually, when good codes are used, the gain on the information rates can be

translated into a gain in the error probability. In this chapter, we analyze the biterror rate (BER) of the schemes in Chapter 4 and reconfirm our results in Chapter

4. We use the method introduced in [33] and revisit various approximations andbounds to the BER of BICM. We observe that our BER analysis matches the

achievable rates analysis and demonstrate that the results on achievable rates arevery good indicators of the error performance of practical systems.

The research on Part I has generated the following papers:

• L. Peng, A. Guillen i Fabregas and A. Martinez, “Mismatched Shaping

Schemes for Bit-Interleaved Coded Modulation”, in Proc. IEEE Interna-tional Symposium on Information Theory, Cambridge, MA, US, 07/2012.

• L. Peng, A. Guillen i Fabregas and A. Martinez, “Mismatched Decoder:

Code Ensembles, Achievable rates and Error Exponents”, to be submittedto 2013 IEEE International Symposium on Information Theory.

2

1.1 Dissertation Outline

• L. Peng, A. Guillen i Fabregas and A. Martinez, “On the Achievable Rates

and Error Exponent of Bit-Interleaved Coded Modulation”, to be submitted

to IEEE Trans. Inf. Theory.

Part II

Part II studies the reliable source transmission over the MIMO block-fading chan-

nel and includes the following three chapters.

Chapter 7: Introduction to the Source Transmission Problem

In this chapter, we introduce the background and motivation of the Part II of the

thesis. We present a brief overview of the different approaches to investigate the

problem of reliable source transmission.

Chapter 8: System Model and Basic Principles

In this chapter, we first introduce the definitions and notations of rate distortion

theory. We then review the source coding problem from the expected distortion

and the excess distortion probability perspectives. In this section, we revisit the

coding theorem and its converse for source coding, and compared the two differ-

ent approaches (expected distortion and excess distortion probability). We then

conclude by introducing the separation and joint source-channel coding model.

Chapter 9: Excess-Distortion Probability in Block-Fading Channels

We first review some information-theoretic material for MIMO block-fading chan-

nel, which includes the concept of mutual information, outage probability and

SNR exponent of the outage probability. We also give a comprehensive literature

review on the different approaches to study the problem. We then introduce the

excess distortion probability as the relevant figure of merit. We start by deriving

a lower bound to the excess distortion probability for any joint source-channel

coding scheme. Then we proceed to analyze the separation scheme and obtain an

upper bound to the excess distortion probability, where we demonstrate the sep-

aration optimality by comparing these two bounds. Finally, we study the excess

distortion probability for the layered source coding scheme.

The research on Part II have generated the following papers:

• L. Peng and A. Guillen i Fabregas, “Distortion Outage Probability in

MIMO Block-Fading Channels”, in Proc. IEEE International Symposium

on Information Theory, Austin, TX, US, 06/2012.

3

1. PRELIMINARIES

• L. Peng and A. Guillen i Fabregas, “Comment on Excess Distortion Proba-bility and Expected Distortion in Block-Fading Channels”, to be submitted

to IEEE Trans. Inf. Theory.

1.2 Notation

We introduce notation that is generally used throughout the dissertation in this

section. Notation which is specific to each chapter is introduced there.We denote random scalars by uppercase letters, e.g. X , and their realizations

by lowercase letters, e.g. x. Random vectors are denoted by uppercase sans serifletters, e.g. X, and their realizations are denoted by boldfaced lowercase letters,

e.g. x. Random matrices are denoted by boldfaced uppercase sans serif letters,e.g. X, and their realizations are denoted by boldfaced uppercase letters, e.g. X .

Sets or events are generally denoted by calligraphic fonts, e.g. X. |X| denotesthe cardinality of the set X (the operator | · | also means the absolute value when

it is used with a scalar and the magnitude of a complex number). For any random

scalar X over an alphabet X, PX(·) denotes the probability mass function (pmf) ifX is discrete and the probability density function (pdf) if X is continuous. Similar

notations are also used for random vectors and matrices.R denotes the set of real numbers, and C denotes the set of complex numbers.

XT denotes the transpose of X , and XH denotes the conjugate transpose of acomplex matrix X . The univariate complex-Gaussian distribution with mean µ

and variance σ2 is denoted by NC (µ, σ2).

E[·] is used to denote the expectation with respect to the random variables in

its arguments. We denote 11· as the indicator function. 11E = 1 if the eventE occurs and 11E = 0 otherwise.

The floor (ceiling) function ⌊x⌋ (⌈x⌉) denotes the largest (smallest) integersmaller (greater) than or equal to x. We use log(·) to denote natural logarithm,

and logr(·) to denote base-r logarithm.Following the definition in [97], the exponential equality f(x)

.= xd indicates

that limx→∞log f(x)log x

= d. Limit and superior are denoted by lim and sup respec-

tively.

4

Part I

Mismatched Shaping Schemes for

Bit-Interleaved Coded

Modulation

5

Chapter 2

Introduction

Bit-Interleaved Coded Modulation (BICM) was introduced by Zehavi [96] as a

pragmatic coding scheme for combining coding and modulation to achieve high

spectral efficiency. It is later extensively studied in [16] under the assumption of

infinite depth interleaving, where it is argued that the system essentially behaves

as a set of parallel independent memoryless binary-input output-symmetric chan-

nels. Therefore, the mutual information of the channel is the sum of the mutual

informations of the parallel independent memoryless binary channels. Similarly,

the error exponent is the sum of the error exponents of the parallel independent

memoryless binary channels.

Recently, the authors in [33, 67] have cast the BICM decoder as a mismatched

decoder [28, 70] and studied its achievable rate and the error exponent. The model

is extended to account for shaping in [44], i.e. the bit or symbol probabilities are

optimized. Interestingly, the restriction of infinite depth interleaving is lifted

with this mismatched decoder framework and the generalized mutual informa-

tion (GMI) is used to measure the achievable rates. When the classical BICM

decoder as in [44, 67] is used, the mismatch solely comes from the assumption of

independent bit metrics. Moreover, it was found in [1, 44] that for the AWGN

channel, this mismatch can be mostly overcome by shaping, virtually closing the

gap that made BICM suboptimal compared to coded modulation (CM) [86] and

multi-level codes (MLC) [47] in terms of information rates. In particular, in the

low signal-to-noise ratio (SNR) regime, shaping makes BICM first and second

order optimal [1, 44].

Motivated by these observations, we are interested to find out when there

is more mismatch in the BICM system, e.g. mismatch of system configura-

tions between the transmitter and receiver, whether the BICM system is still

able to compensate the loss caused by shaping in these cases. The analysis of

7

2. INTRODUCTION

this problem will provide insight into practical scenarios where the hardwareand/or software are mismatched between the transmitter and receiver, and sug-

gest how we should design the communications systems correspondingly. More-over, the cost-constrained i.i.d. ensemble has long been recognized as a method

to improve the performance of random-coding ensemble without cost constraint

[24, 28, 43, 70, 80], and codewords expurgation has long been known as a tech-nique to improve the exponent at low rates [27]. However, BICM has never been

studied under the mismatched decoding model with the cost-constrained i.i.d.ensemble or codewords expurgation in the literature. In this part of the thesis,

we will be able to obtain valuable information related to the performance limitof practical code constructions by studying these factors.

In Part I of the thesis, we will study BICM from three different perspectives:achievable rates, error probability and random coding error exponents. Defini-

tions and system models are introduced in Chapter 3. In Chapter 4 we considera generalized BICM decoder where, at the receiver, one decodes using arbitrary

bit or symbol probabilities and reference constellations instead of the true onesthat are used at the transmitter. The resulting additional source of mismatch

would reduce the complexity of the decoder of BICM shaping schemes in [44],and characterizes the scenario when the base station communicates to terminals

which support less power and computing complexity. We compare the achievable

rates of different schemes and also study the behavior of this doubly-mismatcheddecoder in the low SNR regime analytically, and show that our scheme can also be

made first and second order optimal. We then provide a framework for studyingthe behavior of the GMI for BICM in the high SNR regime.

Chapter 5 studies the error exponents and expurgated error exponent formismatched decoders with the i.i.d ensemble and cost-constrained i.i.d. ensemble.

The results are applied to BICM schemes. Our main goal is to find out how thecost-constrained i.i.d. ensemble improves over the i.i.d. ensemble and how the

pseudo-cost affects the expurgated error exponent. We start with some analyticaland numerical results on the matched decoder (coded modulation). We then move

on to general mismatched decoders and apply the results to BICM.Finally, Chapter 6 is devoted to the analysis of the error probability. various

approximations and bounds to the error probability are introduced. For codingschemes, if the capacity of one scheme is higher than the other, one should expect

the error probability of the former to be lower than the latter when good codes

are used. Hence, we verify our shaping results in Chapter 4 by studying theerror probability of different BICM schemes. As we will see later, the results on

achievable rates are very good indicators of the error performance of practicalsystems.

8

Chapter 3

System Setup and Code

Ensemble

3.1 Channel Model

We consider discrete-time transmission of information with a block code M of

length N and rate R, where R ,log |M|

Nnats. At the transmitter, a message u is

drawn with equal probability from a message set 1, . . . , |M| and then mapped

into a codeword x(u) =(x1(u), . . . , xN (u)

), where xk(u) ∈ X, and X is the

channel input alphabet. In this part of the thesis, we limit our attention to the

case that X is a discrete alphabet. We let M , |X| and m , log2(M) denote thecardinality of X and the number of bits per symbol, respectively.

The output y = (y1, . . . , yN) is a random transformation of the input withtransition probability distribution PY|X(y|x). We assume that the channel is

memoryless, that is

PY|X(y|x) =N∏

k=1

PY |X(yk|xk), (3.1)

where PY |X(y|x) is the per symbol channel transition probability, andX, Y denote

the underlying symbol random variables taking values on the input and outputalphabet X,Y. Similarly, the corresponding random vectors X,Y are drawn from

the sets X , XN , Y , YN respectively.

3.2 Mismatched Decoder

The decoder decides on the estimate of the message u according to a decodingmetric q(x,y), which is considered to be the product of symbol decoding metrics

9

3. SYSTEM SETUP AND CODE ENSEMBLE

q(x, y), i.e.

u = argmaxu

q(x(u),y) = argmaxu

N∏

k=1

q(xk(u), yk). (3.2)

When the metric q(x,y) is a strictly increasing bijective function of PY|X(y|x),the decoder will always select the maximum a posteriori (MAP) codeword x(u).

Otherwise, we have a mismatched decoder [28, 50, 70]. We let Pe denote the

average error probability of a particular code and Pe denote the average error

probability with respect to a given random-coding ensemble. We say a rate R is

achievable if, for every ǫ > 0 and for N sufficiently large, there exists an encoder

and decoder pair such that 1Nlog |M| ≥ R − ǫ and Pe ≤ ǫ. The mismatched

capacity CM is defined as the supremum of all rates R for which there exists a

codebook such that the error probability Pe vanishes for sufficiently large N . The

mismatched capacity is not known in general. However, a number of achievable

rates have appeared in the literature [28, 43, 50, 70], together with converse

theorems under some random coding regime. The author in [7] has shown that in

the binary input case, there is a tight converse for any code, yielding the maximal

transmission rate over any binary-input memoryless channel for a given decoding

rule.

3.2.1 Generalized Mutual Information (GMI)

For the i.i.d. ensemble with input distribution PX , i.e. each codeword in the

code ensemble is composed of symbols generated i.i.d. according to PX(x), The

authors in [28, 50] have derived the achievable rate called Generalized Mutual

Information (GMI). It is given by3.1

I0(PX) , sups≥0

E

[log

q(X, Y )s∑x′ PX(x′)q(x′, Y )s

], (3.3)

The achievability results show that the average error probability over the i.i.d.

ensemble is such that Pe → 0 as N → ∞ when R < I0(PX).

The authors in [28, 50] have also shown a converse theorem for i.i.d. ensemble

for which Pe → 1, as N → ∞ when R > I0(PX). That is I0(PX) is not only an

achievable rate, but it is also the largest achievable rate for the i.i.d. ensemble

with input distribution PX .

3.1 The parameter s is just an optimizing factor that comes up in the derivation using randomcoding technique, it does not bear any significant physical meaning.

10

3.2 Mismatched Decoder

3.2.2 The LM rate

Apart from the i.i.d. ensembles, the constant-composition ensembles and cost-

constrained i.i.d. ensembles have also been considered in the mismatched decod-

ing setting. For the constant-composition ensemble, each codeword has the same

empirical distribution. The codeword distribution is

PX(x) =1

|T (PN)|11x ∈ T (PN), (3.4)

where PN is the most probable type3.2 under PX(x). That is each codeword is

generated uniformly over the type class T (PN), hence each codeword has the

same composition.

For the cost-constrained i.i.d. ensemble, each codeword satisfies a given

pseudo-cost constraint,

Na− δ < a(x) ≤ Na, (3.5)

where a(x) =∑N

k=1 a(xk), a : X → R, is the pseudo-cost function. a = E[a(X)],

and δ > 0 limits the shell on which codewords lie. δ is a constant that does not

depend on N . The pseudo-cost is introduced in order to improve the performance

of the random-coding ensemble [28, 83].

The constant-composition ensemble is only valid when the input is discrete,

whereas the cost-constrained i.i.d. ensemble applies both to continous and dis-

crete inputs. These two ensembles are used to prove a higher achievable rate com-

pared to the GMI. When the a(x) is optimized for both the constant-composition

ensemble and the cost-constrained i.i.d. ensemble, their resulting rate coincides.

This rate is known as the LM rate, and is given by [23, 43, 58, 70]

I1(PX) , sups≥0,a(·)

E

[log

q(X, Y )sea(X)

∑x′ PX(x′)q(x′, Y )sea(x

′)

], (3.6)

where the expectation is carried out over the joint distribution PXPY |X . Since

the GMI can be recovered from the LM rate by choosing a(x) = 0 for x ∈ X, we

have that I0(PX) ≤ I1(PX).

The above achievability result states that for the constant-composition en-

semble and cost-constrained i.i.d. ensemble with symbols follow the distribu-

3.2The set of all probability distributions on an alphabet A is denoted by P(A), and the setof all empirical distributions on a vector in AN is denoted by PN (A). The type of a vector xis denoted by Px. For a given Px ∈ PN (A), the type class T (Px) is defined to be the set of allsequences in XN with type Px. The interested reader is refered to [23, 27] for an introductionon the method of types

11


tion PX(x) for x ∈ X, the average error probability Pe → 0 as N → ∞ when

R < I1(PX). There is also a converse established for the constant-composition

ensemble, which states Pe → 1 as N → ∞ when R > I1(PX).

3.3 Coded Modulation

Coded Modulation (CM) dates back to the pioneering work of Ungerbock [86]. It

merges the coding and modulation in a single entity. The random coding ensemble

corresponding to CM has codewords selected from XN according to a probability

distribution. In the particular case of i.i.d. ensemble, the codewords are selected

i.i.d. from XN according to PX(x), while in the case of cost-constrained i.i.d.

ensemble, the codewords are selected i.i.d. from the ensemble of codewords for

which certain cost constraints are satisfied, and the input distribution is given by

PX(x) = ξ−111 Na− δ < a(x) ≤ NaPX(x), (3.7)

where ξ is a normalization constant to ensure unit probability. The CM decoder

employs the MAP metric, i.e. q(x, y) = PY |X(y|x). It is not difficult to see that

in this case,

I0(PX) = I1(PX) = I(X ; Y ), (3.8)

where I(X ; Y ) is the mutual information between X and Y . Therefore, the largest

information rate that can be achieved with CM with x ∈ X is

Ccm = sup

PX

I(X ; Y ). (3.9)

CM yields the benchmark capacity and error exponent performance for discrete

signaling. However, it is well known that due to the complexity reasons, CM

might not be practical in some cases.

3.4 Bit-Interleaved Coded Modulation

In a BICM scheme, the codewords are obtained as the serial concatenation of a

binary encoder C of length n = mN , a bit-level interleaver3.3 and a binary labeling

function µ : 0, 1m → X which takes blocks of m bits and maps them to signal

constellation symbols x, such that xk = µ(b(k−1)m+1, . . . , bkm

), k = 1, . . . , N . We

3.3This interleaver can be safely ignored in our analysis as it has been absorbed in the de-scription of the random coding ensemble.

12

3.4 Bit-Interleaved Coded Modulation

B1

B2

· · ·

Bm

N Columns

PB1(b1)

PB2(b2)

PBm(bm)

Figure 3.1: BICM random generation of codebook. The length of a BICM codeis mN .

denote the inverse labeling function by bj : X → 0, 1, so that bj(x) is the j-th bit

in the binary label of modulation symbol x, for j = 1, . . . , m. With a slight abuse

of notation, we let B1, . . . , Bm and b1, . . . , bm denote the random variables and

their corresponding realizations of the bits in a given label position j = 1, . . . , m.

In this thesis, unless stated otherwise, we consider the case where the modulation

symbols x are used with probabilities3.4

P bicmX (x) =

m∏

j=1

PBj(bj(x)), (3.10)

where PBj(b) is the probability of the j-th bit being equal to b. Figure 3.1

shows the random generation of the i.i.d. binary codebook C. The labeling rule

modulates the resulting binary codebook column-wise.

Finally, we denote the conditional probability of symbols given that bit b in

the j-th position of the label by PX|Bj(x|b). By construction, PX|Bj

(x|b) is zero

if bj(x) 6= b, and

PX|Bj(x|b) = P bicm

X (x)

PBj(b)

if bj(x) = b. (3.11)

The main difference between CM and BICM is at the decoder end. The BICM

3.4One could also introduce dependencies between bits to allow for all possible P bicm

X(x) ∈

[0, 1], in which case, the BICM code construction would be equivalent to that of CM.

13

Chapter5/Chapter5Figs/bicmcodeensemble.eps


decoder treats each of the m bits in a symbol as independent, yielding

q(x, y) =m∏

j=1

qj(bj(x), y), (3.12)

where the jth bit decoding metric qj(b, y) is given by

qj(b, y) =∑

x′∈X

PY |X(y|x′)QX|Bj(x′|b). (3.13)

Here, X and QX|Bj(x|b) respectively denote the specific reference constellation

and the conditional symbol probabilities used for decoding at the receiver, notnecessarily those used at the transmitter (X and PX|Bj

(x|b)). Mapping is also

considered on X. We denote the reference inverse labeling function at the decoder

by bj : X → 0, 1. For the cases we considered in this thesis, QX|Bj(x|b) is non-

zero only when bj(x) = b. The mapping is kept the same between the transmitterand receiver, i.e. bj(·) = bj(·).

From the mismatched decoder point of view, the BICM system is differentfrom the CM system in two ways. Firstly, the input symbol probability of BICM

is required to be the product of the bit probabilities (as in Eq. (3.10)). Secondly,the BICM uses a mismatched maximum metric decoder whereas the CM uses the

ML decoder.

14

Chapter 4

BICM Achievable Rates with

Probabilistic Shaping

Shaping refers to the use of non-equally spaced (geometrical shaping) and/or

non-uniformly distributed (probabilistic shaping) symbols. The problem has been

studied extensively, cf. [17, 25] and references therein. We limit our attention

to probabilistic shaping in this thesis. Many works in the literature aim to find

an efficient algorithm for finding the capacity-achieving input distributions for

discrete inputs (QAM, PSK, etc.) over various channels. Blahut [11] and Ari-

moto [5] have studied this problem for discrete memoryless channel and proposed

the famous Blahut-Arimoto algorithm. Multilevel codes (MLC) combined with

multistage decoding (MSD) have been proposed in [47, 92] as an efficient method

to attain the channel capacity by using binary codewords. For a fixed input dis-

tribution on the bits, MLC achieve the mutual information both with MAP joint

decoding and with MSD. However, the error exponents of MLC is upper-bounded

by one, making the MLC exponent much worse than that of CM. Wachsmann

et al. [92] have proposed a heuristic method to approach capacity for MLC, for

which they force the channel input to follow the Maxwell-Boltzmann distribution

[55]. For the average power constrained AWGN channel with fixed input constel-

lations, the authors in [87] have proposed a two-stage algorithm, based on the

Blahut-Arimoto algorithm, to find the capacity and capacity-achieving input dis-

tribution. For the multiple-access channel (MAC), the necessity and sufficiency

of the Kuhn-Tucker conditions are proved for channel capacity [93]. Based on

this, a modified Blahut-Arimoto algorithm to find the sum capacity for MAC is

designed in [79].

The idea of probabilistic shaping for BICM was first proposed in [30, 78] in

the context of iterative demodulation. The authors in these references designed

15

4. BICM ACHIEVABLE RATES WITH PROBABILISTICSHAPING

coding schemes such that the BICM codewords are generated with different bit

probabilities in different bit positions. Probabilistic shaping for BICM was stud-

ied from a more information-theoretic point of view in [44], where it was shown

that probabilistic shaping can help BICM bridge the gap that made BICM sub-

optimal in terms of information rate. The problem of shaping for BICM has also

been studied from the geometrical shaping and optimal mapping point of view

by the authors in [1, 2, 3, 42]. However, in this thesis, we will only focus on

the problem of probabilistic shaping for BICM, and henceforth shaping means

probabilistic shaping unless specified otherwise.

The GMI and LM rates of BICM can be obtained by substituting the BICM

decoding metric in Eqs. (3.12) and (3.13) into Eqs. (3.3) and (3.6) respectively.

Moreover, the GMI can be expressed as a sum of information rate quantities with

some manipulation when the symbol probability satisfies Eq. (3.10) [44]. For a

fixed input symbol probability mass function (pmf) P bicmX (x),

Ibicm0 (P bicmX ) = sup

s>0Ibicm0,s (s, P bicm

X ) (4.1)

with

Ibicm0,s (s, P bicmX ) , E

[log

q(X, Y )s∑x′ P bicm

X (x′)q(x′, Y )s

](4.2)

=

m∑

j=1

E

[log

qj(B, Y )s

∑1b′=0 PBj

(b′)qj(b′, Y )s

], (4.3)

where

PY |Bj(y|b) =

∑

x∈XPY |X(y|x)PX|Bj

(x|b). (4.4)

Similarly, for the LM rate,

Ibicm1 (P bicmX ) = sup

s>0,a(·)Ibicm1,s

(s, a(·), P bicm

X

)(4.5)

with

Ibicm1,s

(s, a(·), P bicm

X

), E

[log

q(X, Y )sea(X)

∑x′ P bicm

X (x′)q(x′, Y )sea(x′)

]. (4.6)

Since Eqs. (4.1) and (4.5) give achievable rates for the distribution P bicmX , one

can find the bit input distribution with largest GMI and LM rate respectively,

16

4.1 Discrete Memoryless Channels (DMC)

Table 4.1: BICM Schemes

Schemes QX|Bj(x|b)

BICM1 PX|Bj(x|b) = P bicm

X (x)

PBj(b)

11bj(x) = b

BICM1′P bicmX (x)

1−PBj(b)

11bj(x) = b

BICM22M11bj(x) = b

11 · denotes the indicator function.

Figure 4.1: Example discrete memoryless 4-ary channel. We index the symbolfrom top to bottom as x1, x2, x3, x4. The probability that a transmission erroroccurs for each symbol is p = 0.1748, hence the capacity of this channel withcoded modulation is 1 bit.

resulting in probabilistic shaping, by solving

Cbicm0 = sup

P bicmX

Ibicm0 (P bicmX ) (4.7)

Cbicm1 = sup

P bicmX

Ibicm1 (P bicmX ) (4.8)

The input distribution P bicmX is subject to the constraint in Eq. (3.10), so the max-

imization over P bicmX (x) is equivalent to maximization over PB = PB1(b), . . . , PBm(b).

The GMI and LM rates are non-concave functions of the input distributions, there

is no general simple algorithmic way to find the optimal values of P bicmX , s and

a(·). In the following section, we will study the problem in more detail for both

discrete memoryless and AWGN channels.


In this section, we study the BICM achievable rate for the DMCs. We will first

specify a few BICM schemes of interest based on the models introduced in Section

17

Chapter6/Chapter6Figs/zchannel4ary.eps


3.4, and study these schemes under probabilistic shaping in terms of GMI and

LM rate.

Though one can specify X and QX|Bj(x|b) arbitrarily, we limit our interest

to the schemes shown in Table 4.1 in this section. We choose these schemes,

in particular, BICM2 because the use of uniform conditional symbol probability

at the receiver follows naturally from the conventional BICM. For all schemes

we assume X = X. BICM1 is a BICM scheme with matched conditional symbol

probability at both transmitter and receiver, i.e. QX|Bj(x|b) = PX|Bj

(x|b). In

BICM2, the receiver uses a mismatched conditional symbol probability QX|Bj(x|b)

that assumes equiprobable symbols.

To find Cbicm0 and C

bicm1 for these BICM schemes over DMC, we introduce an

important property of Ibicm0,s (s, P bicmX ) and Ibicm1,s

(s, a(·), P bicm

X

).

Proposition 4.1.1 (Concavity of Ibicm0,s (s, P bicmX ) and Ibicm1,s

(s, a(·), P bicm

X

)) For

a fixed input distribution, Ibicm0,s (s, P bicmX ) is a concave function of s for s ≥ 0, and

Ibicm1,s

(s, a(·), P bicm

X

)is jointly concave in a(x) and s for s ≥ 0 and any a(·).

Proof: See Appendix A.1.

The proposition also holds for continuous channels. In general, we have to search

exhaustively over all possible input distributions. For each fixed input distribu-

tion, we can make use of Proposition 4.1.1 to find the optimal s for Ibicm0,s (s, P bicmX )

and the optimal s and a(x) for Ibicm1,s

(s, a(·), P bicm

X

)using known convex optimiza-

tion methods. However, because of the particular structure of the mismatched

decoder of BICM1, we are able to derive a Blahut-Arimoto (BA) type algorithm

that finds the local optimal P bicmX (x) and guarantees convergence. Though this

algorithm is developed under the DMC setup, it is still applicable to the case

of BICM transmission over continous channels as long as the channel is uncon-

strained. The algorithm is very powerful, as BICM1 is the most conventional

BICM shaping scheme. Details of this algorithm are given in Appendix A.3.

4.1.1 Numerical Results

In this section, we calculate Cbicm0 and C

bicm1 for the channel shown in Figure 4.1.

The channel transition probability matrix is given by

[PY |X(y|x)] =

1 0.1748 0.1748 0.17480 0.8252 0.1748 0.17480 0 0.6504 0.17480 0 0 0.4756

. (4.9)

18


Table 4.2: The GMI and LM rate Simulation Results

GMI Results LM Results

BICM1 BICM2 BICM1 BICM2

Cbicm0 0.886184 0.878373 C

bicm1 0.8873732 0.8872431

s∗ 1 1.18537 s∗ 1.026290 1.361005P ∗B1(0) 0.7237 0.7000 P ∗

B1(0) 0.72 0.72

P ∗B2(0) 0.5250 0.5200 P ∗

B2(0) 0.53 0.53

a∗(x)

0.1501920.0808940.1688070.000107

0.3872040.1903010.060202−0.237706

The capacity of this channel when coded modulation is used is 1 bit. For the

GMI and LM rate, the simulation results are summarized in Table 4.2.

For all experiments, the optimal input distribution, when seen from the equiv-

alent symbol distribution P bicmX (x) =

∏mj=1 PBj

(bj(x)

), decreases from x1 to x4.

This matches the nature of the asymmetric channel in Figure 4.1, where the

symbols at the top are more reliable, hence they should be used more frequently.

In Figure 4.2, we show how well the proposed algorithm in Appendix A.3 works

for finding Cbicm0 for BICM1. We have also compared it with the result given by

the exhaustive search. In general, Cbicm0 can be found in just a few iterations, and

the randomly generated initial distributions all converge to the global optimal

input distribution. Similar behavior was observed in simulations over other types

of channels, which suggests the algorithm is quite efficient. However, one should

still be careful when using the algorithm. Given that convergence to a local

maximum is guaranteed, then multiple initial distributions should be assigned so

that we have a better chance to find the global maximum.

4.1.2 When Is LM Rate Significantly Larger Than the

GMI?

From the results in Table 4.2, we observe that after shaping, the improvement

introduced by calculating the LM rate is marginal. We also observe from all

our experiments that the mismatch in the system largely determines how much

is the gain for the LM rate. We will present some results where the LM rate

be significantly larger than the GMI and illustrate by means of an example for

which we have deliberately introduced some more mismatch to the decoder.

19


1 3 5 7 90.4

0.5

0.6

0.7

0.8

0.9

Iteration n

Ibic

m0

(PB

)

Figure 4.2: The solid line represents the value of Ibicm0 (PB) throughout the it-erations using the proposed algorithm with uniform initial distribution. All thedashed lines represent the value of Ibicm0 (PB) throughout the iterations using theproposed algorithm with a randomly generated initial distribution. The dash-dotted line is the GMI of uniform input distribution. The dotted line is the Cbicm

0

found by exhaustive search.

To answer the question, we will not only compare Cbicm0 and C

bicm1 , but also

Ibicm0 (P bicmX ) and Ibicm1 (P bicm

X ). We observe from Table 4.2 that the improvement of

Cbicm1 over Cbicm

0 is marginal for both BICM1 and BICM2. There is also a generic

mismatched decoder example in [80] where some considerable improvement is

observed for the LM rate over the GMI. Thus, in the case of BICM, we propose

an observation that the more mismatched the system is the larger the gain one

can have by using the LM rate.

To back this observation up with another example, let us consider two types

of mismatched BICM schemes. As in Table 4.1, scheme BICM1 decodes using

20

Chapter6/Chapter6Figs/iteration4ary.eps


the metric given by

q(x, y) =m∏

j=1

∑

x′∈X

PY |X(y|x′)P bicmX (x′)

PBj(b)

11bj(x

′) = b . (4.10)

Scheme BICM1′ in Table 4.1 decodes using the metric given by

q(x, y) =m∏

j=1

∑

x′∈X

PY |X(y|x′)P bicmX (x′)

1− PBj(b)

11bj(x

′) = b , (4.11)

The decoder of BICM1′ is designed to look more mismatched. We have used

1− PBj(b) where it should be PBj

(b).

Figure 4.3 shows the improvement of the LM rate over GMI,Ibicm1 (P bicm


Ibicm0 (P bicmX )

,

for BICM1 for different input distributions. We observe little improvement over

all input distributions.

Figure 4.4 shows the improvement of the LM rate over GMI for BICM1′ for

different input distributions. The decoder is more mismatched and the ratioIbicm1 (P bicm


Ibicm0 (P bicmX )

is much larger, even reaching 4 at PB1(0) = 0.01, PB2(0) =

0.99. For BICM1′ , when PB are uniformly distributed for both B1 and B2, Eq.

(4.11) coincides with Eq. (4.10) and there is no improvement, i.e.

Ibicm1 (P bicmX )− Ibicm0 (P bicm

X )

Ibicm0 (P bicmX )

= 0.

The quantities Cbicm0 and C

bicm1 , as well as the corresponding optimal input dis-

tribution for BICM1′ are given by

Cbicm1 = 0.887373 P ∗

B1(0) = 0.72 P ∗

B2(0) = 0.53 (4.12)

Cbicm0 = 0.839742 P ∗

B1(0) = 0.59 P ∗

B2(0) = 0.50. (4.13)

We observe that even the gap between Cbicm1 and C

bicm0 is larger in case of BICM1′ ,

which is evidence that the more mismatched the decoder is, the more likely it is

that the LM rate improves over the GMI significantly.

4.1.3 Non-product Distribution

In the Section 3.4, we introduced the input distribution of the BICM, P bicmX , as

the product of the corresponding bit probabilities. This constraint is dictated

21


0

0.5

1

0

0.5

10

0.01

0.02

0.03

0.04

0.05

PB2(0)

PB1(0)

Ibicm

1(P

bicm

X)−

Ibicm

0(P

bicm

X)

Ibicm

0(P

bicm

X)

Figure 4.3:Ibicm1 (P bicm


Ibicm0 (P bicmX )


in Eq. (4.10) over all input distributions.

by the fact the we have imposed the independence condition among the bits

from different labeling positions, and it is a practical constraint rather than a

theoretical one.

Clearly, with the product distribution constraint, the possible values of P bicmX (x)

for x ∈ X belong to a subset of all possible PX(x) for x ∈ X with PX(x) ∈ [0, 1]

and∑

x PX(x) = 1. Therefore, once this constraint is removed, i.e., we allow

the bits to be correlated among different labeling positions, we should achieve a

higher achievable rate.

Table 4.3 shows the simulation results of the maximal GMI and the corre-

Table 4.3: Cbicm0 and other optimal parameters for BICM1

P bicmX Type C

bicm0 s∗ [P bicm

X (x)∗]

Non-product P bicmX 0.8938 0.9557 [0.39 0.33 0.17 0.11]

Product P bicmX 0.8862 1 [0.38 0.34 0.13 0.15]

22

Chapter6/Chapter6Figs/diffnormbicm1.eps

4.2 The AWGN Channel

0

0.5

1

0

0.5

10

1

2

3

4

PB2(0)

PB1(0)

Ibicm

1(P

bicm

X)−

Ibicm

0(P

bicm

X)

Ibicm

0(P

bicm

X)

Figure 4.4:Ibicm1 (P bicm


Ibicm0 (P bicmX )


in Eq. (4.11) over all input distributions.

sponding optimal parameters for BICM1 transmission over the same 4-ary chan-

nel in Figure 4.1 with channel transition probability matrix given by Eq. (4.9).

As expected, we observe some improvement of the GMI when the product distri-

bution constraint is removed.


In this section, we study BICM achievable rates for the AWGN channel. In

particular, we will present some numerical results over a wide SNR range, and

some analytical results for both the low and high SNR regimes.

23

Chapter6/Chapter6Figs/diffstrangebicm1.eps


4.2.1 Channel Model and Achievable Rates

We consider transmission over the complex (X ⊂ R,Y = R) AWGN channel for

which

yk =√snr xk + zk k = 1, . . . , N, (4.14)

where zk are realizations of an i.i.d. circularly symmetric complex Gaussian ran-

dom variable with zero mean and unit variance, and snr is the average SNR.Codewords are subject to a general cost constraint EPX

[c(X)] ≤ Γ. In the pres-

ence of the constraint, the problem of probabilistic shaping for BICM GMI is

given by

Cbicm0 (Γ) = sup

P bicmX

: EPbicmX

[c(X)]≤Γ

Ibicm0 (P bicmX ), (4.15)

where Ibicm0 (P bicmX ) is defined in (4.1). Similarly, the problem of probabilistic

shaping for BICM LM rate is given by

Cbicm1 (Γ) = sup

P bicmX : E

PbicmX

[c(X)]≤Γ

Ibicm1 (P bicmX ), (4.16)

where Ibicm1 (P bicmX ) is defined in (4.5).

Though one can specify the cost function c(·) arbitrarily, we focus on the

average power constraint 1N

∑N

k=1 |xk|2 = 1. Hence, Eq. (4.15) becomes

Cbicm0 (snr) = sup

P bicmX

: EPbicmX

[|X|2]≤1

Ibicm0 (snr) (4.17)

with

Ibicm0 (snr) = sups≥0

m∑

j=1

EPBj,X|Bj,Z[fj(ρ,X,Bj , s)] , (4.18)

Table 4.4: BICM Schemes of Interest

Schemes X QX|Bj(x|b)

BICM1 X PX|Bj(x|b) = P bicm

X (x)

PBj(b)

11bj(x) = b

BICM2 X 2M11bj(x) = b

BICM3 X 2M11bj(x) = b

11 · denotes the indicator function.

24


where ρ ,√snr and

fj(ρ, x, b, s) , log

(∑x′∈X e

−|ρ(x−x′)+z|2QX|Bj(x′|b)

)s

∑1b′=0 PBj

(b′)(∑

x′∈X e−|ρ(x−x′)+z|2QX|Bj

(x′|b′))s . (4.19)

Similarly, for the LM rate,

Cbicm1 (snr) = sup

P bicmX : E

PbicmX

[|X|2]≤1

Ibicm1 (snr). (4.20)

Here, Ibicm1 (snr) , Ibicm1 (P bicmX ). We have omitted the dependence of Ibicm0 (snr)

and Ibicm1 (snr) = Ibicm1 (P bicmX ) on P bicm

X for the sake of notational simplicity. We

again limit our attention to the schemes shown in Table 4.4, where X satisfies

∑

x∈X

1

M|x|2 = 1. (4.21)

Note that for the same schemes, the ones in Table 4.4 are slightly different from

those in Table 4.1 by offering an extra degree of freedom to specify the reference

constellation at the receiver X.

When equiprobable symbols are used as channel inputs, BICM1 is the classical

BICM scheme in [16]. When shaping is performed for BICM1 by solving (4.17) for

GMI or solving (4.20) for LM rate, it is the same as the BICM shaping scheme con-

sidered in [1, 41, 44]. In BICM2 and BICM3, both mismatched receivers assume

equiprobable conditional symbol probabilities, i.e. QX|Bj(x|b) = 2

M11bj(x) = b

.

The difference between BICM2 and BICM3 is that the reference constellations X

and X are the same for BICM3, while for BICM2 the reference constellations at

the receiver X are normalized as in Eq. (4.21) (hence the reference constella-

tion and the transmitted constellation are mismatched). Note that BICM2 and

BICM3 are the same only when equiprobable inputs are used.

For shaping on the AWGN channel, the same exhaustive search method as

in Section 4.1 is used. PAM and QAM signal sets are of special interest. When

Gray mapping is used, the 2m-QAM constellation is the Cartesian product of two

2m2 -PAM constellations, one for each of the in-phase and quadrature components

of the channel. Due to this orthogonality, once we obtain the optimal input dis-

tribution of 2m2 -PAM constellation, the corresponding optimal input distribution

for 2m-QAM constellation is automatically solved. Therefore, in our numeri-

cal results, we focus on PAM signaling. Symmetry between the in-phase and

quadrature components and along the zero axis (positive and negative planes

25


−2 0 2 4 6 8 10 12 14 160

0.5

1

1.5

2

2.5

3

Eb

N0

(dB)

Cbic

m

BICM1, equiprobable inputsBICM1, shapingBICM2, shapingBICM3, shapingCM, shaping

Figure 4.5: Comparison of Cbicm0 among different BICM schemes with 8PAM

modulation and Gray labeling [000, 001, 011, 010, 110, 111, 101, 100]. Schemes areshown as in Table 4.4.

have equal probability) dictates that the optimization problems in (4.17) and

(4.20) are simplified to have 2m2−1 − 1 free parameters for 2m-QAM/2

m2 -PAM.

Instead of Monte-Carlo method, we will use Gauss-Hermite quadrature [37] to

compute the achievable rates for the AWGN channel as it is well known for

providing an acurate and fast method for approximating the value of integrals of

the kind∫∞−∞ e−x2

f(x)dx.

Figure 4.5 shows the Cbicm0 performance comparison among the four BICM

schemes featured in Table 4.4, some with shaping and some without. The capac-

ity of CM is also shown for reference. The results illustrate that the mismatched

symbol pmf QX|Bj(x|b) and/or mismatched constellation X are efficiently com-

pensated by shaping at low SNR. On the other hand, in the mid-to-high range

of SNR, BICM2 with shaping performs only slightly better than BICM1 with

equiprobable inputs. We also observe that as soon as the reference constellation

matches the transmit constellation while keeping equiprobable symbol probabil-

ities at the decoder (as in BICM3), the gap can be recovered almost in full by

shaping.

26

Chapter6/Chapter6Figs/bicm_8pam_prime_primenorm_equi.eps


−10 −5 0 5 10 15 20

−6

−4

−2

0

2

4

6

snr (dB)

Opti

mal

Con

stel

lati

on

Active SymbolInactive Symbol

Figure 4.6: Optimal constellation X for different snr for BICM1 with shaping and8PAM modulation with Gray labeling. A symbol is inactive means it is used withzero probability.

Remark 4.2.1 When we consider BICM1 transmission over the AWGN channel

without a power constraint, the BA type algorithm introduced in Appendix A.3

can also be used to find the optimal input distribution and maximal GMI.

Remark 4.2.2 The decoding metrics of BICM1 and BICM3 depend on the input

distribution whereas that of BICM2 does not. This means that when designing the

BICM shaping systems, the receiver does not need to know the input distribution

from the transmitter, one can just fix the decoder to one configuration, and focus

on designing good shaping codes. The resulting rate loss is small, especially at

low SNRs. If the receiver is such that X = X, we then have BICM3 transmission,

and there is almost no loss compared to CM in terms of achievable rate for the

entire SNR range.

Figure 4.6 shows the optimum constellation X for different SNR values for the

27

Chapter6/Chapter6Figs/optimalconstellationgradientmodel1.eps


−10

0

10

20

−5 −4 −3 −2 −1 0 1 2 3 4 5

0

0.2

0.4

0.6

snr (dB)

Optimal Constellation

Pro

bab

ility

Figure 4.7: Optimal constellation X and input distribution for different snr forthe GMI of BICM1 with 8PAM modulation with Gray labeling.

GMI of BICM1 with shaping. Figure 4.7 further shows the probability of each

symbol being used. We can clearly see the evolution of optimal input distribution

as SNR increases (from uniform 2PAM to non-uniform 4PAM to non-uniform

8PAM to uniform 8PAM). The result is simulated by stepsize of 1dB for SNR, and

uniform 2PAM is optimal for snr ≤ −7 dB in this case. Note that in Figure 4.6,

the optimal input distribution at low SNR contains the two outermost symbols

from the 8PAM constellation, i.e.

X = X1 =x7: x ∈ −7,−5,−3,−1, 1, 3, 5, 7

. (4.22)

Thus for the Gray labeling [000, 001, 011, 010, 110, 111, 101, 100] considered, the

28

Chapter6/Chapter6Figs/optimalconstellationgradientmodel13D.eps


000 001 011 010 100 111 101 100

431

2

Figure 4.8: Example of 8PAM constellation with Gray labeling. We have labeledthe pairs with numbers.

corresponding optimal bit probability at low SNR is

PB1(0) = 0.5, PB2(0) = 1, PB3(0) = 1

PB1(1) = 0.5, PB2(1) = 0, PB3(1) = 0. (4.23)

We refer to the inputs with the constellation set in Eq. (4.22) and the proba-

bilities in Eq. (4.23) as (X1, P 1B). It is not the only optimum for (4.17) for BICM1

shaping scheme with 8PAM at low SNR. The same holds when each of the three

remaining pairs as shown in Figure 4.8 are chosen to form the uniform 2PAM.

The second optimal input (X2, P 2B) sets

X =X2 =x5: x ∈ −7,−5,−3,−1, 1, 3, 5, 7

(4.24)

PB1(0) = 0.5, PB2(0) = 1, PB3(0) = 0

PB1(1) = 0.5, PB2(1) = 0, PB3(1) = 1. (4.25)

The third optimal input (X3, P 3B) sets

X =X3 =x3: x ∈ −7,−5,−3,−1, 1, 3, 5, 7

(4.26)

PB1(0) = 0.5, PB2(0) = 0, PB3(0) = 0

PB1(1) = 0.5, PB2(1) = 1, PB3(1) = 1. (4.27)

Finally, the fourth optimal input (X4, P 4B) sets

X =X4 = x : x ∈ −7,−5,−3,−1, 1, 3, 5, 7 (4.28)

PB1(0) = 0.5, PB2(0) = 0, PB3(0) = 1

PB1(1) = 0.5, PB2(1) = 1, PB3(1) = 0. (4.29)

29

Chapter6/Chapter6Figs/bitprobabilityexample.eps


Since the decoder knows the true conditional symbol probability, only the symbols

with non-zero probability affect the calculation of qj(b, y) for BICM1. For all 4

inputs, the symbols with non-zero probability are all −1 and 1, from Eqs. (4.23)–

(4.29). Therefore, as long as uniform 2PAM remains optimal at low SNR (in our

numerical results, this is true for snr ≤ −7 dB), we can choose any of the 4 inputs

to achieve Cbicm0 for the BICM1 shaping scheme. The same holds for BICM2, where

the reference constellation and conditional symbol probability at the decoder are

set to

X =

x√5: x ∈ −7,−5,−3,−1, 1, 3, 5, 7

(4.30)

QX|Bj(x|b) = 2

M11bj(x) = b

, (4.31)

no matter what the inputs are. Hence in the calculation of Ibicm0 (P bicmX ) for

BICM2, the 4 inputs in Eqs. (4.23)–(4.29) lead to the same GMI.

However, when we use a mismatched conditional symbol probability QX|Bj

and a matched reference constellation as in BICM3, the calculation of qj(b, y) will

include the symbols with zero probability, and the symbols are different for the

4 inputs in Eqs. (4.23)–(4.29). Hence, for BICM3 shaping schemes, the 4 pairs

are not equivalent and there is, in general, only one optimum, and it is the one

given in (4.22) and (4.23), i.e. (X, P 1B). We observe from our numerical results

that the GMI decreases as the shaping code uses from the outermost pair to the

innermost pair. However, in the next section, we will see that this difference of

the GMIs produced by the 4 inputs vanishes as snr → 0 for BICM3.

Figure 4.9 shows the BICM LM rate shaping results for the schemes fea-

tured in Table 4.4. We observe negligible improvement of the LM rate over the

GMI for low and high SNRs for all schemes, and for moderate SNRs for BICM1

with equiprobable inputs and BICM3 shaping schemes. We also observe that for

BICM2, the LM rate has more improvement over the GMI for moderate SNRs,

which is in line with the observation in Section 4.1.2 that the more mismatched

the system is, the greater the improvement of the LM rate over the GMI (the

decoder in BICM2 is the most mismatched among those in Table 4.4).

4.2.2 Low SNR Regime for GMI

The GMI of BICM at mid-to-high SNR can only be evaluated by numerical

experiments. On the other hand, at low SNR, the GMI can be analyzed in closed

30


−2 0 2 4 6 8 10 12 14 160

0.5

1

1.5

2

2.5

3

Eb

N0

(dB)

Cbic

m

GMILM

BICM3, shaping

BICM2, shaping

BICM1, equiprobable

Figure 4.9: Comparison of Cbicm1 among different BICM schemes with 8PAM

modulation with Gray labeling [000, 001, 011, 010, 110, 111, 101, 100]. Schemes areshown as in Table 4.4.

form. The GMI admits a Taylor expansion series in terms of snr,

Ibicm0 (snr) = c1snr + c2snr2 + o(snr2). (4.32)

A scheme is said to be first and second order optimal if c1 = 1 and c2 = −12

respectively [89].

The low-snr performance of BICM was studied in [66, Theorem 2], where

expressions of c1 and c2 were given for general mapping rules and equiprobable

signaling with the classical BICM decoder (BICM1 decoder). The result was ex-

ploited in [44] to show that BICM with Gray labeling and shaping is first and

second order optimal in the wideband regime. To generalize the result in [66]

and to determine how the multiple sources of mismatch introduced in this thesis

affect c1 and c2, we start directly with the expression of Ibicm0 (snr) defined in Eq.

(4.18). In order to find c1 and c2, we need the second- and forth-order deriva-

31

Chapter6/Chapter6Figs/GMILMshaping8PAM.eps


tives of fj(ρ, x, b, s) with respect to ρ, respectively. The first-order derivative of

fj(ρ, x, b, s) with respect to ρ is given by

f ′j(ρ, x, b, s) =

−sαj(ρ, x, z, b)

βj(ρ, x, z, b)+

∑1b′=0 PBj

(b′)sβj(ρ, x, z, b′)s−1αj(ρ, x, z, b

′)∑1

b′=0 PBj(b′)βj(ρ, x, z, b′)s

,

(4.33)

where

αj(ρ, x, z, b) ,∑

x′∈X

e−|ρ(x−x′)+z|2QX|Bj(x′|b)γ(ρ, x′, x, z) (4.34)

βj(ρ, x, z, b) ,∑

x′∈X

e−|ρ(x−x′)+z|2QX|Bj(x′|b) (4.35)

γ(ρ, x′, x, z) , 2zr(xr − x′r) + 2zi(xi − x′i) + 2ρκ(x, x′) (4.36)

κ(x, x′) , −2|x− x′|2 (4.37)

and xr, xi and zr, zi denote the real and imaginary parts of x and z respectively.

The second-order derivative of fj(ρ, x, b, s) with respect to ρ is given by

f′′

j (ρ, x, b, s) =sλj(ρ, x, z, b)βj(ρ, x, z, b)− sαj(ρ, x, z, b)

2

βj(ρ, x, z, b)2

+

(∑1b′=0 PBj


′)∑1

b′=0 PBj(b′)βj(ρ, x, z, b′)s

)2

−(∑1

b′=0 PBj(b′)βj(ρ, x, z, b

′)s) (∑1

b′=0 PBj(b′)(s2 − s)βj(ρ, x, z, b

′)s−2αj(ρ, x, z, b′)2)

(∑1b′=0 PBj

(b′)βj(ρ, x, z, b′)s)2

−(∑1

b′=0 PBj(b′)βj(ρ, x, z, b

′)s) (∑1

b′=0 PBj(b′)sβj(ρ, x, z, b

′)s−1λj(ρ, x, z, b′))

(∑1b′=0 PBj

(b′)βj(ρ, x, z, b′)s)2

(4.38)

with

λj(ρ, x, z, b) ,∑

x′∈X

e−|ρ(x−x′)+z|2QX|Bj(x′|b)

(γ(ρ, x′, x, z)2 + κ(x, x′)

). (4.39)

Expanding Eq. (4.18) using Taylor series with Eqs. (4.19),(4.33)-(4.39) and

letting ρ → 0 we have that for general constellations (both transmitted and

32


reference), input pmfs (both transmitted and reference), mapping and value of s

c1(s) = a1s2 − a2(s

2 − s)− a3s, (4.40)

where

a1 ,

m∑

j=1

EPX

[∣∣∣X − EPBjQX|Bj

[X ]∣∣∣2], (4.41)

a2 ,

m∑

j=1

EPX ,PBj

[∣∣∣X − EQX|Bj[X ]∣∣∣2], (4.42)

a3 ,

m∑

j=1

EPBj,QX|Bj

[∣∣∣X − EQX|Bj[X ]∣∣∣2]. (4.43)

By maximizing c1 over s we find that

c1 = sups≥0

c1(s) = − (a2 − a3)2

4(a1 − a2). (4.44)

In order to find c2 in Eq. (4.32), we need the fourth-order derivative of

fj(ρ, x, b, s) with respect to ρ. The derivation and expression of c2 are given

in Appendix A.4. For BICM0 with equiprobable inputs, our expressions of c1 in

Eq. (4.40) and c2 recover the respective expressions provided in [66, Theorem

2]. In [66], the first and second order optimality of BICM1 with shaping and

Gray labeling is shown. With our expressions of c1 and c2, we can also show the

first and second order optimality of BICM2 and BICM3 with shaping and Gray

labeling. Moreover, for both schemes, it is also achieved with 2PAM signaling

by choosing a pair of nested 2PAM constellations of the PAM constellations, and

it does not matter which pair of 2PAM symbols one chooses out of the MPAM

constellations, such as the 4 inputs in Eq. (4.23)–(4.29) for 8PAM (the values of

optimal s are different, see for Table 4.5).

Figure 4.10 shows Cbicm0 and the corresponding wideband regime expansion

(4.32) for four BICM3 configurations (one with optimal parameters and three

with randomly chosen values of s and the bit probabilities). The results confirm

the accuracy of our wideband regime analysis of the general BICM schemes.

Remark 4.2.3 The low-SNR regime of the LM rate can be studied in a simi-

lar way. However, to show the optimality at low-SNR for these BICM shaping

schemes in terms of the LM rate, one just needs to note that the GMI can be made

33


Table 4.5: Optimal s for BICM2 andBICM3 with different inputs at snr →0

(X, PB) s∗(BICM2) s∗(BICM3)

(X1, P 1B) 1.15 1.75

(X2, P 2B) 1.15 1.25

(X3, P 3B) 1.15 0.75

(X4, P 4B) 1.15 0.25

first and second order optimal and that Ibicm1 (P bicmX ) ≥ Ibicm0 (P bicm

X ). Therefore,

we leave the detailed low-SNR analysis of the LM rate for future study.

4.2.3 High SNR Regime for GMI

Both mutual information and GMI are upper bounded by H(X), the input en-

tropy. Furthermore, for high SNR the equivocation H(X|Y ) tends to zero, and

therefore the mutual information of CM saturates at H(X), making equiprobable

inputs optimal at high SNR. In this section, we analyze the high-SNR regime of

the BICM schemes described previously. In particular, we will pay attention to

BICM2, where the reference constellation is different from the transmitted one.

To do this, we rewrite the expression of Ibicm0 (snr) in (4.18) as

Ibicm0 (snr) = sups≥0

H(X)− Hsnr(X|Y ), (4.45)

where we isolated the snr-dependent term, Hsnr(X|Y ),

Hsnr(X|Y ) , −m∑

j=1

EPBj,X|Bj,Z

[log(PBj

(B))+ fj(ρ,X,B, s)

]. (4.46)

We now focus on the high-SNR behavior of Hsnr(X|Y ). When snr → ∞, for

fixed b and y (or x), the channel tends to a noiseless channel, and the limiting

value of the fj(ρ, x, b, s) term in (4.46) can be calculated as

limsnr→∞

fj(ρ, x, b, s) = logQX|Bj

(xj,b(x)|b

)se−s|x−xj,b(x)|2

∑1b′=0 PBj

(b′)QX|Bj

(xj,b′(x)|b′)

)se−s|x−xj,b′(x)|2

, (4.47)

34


−2 0 2 4 6 8 10 12 14 160

1

2

3

4

5

6

Eb

N0

(dB)

GM

I

8PAM P ∗

B , s∗

64QAM PB1(0) = 0.45, PB2

(0) = 0.39,PB3

(0) = 0.56, s = 0.8

8PSK PB1(0) = 0.3, PB2

(0) = 0.9,PB3

(0) = 0.56, s = 1.5

8PSK PB1(0) = 0.45, PB2

(0) = 0.39,PB3

(0) = 0.56, s = 0.8

Figure 4.10: BICM performance in the wideband regime. Four different BICM3

configurations with Gray labeling are considered. P ∗B and s∗ denote the optimal

bit probabilities and s. For 64QAM, we assume symmetry between in-phase andquadrature, i.e. PB1(0) = PB4(0), PB2(0) = PB5(0), PB3(0) = PB6(0).

where

xj,b(x) , argminx′∈X,bj(x′)=b

|x− x′|2. (4.48)

When snr → ∞, the term Hsnr(X|Y ) in (4.46) converges to

limsnr→∞

Hsnr(X|Y ) =

−m∑

j=1

1∑

b=0

PBj(b)∑

x∈XPX|Bj

(x|b) log PBj(b)QX|Bj

(xj,b(x)|b

)se−s|x−xj,b(x)|2

∑1b′=0 PBj

(b′)QX|Bj

(xj,b′(x)|b′

)se−s|x−xj,b′(x)|2

.

(4.49)

Substituting Eq. (4.49) into Eq. (4.45), we can characterize the saturating

value of the GMI for high SNR for a general reference constellation. In particular,

35

Chapter6/Chapter6Figs/bicm_wideband.eps


for schemes with equal transmit and reference constellation (i.e. X = X), Eq.

(4.47) tends to 0 at high SNR, hence Eq. (4.49) also tends to 0 at high SNR.

Hence Ibicm0 (snr) saturates at H(X). However, when X 6= X, Eq. (4.47) may not

be 0, and Ibicm0 (snr) in Eq. (4.45) does not necessarily converge to H(X).

Proposition 4.2.1 limsnr→∞ Hsnr(X|Y ) is a convex function of s.

Proof: See Appendix A.5

It follows from Proposition 4.2.1 that simple numerical methods can be used to

optimize Eq. (4.45) for fixed P bicmX (x) over s. Optimization over P bicm

X (x) can

then be performed with an exhaustive search.

Note that by using this analysis, we have swapped the limsnr→∞ with the

sups,PBin Eq. (4.45). The following property shows that for the case we consider,

the result will not be affected.

Proposition 4.2.2 Ibicm0 (s, P bicmX ; snr) is an increasing (weakly increasing) func-

tion of snr for the BICM schemes we have considered in this dissertation, and

limsnr→∞

sups,P bicm

X

Ibicm0 (s, P bicmX ; snr) = sup

s,P bicmX

limsnr→∞

Ibicm0 (s, P bicmX ; snr). (4.50)

Proof: See Appendix A.5.

Figure 4.11 shows numerical results which support the above analysis. In

particular, we consider 8-PAM with X = X, where X satisfies Eq. (4.20). As

we can see, in some cases, the resulting GMI does not saturate at the integer-

valued bits, but rather to the values predicted by our previous analysis. We can

interpret the optimized GMI for the BICM2 shaping scheme (thin solid line) as

the envelope of all possible GMI values with a fixed input distribution.

4.2.4 Non-product Distribution

Under the same rationale in Section 4.1.3, we show in Figure 4.12 Cbicm0 for both

BICM1 and BICM2 transmission over AWGN channel with and without the prod-

uct distribution constraint. For both cases (BICM1 and BICM2), by allowing

correlations among the bits from different labeling positions, one can obtain a

higher achievable rate. However, the improvement is small over all snr, implying

that the independent bits constraint does not result in any significant rate loss.

36

4.3 Chapter Review and Conclusion

0 10 20 30 40 500

0.5

1

1.5

2

2.5

3

Eb

N0

(dB)

GM

I

1.993

1.584

1.095

1

3

Figure 4.11: High SNR analysis of different (over different subsets of the proba-bility space) BICM2 8PAM shaping schemes with Gray labeling. The decoding

reference constellation X = X satisfies (4.21) for all schemes. The solid linecorresponds to BICM2 shaping scheme introduced in Section 4.2.1. The thicksolid line corresponds to fixing PB1(0) = 0.5, PB2(0) = 1, PB3(0) = 0.5 and max-imizing s. The dotted line corresponds to fixing PB2(0) = 1, PB3(0) = 1 andmaximizing PB1(0) and s. The dashed line corresponds to fixing PB2(0) = 1and maximizing PB1(0), PB3(0) and s. The dash-dotted line corresponds to fix-ing PB2(0) = 0 and maximizing PB1(0), PB3(0) and s. The horizontal lines plotsups,PB

limsnr→∞(H(X) − Hsnr(X|Y )

), and indicates the saturating value of the

GMI for snr → ∞.


In this chapter, we have studied the achievable rates of BICM shaping schemes

from the mismatched decoding point of view. We have considered a BICM mis-

matched decoding model where we have the freedom to arbitrarily choose the

reference constellation and conditional symbol probabilities at the receiver. In

particular, we have studied schemes where the decoder employs a reference con-

37

Chapter6/Chapter6Figs/bicm_8pam_highsnr.eps


−5 0 5 10 15 20 250

0.5

1

1.5

2

2.5

3

Eb

N0

(dB)

Cbic

m0

5.4 5.6 5.8

1.75

1.8

1.85

1.9

1.95

Product DistributionNon−product Distribution

BICM1, shaping

BICM2, shaping

Figure 4.12: Cbicm0 for BICM1 and BICM2 transmissions over AWGN channel

using 8PAM as channel inputs. The solid line plots Cbicm0 under product distri-

bution, and the dotted line plots Cbicm0 under non-product distribution. Gray

labeling is used, and the schemes are described in Table 4.4.

stellation that assumes the symbols have been used with equal probability.

For discrete memoryless channels and continuous channel without power con-

straint, we have designed a Blahut-Arimoto type algorithm to find the Cbicm0 and

optimal input distributions for the BICM1 shaping scheme in Table 4.1, and the

algorithm guarantees local optimality. The LM rate shaping is also performed for

different BICM schemes and compared with the GMI shaping results. We ob-

serve that the more mismatched the system is, the more improvement we obtain

by considering the LM rate rather than the GMI.

We have performed a similar analysis for BICM transmission over AWGN

channels, where we have shown that for all schemes, shaping can help us tremen-

dously in recovering the loss which made BICM suboptimal in terms of achievable

rates compared to CM and MLC. More specifically, the results illustrate that the

38

Chapter6/Chapter6Figs/productornot.eps


mismatched conditional symbol probability and/or mismatched constellation areefficiently compensated by shaping at low SNR. Furthermore, when the reference

constellation at the receiver matches the transmitter constellation while keepingequiprobable conditional symbol probabilities at the receiver, the gap is closed for

the entire SNR range. Finally, we have analytically studied the BICM shaping

schemes in the low and high SNR regime, proved the first and second order opti-mality in the low SNR regime for the BICM shaping schemes we proposed, and

put forward a framework for studying the saturation regime of BICM schemes.

39


40

Chapter 5

Random Coding Error Exponent

of BICM

5.1 Introduction

It is well known that random coding techniques can be used to prove the achiev-

ability theorem of Shannon’s channel coding and joint source-channel coding theo-

rems and characterize the exponential decay (error exponent) of the average error

probability of the code ensemble for a range of rates under maximum-likelihood

(ML) decoding [27]. Moreover, in the literature, different types of random coding

ensembles [23, 27, 28] and bounding techniques [23, 27] are used to improve the

behavior of error exponents. In practice, due to the complexity of ML decod-

ing, mismatched decoding is often used. BICM is known to be one of the most

important practical mismatched decoding schemes. However, except in the case

that the i.i.d. random coding ensemble is used, little is known on the theoretical

behavior of the error exponent for BICM. In this chapter, we study random cod-

ing error exponents for different ensembles, and consider the BICM as one of the

special cases.

The behavior of the average error probability of a family of randomly gener-

ated i.i.d. codes, decoded with a maximum metric decoder,

u = argmaxu

q(x(u),y) = argmaxu

N∏

k=1

q(xk(u), yk). (5.1)

was studied in [33, 50, 75], and can be upper-bounded by

Pe ≤ e−NEr,0(R), (5.2)

41

5. RANDOM CODING ERROR EXPONENT OF BICM

where

Er,0(R) , maxPX

sup0≤ρ≤1s≥0

E0(ρ, s)− ρR (5.3)

is the random coding error exponent of the i.i.d. ensemble, and

E0(ρ, s) , − logE

[(∑

x′

PX(x′)q(x′, Y )s

q(X, Y )s

)ρ](5.4)

is the generalized Gallager function. In the special case of q(x, y) = PY |X(y|x), i.e.the maximum likelihood decoder is used, the optimal s∗(ρ) in (5.3) for maximum

likelihood decoder is s = 11+ρ

[27] and Eqs. (5.3) to (5.4) become

Er,0(R) , max0≤ρ≤1

E0

(ρ,

1

1 + ρ

)− ρR (5.5)

is the random coding exponent, and

E0(ρ) , E0

(ρ,

1

1 + ρ

)= − logE

[(∑

x′

PX(x′)PY |X(Y |x′)

11+ρ

PY |X(Y |X)1

1+ρ

)ρ](5.6)

is the Gallager function.

The GMI can be recovered from the generalized Gallager function E0(ρ, s).

We let s∗(ρ) achieve the supremum in

sups≥0

E0(ρ, s). (5.7)

We have (details are in Appendix B.1)

I0(PX) = sup0≤ρ≤1s≥0

E0

(ρ, s)

ρ(5.8)

= sups≥0

sup0≤ρ≤1

E0

(ρ, s)

ρ= sup

0≤ρ≤1sups≥0

E0

(ρ, s)

ρ(5.9)

= sups≥0

limρ→0

E0

(ρ, s)

ρ(5.10)

= limρ→0

E0 (ρ, s∗(ρ))

ρ(5.11)

42

5.1 Introduction

=dE0 (ρ, s

∗(ρ))

dρ

∣∣∣∣ρ=0

. (5.12)

We have also mentioned that the constant-composition and cost-constrained

i.i.d. ensembles can be used to achieve the LM rate in Section 3.2. In this thesis,

we focus on the cost-constrained i.i.d. ensemble, due to the fact that constant-

composition code is only valid with discrete alphabets. The authors in [63, 80]

have derived an achievable error exponent for this type of ensemble for fixed input

distribution PX , and it is given by

Er,1(R) , sup0≤ρ≤1

s≥0,r,r,a(·)

E1

(ρ, s, r, r, a(·)

)− ρR, (5.13)

where

E1

(ρ, s, r, r, a(·)

), − logE

[era(X)

era

(∑

x′

PX(x′)q(x′, Y )sera(x

′)

q(X, Y )sera

)ρ]. (5.14)

Recall that a(·) is introduced in Section 3.2.2 as pseudo-cost function to improve

the performance of the random-coding emsemble and a = E[a(X)]. This exponent

is a refinement of the following LM exponent given by Shamai and Sason [83],

Er,1′(R) , sup0≤ρ≤1s≥0,a(·)

E1′(ρ, s, a(·)

)− ρR, (5.15)

where

E1′(ρ, s, a(·)

), − logE

[(∑

x′

PX(x′)q(x′, Y )s

q(X, Y )sea(x

′)

ea(X)

)ρ]. (5.16)

When r = −ρ and r = 1, we have

E1

(ρ, s,−ρ, 1, a(·)

)= E1′

(ρ, s, a(·)

). (5.17)

Thus [80],

Er,1′(R) ≤ Er,1(R) (5.18)

always holds. Similarly to the argument behind Eqs. (5.8) to (5.12), the expres-

43


sion of LM rate can be recovered as

I1(PX) = sups≥0,a(·)0≤ρ1

E1′(ρ, s, a(·)

)

ρ= sup

0≤ρ1s≥0,a(·)

E1

(ρ, s,−ρ, 1, a(·)

)

ρ(5.19)

= sups≥0,a(·)

sup0≤ρ≤1

E1′(ρ, s, a(·)

)

ρ= sup

s≥0,a(·)sup

0≤ρ≤1

E1

(ρ, s,−ρ, 1, a(·)

)

ρ(5.20)

= sups≥0,a(·)

limρ→0

E1′(ρ, s, a(·)

)

ρ= sup

s≥0,a(·)limρ→0

E1

(ρ, s,−ρ, 1, a(·)

)

ρ(5.21)

= limρ→0

E1′(ρ, s∗(ρ), a∗(ρ)

)

ρ= lim

ρ→0

E1

(ρ, s∗(ρ),−ρ, 1, a∗(ρ)

)

ρ(5.22)

=dE1′

(ρ, s∗(ρ), a∗(ρ)

)

dρ

∣∣∣∣∣ρ=0

=dE1

(ρ, s∗(ρ),−ρ, 1, a∗(ρ)

)

dρ

∣∣∣∣∣ρ=0

, (5.23)

where s∗(ρ) and a∗(ρ) achieve the supremum in

sups≥0,a

E1

(ρ, s,−ρ, 1, a(·)

). (5.24)

We will give some properties on the special case of maximum likelihood de-

coder in Section 5.2, and before going into our main results, we first introduce

the following general properties

Proposition 5.1.1 (Concavity of E0(ρ, s) [6]) For a fixed input distribution,

the generalized Gallager function, E0(ρ, s), is a concave function of s for s ≥ 0

and of ρ for 0 ≤ ρ ≤ 1.

Proposition 5.1.2 (Concavity of E1′(ρ, s, a(·)

)and E1

(ρ, s, r, r, a(·)

)) For a

fixed input distribution, function E1′(ρ, s, a(·)

)is jointly concave in s ∈ [0,∞)

and a(x) for x ∈ X, and is a concave function of ρ for 0 ≤ ρ ≤ 1. The function

E1

(ρ, s, r, r, a(·)

)is jointly concave in s ∈ [0,∞) and a(x) for x ∈ X, jointly

concave in s ∈ [0,∞), r ∈ [0,∞) and r ∈ [0,∞), and concave in ρ for 0 ≤ ρ ≤ 1.

Proof: See Appendix B.2.

For simplicity, in the numerical results of this section, we only use the uniform

input distribution.

44

5.2 Maximum Likelihood Decoder

0 0.3 0.6 0.9 1.2 1.50

0.2

0.4

0.6

0.8

1

1.2

R (nats/channel use)

Err

orE

xpon

ent

Er,1(R)Er,0(R)

10 dB

5 dB

0 dB

Figure 5.1: Error exponents Er,0(R), Er,1(R) for 8PAM Coded Modulation overthe AWGN channel. The uniform input distribution is used. The exponents arecalculated for snr = 0, 5, 10 dB, and the corresponding I(PX) are marked withdots on the R axis.


For the maximum likelihood decoder, the random coding error exponents are

given by replacing the decoding metric q(x, y) with PY |X(y|x).

Proposition 5.2.1 (Optimal System Parameters) For the exponents Er,0(R),

Er,1′(R) and Er,1(R), when q(x, y) = PY |X(y|x), the optimal value of s is equal

to 11+ρ

for all three exponents. The optimal values of r, r satisfy r = r for

E1

(ρ, s, r, r, a(·)

). Furthermore, we have

Er,1′(R) = Er,0(R). (5.25)


45

Chapter8/Chapter8Figs/exponentCM.eps


The two exponents in (5.25) coincide with the Gallager’s random coding error

exponent. Therefore, in this section, we will only study one of the two exponents.

For simplicity, we choose Er,0(R). For the maximum likelihood decoder, the

following equality holds.

I(PX) = Icm0 (PX) = Icm1 (PX), (5.26)

where Icm0 (PX) and Icm1 (PX) are obtained by substituting q(x, y) = PY |X(y|x)into the expression of the GMI and the LM rate.

Proposition 5.2.1 also shows some insights on the optimal system parameters,

particularly on s, r and r. In turn, it gives rise to the following lemma that

further simplifies the expression of E1

(ρ, s, r, r, a(·)

)for q(x, y) = PY |X(y|x).

Lemma 5.2.1 When q(x, y) = PY |X(y|x), we let τ(x) = a(x)− a for x ∈ X, we

then have

sups≥0

r,r,a(·)

E1

(ρ, s, r, r, a(·)

)= sup

s≥0,τ(·):E[τ(X)]=0

− logE[eτ(X)−ρis,τ (X;Y )

], (5.27)

where

is,τ(x; y) , logPY |X(y|x)s

EX′

[eτ(X

′)PY |X(y|X ′)s] (5.28)

and optimal s∗ for the RHS of Eq. (5.27) satisfies s∗ = 11+ρ

.


For the LHS of Eq. (5.27), the maximization is over s ≥ 0 and zero-mean

function τ(·) or a(·). We know that the optimal s∗ = 11+ρ

for all ρ, and it is

difficult to give the expression of the pseudo cost function a(·) for general valuesof ρ. However, this will not be a problem if we narrow the region of analysis to

near ρ→ 0. Thus, we have the following result near ρ→ 0

Theorem 5.2.1 When ρ→ 0, the maximizing a(·) for

G(ρ, s, a(·)

), − logE

[ea(X)−ρis,a(X;Y )

], (5.29)

under the constraint E[a(X)] = 0 is such that

a(x) = 0 for all x. (5.30)

46


−1.5 −1 −0.5 0 0.5 1 1.50

0.01

0.02

0.03

0.04

0.05

0.1

0.16

X

a(x

)−E

[a(X

)]i(

x)−

I(P

X)

ρ = 0.01

ρ = 0.02

ρ = 0.03

ρ = 0.04

ρ = 0.1

ρ = 0.2

ρ = 0.05

Figure 5.2: The value of a(x)−E[a(X)]i(x)−I(PX)

for 8PAM CM transmission over AWGNchannel for snr = −5dB. The markers ’·’ denote the 8PAM constellation. Thevalues of a(x) are obtained using the gradient descent type convex optimizationmethod.

Moreover, the derivative of a(x) with respect to ρ5.1 evaluated at ρ → 0 is given

by

a(x)′ =da(x)

dρ= i(x)− I(PX), (5.31)

where I(PX) is the mutual information for a given input distribution PX , and

i(x) , EPY |X=x[i(x; Y )] (5.32)

i(x; y) , logPY |X(y|x)

EX′

[PY |X(y|X ′)

] (5.33)

5.1Note that we do not always write a(x) as an explicit function of ρ, but the reader shouldbear in mind that the optimal a(x) is a function of ρ.

47

Chapter8/Chapter8Figs/axixratio.eps


.0 0.2 0.4 0.6 0.8 1

1

1.05

1.1

1.15

1.2


Er,1

(R)

Er,0

(R)

0 0.2 0.4 0.6 0.8 10

0.005

0.01

0.015

0.02

0.025

Er,

1(R

)−

Er,

0(R

)Figure 5.3: The ratio and difference between the LM exponent Er,1(R) and GMIexponent Er,0(R) for 8PAM CM transmission over the AWGN channel. The solidline is the ratio of the two exponents and the dash-dotted line is the differenceof the exponents. The dotted line indicates the largest achievable rate at thisSNR. The marker is the analytical limit of the ratio

Er,1(R)

Er,0(R)as R → I(PX) given

by Theorem 5.2.2. We use snr = 5 dB.

is the information density function.

Proof: See Appendix B.5Figure 5.1 shows the exponents calculated for 8PAM CM transmission over the

AWGN channel for snr values at 0, 5, 10 dB. A uniform input distribution is usedand the I(PX) at different SNR values are marked on the R axis. The optimization

in the evaluation of the exponents makes uses of the Propositions 5.1.1-5.2.1directly for Er,0(R). The optimal parameters

(s, r, r, a(·)

)for Er,1(R) with fixed

ρ are chosen using an alternating optimization among s, a and (r, r), and agradient descent type of convex optimization method is used at each optimization

step. We observe some improvement of the LM exponent Er,1(R) over the GMIexponent Er,0(R) (thus also over the LM exponent Er,1′(R)). This highlights that

imposing a cost constraint a(·) on the codewords can actually improve the random

coding ensemble in terms of the error exponent, though it can not improve theinformation rate.

48

Chapter8/Chapter8Figs/CMexponentratiodiffsnr5.eps


In our simulation results on Er,1(R) with ML decoder, we observe the optimal

values of r and r satisfy r = r in all cases, which is in line with Proposition 5.2.1.

Figure 5.2 shows the value of a(x)−E[a(X)]i(x)−I(PX)

for 8PAM CM transmission over AWGN

channel for snr = −5dB. The values of a(x) are optimized for the corresponding

value of ρ. We observe that for small ρ, the optimal a(x) satisfies a(x)−E[a(X)] =

ρ(i(x)− I(PX)

). This is in line with the result in Theorem 5.2.1.

One way to study how the error exponents are improved by using cost-

constrained i.i.d. code ensemble is to look at the ratio Er,1(R)

Er,0(R)and the difference

Er,1(R)−Er,0(R). In Figure 5.3, we plot how much Er,1(R) improves over Er,0(R)

in terms of their ratio as well as the difference for 8PAM CM transmission over

AWGN channel at snr = 5 dB. We observe that the improvement can be as high

as 18%. The difference is a constant for rates smaller than the critical rate derived

from dE0(ρ,s∗(ρ))dρ

∣∣∣ρ=1

.

It is difficult to study the ratio of improvement analytically at all rates. How-

ever, in the region where R → I(PX), we are able to do some asymptotic analysis,

and we obtain the following theorems.

Theorem 5.2.2 For the maximum likelihood decoder, i.e. q(x, y) = PY |X(y|x),the ratio between the LM exponent Er,1(R) and GMI exponent Er,0(R) evaluated

at R → I, limR→IEr,1(R)

Er,0(R), is given by

limR→I

Er,1(R)

Er,0(R)=E

′′

0 (ρ)|ρ=0

E′′

1 (ρ)|ρ=0

, (5.34)

where I , I(PX) and

E′′

0 (ρ) =d2 sups≥0E0(ρ, s)

dρ2(5.35)

E′′

1 (ρ) =

d2 sups≥0,rr,a(·)

E1

(ρ, s, r, r, a(·)

)

dρ2. (5.36)

Proof: Both exponents Er,0(R) and Er,1(R) admit a Taylor expansion at

R → I(X ; Y ),

Er,0(R) = Er,0(I) +dEr,0(R)

dR

∣∣∣∣R=I

(R− I) +1

2

d2Er,0(R)

dR2

∣∣∣∣R=I

(R− I)2

+ o((R− I)2

)(5.37)

49


Er,1(R) = Er,1(I) +dEr,1(R)

dR

∣∣∣∣R=I

(R− I) +1

2

d2Er,1(R)

dR2

∣∣∣∣R=I

(R− I)2

+ o((R− I)2

)(5.38)

Note that

Er,0(I) = 0 (5.39)

Er,1(I) = 0 (5.40)

Let ρ(R) denotes the maximizing ρ for Er,0(R), i.e.

Er,0(R) = sups≥0

E0(ρ(R), s)− ρ(R)R (5.41)

HencedEr,0(R)

dR=∂ sups≥0E0(ρ(R), s)

∂ρ

dρ

dR− dρ

dRR − ρ(R) (5.42)

Since E0(ρ, s) is a concave function of ρ ∈ [0, 1], for R → I, the ρ that maximizes

sups≥0E0(ρ, s)− ρR for a fixed R satisfies

d sups≥0E0(ρ, s)− ρR

dρ=d sups≥0E0(ρ, s)

dρ− R = 0 (5.43)

Substituting (5.43) into (5.42), we have

dEr,0(R)

dR= −ρ(R). (5.44)

Hence,dEr,0(R)

dR

∣∣∣∣R=I

= −ρ(I) = 0. (5.45)

Similarly, we can obtain that

dEr,1(R)

dR= −ρ′(R) (5.46)

dEr,1(R)

dR

∣∣∣∣R=I

= −ρ′(I) = 0 (5.47)

50


where ρ′(R) denotes the maximizing ρ for Er,1(R), i.e.

Er,1(R) = sups≥0,r,r,a(·)

0≤ρ≤1

E1

(ρ, s, r, r, a(·)

)− ρR (5.48)

= sups≥0,rr,a(·)

E1

(ρ′(R), s, r, r, a(·)

)− ρ′(R)R (5.49)

From (5.44) and (5.46), we have

d2Er,0(R)

dR2= −dρ(R)

dR(5.50)

d2Er,1(R)

dR2= −dρ

′(R)

dR(5.51)

Due to (5.43), we have

R(ρ) =d sups≥0E0(ρ, s)

dρ(5.52)

where R(ρ) is the inverse function of ρ(R) and

dR(ρ)

dρ=d2 sups≥0E0(ρ, s)

dρ2(5.53)

Combining (5.50), (5.52) and (5.53) we have

d2Er,0(R)

dR2= − 1

d2 sups≥0 E0(ρ,s)

dρ2

. (5.54)

Similarly we have

d2Er,1(R)

dR2= − 1

d2 sups≥0,rr,a(·)

E1

(ρ,s,r,r,a(·)

)

dρ2

. (5.55)

The theorem then follows by substituting Eqs. (5.39), (5.40), (5.45), (5.47), (5.54)

and (5.55) into Eqs. (5.37) and (5.38).

The second order derivatives of sups≥0E0(ρ, s) and sups≥0,r,r,a(·)E1

(ρ, s, r, r, a(·)

)

with respect to ρ is difficult to evaluate and does not give us much information

about the optimal values of the system parameters. The following theorem is

introduced to facilitate the calculation of Eqs. (5.35) and (5.36).

51


Theorem 5.2.3

−E ′′

0 (ρ)|ρ=0 = var[i(X ; Y )

](5.56)

−E ′′

1 (ρ)|ρ=0 = E

[var[i(X ; Y )|X

]]= var

[i(X ; Y )

]− var

[i(X)

], (5.57)


By definition, we have

E

[var[i(X ; Y )|X

]]≤ var

[i(X ; Y )

]. (5.58)

−20 −10 0 10 20 30 401

1.02

1.04

1.06

1.08

1.1

1.12

1.14

1.16

1.18

snr (dB)

Er,1

(I)

Er,0

(I)

4PAM8PAM16PAMM → ∞

Figure 5.4: The limiting value ofEr,0(R)

Er,1(R)as R → I for 4PAM, 8PAM and 16PAM

CM transmission over AWGN channel for different snr. The case M → ∞ is alsoshown by using a continuous uniform distribution between [−

√3,√3]

The marker in Figure 5.3 plots the ratio of the two exponent at R → I ob-tained by Theorem 5.2.2 and 5.2.3. We observe that the analysis is consistent with

52

Chapter8/Chapter8Figs/CMexponentratio.eps

5.3 Mismatched BICM Decoder

the simulation. We also show in Figure 5.4 the limiting value ofEr,0(R)

Er,1(R)as R → I

for 4PAM, 8PAM and 16PAM CM transmission over AWGN channel across theSNRs. The gain is most prominent in the SNR range [0, 5] dB, and there is

actually no gain at very low or very high SNRs by using the cost-constrained en-

semble at R → I. Moreover, for larger constellations, the improvement of the LMexponent Er,1(R) is bigger and nearly reaches 20%. This translates into a 20% re-

duction of the block length when using constant-composition or cost-constrained

i.i.d. random coding ensembles with respect to i.i.d. random coding ensemble5.2.

This signifies the benefit of using cost constrained i.i.d. ensemble. Finally, fromthe figure we observe that as the constellation grows even larger, the improve-

ment introduced by increasing the constellation size will become a constant, i.e.

the gain limR→IEr,0(R)

Er,1(R)saturates as M → ∞. We have approximated M → ∞

by a uniform distribution between [−√3,√3] as an indirect way of showing this.

The interval is chosen such that the inputs have unit energy. The dotted line in

Figure 5.4 plots the ratioEr,0(R)

Er,1(R)as R → I for this type of input. We observe that

it is slightly above the curve for 16PAM, and that it matches our observation.


When the decoder is mismatched, Proposition 5.2.1 as the base of many theoremsin the previous section is no longer true in general. Also, since Ibicm0 (P bicm

X ) and

Ibicm1 (P bicmX ) are not equal in general, the analysis of the previous section is no

longer directly applicable. We already know from Chapter 4 that the LM rate

Ibicm1 (P bicmX ) only marginally improves over the GMI Ibicm0 (P bicm

X ) for well-behavedBICM decoders (the BICM decoders we have considered). Similar to CM, we will

show in this section that the corresponding exponents for the LM rate Er,1(R)

and Er,1′(R) does improve on the GMI exponent by a considerable margin even

for well-behaved BICM decoders, especially for the LM exponent Er,1(R).We plot the three exponents for BICM1 with equiprobable input distribution

(as in Table 4.4) over an AWGN channel in Figure 5.5 at snr = 5 dB. The

exponents of CM transmission are also shown for reference. The markers on

the R axis show I(PX), Ibicm0 (P bicm

X ) and Ibicm1 (P bicmX ). In this example, unlike

Proposition 5.2.1 for the matched decoder, we observe that the LM exponent

Er,1′(R) gives a higher exponent than the GMI exponent Er,0(R). We also see

that though the LM rate improves on the GMI marginally, both LM exponents are

5.2According to the random coding upper bound for error probability by Gallager, the averageerror probability is upper-bounded by the exponential of the negative product of block lengthand the error exponent. If the error exponent is increased by a certain amount, for the sameerror probability level, the required block length can be reduced by the same amount

53


0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8


Err

orE

xpon

ent

Er,1(R) (CM)Er,0(R) (CM)Er,1(R) (BICM1)Er,0(R) (BICM1)Er,1′(R) (BICM1)

Figure 5.5: GMI and LM rate error exponents for BICM0 (as in Tab. 4.4) 8PAMtransmission over AWGN channel at snr = 5 dB. The exponents of CM trans-mission are also shown as reference. The corresponding I(PX), I

bicm0 (P bicm

X ) andIbicm1 (P bicm

X ) are marked on the R axis.

indeed improved. Similar to the observation by Scarlett [80], the LM exponent

Er,1(R) has a significant improvement over LM exponent Er,1′(R). Finally, we

also observe that the BICM error exponents are close to those of CM.

A known drawback of the i.i.d. random coding ensemble with discrete alpha-

bet is that at low rates it suffers from the effect of some exceedingly bad codes

having two or more code vectors which are identical. Expurgation often is used

to mitigate this effect and to improve the bound at low rates. We will generalize

the expurgated random coding upper bound [27] for mismatched decoding for

both the i.i.d. and cost-constrained i.i.d. ensembles in this section.

Consider a particular code with A′ codewords x1, . . . ,xA′ in the cost con-

strained i.i.d. random coding ensemble, where the length N codewords satisfying

54

Chapter8/Chapter8Figs/exponentbicm0.eps


the cost constraint

Na− δ < a(x) ≤ Na (5.59)

where a(x) =∑N

k=1 a(xk) is the pseudo-cost function, and a = E[a(X)] and δ > 0

limit the shell on which codewords lie. Let X be an i.i.d. random vector with

distribution PX satisfying the cost, i.e., EX∼PX[a(X)] = Na. We construct a new

codebook by randomly selecting codewords according to a distribution PX(x) in

the shell

PX(x) = ξ−111 Na− δ < a(x) ≤ NaPX(x), (5.60)

where ξ is a normalization constant. This is Gallager’s shell construction [27].

Since

11 Na− δ < a(x) ≤ Na ≤ er(a(x)−Na+δ

)for r ≥ 0, (5.61)

for general f(x), we have that

PX(x) ≤1

ξPX(x)e

r

(a(x)−Na+δ

)(5.62)

EX∼PX

[f(X)] ≤ EX∼PX

[f(X)er

(a(X)−Na+δ

)]. (5.63)

We can now easily derive the expurgated error exponent for the cost-constrained

i.i.d. code ensemble. When message u is transmitted and maximum metric

decoding is used, the error probability Pe,m is given by

Pe,u = EPY|X[11 Y ∈ Y

cu] , (5.64)

where the expectation is over PY|X(y|xu) and Ycu is the region where q(xu,y) ≤

q(xu′ ,y) for all u′ 6= u. For any y ∈ Ycu, we can upper bound PY|X(y|xu) by

PY|X(y|xu) ≤ PY|X(y|xu)q(xu′,y)s

q(xu,y)sany u′ 6= u, s ≥ 0. (5.65)

Eq. (5.65) gives an upper bound to Pe,m by using the union bound

Pe,u ≤∑

u′ 6=u

EPY|X=xu

[q(xu′,Y)s

q(xu,Y)s

]. (5.66)

Now let γ be restricted to 0 < γ ≤ 1. Using the standard inequality ((∑ai)

γ ≤

55


∑aγi for ai > 0, see for [27, Problem 4.15f]), we have

(Pe,u)γ ≤

∑

u′ 6=u

(EPY|X=xu

[q(xu′,Y)s

q(xu,Y)s

])γ

0 < γ ≤ 1. (5.67)

Consider an ensemble of codes in which each codeword is selected independently

with the probability assignment PX(x).

Pe,u ≤∑

u′ 6=u

∑

xu

∑

xu′

PX(xu)PX(xu′)

(EPY|X=xu

[q(xu′ ,Y)s

q(xu,Y)s

])γ

. (5.68)

Before going any further, we present the following lemma,

Lemma 5.3.1 (Gallager [27]) For any γ > 0, there is at least one code in

the ensemble of codes with A′ = 2A − 1 codewords for which at least A of the

codewords satisfy

Pe,u < 21γ

(Pe,u

)γ(5.69)

Since in Eq. (5.68), xu′ is a dummy variable of summation in Eq. (5.68), the

term inside the largest bracket is independent of u′ and we have A′−1 = 2(A−1)

identical terms. By applying Lemma 5.3.1, we have that at least there is one code

with A codewords whose error probability when message u is transmitted satisfies

Pe,u < 21γ

(2(A− 1)

∑

x

∑

x′

PX(x)PX(x′)

(EPY|X=x

[q(x′,Y)s

q(x,Y)s

])γ) 1

γ

. (5.70)

Defining ρ = 1γ, we use a change of variables in Eq. (5.70) to obtain

Pe,u <(4(A− 1)

)ρ(∑

x

∑

x′

PX(x)PX(x′)

(EPY|X=x

[q(x′,Y)s

q(x,Y)s

]) 1ρ

)ρ

. (5.71)

Eq. (5.71) is valid for each message in the code. Substitute Eqs. (5.62) and

(5.63), we have

Pe,u <

(4(A− 1)

ξ2e−δ(r+r)

)ρ(∑

x

∑

x′

PX(x)er(a(x)−Na)

e−r(a(x′)−Na)PX(x

′)

(EPY|X=x

[q(x′,Y)s

q(x,Y)s

]) 1ρ

)ρ

.

(5.72)

56


For a memoryless channel and distribution PX satisfying

PX(x) =

N∏

k=1

PX(xk), (5.73)

and cost constraint satisfying a(x) =∑N

k=1 a(xk). Similar to [27, Section 5.7], we

can expand the product in Eq. (5.72) and have the following.

Theorem 5.3.1 For an arbitrary memoryless channel, let N be any positive in-

teger and R be any positive number. There exist codes from the cost-constrained

i.i.d. random coding ensemble with length N , rate R and pseudo-cost function

a(·) for which, for a given input distribution PX and all messages

Pe,u ≤ e−NEx

r,1

(

R+ 1N

log 4eδ(r+r)

ξ2

)

, (5.74)

where the function Exr,1(·) is given by

Exr,1(R) , sup

s≥0,ρ≥1r,r,a(·)

Ex1

(ρ, s, r, r, a(·)

)− ρR (5.75)

Ex1

(ρ, s, r, r, a(·)

), −ρ log

∑

x

∑

x′

PX(x)era(x)−raPX(x

′)era(x)−ra

·(EPY |X=x

[q(x′, Y )s

q(x, Y )s

]) 1ρ

. (5.76)

For the i.i.d. code ensemble without the cost constraint, the corresponding

theorem can be obtained by just ignoring the constraint.

Theorem 5.3.2 For an arbitrary memoryless channel, let N be any positive in-

teger and R be any positive number. There exist codes from the i.i.d. random

coding ensemble with length N and rate R for which, for a given input distribution

PX and all messages

Pe,u ≤ e−NExr,0(R+ log 4

N ), (5.77)


Exr,0(R) , sup

s≥0,ρ≥1Ex

0 (ρ, s)− ρR (5.78)

Ex0 (ρ, s) , −ρ log

∑

x

∑

x′

PX(x)PX(x′)

(EPY |X=x

[q(x′, Y )s

q(x, Y )s

]) 1ρ

. (5.79)

57


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

R (nats/Channel use)

Err

or

Exponen

t

Exr,0(R)

Exr,1(R)

Er,0(R)Er,1(R)

Figure 5.6: The GMI and the LM error exponents and expurgated error exponentsfor BICM1 with equiprobable input distribution (as in Tab. 4.4), using 8PAMtransmission over the AWGN channel at snr = 5 dB.

Corollary 5.3.1

Exr,1(R) ≥ Ex

r,0(R). (5.80)

Proof: The expression of Exr,0(R) can be recovered from Ex

r,1(R) by letting

a(x) be constant.

Corollary 5.3.2 For a given rate R, if the optimal values of ρ for the error

exponents Er,0(R), Er,1(R) and expurgated error exponent Exr,0(R), E

xr,1(R) are

all ρ = 1, we have

Er,0(R) = Exr,0(R) (5.81)

Er,1(R) = Exr,1(R). (5.82)

Figure 5.6 plots the error exponents Er,0(R), Er,1(R) and the expurgated errorexponents Ex

r,0(R), Exr,1(R) for the i.i.d. random coding ensemble and the cost-

58

Chapter8/Chapter8Figs/expurexponentbicm0.eps


constrained i.i.d. random coding ensemble for BICM1 transmission with 8PAM

signaling over the AWGN channel with equiprobable input distribution. We

observe that for rates where the optimal ρ equals one for both the error exponent

and the expurgated error exponent for an input ensemble with or without a cost-

constraint, the two exponents of each code ensemble coincide. For smaller rates

where the optimal ρ is greater than one for the expurgated error exponent, there

is a significant improvement when the bad codes are expurgated for both types

of inputs. On the other hand, we observe that the cost-constraint on the code

ensemble helps us improve the expurgated error exponent for almost all rates,

though this improvement vanishes as R → 0.

Theorem 5.3.3

limR→0

Exr,0(R) = lim

R→0Ex

r,1(R) = sups≥0

−EX′X

[log

(EPY |X

[q(X ′, Y )

q(X, Y )

∣∣∣∣X,X′])]

.

(5.83)

Before proving the theorem, we first present the following proposition.

Proposition 5.3.1

limR→0

sups≥0,ρ≥1

Ex0 (ρ, s)− ρR = sup

s≥0,ρ≥1limR→0

Ex0 (ρ, s)− ρR (5.84)

limR→0

sups≥0,ρ≥1r,r,a(·)

Ex1

(ρ, s, r, r, a(·)

)− ρR = sup

s≥0,ρ≥1r,r,a(·)

limR→0

Ex1

(ρ, s, r, r, a(·)

)− ρR.

(5.85)


We now proceed to show the proof of Theorem 5.3.3

Proof: Proposition 5.3.1 shows that

limR→0

Exr,0(R) = sup

s≥0,ρ≥1Ex

0 (ρ, s) (5.86)

limR→0

Exr,1(R) = sup

s≥0,ρ≥1r,r,a(·)

Ex1

(ρ, s, r, r, a(·)

). (5.87)

For fixed s, according to Lemma B.1.2,

Ex0 (ρ, s) = −ρ log

∑

x

∑

x′

PX(x)PX(x′)

(EPY |X=x

[q(x′, Y )s

q(x, Y )s

]) 1ρ

(5.88)

59


is a nondecreasing concave function with ρ for ρ ≥ 0. Hence the supremum of

Ex0 (ρ, s) for fixed s over ρ ≥ 1 is at ρ→ ∞. Evaluating this limit by L’Hopital’s

rule, we obtain

limR→0

Exr,0(R) = sup

s≥0limρ→∞

Ex0 (ρ, s) (5.89)

= sups≥0

−∑

x

∑

x′

PX(x)PX(x′) log

(EPY |X=x

[q(x′Y )s

q(x, Y )s

])(5.90)

For sups≥0,ρ≥1r,r,a(·)

Ex1

(ρ, s, r, r, a(·)

), we first do some changes of variables as fol-

lowing

sups≥0,ρ≥1r,r,a(·)

Ex1

(ρ, s, r, r, a(·)

)(5.91)

= sups≥0,ρ≥1r,r,a(·)

−ρ log∑

x

∑

x′

PX(x)era(x)−raPX(x

′)era(x)−ra

(EPY |X=x

[q(x′, Y )s

q(x, Y )s

]) 1ρ

(5.92)

= sups≥0,ρ≥1r,r,a(·)

−ρ log∑

x

∑

x′

PX(x)PX(x′)

(erρa(x)−rρa+rρa(x)−rρa

EPY |X=x

[q(x′, Y )s

q(x, Y )s

]) 1ρ

(5.93)

= sups≥0,ρ≥1r′,r′,a(·)

−ρ log∑

x

∑

x′

PX(x)PX(x′)

(er

′a(x)−r′a+r′a(x)−r′ aEPY |X=x

[q(x′, Y )s

q(x, Y )s

]) 1ρ

,

(5.94)

where we have defined r′ = rρ and r′ = rρ. For fixed s, r′, r′, a, the term inside the

sup of Eq. (5.94) is a nondecreasing concave function of ρ with ρ > 0 according

to Lemma B.1.2. Hence the supremum for fixed s, r′, r′, a over ρ ≥ 1 is at ρ→ ∞.

60

5.4 Multiple Costs

Evaluating this limit by L’Hopital’s rule, we obtain

limR→0

Exr,1(R) = sup

s≥0r,r,a(·)

limρ→∞

Ex1

(ρ, s, r, r, a(·)

)(5.95)

= sups≥0

r,r,a(·)

−∑

x

∑

x′

PX(x)PX(x′) log

(er

′a(x)−r′a+r′a(x)−r′aEPY |X=x

[q(x′Y )s

q(x, Y )s

])

(5.96)

= sups≥0

−∑

x

∑

x′

PX(x)PX(x′) log

(EPY |X=x

[q(x′Y )s

q(x, Y )s

]), (5.97)

which coincides with Eq. (5.90). This concludes the proof.

It follows from the above theorem that indeed at R → 0, the improvement

introduced by using a cost-constrained ensemble vanishes for the expurgated error

exponent.

5.4 Multiple Costs

In [80], the authors have introduced the cost-constrained i.i.d. ensemble with L

cost constraints to improve on Er,1(R). They use the fact that the distribution

of the cost-constrained codewords is given by

PX(x) =11 Nak − δk < ak(x) ≤ Nak, k = 1, . . . , LPX(x)

ζ

≤ ζ−1PX(x)e∑L

k=1 rk(ak(x)−Nak+δk). (5.98)

and that ζ decays polynomially in N to show that the corresponding error expo-

nent is given by

Er,L(R) , sup0≤ρ≤1,s≥0

rk,rk,ak(·)

EL

(ρ, s, rk, rk, ak(·)

)− ρR, (5.99)

where

EL


), − logE

E

[q(X ′, Y )se

∑Lk=1 rk(ak(X

′)−ak)|Y]

q(X, Y )se−∑L

k=1rkρ(ak(X)−ak)

ρ ,

(5.100)

61


ak = E[ak(X)], ak(·) denotes a1(·), . . . , aL(·), and similarly for rk and rk.The authors in [80] have shown that the L cost-constrained ensemble contains

the constant composition ensemble as a special case, and they have shown thatthe constant composition ensemble exponent can be recovered using the cost-

constrained i.i.d ensemble with at most two cost constraints.

Similarly, we can generalize the results in the previous section for the mismatched-decoding expurgated exponent for the L-cost-constrained i.i.d. ensemble.

Theorem 5.4.1 For an arbitrary memoryless channel, there exist codes from the

expurgated cost-constrained i.i.d. random coding ensemble with length N , rate R

and pseudo-cost function ak(·) with k = 1, . . . , L for which, for a given input

distribution PX and all messages

Pe ≤ e−NEx

r,1

(

R+ 1N

log 4ζ2

e∑L

k=1 δk(rk+rk)

)

, (5.101)


Exr,1(R) , sup

s≥0,ρ≥1,rk,rk,ak(·)

Ex1


)− ρR (5.102)

Ex1


), −ρ log

∑

x

∑

x′

PX(x)

· PX(x′)e∑L

k=1 rkak(x)−rkak

e−∑L

k=1 rkak(x)−rk ak

(EPY |X=x

[q(x′, Y )s

q(x, Y )s

]) 1ρ

. (5.103)

The Theorem 5.3.3 can also be generalized as

Theorem 5.4.2 The expurgated exponents have the following property at R → 0

limR→0

Exr,0(R) = lim

R→0Ex

r,1(R) = sups≥0

−E

[log

(E

[q(X ′, Y )s

q(X, Y )s

∣∣∣∣X,X′])]

. (5.104)

Proof: This is theorem can be proved easily by following the proof ofTheorem 5.3.3.


We have studied the random coding error exponent for matched CM and mis-matched BICM. Several ensembles have been considered, namely the i.i.d. ensem-

62


ble and cost-constrained i.i.d. ensemble. For the maximum likelihood decoder,we have shown that the LM exponent Er,1′(R) is equal to the Gallager’s expo-

nent Er,0(R), whereas the LM exponent Er,1(R) [63, 80] generally improves onEr,0(R). In particular, we have investigated the optimal system parameters near

ρ→ 0, we also have given an asymptotic analysis on the ratio of the improvement

of Er,1(R) over Er,0(R) at R → I and shown that it is equal to the inverse ofthe ratio of the second derivatives of the E1(·) and E0(·) functions with respect

to ρ. Moreover, we have drawn the connection between the second derivativesof the E1(·) and E0(·) functions with respect to ρ and the information density

function, which results in an easier term to evaluate. For the error exponent atR → I, we have shown that the improvement limR→I

Er,1(R)

Er,0(R)introduced by using

cost-constrained ensemble is the most prominent in the SNR range of [0, 5] dB,

while for very low and high SNRs, there is negligible improvement.

For mismatched BICM decoder, we have shown some numerical results onthe BICM error exponent Er,0(R), Er,1′(R) and Er,1(R). Unlike the maximum

likelihood decoding case, Shamai’s LM exponent Er,1′(R) does improve on thegeneralized Gallager exponent Er,0(R). We have then focused on how the ex-

purgation of codewords affects the performance of the i.i.d. and cost-constrainedi.i.d. random coding ensembles for general mismatched decoders (hence including

BICM). For rates where the optimal ρ equals one for both expurgated error ex-ponents and error exponents, the two types of error exponents coincide for both

the i.i.d. and the cost-constrained i.i.d. ensembles. However, for smaller rateswhere the optimal ρ is greater than one for expurgated error exponents, there

is a significant improvement, since the expurgation removes bad codes from theinput ensembles. Finally, our results show that the pseudo-cost on the inputs

helps us improve the expurgated error exponents for almost all rates. However,this improvement vanishes at R → 0, i.e. limR→0E

xr,0(R) = limR→0 E

xr,1(R).

63


64

Chapter 6

Error Probability Analysis

This chapter is motivated by our result in Chapter 4 that BICM schemes with a

mismatched reference constellation and conditional symbol probabilities can still

be asymptotically optimal in the low-SNR or low-rate regime when the input

distribution is optimized. This is shown from an information-theoretic point of

view by studying the achievable rates, usually when good codes are used, the gain

on the information rates can be translated into a gain in the error probability.

In this chapter, we study mismatched BICM schemes (particularly BICM2 and

BICM3 in Table 4.4) from a more practical point of view. In particular, we

analyze and compare the error probability of BICM schemes using convolutional

codes. This results match the information-theoretic prediction in the Chapter 4.

We will first introduce some preliminaries related to the error probability

analysis for BICM, and then we will illustrate how to build a BICM scheme

with shaping codes for the cases we are interested in. Finally, analytical bounds,

approximations and simulations will be presented.

6.1 Preliminaries

The error probability analysis of BICM1 with equiprobable inputs has been stud-

ied extensively in [33, 64, 68]. In these references, the union bound technique is

used to obtain a tight upper bound for the BER of BICM1 at high-SNR. The

upper bound is expressed in terms of the pairwise error probability by

Pe ≤1

|M|

|M|∑

u=1

∑

u′ 6=u

PEP(b(u′), b(u)

), (6.1)

65

6. ERROR PROBABILITY ANALYSIS

where

PEP(b(u′), b(u)

),Pr

q(µ (b(u′)) ,y

)> q(µ (b(u)) ,y

)(6.2)

=Prq(x(u′),y

)> q(x(u),y

). (6.3)

We define the pairwise score as

Ξpw ,

N∑

k=1

m∑

j=1

logqj(bm(k−1)+j(u

′), yk)

qj(bm(k−1)+j(u), yk). (6.4)

The pairwise score can be expressed as

Ξpw =

N∑

k=1

Ξsk =

N∑

k=1

m∑

j=1

Ξbk,j, (6.5)

where

Ξsk ,

m∑

j=1

logqj(bm(k−1)+j(u

′), yk)

qj(bm(k−1)+j(u), yk)(6.6)

is the k-th symbol score and

Ξbk,j , log

qj(bm(k−1)+j(u′), yk)

qj(bm(k−1)+j(u), yk)(6.7)

is the bit score corresponding to the j-th bit of the k-th symbol. These scores are

random variables depending on the transmitted bits, their position in the symbol

and the bit pattern, and they are difficult to analyze. The random coset code

technique is used in [8, 51] to remove this dependence, and using random coset

code is equivalent to scrambling the output of the encoder b ∈ C by a sequence r ∈0, 1n known at the receiver. This guarantees that the symbols corresponding

to two m-bit sequences are mapped to all possible pairs of modulation symbols

differing in a given Hamming distance.

When shaping is performed, the use of random coset coding will change the

bit probabilities PB, no matter what it is, to uniform, and hence shaping will

be compromised. While the general method for analyzing the BER of BICM

with shaping is still unknown, we are going to show in this chapter that one can

analyze the BER of BICM shaping schemes using the method in [33, 64, 68] for

cases when the optimal input distribution effectively leads to 2PAM signaling for

an M-PAM reference constellation or QPSK signaling for an M-QAM reference

constellation.

66

6.1 Preliminaries

Recalling the results we obtained on BICM achievable rates with shaping inChapter 4, we have shown that in the low-SNR or the low-rate regime, the optimal

input distribution (as a result of shaping) for the systems we considered is theone that picks a pair of nested 2PAM symbols out of an M-PAM constellation

or QPSK symbols out of an M-QAM constellation. We analyze the BER of

these BICM shaping schemes in the low-rate regime, relevant in several practicalscenarios. As we have seen in Chapter 4, in this regime, mismatched BICM

virtually recovers all the loss by shaping, and the optimal input distributionsare effectively uniform 2PAM/QPSK. The BER analysis in this regime can be

done by using the method in [33, 64, 68], helps us draw connections betweenerror performance and information rate results, understand more about binary

transmission and provide guidelines for practical system designs. Therefore, inthe remainder of this chapter, we focus on the BICM shaping schemes in the

low-rate regime.

6.1.1 BICM Schemes and Its Realizations

Note that with Gray labeling, the analysis of real modulation (PAM) can bereadily extended to complex modulation (QAM), so we will henceforth only study

PAM signaling. We first define the four BICM schemes we are going to study.

Definition 6.1.1 (Scheme BICM1(2PAM)) This scheme uses BICM1 in Ta-

ble 4.4 with a uniform 2PAM X and X.

This scheme is equivalent to a 2PAM coded modulation system, and sets the

benchmark for our analysis.

Definition 6.1.2 (Scheme BICM1(MPAM) for M > 2) This scheme uses BICM1

in Table 4.4 with uniform MPAM X and X.

Definition 6.1.3 (Scheme BICM2(2PAM-MPAM)) This scheme uses BICM2

in Table 4.4 where the decoder assumes equiprobable conditional symbol probabil-

ities and the MPAM reference constellation X satisfies Eq. (4.21). At the trans-

mitter we use shaping codes which effectively result in 2PAM modulation even

though the constellation at the transmitter, X, is MPAM.

There are many ways to set the bit/symbol probability such that the shaping

code is effectively mapped to a uniform 2PAM constellation after modulation.However, only symbols −1 and 1 are used with uniform probability among the

reference constellation for each case and the decoder in this scheme does not

depend on the P bicmX and X at the transmitter at all. Therefore, there is no

difference in their performance (in terms of Cbicm0 , Cbicm

1 and BER).

67


Definition 6.1.4 (Scheme BICM3(2PAM-MPAM)) This scheme uses BICM3

scheme in Table 4.4 where the decoder assumes X = X and equiprobable condi-

tional symbol probabilities.

The decoder in this scheme depends on X at the transmitter, and hence the

performance will be different depending on which pair of 2PAM symbols it chooses

after modulation. From our previous shaping result in the low-SNR regime, for

8PAM, the optimal rate Cbicm0 of BICM3 is achieved when the inputs (X1, P 1

B) as

in Eq. (4.23) are used. The GMI decreases as the inputs change from (X1, P 1B)

to (X4, P 4B), but the difference among them is quite small. According to this, we

believe that the BERs should behave in the opposite way, i.e. increasing as the

inputs change from the outermost to the innermost pair of nested 2PAM signal

set.

In the previous chapter, we learnt that in the low rate regime, the GMI of the

BICM2 and BICM3 shaping schemes outperform that of BICM1 with equiprob-

able inputs. Hence, when a good low rate code (i.e. a capacity approaching

or convolutional low rate code) is used, one should expect better BER per-

formance of BICM2(2PAM-MPAM) and BICM3(2PAM-MPAM) compared to

BICM1(MPAM). In fact, their BER performance should be very close to that of

BICM1(2PAM).

BICM2(2PAM-MPAM) and BICM3(2PAM-MPAM) will be the focus of our

BER analysis. To use such schemes, we are going to design a binary code C whose

codebook induce the following bit probabilities (but not restrict to)6.1

PB1(0) = 0.5, PB2(0) = 1, · · · , PBm(0) = 1

PB1(1) = 0.5, PB2(1) = 0, · · · , PBm(1) = 0. (6.8)

In general, there is no good code that has asymmetric bit probabilities as in

Eq. (6.8). However, since the zeros and ones are uniformly distributed in the first

labeling positions and the rest of the labeling positions have fixed bits in Eq. (6.8),

we argue that using the transmitter of the system diagram shown in Figure 6.1

is equivalent to having a shaping code with the required bit probabilities in Eq.

(6.8). The required bit probabilities are achieved by using an Appender that

appends m − 1 zeros after each bit at the output of the interleaver. Note that

when we choose bit probabilities differently from Eq. (6.8), we also have to change

what we append accordingly.

6.1As we have pointed out that there are many ways to set the bit/symbol probabilities for theshaping code such that effectively 2PAM signaling is used for M -PAM reference constellationduring the modulation.

68

6.1 Preliminaries

DEC

ENC APPEND MOD

DEMDISCARD

m bmπ

π(bm)

x(m)

PY|X(y|x)

yΛ

π−1π−1(Ξb)m

cm

Ξb

Figure 6.1: System diagram of BICM2(2PAM-MPAM) and BICM3(2PAM-MPAM) transmission scheme with shaping code, whose bit probability isPB1(0) = 0.5, PB2(0) = 1, . . . , PBm(0) = 1. This shaping code is obtained byusing an Appender that append m − 1 ‘0’ after each bit at the output of theinterleaver. And at the receiver, the LLRs calculated for each 2nd until m-thbit positions are discarded as we are only interested in the ones for the 1st bitposition

At the receiver, LLRs are calculated and only the LLRs corresponding to the

first bit labeling positions are kept for decoding. When maximum metric decoding

is used, the performance of this decoder will not be compromised, because the

bits corresponding to the second until m-th labeling position are fixed in the

codebook (to ‘0· · ·0’ in this case). That is,

u = argmaxu

q(x(u),y) = argmaxu

N∏

k=1

q(xk(u), yk)

= argmaxu

N∏

k=1

m∏

j=1

qj(bj(xk(u)), yk)

= argmaxu

N∏

k=1

q1 (b1(xk(u)), yk) · q2(0, yk) · · · qm(0, yk). (6.9)

Thus, only the LLRs in the 1st labeling position will contribute to the metric

effectively,

u = argmaxu

N∏

k=1

q1(b1(xk(u)), yk). (6.10)

69

Chapter7/Chapter7Figs/BICMBERAnalysisDiagram1.eps


The transmitter in Figure 6.1 can actually be simplified by replacing the Appender

and modulator with 2PAM modulator. We choose to use the more complex setup

as it is a better representation of the usage of probabilistic shaping codes and

easier to generalize.

6.2 Error Probability of Maximal Metric De-

coding

We can upper-bound the average error probability of the BICM scheme shown in

Figure 6.1 using the standard union bound technique, yielding

Pe =1

|M|

|M|∑

u=1

Pe(u), (6.11)

where

Pe(u) ≤∑

u 6=u′

PEP(b(u′), b(u)), (6.12)

PEP(b(u′), b(u)) , Prq(µ(b(u′)),y

)> q(µ(b(u)),y

), (6.13)

and the decoding metric can be written as

q(µ(b(u)),y

)=

N∏

k=1

q(xk(u), y

)=

N∏

k=1

q1(bm(k−1)+1(u), yk

). (6.14)

The average error probability of message u (6.12) depends on u through the

transmitted bits, their positions in the symbol and the bit patterns. This depen-

dency can again be removed by introducing coset coding without compromising

the performance of the system. The output of the encoder will be fed into a

scrambler and scrambled by a sequence known at the receiver, and the output

of the de-interleaver will be de-scrambled, hence making the equivalent channel

between scrambler and de-scrambler symmetric. The error probability computed

this way gives an average over all possible scrambling sequences.

If the underlying binary code C is linear and the channel is symmetric, the

pairwise error probability depends on the transmitted codeword b(u) and the

competing codeword b(u′) only through their respective Hamming distance d

[91]. Due to (6.12)-(6.13), we may rewrite the average error probability similarly

70

6.2 Error Probability of Maximal Metric Decoding

to [32, 33, 64] as

Pe ≤∑

d

AdPEP(d), (6.15)

where Ad is the weight enumerator (i.e. the number of codewords of C with

Hamming weight d), and PEP(d) denotes the error probability of a pairwise error

event of two competing codewords of Hamming distance d. Viterbi and Omura

argue in [91] that the union bound in Eq. (6.15) accurately characterizes the

error probability in the region above the cutoff rate.

The bit error probability can then be simply bounded by

Pb ≤∑

d

A′dPEP(d), (6.16)

where

A′d ,

∑

i

i

nAi,d, (6.17)

where Ai,d is the input-output weight enumerator, i.e. the number of codewords

of C of Hamming weight d generated with an input message of Hamming weight

i. The transfer function technique is used in [91] to give an upper bound on Ai,d

for convolutional codes.

The Pairwise Error Probability (PEP) (6.13) can take the form

PEP(b(u′), b(u)) = PEP(d) = Pr Ξpw > 0 , (6.18)

where Ξpw is the pairwise score,

Ξpw =

N∑

k=1

Ξbk, (6.19)

and Ξbk is the bit score corresponding to the k-th symbol,

Ξbk = log

q1(bm(k−1)+1(u′), yk)

q1(bm(k−1)+1(u), yk). (6.20)

The codewords differ by d bits, only the bit indices i for which bi(u) 6= bi(u′) need

to be considered. Because the bits on 2 to (m − 1)-th labeling position of each

symbol are the same in the codebook by construction, the error bits can only be

on the first labeling positions, and come from d different symbols. The PEP can

71


−15 −10 −5 0 5 1010

−4

10−3

10−2

10−1

100

Ξb

Figure 6.2: Simulated density of Ξb for BICM2(2PAM-MPAM) with the setup inFigure 6.1. Gray mapping and the AWGN channel are used. We set snr = 0 dB.The solid line shows the simulated density and the dashed line plots the densityapproximated using the Gaussian approximation method.

then be written as

PEP(d, snr) = Pr

d∑

i=1

Ξbi > 0

. (6.21)

Due to the presence of the interleaver, we argue that Ξbi can be considered as

i.i.d.

6.3 Cumulant Transform

The cumulant transform is shown in [33, 64, 68] as a convenient tool for accurately

approximating the pairwise error probability with the saddlepoint approximation

[14, 49].

Let U be a random variable. Then for s ∈ C, the cumulant transform of U is

72

Chapter7/Chapter7Figs/BICM_BPSK8PAM_llr_0dB_GaussApprox.eps

6.3 Cumulant Transform

−40 −30 −20 −10 0 1010

−4

10−3

10−2

10−1

100

Ξb

Figure 6.3: Simulated density of Ξb for BICM2(2PAM-MPAM) with the setup inFigure 6.1. Gray mapping and the AWGN channel are used. We set snr = 5 dB.The solid line shows the simulated density and the dashed line plots the densityapproximated using the Gaussian approximation method.

defined as [14, 49]

κ(s) , logE[esU ] (6.22)

Note that one can recover the probability density function (pdf.) PU(u) of the

random variable by the inverse Fourier transform of κ(s) [10]. The cumulant

transform of the pairwise score Ξpw is then given by

κpw(s) = logE[esΞb

]d (6.23)

= dκb(s) (6.24)

where (6.23) is because Ξbi is i.i.d., and

κb(s) = logE

[E

[q1(B, Y )

s

q1(B, Y )s

∣∣∣∣B]]

(6.25)

73

Chapter7/Chapter7Figs/BICM_BPSK8PAM_llr_5dB_GaussApprox.eps


Table 6.1: PrΞb > 0 with computer simulated density and Gaussian approxi-mation density.

Density Function 0dB 2dB 5dB 10dB

Computer Simulated 0.07812 0.03748 0.00596 5.00× 10−6

Gaussian Approximation 0.07828 0.03756 0.00612 6.15× 10−6

where b = b⊕ 1 is the binary complement of b.

Figure 6.2 and Figure 6.3 show, for snr = 0dB and snr = 5dB respectively,

the computer-simulated density (solid line) of Ξb for the BICM2(2PAM-8PAM)

depicted in Figure 6.1, with a shaping code whose bit probabilities satisfy

PB1(0) = 0.5, PB2(0) = 1, PB3(0) = 1

PB1(1) = 0.5, PB2(1) = 0, PB3(1) = 0 (6.26)

The channel is AWGN and Gray mapping is used. We have also shown with the

dashed line the distribution of a Gaussian random variable of mean −4snreq and

variance 8snreq, where snreq = −κb(s) is the equivalent signal-to-noise ratio, and sis the saddlepoint that makes the first order derivative of the cumulant transform

κb(s) equal to zero, i.e.

κ′b(s) = 0. (6.27)

This Gaussian approximation is introduced heuristically in [32]. At first, it seems

that unlike in [33, 64], the Gaussian approximation does not track the shape of the

tail of the true distribution. However, when the tail probability of PrΞb > 0

is

calculated using the Gaussian approximation density and the computer simulated

density, we found that the tails are matched surprisingly well (see Table 6.1).

Gaussian Approximation [32]: The pairwise error probability can be approxi-

mated by

PEP(d, snr) ≃ Q(√

−2dκb(s))

(6.28)

The approximation in (6.28) corresponds as well to the zero-th order term in the

Lugannani-Rice formula [33, 61]

Chernoff Bound [33]: The Chernoff bound to the pairwise error probability is

PEP(d, snr) ≤ eκb(s) (6.29)

Unlike the Gaussian approximation, the Chernoff bound is a true bound and it is

known to correctly give the asymptotic exponential decay of the error probability

for large d and SNR. However, it is not always tight.

74

6.4 Saddlepoint Approximation (SPA)

6.4 Saddlepoint Approximation (SPA)

The saddlepoint approximation method is introduced in [33, 64] in order to ac-curately analyze the error rate of BICM. As shown in these works, saddlepoint

approximation works well in analyzing the pairwise error probability, approxi-

mating it by.

PEP(d, snr) =1√

2πdκ′′

b(s)sedκb(s)

1 +O(dκ

′′

b(s))−1

(6.30)

where s is the saddlepoint that makes the first order derivative of the cumulant

transform κb(s) equal to zero. κ′′

b(s) is the second derivative of κb(s). The termO(κ

′′

b(s))−1 decays as fast as the inverse of κ

′′

b(s), and it is a correction term which

is found to be negligible in practical calculations [33, 64]. Therefore, we drop the

O(·) term when we estimate the pairwise error probability using (6.30).

Reference [33] has shown that when qj(b, y) is proportional to PY |Bj(y|b) (i.e.

the classical BICM bit metric), we always have s = 12. However, for the metric

we consider here and other general metrics, s needs to be calculated numerically.

The saddlepoint approximation can be seen as an extension of the Chernoff boundby including a scaling coefficient in front of the exponential term.

6.5 Results: BICM Schemes with Convolutional

Codes and 8PAM Modulation

In this section, we show some examples to illustrate the accuracy of the bounds

and approximations discussed in the previous section for convolutional codes. Inparticular, we simulate the bit error probability of the four schemes of interest

in Section 6.1.1 using a low rate convolutional code over the AWGN channel.

The simulated BER is then compared with the three approximations and boundsintroduced.

6.5.1 BICM2(2PAM-8PAM)

Figure 6.4 and Figure 6.5 show the BER and its approximations as a function ofEb

N0for the schemes BICM2(2PAM-8PAM) and BICM1(8PAM) over the AWGN

channel with Gray mapping. An 4-state, rate 18convolutional code with generator

(7, 7, 5, 5, 5, 7, 7, 7) is used for all simulations. Note that for BICM2(2PAM-8PAM)and BICM1(2PAM), the overall coding rate is 1

8, whereas for BICM1(8PAM), the

overall coding rate is 38. Therefore, though BICM2(2PAM-8PAM) outperforms

75


0 3 6 9 12 15

10−5

10−1

10−3

10−7

10−9

Eb

N0

(dB)

BE

R

BICM1(8PAM)BICM2(2PAM-8PAM)Chernoff BoundSPAGA

Figure 6.4: Comparison of BER simulation results between BICM2(2PAM-8PAM) and BICM1(8PAM). Chernoff union bound, saddlepoint approximationand Gaussian approximation of the BER are also shown. All results are withGray mapping and AWGN channel with rate 1

8convolutional codes.

BICM1(8PAM) in terms of BER, we can not justify the comparison between

BICM2(2PAM-8PAM) and BICM1(8PAM) as fair. However, from Figure 6.5 we

can see that the performance of BICM2(2PAM-8PAM) and BICM1(2PAM) are

nearly the same. This is intuitively in line with our achievable rate result where

the Cbicm0 of BICM2 is as good as that of BICM1 in the low SNR regime. In fact,

both schemes achieve the first and second order optimality, so one would expect

that with the same good binary code, the BER performance of BICM2(2PAM-

8PAM) is as good as that of BICM1(2PAM). Moreover, the Gaussian approxi-

mation and sadddlepoint approximation of the BER tracks the simulated BER

very well at high SNR, which is also consistent with the result in [33, 64] and our

result in Table 6.1. The Chernoff bound gives the correct asymptotic exponential

decay.

From our numerical results in Figure 4.5, the value of Cbicm0 for BICM2 is still

76

Chapter7/Chapter7Figs/BICMprime_8PAM_AWGN_one_eighth.eps

6.5 Results: BICM Schemes with Convolutional Codes and 8PAMModulation

0 3 6 9

10−5

10−1

10−3

10−7

10−9

Eb

N0

(dB)

BE

R

BICM1(2PAM)BICM2(2PAM-8PAM)BPSK Closed-Form ApproximationChernoff BoundSPAGA

Figure 6.5: Comparison of BER simulation results between BICM2(2PAM-8PAM) and BICM1(2PAM). 2PAM closed-form approximation [65], Chernoffunion bound, saddlepoint approximation and Gaussian approximation of the BERare also shown. All results are with Gray mapping and AWGN channel with rate18convolutional codes.

as good as that of BICM1 at rate 12, and the optimal input distribution for this

rate is again one that picks 2PAM out of M-PAM for both schemes. Hence we

simulate the error probability again at rate 12in the following way to create a

fairer comparison. An 4-state, rate 12convolutional code with generator (5, 7) is

used for BICM2(2PAM-8PAM) and BICM1(2PAM), and an 4-state, rate 16code

with generator (7, 7, 7, 7, 5, 5) is used for BICM1(8PAM). The results are shown

in Figure 6.6.

This comparison is fairer in the sense that the overall coding rate is 12for

all schemes. Moreover, the complexity of the convolutional code only grows

exponentially with the constraint length of the convolutional code generator

[91]. Therefore, the rate 12and 1

6convolutional codes used in this simulation,

whose constraint length is 3, are of the same complexity. Figure 6.6 shows that

77

Chapter7/Chapter7Figs/BICMprime_BPSK_AWGN_one_eighth.eps


0 5 10 15

10−5

10−1

10−3

10−7

10−9

Eb

N0

(dB)

BE

R

BICM1(2PAM)BICM1(8PAM)BICM2(2PAM-8PAM)BPSK Exact BoundChernoff BoundSPAGA

Figure 6.6: Comparison of BER simulation results between BICM2(2PAM-8PAM) (rate 1

2convolutional code), BICM1(8PAM) (rate 1

6convolutional code)

and BICM1(2PAM) (rate 12convolutional code). 2PAM closed-form approxima-

tion [65], Chernoff union bound, saddlepoint approximation and Gaussian ap-proximation of the BER are also shown. All results are with Gray mapping andAWGN channel.

apart from observing the same relationship between BICM2(2PAM-8PAM) and

BICM1(2PAM) as in the last simulation (for the rate 18convolutional code), we

can also see that BICM2(2PAM-8PAM) indeed outperforms BICM1(8PAM) in

terms of BER.

The authors in [33] find that BICM preserves the properties of the underlying

binary code, and as a result of an asymptotic analysis of the cumulant transform,

it behaves like a binary modulation with distance d2X,min for large SNR, where

d2X,min , minx,x′

|x− x′|2 (6.31)

is the minimum squared Euclidean distance of the signal constellation X. This

suggests a way to study the BER gap between different BICM schemes at high

78

Chapter7/Chapter7Figs/BICMprime_8PAM_vs_BPSK_one_second.eps


0 5 11 15 19.14 2510

−15

10−10

10−5

Eb

N0

(dB)

SPA

BE

R

Figure 6.7: Saddlepoint approximation of the error probability for BICM2(2PAM-8PAM) (rate 1

2convolutional code) and BICM1(8PAM) (rate 1

6convolutional

code) with Gray mapping for the AWGN channel.

SNR. For large SNR, the pairwise error probability of BICM decays exponentially

with snr as [33]

PEP(d) ≃ κe−14snrd2

X,mind. (6.32)

Therefore, the BER is asymptotically dominated by the pairwise error probabilityof the minimum distance term PEP(dfree). For a rate 1

2convolutional code with

generator (5, 7), the minimum distance is 5, and for a rate 16convolutional code

with generator (7, 7, 5, 5, 5, 7, 7, 7), the minimum distance is 16. Thus, we havefor high SNR

• Scheme BICM2(2PAM-8PAM): dfree = 5, d2X,min = 4, Pb ≃ κe−5snr.

• Scheme BICM1(8PAM): dfree = 16, d2X,min =421, Pb ≃ κ′e−

1621

snr.

Hence when BER is plotted on a log-log scale, at large SNR, the gap between two

79

Chapter7/Chapter7Figs/BERasymptotic.eps


0 2 4 6 8 10

10−8

10−6

10−4

10−2

Eb

N0

(dB)

BE

R

7 7.02

10−6.46

10−6.45

7 7.02

10−5.32

10−5.315CUB

GA

Figure 6.8: Comparison between the Chernoff union bound (CUB) and Gaussianapproximation (GA) for BICM3(2PAM-8PAM). The AWGN channel and Graymapping are used. The Thick-solid line represents bounds for BICM3(2PAM-8PAM) BER with input (X1, P 1

B). The Dashed line represents the bounds forBICM3(2PAM-8PAM) BER with input (X2, P 2

B). The Dash-dotted line representsthe bounds for BICM3(2PAM-8PAM) BER with input (X3, P 3

B). The Dotted linerepresents the bounds for BICM3(2PAM-8PAM) BER with input (X4, P 4

B). TheSolid line with marker ’·’ represents the BER of BICM2(2PAM-8PAM).

schemes should be 10 log10(10516) ≈ 8.17 dB. It is difficult to simulate the exact BER

for large SNR. However, the saddlepoint approximation to the error probability

can be plotted easily at high SNR. Figure 6.7 shows the high SNR saddlepoint

approximation to the error probability for BICM2(2PAM-8PAM) (rate 12con-

volutional code with generator (5, 7)) and BICM1(8PAM) (rate 16convolutional

code with generator (7, 7, 7, 7, 5, 5)) with Gray mapping for AWGN channel. We

observe that when the error probability is on the order of 10−15, the SNR gap

between BICM2(2PAM-8PAM) and BICM1(8PAM) is roughly 8.14 dB, which is

very close to our analytical prediction of 8.17 dB.

80

Chapter7/Chapter7Figs/BICMprime_AWGN_oneeighth_scaled_GP_BU.eps


6.5.2 BICM3(2PAM-8PAM)

As we have pointed out earlier, there are many ways to set the bit/symbol proba-bilities such that 2PAM is effectively picked out of theM-PAM constellation, and

unlike BICM2, the GMI and the BER depend on the choice of input distributionfor BICM3.

Figure 6.8 shows the Chernoff union bound (CUB) and Gaussian approxi-mation (GA) of the BER as a function of Eb

N0for BICM3(2PAM-8PAM) over

the AWGN channel with Gray mapping. We observe that both the boundand the approximation for BER increase as the inputs change from (X1, P 1

B)

to (X4, P 4B), which verifies our previous observation in Section 6.1.1. The CUB

and GA for BICM2(2PAM-8PAM) are also shown in Figure 6.8 for compari-son. Note that the scheme BICM3(2PAM-8PAM) with different inputs (e.g.

(X1, P 1B), (X

2, P 2B) etc.) essentially uses 2PAM modulation at the transmitter

and 8PAM soft-demodulation at the decoder with reference constellations scaled

differently. This means that for the BICM3(2PAM-8PAM) scheme, if we letΩ = −7,−5,−3,−1, 1, 3, 5, 7, then the scheme with input (X2, P 2

B) (represented

by the dashed line in Figure 6.8) decodes using the reference constellation X2. Itcan be obtained by scaling each element of Ω by 1

5. Also, the scheme with input

(X3, P 3B) (represented by the dashed-dotted line in Figure 6.8) decodes using the

reference constellation X3. It can be obtained by scaling each element of Ω by 13.

Similarly, BICM2(2PAM-8PAM) decodes using reference constellation X, whichcan be obtained by scaling each element of Ω by 1√

5. We observe that 1√

5is in

between 13and 1

5, and correspondingly, the BER curve for BICM2(2PAM-8PAM)

is between the BER curves of BICM3(2PAM-8PAM) with inputs (X2, P 2B) and

(X3, P 3B).

Figure 6.9 shows the simulated BER and its saddlepoint approximation asa function of Eb

N0for BICM3(2PAM-8PAM) over the AWGN channel with Gray

mapping. An 4-state, rate 18convolutional code with generator (7, 7, 5, 5, 5, 7, 7, 7)

is used for the simulation. To avoid overcrowding the figure, we only plot theschemes where the outermost pair and innermost pair of 2PAM are used. Despite

the limited simulation precision, we can still observe that, in general, the BERof the scheme for which the outermost pair is chosen ((X1, P 1

B) as input) slightly

outperforms the scheme for which the innermost pair is chosen ((X4, P 4B) as input).

We also observe that the saddlepoint approximation estimates the BER quite well

at high SNR. Though it is not shown in this figure, the Gaussian approximation,

saddlepint approximation and Chernoff union bound give the correct asymptoticdecay of the BER, and the Gaussian approximation is nearly as good as the

saddlepoint approximation.

81


0 1 2 3 4 5 6 710

−6

10−5

10−4

10−3

10−2

10−1

Eb

N0

(dB)

BE

R

000, 100010, 110SPA with 000, 100SPA with 010, 110

Figure 6.9: Simulation results and saddlepoint approximation of BER forBICM3(2PAM-8PAM) transmission over the AWGN channel with Gray map-ping. A rate 1

8convolutional codes is used. To avoid overcrowding the figure,

we only show two cases here (when the outermost pair is chosen, and when theinnermost pair is chosen.)

82

Chapter7/Chapter7Figs/BICMprime_AWGN_oneeighth_scaled_SP_simu.eps



We have analyzed the error probability of the mismatched BICM shaping schemes

introduced in Chapter 4. In particular, we have analyzed the schemes defined inTable 4.4 where the conditional symbol probability and/or reference constellations

are mismatched between the transmitter and receiver. We have carefully designedthe shaping codes by introducing an appender/discarder pair. This allows us

to make the channel symmetric by using random coset codes even when theinputs are not used with uniform probability. Hence, we can still use the error

probability analysis introduced in [33, 64]. We have also presented a number ofbounds and approximations to the error probability based on a union bound

approach. Amongst these bounds and approximations, we have shown that thesaddlepoint approximation yields accurate results and is easy to compute.

We have also seen from a more practical point of view that the mismatchedBICM shaping schemes we have considered are optimal when low rate shaping

codes are used, and the gain in achievable rates can indeed translate into a gain in

the error probability when good codes are used. In particular, we have simulatedthe error probability of BICM1, BICM2 and BICM3 schemes with convolutional

codes and shown that BICM2 and BICM3 can achieve the benchmark error perfor-mance of a 2PAM coded modulation scheme even under strong mismatch between

the transmitter and receiver (the mismatch is effectively due to uniform 2PAMmodulation at the transmitter and uniform MPAM demodulation at the receiver).

We have also discussed how the BICM3 scheme, where only conditional symbolprobabilities are mismatched, is different from BICM2, where both conditional

symbol probabilities and reference constellations are mismatched.

83


84

Part II

Reliable Source Transmission

over MIMO Block-Fading

Channels

85

Chapter 7

Introduction to the Source

Transmission Problem

Communication is a process that allows one to convey information from a source

to a destination over a noisy medium. One of the most common mediums is

the wireless channel. It captures the attention of the public and research com-

munities because wireless communication systems are more flexible and easier to

deploy. However, the wireless channel also poses a severe challenge for reliable

high speed communications, since it is susceptible to noise, interference, fading

and other channel disturbances. These problems change over the distribution of

transmission in a unpredictable way because of the randomness of the movement

of the objects and the wireless environment. Fading is mainly due to the small-

scale effect of multi-path propagation and larger-scale effects such as path loss

via distance attenuation, user mobility and shadowing by obstacles.

In Part II of the thesis, we study data transmission over a very simple but in-

teresting type of fading channel, namely the block-fading channel. The duration

of the block-fading period is determined by the channel coherence time, and each

transmitted packet (also referred to as a codeword) spans only a finite and fixed

number of independent fading blocks. The channel fading gain remains constant

within a block-fading period. This model was first introduced in [76] to model

delay-limited transmission over channels subject to small temporal fading varia-

tions. Practical scenarios include OFDM and frequency hopping for low-mobility

wireless communications [9]. Under this setup, the channel is non-ergodic, and

the Shannon capacity of the channel is zero since at a given fixed rate, there

does not exist an encoder/decoder pair which can make the error probability ar-

bitrarily small [9, 76]. In particular, a communication outage occurs whenever

the instantaneous mutual information is less than the target data rate we wish

87

7. INTRODUCTION TO THE SOURCE TRANSMISSIONPROBLEM

to communicate at [9, 76]. As shown in [62], the information outage probability

is the natural fundamental limit of the channel.

The problem of reliable source transmission over the block-fading channel

has been studied with the end-to-end distortion metric. Inspired by the work of

Laneman et al. [56], the end-to-end expected distortion has been studied to char-

acterize the performance of continuous source transmission over outage-limited

multiple-antenna fading channels [15, 34, 35, 40, 56, 57]. These works consider

the SNR exponent of the end-to-end expected distortion (where the expectation

is also taken over the fading) as a performance metric for a number of joint

source-channel coding (JSCC) schemes. In particular, when the expected dis-

tortion is considered, these works illustrate the suboptimality of source-channel

separation. The end-to-end performance can be improved by using a number of

joint source-channel coding schemes based on hybrid analogue-digital transmis-

sion, multi-layered coding or broadcast coding.

The expected distortion is the natural performance metric for ergodic fad-

ing channels because when the channel is stationary and ergodic, there always

exists a long enough code that achieve the desired level of end-to-end expected

distortion with probability one, for any source sequence. However, for outage-

limited channels or non-ergodic fading channels, each source sequence is likely to

undergo a different channel realization, hence the instantaneous distortion (ex-

pectation over source distribution only) is a random variable depending on the

channel. Thus, its expectation over the fading coefficients fails to characterize

the true end-to-end performance of the wireless systems. This motivates us to

find a more relevant criterion for such channel.

Meanwhile, a more stringent criterion is that of the excess-distortion proba-

bility (c.d.f. of the distortion measure, e.g. see [22, 23, 36, 46, 52, 54, 69, 99]).

The error exponent of this quantity has been studied extensively in terms of the

error exponent in some memoryless channels but never under the non-ergodic

block-fading channel setup. In general, the excess-distortion probability contains

more information about the source, the distortion measure and the channel dy-

namics than the expected distortion. Mirroring the work of information outage

probability from channel coding problems for non-ergodic block-fading channels,

we introduce the excess distortion probability criterion to the end-to-end anal-

ysis of source transmission over non-ergodic block-fading channels in Part II of

this thesis. We aim to provide the tools for studying this more relevant figure of

merit, uncover the fundamental limit of source transmission systems under this

criterion, and compare the results with that of the expected distortion criterion.

The results show that the excess distortion probability is the right figure of merit

88

under the non-ergodic block-fading channel setup, because it has the correspond-ing coding theorem and converse theorem while the expected distortion does not.

Another simple consequence of our findings is that separation optimality holdsunder the excess distortion criterion.

Using a similar idea, the distortion versus outage criterion was considered in

[59, 60], where it was shown that separation holds under this metric. Our workhas several differences from distortion versus outage. First of all, the authors in

[59, 60] defined distortion vs. outage as the infimum of the set of outage-q achiev-able D, which is the set ofD satisfying certain information outage probability and

conditional excess distortion probability conditions. The distortion vs. outage isa term rather difficult to evaluate directly. On the contrary, the metrics we work

with can be evaluated directly and conveniently. Furthermore, with our frame-work, it is possible to study continuous and discrete channel inputs as well as more

complex layered source coding schemes. In addition, we work directly with con-tinuous memoryless sources with unbounded distortion measures, whereas Liang

proved their result for discrete memoryless systems using the method of types,and claimed that their results can be generalized to continuous alphabet using the

technique of [27, Chapter 7] which requires quantizing the source and channel.

89

7. INTRODUCTION TO THE SOURCE TRANSMISSIONPROBLEM

90

Chapter 8

System Model and Basic

Principles

This chapter reviews some background materials on the reliable source trans-

mission, more particularly, the rate distortion theory and joint source-channel

coding model. Rate distortion theory studies the following problem: Given a

source distribution and a distortion measure, what is the minimum expected dis-

tortion achievable at a particular rate? Or, equivalently, what is the minimum

rate description required to achieve a particular distortion? We will present the

definitions of the rate distortion problem, and review the source coding theorems

and source coding with a fidelity criterion results in Section 8.1.

Shannon [81] showed that one can reliably transmit a discrete memoryless

source (DMS) over a discrete memoryless channel (DMC) with long enough codes

and vanishing error probability as long as the achievable rate of the source is

strictly less than the capacity of the channel, the system is stationary and ergodic.

This implies that a two-stage separate source coding and channel coding process

is as good as any one stage process under certain source and channel conditions

[81]. However, when these conditions mentioned above are not met, the conditions

for reliable source transmission changes and it is studied in [36, 88, 90]. Apart

from transmissibility, the problem of joint source-channel coding (JSCC) is also

studied with a distortion measure, where particular JSCC transmission schemes

are proposed and the end-to-end distortions are computed for comparison [22, 39].

91

8. SYSTEM MODEL AND BASIC PRINCIPLES

8.1 End-to-End Systems: The Rate Distortion

Theory Paradigm

In this section, we will first define several basic concepts in rate distortion theory

and then review the problem of source coding and source coding with a fidelitycriterion. We will provide the general definition following the approach of [21,

27, 91].We denote the source alphabet by V and let V , VK be the K-Cartesian

product of the source alphabet V. The source outputs the vector v ∈ VK oflength K with probability PV(v). We limit our attention to memoryless sources

where

PV(v) =

K∏

k=1

PV (vk), v ∈ V. (8.1)

Definition 8.1.1 A distortion function (or distortion measure) is a mapping

d : V× V → R+, (8.2)

from the set of source reconstruction pairs into the set of non-negative real num-

bers.

d(v, v) models how well the reconstructed message v approximates the original

one v. The distortion measure of the source vector is then (with a slight abuseof notation)

d(v, v) =1

K

K∑

k=1

d(vk, vk). (8.3)

Examples of common distortion measures are

• Hamming distortion (probability of error):

d(v, v) = 11 v 6= v , (8.4)

which after taking the expectation over all possible v, results in a average

error probability distortion, i.e. E[d(V, V )] = PrX 6= X

.

• Squared error distortion:

d(v, v) = (x− x)2. (8.5)

which is commonly used for continuous sources.

92

8.1 End-to-End Systems: The Rate Distortion Theory Paradigm

SourceEncoder

NoiselessChannel

SourceDecoder

v u u v

Figure 8.1: Diagram of source coding.

A distortion measure is said to be bounded if the maximum value of the distortionis finite. In general, the distortion measure for any discrete source is bounded.

Definition 8.1.2 A (2KRs, K) rate distortion code (or source code) S consists

of an encoding function φs : VK → 1, . . . , |S| and a decoding function ψs :

1, . . . , |S| → VK . The distortion associated with this code is defined as

d(S) = EV[d(V, ψs(φs(V))] (8.6)

where the expectation is over the source vector random variable V. The source

coding rate is denoted by Rs =1Klog |S|8.1.

Definition 8.1.3 A rate distortion pair (Rs, D) is said to be achievable if there

exists a source code S with limK→∞EV[d(V, ψs(φs(V)))] ≤ D.

Definition 8.1.4 The rate distortion region for a source is the closure of the set

of achievable rate distortion pairs (Rs, D).

Definition 8.1.5 The rate distortion function Rs(D) is the infimum of rates Rs

such that (Rs, D) is in the rate distortion region of the source for a given distortion

D.

Definition 8.1.6 The distortion rate function D(Rs) is the infimum of all dis-

tortion values D such that (Rs, D) is in the rate distortion region of the source

for a given rate Rs.

D(Rs) is an inverse function of Rs(D). Clearly, Rs(D) and D(Rs) depend on thesource statistics and the distortion measure. The definitions of Rs(D) and D(Rs)

do not allow direct evaluation. However, for stationary ergodic sources with asingle-letter distortion measure, Rs(D) can be expressed as a minimum mutual

information function term, which is much easier to work with. We will presentthe results in Section 8.1.1.

8.1We now consider both channel coding and source coding in this part of the thesis, so wedenote the channel coding rate as Rc and source coding rate as Rs.

93

Chapter2/Chapter2Figs/sourcecoding.eps


8.1.1 The Source Coding Problem

The source coding theorem can be stated as follows for discrete memorylesssources [91].

Theorem 8.1.1 (Viterbi and Omura [91]) For any block length K and rate

Rs, there exists a source block code S with average distortion d(S) satisfying

d(S) = EV[d(V, ψs(φs(V))] ≤ D + d0e−KEs(Rs,D), (8.7)

where

Es(Rs, D) > 0 for Rs > minPV |V

:EV,V

[d(V,V )]≤D

I(V, V ). (8.8)

Es(Rs, D) is the source random coding function, we limit our attention to its

property in Eq. (8.8) in this part of the thesis. Its expression can be found in [91,Chapter 7]. In [91], Theorem 8.1.1 was generalized to sources with continuous

amplitude and unbounded distortion measures that satisfy

∫ ∞

−∞PV(v)d

2(v, 0)dv ≤ d20. (8.9)

Eq. (8.9) is satisfied in most cases of interest. The converse theorem is as follows.

Theorem 8.1.2 (Viterbi and Omura [91]) For any source encoder-decoder pair

it is impossible to achieve average distortion less than or equal to D if the rate

Rs satisfies Rs < minPV |V :E

V,V[d(V,V )]≤D I(V, V ).

According to Definition 8.1.5, Theorem 8.1.1 and Theorem 8.1.2 together is equiv-alent to the following theorem,

Theorem 8.1.3 (Cover and Thomas [21]) The rate distortion function for

an i.i.d. source V with distribution PV (V ) and distortion measures satisfies Eq.

(8.9) is given by

Rs(D) = minPV |V :E

V,V[d(V,V )]≤D

I(V, V ). (8.10)

Thus, Rs(D) is the minimum achievable rate at distortion D.

Theorem 8.1.1, 8.1.2 and 8.1.3 gives the operational definition of the rate distor-

tion function, and the closed-form expression of Rs(D) is known for a few types of

94

8.1 End-to-End Systems: The Rate Distortion Theory Paradigm

sources and distortion measures, such as binary sources with Hamming distortion

and Gaussian sources with squared error distortion. The closed-form expression

of Rs(D) allows one to find the close-form expression ofD(Rs), which is often used

to calculate the end-to-end expected distortion of a source transmission scheme.

We will discuss this in detail in Section 9.1.

8.1.2 Excess Distortion Probability and its Error Expo-

nent

In the previous section, we discussed the source coding problem with a focus on

the expected distortion d(S) of the code. It is also interesting to look at the same

problem from the excess distortion probability point of view.

Definition 8.1.7 The excess distortion probability of a code S for a given dis-

tortion threshold D > 0 is given by [22, 23, 36, 45, 46, 52, 54, 69, 99]

Pe(D, S) , Pr d(V, ψs(φs(V))) > D . (8.11)

In [22, 45, 46, 69, 99], the authors have derived an upper bound for the excess

distortion probability and its error exponent

Pe(D, S) ≤ e−KFr(Rs;D), (8.12)

where Fr(Rs;D) is the random coding error exponent of the source. Marton

has expressed the exponent in terms of the Kullback-Leibler divergence between

two probability distributions for finite-alphabet DMSs with arbitrary distortion

measures [69, Theorem 1]. The generalisation to the memoryless Gaussian source

(MGS) with squared error distortion measure was given by Ihara et al. [46]. The

authors in [98] later generalized the exponent to memoryless Laplacian sources

with magnitude error distortion measure, and the authors in [45] generalized it

again to stationary memoryless sources. In this thesis, We are only interested in

the following behavior of Fr(Rs;D) [46, 69]; the exact form of Fr(Rs;D) is beyond

the scope of this thesis.

Fr(Rs;D) > 0 whenever Rs > Rs(D). (8.13)

From Eqs. (8.13) and (8.12), we have that for Rs > Rs(D)

limK→∞

Pe(D, S) = 0. (8.14)

95


Also note that in the limit of K → ∞, for stationary ergodic sources and vector

distortion measures satisfy Eq. (8.3), we have

limK→∞

d(v, ψs(φs(v))) = EV [d(V, ψs(φs(V )))]. (8.15)

Eqs. (8.11), (8.14) and (8.15) give, for Rs > Rs(D),

limK→∞

Pe(D, S) = limK→∞

Pr d(V, ψs(φs(V))) > D (8.16)

= limK→∞

Pr EV [d(V, ψs(φs(V )))] > D (8.17)

= 0. (8.18)

It is easy to see that Eq. (8.18) is equivalent to the result in Theorem 8.1.1. Hence,

when ergodicity holds, studying the expected distortion leads to the same result

as studying the excess distortion probability atK → ∞. However, we will show in

Chapter 9 that once we introduce non-ergodicity (specifically, transmission over

the block fading channel), the conclusions of the source transmission problem for

these two perspectives can be different. The excess distortion probability is

appealing to use as the natural fundamental limit because it has the associated

coding theorem and converse.

8.2 Separation and Joint Source-Channel Cod-

ing

In the previous section we have only considered source coding in the transmission,

we now include channel coding in the transmission and give the relationship with

channel coding.

8.2.1 The Separation Scheme

A source-channel separation scheme for transmitting DMS over DMC is given

in Figure 8.2. The DMS emits a symbol every Ts seconds and that we have

a source encoder and decoder for a source block code, S, of block length K

and rate Rs nats per symbol of duration Ts seconds. For each KTs seconds, a

minimum distortion codeword in S = v1, . . . , v|S| is chosen to represent the

source sequence and codeword index u is sent over the channel. Hence, every

KTs seconds, one of |S| = eKRs messages is sent over the channel. We assume

the memoryless channel is used once every Tc seconds and has a channel capacity

96

8.2 Separation and Joint Source-Channel Coding

DMSV, Ts

v

VK

Search fora codeword

vu ∈ S

Source encoder

u ∈ 1, 2, . . . , |S|

Rs bits every Ts seconds

Choose acodewordxu ∈ C

Channel encoder

xu

XN Rc bits everyTc seconds

DMCTc, CPr E = Pr u 6= u

y

YN

Search fora codewordxu ∈ C

Channel decoder

u ∈ 1, 2, . . . , |S|Choose acodewordvu ∈ B

Source decoder

vu

VKUSER

Figure 8.2: Source-channel separate coding system diagram.

of C nats per channel use. The channel encoder and decoder use a channel block

code, C, of block length N and rate Rc where the bandwidth ratio is defined as

b ,N

K=Rc

Rs

=Ts

Tc. (8.19)

In [91], the achievability and converse theorem on source-channel coding have

been introduced.

Theorem 8.2.1 (Achievability [91]) For a combined source and channel cod-

ing scheme, there exists a source code S of rate Rs and block length K and a

channel code C of rate Rc and block length N satisfying

Rs

Rc

=K

N, (8.20)

such that the average distortion is bounded by

d(S,C) ≤ D + d0e−KEs(Rs,D) + d0e

−NEc(Rc), (8.21)

where

Es(Rs, D) > 0 and Ec(Rc) > 0 (8.22)

for Rs,Rc satisfying

Rs(D) < Rs <K

NRc. (8.23)

d0 is the maximal distortion between source symbol and reconstruction symbol

97

Chapter2/Chapter2Figs/aaa.eps


when the source is discrete and satisfies

∫ ∞

−∞PV (v)d

2(v, 0)dv ≤ d20 (8.24)

when the source has continuous amplitude.

Theorem 8.2.2 (Converse [91]) It is impossible to reproduce the source in

Theorem 8.2.1 with fidelity D at the receiving end of any memoryless channel

of capacity C < Rs(D) nats per source letter.

The achievability uses the separation scheme and when K → ∞ and N → ∞and Eq. (8.23) is satisfied, the average distortion satisfies

d(S,C) ≤ D. (8.25)

Moreover, the converse is true regardless of what type of encoders and decoders

are used. Thus, Theorem 8.2.1 and 8.2.2 together suggest that the separation can

achieve optimality as long as Rs(D) < C.

8.2.2 Joint Source-Channel Coding

Naturally, one might also think of jointly designing the source and channel codes.

Under the finite-length consideration, the authors in [38, 39] have studied the

tradeoff between source and channel coding for binary symmetric channel (BSC)

and Gaussian channel with asymptotically high source dimension. They have es-

tablished the optimal relationship between source vector dimension and channel

coding rate. Moreover, from the perspective of the interaction between source

and channel coding, we know that the process of source coding aims to reduce

the redundancy of the source while still maintaining the prescribed fidelity cri-

terion, whereas channel coding adds redundancy to the transmitted message to

increase the robustness of transmission. Obviously, seen from this perspective,

there is a tradeoff between source coding and channel coding. If the channel code

can exploit the redundancy of the source, then jointly designing the source and

channel codes may be even preferable. For example, one can optimally transmit

a Gaussian source over a Gaussian channel and a binary symmetric source over a

BSC without any coding [29, 48] when the source, the channel and the distortion

metric are matched.

Consider a memoryless source that outputs a vector v ∈ VK . A K-to-N joint

source-channel encoder is a mapping φ : RK → CN that maps blocks of K source

98

8.2 Separation and Joint Source-Channel Coding

Source-ChannelEncoder

ChannelPY|X

Source-ChannelDecoder

v x y vSourcePV

SourceReconstruction

Figure 8.3: Block diagram of joint source-channel coding scheme.

symbols v ∈ VK onto length N channel codewords x ∈ CN . The codewords are

sent through a memoryless channel characterized by the transition probabilityPY|X(y|x). At the receiver end, the corresponding joint source-channel decoder

ψ : CN → RK outputs a source estimate v = ψ(y) by processing the channeloutput y. The bandwidth ratio b of the code is defined as

b ,N

K. (8.26)

The block diagram of the joint source-channel coding scheme is shown in

Figure 8.3. The scheme can be characterized by the Markov chain

V → X → Y → V. (8.27)

From the data processing inequality, we have

I(X;Y) ≥ I(V; V) (8.28)

We will introduce the channel model for the MIMO block-fading channel in Sec-

tion 9.1.1, in which case, ergodicity fails to hold.

99

Chapter2/Chapter2Figs/jsc.eps


100

Chapter 9

Excess-Distortion Probability in

Block-Fading Channels

In this chapter we consider point-to-point transmission of a (possibly continuous)

source over a outage-limited MIMO block-fading channel. We first introduce

the channel model and previous works in Section 9.1. In Section 9.2, we define

the problem. We introduce the excess distortion probability metric into this

channel setup and define its SNR exponent. In Section 9.3, we use the information

inequality to derive a lower bound to the excess distortion probability for any joint

source-channel coding scheme, which coincides with a benchmark known as the

transmitter informed bound [15, 34, 35]. In Section 9.4, we proceed to analyze the

separation scheme and obtain an upper bound to the excess distortion probability.

A simple consequence, after optimizing the upper bound, is separation optimality

under the excess distortion metric. Finally, in Section 9.5, we study the excess

distortion probability of a very important transmission scheme, layered source

coding, and show that multiple layers are not desirable when excess distortion

probability is the figure of merit.

9.1 PreviousWork on Joint Source-Channel Cod-

ing over Non-Ergodic Channels

9.1.1 MIMO Block-Fading Channel

Despite the simplicity of block-fading channel model, it is pragmatically relevant

in wireless communication systems involving slow time-frequency hopping such as

GSM and EDGE or multicarrier modulation using orthogonal frequency division

101

9. EXCESS-DISTORTION PROBABILITY IN BLOCK-FADINGCHANNELS

.

.

.

.

.

.TX RX

Figure 9.1: Transmission and reception with multiple antennas. The channelcoefficients are described by the nr × nt matrix H .

multiplexing (OFDM). This is because that all these systems share a common

feature that they suffer from different fading conditions periodically. In particular,

we consider multiple-input multiple-output (MIMO) single user point to point

block-fading channels. We will review the channel model and its assumptions

and then present the information-theoretic limits in this Section. The use of

multiple antennas at the transceivers in wireless systems, popularly known as

MIMO technology (see Figure 9.1), has rapidly gained popularity since the work

by Telatar [85] and by Foschini et al. [26], due to its powerful performance

enhancing capabilities. A new type of channel coding scheme, known as space-

time coding, has been designed to exploit the degrees of freedom in both space

and time of the MIMO channel [4, 84, 97].

We consider a MIMO block-fading channel with B fading blocks, nt transmit

and nr receive antennas. Each block spans a length of L channel uses. The fading

coefficients are i.i.d. from block to block and from codeword to codeword, and

are constant inside a block. This channel model can be expressed as

Y k =

√snr

ntHkXk +Zk k = 1, . . . , B (9.1)

where Hk ∈ Cnr×nt,Xk ∈ C

nt×L,Y k ∈ Cnr×L, and Zk ∈ C

nr×L are the fad-

ing channel matrix, transmitted, received and noise signals corresponding to

block k, respectively. The entries of the channel matrices and noise matrices

are assumed to be i.i.d. NC(0, 1). We also assume that the codewords X =

[X1, . . . ,XB]T ∈ C

ntB×L are normalized in energy, i.e. 1ntBL

trace(E[XHX]

)= 1.

We define H = diag(H1, . . . ,HB), and we assume the channel is normalized so

102

Chapter3/Chapter3Figs/mimodia.eps

9.1 Previous Work on Joint Source-Channel Coding overNon-Ergodic Channels

ChannelEncoder

ChannelChannelDecoder

u X Y u

Figure 9.2: Space-time coding channel model. For convenience of presentationwe have omitted the dependence of x and x on u and u.

that 1ntnr

trace(E[HH

k Hk])= 1 for 1 ≤ k ≤ B. The average signal-to-noise ra-

tio (SNR) per receive antenna is snr and the average SNR per transmit antenna

is snr

nt. The fading matrix H is assumed to be known perfectly at the receiver,

whereas the transmitter only knows its statistics. Defining Y = [Y 1, . . . ,Y B]T

and Z = [Z1, . . . ,ZB]T , the channel (9.1) can be expressed more compactly as

Y =

√snr

ntHX +Z. (9.2)

Let each symbol of the channel input and output belong to the alphabets X,Y

respectively. A space-time channel coding scheme C over X is the collection of all

possible transmitted codeword X. We consider both random codes constructed

using Gaussian and discrete channel inputs (PSK, QAM), and for discrete channel

inputs we define m = log |X|. We let φc : 1, . . . , |C| → XntB×L denote the

channel encoding function and ψc : YnrB×L ×CBnr×Bnt → 1, . . . , |C| denote the

channel decoding function. The rate of the code is defined as

Rc =log |C|BL

. (9.3)

The average error probability is Pe(snr,C) = Pr u 6= u (see the block diagram

in Figure 9.2).

9.1.2 Fundamental Principles

The transmission of equiprobable messages over this channel model has been

extensively studied [9, 31, 53, 62, 73, 74, 76, 97]. In particular, when the ergodicity

assumption cannot be invoked, the channel capacity may be viewed as a random

variable of the instantaneous channel fading process. In fact, there is a non-

vanishing probability that the value of the actual transmitted rate, no matter

how small, exceeds the instantaneous mutual information. This probability is

termed the information outage probability. We first introduce the concept of the

103

Chapter3/Chapter3Figs/spacetimecoding.eps


instantaneous mutual information for the MIMO block-fading channel and then

give the definition of information outage probability. The vector instantaneous

mutual information for a given channel realization H can be written as

IH(X;Y) = BLIH(X;Y) = EX,Y

[log

PY|X,H(Y|X,H)

PY|H(Y|H)

∣∣∣∣H = H

](9.4)

IH(X;Y) =1

B

B∑

k=1

EXk ,Yk

[log

PYk|Xk,Hk(Yk|Xk,Hk)

PYk|Hk(Yk|Hk)

∣∣∣∣Hk = Hk

]. (9.5)

For simplicity, we let IH(snr) = IH(X;Y). In doing so, we have made explicit the

dependency on snr. The information outage probability is then defined as

Pout(snr, Rc) , Pr IH(snr) < Rc . (9.6)

The outage probability is a widely accepted figure of merit for slowly varying

fading channels and it has been shown [62] that the outage probability is the

natural fundamental limit of the block-fading channel: for a fixed B, there exists

a code C of rate Rc and length BL whose average error probability is such that

limL→∞

Pe(snr,C) = Pout(snr, Rc). (9.7)

Definition 9.1.1 (outage diversity) The Outage diversity or the SNR expo-

nent of information outage probability, δout, is defined as the asymptotic slope of

the information outage probability with snr.

δout(Rc) = − limsnr→∞

logPout(snr, Rc)

log snr. (9.8)

Following the method in [31], Nguyen et al [6, 73, 74] have studied the outage

diversity for MIMO block-fading channel and obtained the following theorem.

Theorem 9.1.1 (Nguyen et al. [73, 74]) Consider fixed-rate transmission with

rate Rc over the MIMO block-fading channel using Gaussian and discrete constel-

lation X of size 2m. For large snr, we have that

Pr IH(snr) < Rc = Pout(snr, Rc).= snr

−δout(Rc) (9.9)

Pr IH(snr) ≤ Rc .= snr

−δout(Rc), (9.10)

104


where δout(Rc) is bounded by δout(Rc) ≤ δout(Rc) ≤ δout(Rc), and

δout(Rc) = δout(Rc) = Bnrnt, (9.11)

for Gaussian channel inputs while for discrete channel inputs

δout(Rc) , nr

(1 +

⌊B

(nt −

Rc

m

)⌋)(9.12)

δout(Rc) , nr

⌈B

(nt −

Rc

m

)⌉. (9.13)

As we shall see later, the end-to-end distortion measures we study are relatedto the information outage probability and outage diversity, the results mentioned

above will be useful for the analysis.Central to joint source-channel coding is the idea of seeking a practical scheme

that can minimize the end-to-end distortion metric. As we have mentioned before,the optimal source-channel coding schemes largely depends on the tradeoff be-

tween source and channel coding, i.e. how much effort we should put into sourcecoding in order to obtain a high resolution version of the original source and

how much we should put into channel coding so that the transmission throughthe wireless environment is sufficiently robust. In the following sections, we will

illustrate the results on this topic from two perspectives, the expected distortion

and distortion vs. outage, and explain why they are not appropriate metrics forthe problem we consider.

9.1.3 Expected Distortion Perspective

The vast majority of the work in the literature study the expected distortion innon-ergodic block-fading channels (see e.g. [15, 34, 35, 56, 57, 71]). In particular,

the authors in these references study the distortion exponent which characterizethe expected distortion in the limit of high SNR.

Definition 9.1.2 The expected distortion SNR exponent of a joint source-channel

coding scheme is defined as [56]

δexp , − limsnr→∞

logEV [d(V, ψ(φ(V )))]

log snr(9.14)

The authors in [15, 35] first established an upper bound with the assumption

that the transmitter has perfect channel state information (CSI). They assume ascheme that, for any channel realization, chooses the channel coding rate equal

105


0 1 2 3 4 50

0.5

1

1.5

2

2.5

3

3.5

4

Bandwidth ratio, b

Dis

tort

ion

expon

ent,

δexp

Upper boundLS (infinite layers)HLS/HDA (1 layer)HLS (infinite layers)BS/LS (1 layer)BS (infinite layers)SEP

Figure 9.3: Distortion exponent as a function of bandwidth ratio for 4×1 MIMO,N = 1.

to or slightly below the MIMO channel mutual information for the same given

channel realization, and the source coding rate is decided by the channel coding

rate and bandwidth ratio through Eq. (8.19). It is clear that this scheme is point-

wise optimal for each H and therefore its expected distortion is a lower bound on

the minimum achievable expected distortion for any system without perfect CSI

at the transmitter. It yields an upper bound on the achievable SNR exponent (see

Figure 9.3). A number of joint source-channel coding schemes are then analyzed

to approach this upper bound. Since only separation and layered source coding

are considered in Chapter 9, we will outline the other joint source-channel coding

schemes briefly.

Hybrid digital-analog (HDA) [15, 71]: The idea of HDA makes use of the

results in [29] that uncoded transmission is optimal in certain cases, and combines

coded transmission (digital part) with uncoded transmission (analog part). Its

expected distortion SNR exponent is studied in [15] and it shows that the HDA

scheme is asymptotically optimal for small bandwidth ratios (see Figure 9.3).

106

Chapter3/Chapter3Figs/expcom.eps


L channel uses

t1LtlLt2L

Rc,1 Rc,2 Rc,l. . .

Figure 9.4: One block of the channel codewords for layered source coding scheme.

Layered source coding (LS) [34, 35]: This joint source-channel coding scheme

compresses the source in layers, and encodes each layer using successive refinement

codes. Suppose that the source encoder has l layers and the BL channel uses are

divided among these layers such that layer k is transmitted in tkBL channel

uses, where tk ≥ 0 is the fraction of channel uses per layer, and∑l

k=1 tk = 1.

Figure 9.4 shows a particular block of the channel codeword of the layered source

coding scheme. The values Rc,k for k = 1, . . . , l denote the channel coding rate

for the specific part of the codeword, and satisfy

Rc,1 ≤ Rc,2 ≤ · · · ≤ Rc,l. (9.15)

For each layer, the corresponding source coding rate is btkRc,k nats per source

sample. The scheme further assumes that tkL is large enough so that the mes-

sage is reliably transmitted as long as the mutual information is larger than the

coding rates. In case of an outage, the decoder simply outputs the mean of the

source which results in the highest possible distortion of 1 due to the unit vari-

ance assumption of the source. Also note that due to the successive refinement

nature of the source code, layers k+1, . . . , l are useless when any of the previous

layers is missing. This classical idea, also referred to as progressive coding, has

been used to some extent in image and video standards such as JPEG2000 and

MPEG-4. The authors in [35] have studied this scheme and showed that layered

source coding can improve the expected distortion performance and can be easily

implemented by concatenating a layered source encoder with a MIMO channel

encoder that time-shares among different code rates.

Hybrid digital-analog transmission with layered source (HLS) [35]: Since both

HDA and layered source coding schemes can improve the overall expected distor-

tion SNR exponent, Gunduz et al. put forward a combined scheme of HDA with

LS coding. They show that the introduction of analog transmission improves

the distortion exponent compared to LS with the same number of digital layers.

107

Chapter3/Chapter3Figs/block_layer.eps


However, the improvement becomes insignificant as the number of layers or the

bandwidth ratio increases (see Figure 9.3).

Broadcast strategy with layered source (BS) [34, 35, 82]: The broadcast ap-

proach is introduced in [82] for transmitting messages to multiple receivers, where

each receiving user’s channel corresponds to a different channel realization. For

this kind of channel, it has been shown that superposition coding performs better

than coding the source into layers successively in time [20]. Thus, the authors

in [35] consider superimposing multiple source layers with a broadcast strategy,

and show that for an infinite number of superposition layers, the BS scheme can

achieve the optimal expected distortion SNR exponent (see Figure 9.3).

As we have mentioned previously, for ergodic channels, studying the expected

distortion is equivalent to studying the excess distortion probability at K →∞. However, for non-ergodic block fading channels, the expected distortion for

each codeword is a random variable of the channel realization H , and there do

not exists code with bounded expected distortion (or vanishing excess distortion

probability as in Eq. (8.18)) unless B → ∞ (ergodic channel). Therefore, the

mean of d(v, v) over the source distribution and fading (expected distortion) gives

limited information on how well the scheme performs compared to a metric that

studies its distribution directly (excess distortion probability). This is the main

motivation for the Part II of this thesis.

9.1.4 Distortion vs. Outage

A more general variety of end-to-end distortion metrics for non-ergodic composite

channels are studied in [59, 60], where the channel is a collection of component

channels WF : F ∈ F parameterized by F , and the random variable F is chosen

according to some some distribution at the beginning of the transmission and then

held fixed. In particular, they have studied distortion vs. outage as an end-to-end

distortion metric.

Definition 9.1.3 A distortion level D is outage-q achievable if

limL→∞

Pout(snr, Rc) ≤ q, (9.16)

and

limL→∞

Pr(V, V) : d(V, V) > D|no outage

= 0. (9.17)

Definition 9.1.4 The distortion vs. outage Dq is the infimum over all outage-q

achievable distortions.

108

9.2 Problem Formulation

This is the first time that distortion has been studied with the concept of outage

probability for non-ergodic channels. They have also noticed the limit of the

expected distortion metric under a non-ergodic channel setup. They show sep-

aration optimality under the distortion vs. outage metric for both lossless and

lossy transmissions of ergodic sources over non-ergodic channels, while separation

is clearly suboptimal under the expected distortion metric.

However, there are several issues with this metric. First of all, distortion vs.

outage defined as an infimum of all outage-q achievable distortions is a quantity

which is rather difficult to work with. The authors in [59, 60] focus on the optimal-

ity of separation under the distortion vs. outage metric, but it is also important

to provide a more appropriate framework for evaluating different source transmis-

sion systems over non-ergodic block-fading channels. Secondly, they prove their

result using the method of types for discrete memoryless systems, and claim that

to generalize their result to continuous-amplitude sources and channels, one just

need to use the technique in [27, Chapter 7]. However, this technique requires the

quantization of the source and channel. In the next chapter, we will present our

work in excess distortion probability in the block-fading channel, which overcomes

the aforementioned problems.

9.2 Problem Formulation

The performances of certain systems vary according to the metrics being used

to evaluate them. To better characterize the performance of a system, we need

to use an appropriate performance metric. From our previous analysis, the ex-

pected distortion fails to be the appropriate figure of merit for non-ergodic chan-

nels. Mirroring the idea of studying information outage probability instead of

channel capacity for non-ergodic block-fading channels, we introduce the excess

distortion probability as the relevant figure of merit for the end-to-end distortion

performance of non-ergodic block-fading channels. We will focus on continuous

sources and AWGN channel with Rayleigh fading.

The performance of the system is evaluated using a distortion measure d :

vK × vK → R+. The distortion between the transmitted source vector v and its

estimate at the decoder v is d(v, v = ψ(φ(v),H , snr)). The distortion measure

satisfies

d(v, v) =1

K

K∑

k=1

d(vk, vk). (9.18)

109


For K → ∞, fixed H and ergodic sources we have

limK→∞

d(v, v = ψ(φ(v),H , snr)) = EV [d(V, V )|H ], almost surely. (9.19)

Finally, we recall the definition of excess distortion probability introduced in

the previous chapter,

Pe(D, snr) , Prd(V, V) > D

. (9.20)

We are also interested in the high SNR behavior of Pe(D, snr) and we define the

SNR exponent of excess distortion probability as

δe = − limsnr→∞

logPe(D, snr)

log snr. (9.21)

In the following sections of the chapter, we will study the excess distortion

probability and its SNR exponent for the problem of reliable source transmission

over MIMO block-fading channel. In particular, we will find the system limits

for general joint source-channel codes and study how conventional schemes such

as separation and layered source coding perform under this new figure of merit

in non-ergodic channels. We will also make comparisons of our results from an

excess distortion perspective with that from the expected distortion perspective,

as well as that from the distortion vs. outage perspective.

9.3 Lower Bound to Excess Distortion Proba-

bility

In this section, we show the lower bound to the excess distortion probability.

For a fixed channel realization H , mimicking standard steps we have that for

memoryless sources, every joint source-channel rate-distortion code must satisfy

[21, 91]

Rs(D) = minPV |V

:E[d(V,V )]≤D

I(V ; V ). (9.22)

From Eq. (9.22), we have

I(V ; V ) ≥ Rs(D) (9.23)

I(V; V) ≥ KRs(D), (9.24)

110

9.3 Lower Bound to Excess Distortion Probability

where Eq. (9.24) is due to the memoryless nature of the source. As introduced

in Section 8.2.2, we have the Markov Chain

V → X → Y → V. (9.25)

Hence, by the data processing inequality, we have

I(V; V) ≤ IH(X;Y) (9.26)

≤ BLIH(snr) (9.27)

where

IH(X;Y ) = EX,Y

[log

PY|X,H(Y|X,H)

PY|H(Y|H)

∣∣∣∣H = H

](9.28)

IH(snr) =1

B

B∑

k=1

EXk,Yk

[log

PYk|Xk,Hk(Yk|Xk,Hk)

PYk|Hk(Yk|Hk)

∣∣∣∣Hk = Hk

]. (9.29)

denote the vector and per-symbol mutual information for a fixed channel realiza-

tion H respectively. Therefore, for K → ∞, every joint source-channel code that

meets the target distortion condition EV [d(V, V )] ≤ D (equivalently d(v, v) ≤ D)

must satisfy

BLIH(snr) ≥ KRs(D). (9.30)

This implies that the code will only be able to meet the target distortion crite-

rion for fading realizations which meet Eq. (9.30). When the condition (9.30)

is violated, the code will exceed the target distortion. This suggests that for

sufficiently long codes the excess distortion probability satisfies

limL→∞

Pe(D, snr) = Prd(V, V) > D

(9.31)

≥ Pr

IH(snr) <

Rs(D)

b

. (9.32)

Remark 9.3.1 Though the above results are derived for continuous sources and

the AWGN channel with Rayleigh fading, it is also true for general memoryless

sources and channels.

Remark 9.3.2 We could have also used lower bounds on the excess distortion

probability of the best code to obtain the result in Eq. (9.32). In particular, the

lower bound for any joint source-channel rate distortion codes [22, 99] states that

111


the excess distortion probability of the best code is lower-bounded as

Pe(D, snr) = E[Pe(D, snr|H)] ≥ E

[e−BLminRc [ 1bFr(Rcb;D)+Esp(Rc,H)]

∣∣∣H], (9.33)

where Fr(Rcb;D) is the random coding error exponent of the source [22, 45, 46,

69, 99], and Esp(Rc,H) is the sphere-packing exponent [27]. For the purposes of

this thesis, the only relevant property of this lower bound is that the exponent

minRc

[1bFr(Rcb;D) + Esp(Rc,H)

]is positive whenever Rs(D)

b< IH(Rc) and zero

otherwise. Therefore, for large L,

Pe(D, snr) ≥ E

[11

IH(snr) <

Rs(D)

b

]= Pout

(snr, Rc =

Rs(D)

b

). (9.34)

Remark 9.3.3 The above lower bound coincides with the so-called transmitter

informed lower bound in the literature [15, 34, 35]. The transmitter informed

lower bound is an achievability scheme that assumes that the transmitter has

access to the channel realization. Under these circumstances, the transmitter

may adapt the rate of the joint source-channel rate distortion code as a function

of the channel realization. That is, it can choose the channel coding rate Rc(H)

equal to or slightly smaller than the instantaneous mutual information of the

channel. The source coding rate satisfies Rs(H) = Rc(H)b. By doing this,

the channel is never in outage and for sufficiently long codes, the instantaneous

distortion is given by

D(Rs = bIH(snr)), (9.35)

where D(Rs) is the distortion rate function of the source. Therefore, the proba-

bility that this distortion exceeds a target threshold is

Pr D(Rs = bIH(snr)) > D = Pr

IH(snr) <

Rs(D)

b

. (9.36)

Eq. (9.36) is due to the fact that D(Rs) is the inverse function of Rs(D) and

Rs(D) is a monotonic decreasing function of D. This shows that the transmitter

informed lower bound has a matching converse and is not only a benchmark

achievability scheme.

112

9.3 Lower Bound to Excess Distortion Probability

9.3.1 SNR Exponent

We first note that

Pr

IH(snr) <

Rs(D)

b

(9.37)

is exactly the information outage probability for channel coding [9, 31, 53, 62, 74,76, 97] evaluated at rate Rs(D)

b. Therefore, since Rs(D)

bdoes not depend on snr,

the exponents of the information outage probability for channel coding problemsare also those of this problem, replacing Rc by

Rs(D)b

. More precisely, we have the

following

Theorem 9.3.1 The SNR exponent for the excess distortion probability of any

joint source-channel coding scheme is upper-bounded by

δe = ntnrB, (9.38)

for Gaussian channel inputs, and by the Singleton bound

δe = nr

(1 +

⌊B

(nt −

Rs(D)

bm

)⌋)(9.39)

for discrete input constellations with size of 2m.

In Figure 9.5, we plot the excess distortion probability SNR exponent upper

bound with both Gaussian random codes and discrete inputs. We observe thatfull diversity (Bnrnt) is achieved for all inputs when the bandwidth ratio b is

large. For discrete inputs, schemes with larger target distortion D have larger

SNR exponent over all range of bandwidth ratio. In Figure 9.6 we compare δe fordifferent discrete inputs, and we observe that a larger constellation size results in

a larger support with full diversity. Note that the Singleton bound is valid whenRc ≤ m, and we can therefore obtain a threshold on b, which is bt = −Rs(D)

m. For

b ≤ bt, δe = 0.

Remark 9.3.4 (Comparison with Expected Distortion) Most previous works

on the expected distortion of various joint source-channel coding schemes over

block-fading channels have focused on the SNR exponent. The SNR exponent

of expected distortion for the transmitter informed lower bound with Gaussian

channel inputs and a Gaussian source is given by [15]

δtxexp = B

min(nr,nt)∑

k=1

min

2b

B, 2k − 1 + |nt − nr|

. (9.40)

113


0 0.5 1 1.5 2 2.5 30

5

10

15

20

25

30

35

Bandwidth Ratio, b

δ e

Gaussian Channel Inputs

D = 0.6, QPSK

D = 0.05, QPSK

Figure 9.5: Excess distortion probability SNR exponents upper bound δe in a4×4 MIMO block-fading channel with B = 2. Different channel inputs andtarget distortion D are considered.

Note that for discrete channel inputs we always have 0 ≤ IH(snr) ≤ m, hence for

a zero mean variance σ2 Gaussian source whose distortion rate function is of the

form

D(Rs) = σ2e−2bRc , (9.41)

we have for any joint source-channel coding scheme with a discrete channel input

that

σ2e−2bm ≤ EV [d(V, V )] ≤ σ2, (9.42)

where the term on the LHS, σ2e−2bm, is derived from transmitter informed scheme

and it does not depend on snr. Therefore, according to the definition of the

expected distortion SNR exponent, we have that for discrete channel inputs, the

expected distortion SNR exponent of any joint source-channel coding schemes

with discrete channel inputs satisfies

δtxexp , − limsnr→∞

logEV [d(V, V )]

log snr= 0. (9.43)

114

Chapter3/Chapter3Figs/exponentGaussiandiscrete.eps

9.4 Upper bound to the Excess Distortion Probability for Separation

0 0.5 1 1.5 2 2.5 30

5

10

15

20

25

30

35

Bandwidth Ratio, b

δ e

16QAMQPSK

Figure 9.6: Excess distortion probability SNR exponents upper bound δe in a4×4 MIMO block-fading channel with B = 2. Target distortion D = 0.05 areconsidered.

When the instantaneous distortion is a random variable of the channel for non-

ergodic channels, the expected distortion only gives limited information about

the mean of distortion. Moreover, the expected distortion SNR exponent also

fails to provide insight to the performance of different transmission schemes with

discrete channel inputs (which is of more practical interest). In contrast, Theo-

rem 9.3.1 gives us the tool to analyze the transmission of sources using discrete

channel inputs under the excess distortion probability metric. It shows that

SNR exponent of excess distortion probability will be affected by the number

of transmitter/receiver antennas, the number of fading blocks, bandwidth ratio,

constellation size and rate distortion function of the source.

9.4 Upper bound to the Excess Distortion Prob-

ability for SeparationA source-channel coding separation scheme consists of the concatenation of a

fixed-length block source encoder φs : VK → VK , of rate Rs nats per source

sample, with a space-time channel encoder φc : VK → CntB×L of rate Rc nats

115

Chapter3/Chapter3Figs/exponentqpsk16qam.eps


per channel use. The source and space-time coding rates are related through the

bandwidth ratio as Rs = bRc. The excess distortion probability can be written

as

Pe(D, snr) = E

[Prd(V, V) > D|H

](9.44)

= E

[Prd(V, V) > D, no channel error|H

+ Pr

d(V, V) > D, channel error|H

]

(9.45)

≤ E

[Prd(V, V) > D|no channel error,H

+ Pr channel error|H

](9.46)

= E

[Prd(V, V) > D|no channel error

+ Pr channel error|H

], (9.47)

where the expectation is over the fading distribution. We note that the first

term is the excess distortion probability of the rate distortion code alone, while

the second term is the error probability of the channel code. By using classical

bounds, we obtain that for our case,

Pe(D, snr) ≤ E[e−KFr(Rs;D)

]+ E

[e−BLEr(Rc;H)

], (9.48)

where Fr(Rs;D) is the random coding error exponent of the source [22, 45, 46,

69, 99] and Er(Rc;H) is the random coding error exponent of the channel [27]

for channel realization H . Once again, since we are interested in the limiting

behavior for L → ∞ (K and L goes to ∞ with the ratio K = BLb). We know

that Fr(Rs;D) > 0 whenever Rs > Rs(D) and Er(Rc;H) > 0 for Rc < IH(snr).

Thus,

limL→∞

Pe(D, snr) ≤ 11 Rs ≤ Rs(D)+ E[11 Rc ≥ IH(snr)], (9.49)

where the expectation is over the probability is again over the fading distribution.

Assuming the source is encoded with rate Rs > Rs(D), we have

limL→∞

Pe(D, snr) ≤ Pr IH(snr) ≤ Rc (9.50)

for Rc >Rs(D)

b.

116


9.4.1 Separation Optimality

In Section 9.3, we derived the lower bound to the excess distortion probability

P lbe for sufficiently long joint source-channel coding schemes, given by

limL→∞

Pe(D, snr) ≥ P lbe = Pr

IH(snr) <

Rs(D)

b

. (9.51)

For the upper bound given in Eq. (9.50), we pick the channel coding rate Rc =Rs(D)

b+ ǫ for any ǫ > 0. We have

limL→∞

Pe(D, snr) ≤ P ube = Pr

IH(snr) ≤

Rs(D)

b+ ǫ

(9.52)

= Pr

IH(snr) ≤

Rs(D)

b

+ ǫ′ (9.53)

where ǫ′ is some positive number. For the AWGN channel with Rayleigh fading,

the information outage probability is a continuous function of channel coding

rate Rc. Hence, the infimum of the excess distortion probability upper bound

over ǫ′ > 0 for the separation scheme is given by

infǫ′>0

P ube = Pr

IH(snr) ≤

Rs(D)

b

. (9.54)

Eqs. (9.51) and (9.54) simply show the lower bound and upper bound are

matched, hence separation scheme can be made optimal by optimizing the chan-

nel coding rate or source coding rate when the excess distortion probability is the

figure of merit.

Remark 9.4.1 Obviously, joint source-channel codes also achieve the informa-

tion outage probability in the limit of large block length. In particular, the

existence of length BL block codes (φ, ψ) whose excess distortion probability

satisfies

Pe(D, snr) =E[Pr d(v, v) > D|H] (9.55)

≤E

[e−BLminRc [ 1bFr(bRc;D)+Er(Rc;H)]

∣∣∣H]. (9.56)

is proved in [22, Theorem 2],[99, Theorem 2]. They also show that the above

exponent is positive if and only if IH(snr) > Rs(D)b

. This shows the existence of a

117


joint source-channel code whose excess distortion probability satisfies

Pe(D, snr) ≤ Pr

IH(snr) ≤ Rs(D)

b

. (9.57)

9.4.2 SNR Exponent

The following result regarding the SNR exponent follows easily from [74].

Theorem 9.4.1 The SNR exponent achieved by a random coding separation

scheme with channel coding rate Rc is given by

δe = ntnrB (9.58)

for Gaussian inputs, and by

δe = nr

⌈B

(nt −

Rc

m

)⌉(9.59)

for discrete input constellations with 2m points.

In Figure 9.7, we illustrate the SNR exponents of the excess distortion probabilityand expected distortion for Gaussian channel inputs. We observe that the upper

bound to the excess distortion probability exponent for any joint source-channel

schemes matches the achievable lower bound to the excess distortion probabilityexponent with the separation scheme for Gaussian channel inputs. This shows

the optimality of the excess distortion exponent for separation. On the otherhand, the expected distortion exponent for separation δsepexp is largely suboptimal

[15].

Remark 9.4.2 There is a minor difference between the upper bound to the

separation scheme (achievability) and lower bound for any joint source-channel

schemes (converse): The converse excess distortion probability has a strict in-

equality inside the braces, while the achievability includes the case of equality.

This causes a discrepancy between the upper and lower bounds to the SNR ex-

ponent with discrete constellations and is a known pathology of random codes

[31, 73]. In terms of the exponent, this translates into random codes not being

able to achieve the points where the Singleton bound (9.39) is discontinuous.

There are explicit code constructions [13, 31, 53, 62] that can in practice achieve

these SNR exponents, and therefore this is a technical issue related to random

codes rather than a practical one. In any case, random codes can achieve the

optimal SNR exponent for all rates for which the Singleton bound is continuous.

118


0 1 2 3 4 5 6 7 80

5

10

15

20

25

30

35

Bandwidth Ratio, b

δ

δe, δe

δtxexp

δsepexp

Figure 9.7: SNR exponents of the excess distortion probability and expecteddistortion in a 4 × 4 MIMO block-fading channel with B = 2, Gaussian inputsand D = 0.05. The thick solid line corresponds to the excess distortion exponent,while the dashed and dash-dotted lines correspond to the expected distortionexponent.

Remark 9.4.3 Channel adaptive methods such as power control and ARQ have

the potential to significantly improve the performance of the wireless systems,

particularly the information outage probability and its SNR exponent [9, 19, 34,

72, 74]. From our analysis the optimal excess distortion probability for separation

is actually the information outage probability for a given Rc. Hence, all techniques

that can be employed to improve the information outage performance will improve

the excess distortion performance.

In Figure 9.8, we show the excess distortion probability of a 2 × 2 MIMO

block-fading channel with i.i.d. Rayleigh fading, N = 1, 2 and D = 0.05, for a

Gaussian source and Gaussian channel inputs. As predicted by our analysis, the

lower bound for excess distortion probability for any joint source-channel schemealways has a slope that equals ntnrB and is independent of D and b. This

figure also validates our results that the separation scheme can achieve an excess

119

Chapter3/Chapter3Figs/exponentGaussian.eps


−5 0 5 10 15

10−4

10−3

10−2

10−1

100

snr (dB)

Pe(D

,snr)

Lower boundRc = Rs(D)

bRc = 1.5Rc = 2

N = 2

N = 1

Figure 9.8: Lower bound to excess distortion probability and separation achiev-ability upper bound with zero mean unit variance Gaussian source and Gaussianchannel inputs, bandwidth ratio b = 2, target distortion D = 0.05 in a 2 × 2MIMO system. In this case Rs(D)

b≈ 1.080.

distortion probability SNR exponent of ntnrB for Gaussian channel inputs. When

the channel coding rate is chosen arbitrarily close to Rs(D)b

, the resulting excess

distortion probability upper bound for separation is arbitrarily close to the system

lower bound. We have also shown in Figure 9.9 the excess distortion probability

of a 2× 2 MIMO block-fading channel with B = 2, D = 0.06 for BPSK channel

inputs. We observe an exponent of 4 when Rc = Rs(D)b

, and 2 when Rc = 1.7,

as predicted by the Singleton bound. We remark that for high Rc, the excess

distortion probability upper bound moves away from the system lower bound

(not only in probability gain, but also in exponent) due to the Singleton bound.

9.4.3 Connection with the Work of Liang et al. [59, 60]

Liang et al. [59, 60] have studied the problem of joint source-channel coding in

the non-ergodic composite channel, and shown that separation is optimal under

120

Chapter3/Chapter3Figs/pesimgaussian.eps


−5 0 5 10 15 20

10−3

10−2

10−1

100

snr (dB)

Pe(D

,snr)

Rc = 1.7Rc = Rs(D)

b

Lower bound

Figure 9.9: Lower bound to excess distortion probability and separation achiev-ability upper bound with zero mean unit variance Gaussian source and BPSKchannel inputs, bandwidth ratio b = 1.5, number of blocks B = 2, target distor-tion D = 0.06 in a 2× 2 MIMO system. In this case Rs(D)

b≈ 1.353.

the distortion vs. outage metric. Our results are related to their results by thefollowing theorem

Theorem 9.4.2 For separation, the distortion vs. outage Dq is exactly the D

for which the minimum excess distortion probability satisfies

limL→∞

Pe(D, snr) = limL→∞

Pr

IH(snr) ≤

Rs(D)

b

= q. (9.60)

Proof: Recall that by definition, a distortion level D is outage-q achievableif limL→∞ Pout(snr, Rc) ≤ q and

limL→∞

Pr(V, V) : d(V, V) > D|no outage

= 0, (9.61)

the distortion vs. outage Dq is the infimum over all outage-q achievable distor-tions.

121

Chapter3/Chapter3Figs/pesimdiscrete.eps


We first show that any D satisfying Eq. (9.60) is outage-q achievable. For

the optimized separation scheme, the source (channel) coding rate is chosen ar-

bitrarily close to Rs(D) (Rs(D)b

). Therefore, according to the definition of outage

probability

limL→∞

Pout(snr, Rc) = limL→∞

Pr

IH(snr) ≤

Rs(D)

b

= q. (9.62)

Moreover, for this optimal separation scheme, when there is no outage and suffi-

ciently long codes are used, the channel inputs can always be received and decoded

reliably. Hence, the channel can be seen as a noiseless channel in this case and

according to Theorem 8.1.1 in Section 8.1.1 and Eq. (8.14) in Section 8.1.2, we

obtain Eq. (9.61). Therefore, any D satisfying Eq. (9.60) is outage-q achievable.

We now proceed to show that any D′ smaller than the D satisfies Eq. (9.60) is

not outage-q achievable. In fact, this is very easy to see. The outage probability

for the optimal separation scheme PrIH(snr) ≤ Rs(D)

b

is a decreasing function

of D. Any D′ smaller than the D satisfies Eq. (9.60) will result in a outage

probability larger than q which makes D′ not outage-q achievable. Hence, the

theorem is proved.

9.5 Layered Source Coding

Gunduz et al. [34, 35] studied a simple joint source-channel coding scheme which

is based on compression of the source in layers, where successive refinement codes

are used in each layer. They have shown that layered source coding improves the

expected distortion exponent significantly. However, we will show that layering

is not necessary when the excess distortion probability is considered as the figure

of merit.

Recall the setup of layered source coding in Section 9.1.3. Figure 9.4 shows a

particular block of codeword of the scheme. We assume that there are l layers in

total, each transmitting over the channel at rate Rc,k nats per channel use and

occupying a fraction tk of the L channel uses with∑l

k=1 tk = 1. The layered rates

satisfy

Rc,1 ≤ Rc,2 ≤ · · · ≤ Rc,l. (9.63)

For each layer, the corresponding source coding rate is btkRc,k nats per source

sample. We further assume that tkL is large enough so that the message is reliably

transmitted as long as the mutual information is larger than the coding rates, and

in case of an outage, the decoder simply outputs the mean of the source which

122

9.5 Layered Source Coding

results in the highest possible distortion of 1 due to the unit variance assumption

of source.

We define DLSk as the distortion achieved when the first k layers are successfully

received and decoded,

DLSk = D

(b

k∑

i=1

tiRc,i

)= e−2b

∑ki=1 tiRc,i . (9.64)

Eq. (9.63) implies that

DLS1 ≥ D

LS2 ≥ · · · ≥ D

LSl . (9.65)

We now find the instantaneous distortion d(v, v|H). For a given channel realiza-

tion H , we have that for some k∗ the instantaneous mutual information will lie

in the interval

Rc,k∗ < IH(snr) ≤ Rc,k∗+1, (9.66)

for a k∗ ∈ 0, 1, . . . , l with Rc,0 = 0 and Rc,l+1 = ∞. Hence, the instantaneous

distortion d(v, v|H) is exactly DLSk∗ . Therefore, for L → ∞, the corresponding

excess distortion probability is given by

limL→∞

Pe(D, snr) = Pr IH(snr) ≤ Rc,k∗+1 , (9.67)

where k∗ is the largest i = 0, 1, . . . , l such that DLSi is larger than D. To illustrate

this, Figure 9.10 shows a particular example when the target distortion D lies

between DLSk∗ and D

LSk∗+1. We observe that as long as IH(snr) ≤ Rc,k∗+1, the

channel can only support up to layer k∗ resulting in an instantaneous distortion of

d(v, v|H) = DLSk∗ , and the transmission will be in the situation of d(v, v|H) > D.

However, as soon as IH(snr) > Rc,k∗+1, layer k∗ + 1 can be decoded successfully.

In this case, the instantaneous distortion will not exceed the target level.

We now have the expression for the excess distortion probability of the layered

source coding scheme as Eq. (9.67). We will find the values of tk and Rc,k for

k = 1, 2, . . . , l leading to the minimum excess distortion probability. Following

Eq. (9.67) and the discussion above, we have

DLSk∗+1 ≤ D < D

LSk∗ . (9.68)

Therefore, from the definition of DLSk , we obtain

DLSk∗+1 = e−2b

∑k∗+1i=1 tiRc,i ≤ D, (9.69)

123


D

DLS2 D

LSk∗ D

LSk∗+1 D

LSlD

LS1

Rc,1 Rc,k∗ Rc,lRc,k∗+1Rc,2 · · ·

Figure 9.10: Example of layered source coding, with pre-assigned channel codingrate Rc,k, k = 1, 2, . . . , l. The horizontal axis is the rate axis and the verticalaxis denotes the distortion level. The dotted horizontal line denotes the targetdistortion, which lies in the interval [DLS

k∗+1,DLSk∗ ).

which gives

Rc,k∗+1 ≥Rs(D)

b−∑k∗

i=1 tiRc,i

tk∗+1(9.70)

≥Rs(D)

b−Rc,k∗

∑k∗

i=1 ti

tk∗+1(9.71)

≥Rs(D)

b−Rc,k∗

∑k∗

i=1 ti

1−∑k∗

i=1 ti(9.72)

=Rs(D)

b− Rc,k∗

1−∑k∗

i=1 ti+Rc,k∗ (9.73)

≥ Rs(D)

b(9.74)

where Eq. (9.71) is due to Eq. (9.63), and Eqs. (9.72) and (9.74) are due to

124

Chapter3/Chapter3Figs/distortionlayer.eps


∑l

k=1 tk = 1 and tk ≥ 0 for k = 1, . . . , l. The equalities of Eqs. (9.71)-(9.74) are

simultaneously achieved when only one layer is used, i.e. tk∗+1 = 1. The resulting

minimum excess distortion for layered source coding is therefore given by

Pe(D, snr) = Pr

IH(snr) ≤

Rs(D)

b

, (9.75)

which achieves the lower bound of excess distortion probability. This is to say

that to minimize the excess distortion probability we should not do any layering.


In this chapter, we revisited the problem of reliable source transmission over a

MIMO block-fading channel. From our analysis, there do not exist sufficiently

long codes with bounded expected distortion (or vanishing excess distortion prob-

ability) unless B → ∞ (ergodic channel). To study the source transmission prob-

lem under a non-ergodic setup using expected distortion (expectation also over

fading) fails to characterize the true performance of the system. This motivates

us to introduce the excess distortion probability, which studies the distribution

of the instantaneous distortion.

First, we have derived a lower bound to the excess distortion probability

for any joint source-channel coding scheme and the corresponding upper bound

to its exponent. We have shown this lower bound is equal to the information

outage probability evaluated at rate Rc =Rs(D)

b, and an upper bound to the SNR

exponent is given for Gaussian channel inputs and discrete constellations.

Then we have found the upper bound to the excess distortion probability that

is achievable for separation, and the corresponding lower bound to its exponent.

By choosing the channel coding rate arbitrarily close to Rs(D)b

, we can obtain

the tightest upper bound to the excess distortion probability. By doing this we

have shown that separation can achieve the system lower bound for the excess

distortion probability. In terms of the exponent, separation can be made optimal

for both Gaussian and discrete channel inputs.

Due to the importance of layered source coding as a joint source-channel

scheme, we have also analyzed a layered source coding system using excess dis-

tortion probability. We have shown that layering should not be considered under

the excess distortion setup and reconfirmed our result on separation optimality.

Our results on separation optimality are in line with the results of [59, 60].

However, compared to the distortion vs. outage metric, the excess distortion

probability metric is more direct and gives an expression for easy evaluation. In

125


fact, we have shown for separation that distortion vs. outage Dq is actually thevalue of target distortion D for which the excess distortion probability of optimal

separation scheme satisfies

Pe(D, snr) = Pr

IH(snr) <

Rs(D)

b

= q. (9.76)

Moreover, we have worked directly with continuous sources over the AWGN chan-

nel with an unbounded distortion measure whereas the result in [60] on discretememoryless systems does not readily generalize to the continuous case.

The conclusion from these results is that the excess distortion probability is

the relevant figure of merit for continuous source transmission over non-ergodicblock-fading channels, because there are achievability and converse associated

with excess distortion probability. Separation optimality is a simple consequenceof this observation. The studies of the expected distortion are a more natural

match to the ergodic fading dynamics.

126

Chapter 10

Conclusion and Future Work

We summarize in this chapter the main contributions of this dissertation and

describe some lines of future research.

10.1 Main Contribution

The main contributions of our work are summarized as follows:

10.1.1 Part I

• Extension of the BICM mismatched decoding model to a scenario where

the conditional symbol probability and reference constellations can be mis-

matched between the transmitter and receiver.

• Characterization of the the GMI for the new BICM schemes with proba-

bilistic shaping. The final results show that shaping can help us tremen-

dously in recovering the loss which made BICM suboptimal in terms the

GMI compared to CM and MLC. Moreover, the results hold even when we

do not know the correct conditional symbol probability and/or reference

constellations (the ones used at the transmitter) at the receiver.

• Formulation of a Blahut-Arimoto type algorithm for finding the maximizing

input distributions for the GMI of BICM1 transmission.

• Analytical and numerical evaluation of the error exponents and expurgated

error exponents for the i.i.d. ensemble and the cost-constrained i.i.d. en-

semble. The results for the ML decoder show that the cost constraints on

127

10. CONCLUSION AND FUTURE WORK

the input improve the error exponent for all rates in general. If we par-

ticularly look at rates near the mutual information, we find that the cost

constraints on the input improve the error exponent for moderate SNRs,

whereas the improvement is negligible for low and high SNRs. The results

for the mismatched decoder show that expurgating the bad codes from the

code ensemble also improves the error exponent for mismatched decoder.

Moreover, the cost constraints on the input improve the expurgated error

exponent, but the improvement vanishes as rate goes to zero.

• Analytical and numerical evaluation of the error probability of the mis-

matched BICM shaping schemes which are considered in Chapter 4. The

results reconfirm the achievable rates results and demonstrate the link be-

tween the error probability and achievable rates.

10.1.2 Part II

• Identification of the limit of the performance metric used for studying the

joint source-channel coding problem over block-fading channels.

• Extension of excess distortion probability analysis to MIMO block-fading

channels. Converse lower bound is derived for any joint source-channel

coding schemes, and an upper bound that is achievable for separation is

also derived. A simple consequence after optimizing the upper bound is the

separation optimality under excess distortion probability metric.

• Analytical evaluation and optimization of the excess distortion probability

for layered source coding.

• Comparison between our work and the work in [59, 60].

10.2 Future Work

In Part I, we have developed an algorithm for finding the maximizing input distri-

butions for the GMI of BICM1. The algorithm only guarantees local optimality,

and only works for BICM1 without power constraint. Finding the algorithm

that works for other BICM schemes with power constraint can be considered for

further research.

Our BICM low SNR optimality results in Part I are done analytically for

the GMI, but only qualitatively for the LM rate. Using the same method, one

128

10.2 Future Work

can investigate the LM rate for BICM in the low SNR analytically and obtainclosed-form expression for the coefficients c1 and c2.

BICM is known to preserve the properties of the underlying binary code, andis a perfect coding scheme to combat fading. In this dissertation, we have only

considered the AWGN channels without fading and the performance of BICM

shaping schemes over the fading channel shall be considered in future research.So far we have only considered shaping for the achievable rates. Compared to

achievable rates, shaping for the error exponents is a less studied topic in general.To what extent would shaping improve the error exponents and how it affects

different types of code ensembles are potential problems that shall be solved inthe future research.

We highlighted in Part I that theoretically shaping can close the gap betweenBICM and CM in terms of achievable rates. As the majority of the known good

codes are uniformly distributed among their alphabets, we have yet to designpractical shaping codes that can actually do this. This is highly relevant for

middle SNR range.In Part II, we have show the connections between excess distortion probability

and information outage probability for separation and layered source coding overblock-fading channels. It is well known that there exist a number of channel

adaptive methods such as power control and automatic repeat request (ARQ)

that can significantly reduce the information outage probability. Future researchmay consider examining these channel adaptive methods for excess distortion

probability.We have mostly resorted to asymptotic analysis in the limit of large block

length in Part II. A more practical and interesting area for future research mayinvolve some theoretical analysis in the finite block length regime. Classic rate

distortion theory is proved in the limit of large block length due to the applicationof the law of large numbers, ergodicity and the large deviation theorem etc.

Therefore, studying the reliable source transmission problem in the finite blocklength regime requires new techniques such as information-spectrum method [36].

The authors in [18, 36, 77, 88] have studied the source-channel separation problemin the finite block length regime, and gave new necessary and sufficient condition

for the source transmissibility and new bounds for joint-source channel coding.Many topics including excess distortion probability in the finite block length

regime can be further explored.

129

10. CONCLUSION AND FUTURE WORK

130

Appendix A

A.1 Ibicm0,s (s, P bicmX ) and Ibicm1,s

(s, a(·), P bicm

X

)Proper-

ties

A.1.1 Concavity of Ibicm0,s (s, P bicmX )

Fix the input distribution P bicmX , without loss of generality, we choose 0 ≤ s1 ≤ s2,

λ ∈ [0, 1] and s′ = λs1 + (1 − λ)s2. Then applying Holder’s inequality, we have

that

∑

x′

P bicmX (x′)q(x′, y)s

′ ≤(∑

x′

P bicmX (x′)q(x′, y)s1

)λ(∑

x′

P bicmX (x′)q(x′, y)s2

)1−λ

.

(A.1)

Substituting (A.1) into Ibicm0,s (s, P bicmX ) in (4.2), we have

Ibicm0,s (s′, P bicmX ) = E

[log

q(X, Y )s′

∑x′ P bicm

X (x′)q(x′, Y )s′

](A.2)

≥E

[log

(q(X, Y )s1∑

x′ P bicmX (x′)q(x′, Y )s1

)λ(q(X, Y )s2∑

x′ P bicmX (x′)q(x′, Y )s2

)1−λ]

(A.3)

=λIbicm0,s (s1, PbicmX ) + (1− λ)Ibicm0,s (s2, P

bicmX ) (A.4)

which proves the concavity of Ibicm0,s (s, P bicmX ) for s ≥ 0.

131

A.

A.2 Concavity of Ibicm1,s

(s, a(·), P bicm

X

)

For fixed input distribution P bicmX , instead of treating a(·) as a function on X

of particular form, we take it as M discrete parameters. Hence without loss of

generality, we choose 0 ≤ s1 ≤ s2, λ ∈ [0, 1], s′ = λs1 + (1 − λ)s2, a1(x) ∈R, a2(x) ∈ R for each x ∈ X and a′(x) = λa1(x) + (1 − λ)a2(x). Then apply

Holder’s inequality, we have that

∑

x′

P bicmX (x′)q(x′, y)s

′

ea′(x) ≤

(∑

x′

P bicmX (x′)q(x′, y)s1ea1(x)

)λ

(A.5)

×(∑

x′

P bicmX (x′)q(x′, y)s2ea2(x)

)1−λ

. (A.6)

Substitute (A.6) into Ibicm1,s

(s, a(·), P bicm

X

)in (4.6), similarly we have

Ibicm1,s

(s′, a′(·), P bicm

X

)≥ λIbicm1,s

(s1, a1(·), P bicm

X

)+ (1− λ)Ibicm1,s

(s2, a2(·), P bicm

X

)

(A.7)

which proves the joint concavity of Ibicm1,s

(s, a(·), P bicm

X

)for s ≥ 0 and any real

function a(·)

A.3 Shaping Algorithm for BICM1 GMI over

DMC

For BICM1, the supremum over s is achieved at s = 1, hence, with a slight abuse

of notation, the GMI can be rewritten as [44]

Ibicm0 (PB) =m∑

j=1

I(Bj; Y ) (A.8)

where PB denotes the input distribution of each bit, i.e., PB1(b), . . . , PBm(b). We

define function I0(PB,WBj |Y ) for input distribution PB and conditional probabil-

ity WBj |Y as

I0(PB,WBj |Y ) =

m∑

j=1

1∑

b=0

PBj(b)

(log2

1

PBj(b)

+∑

y

Pj(y|b) log2WBj |Y (b|y))

(A.9)

132

A.3 Shaping Algorithm for BICM1 GMI over DMC

And the GMI (A.8) can be written as

Ibicm0 (PB) =

m∑

j=1

1∑

b=0

PBj(b)

(log2

1

PBj(b)

+∑

y

Pj(y|b) log2 PBj |Y (b|y))

(A.10)

where PBj |Y (b|y) is the reverse channel transition probability of PY |Bj(y|b)

PBj |Y (b|y) =PY |Bj

(y|b)PBj(b)

∑1b′=0 PBj

(b′)PY |Bj(y|b′)

. (A.11)

We observe that I0(PB,WBj |Y ) = Ibicm0 (PB) when WBj |Y satisfies (A.11). As a

matter of fact, we can show that I0(PB,WBj |Y ) is upperbounded by Ibicm0 (PB).

Let Pj(y) =∑1

b=0 PBj(b)PY |Bj

(y|b) so that we have

PBj(b)PY |Bj

(y|b) = Pj(y)PBj |Y (b|y) (A.12)

Then

Ibicm0 (PB)− I0(PB,WBj |Y ) =

m∑

j=1

1∑

b=0

∑

y

PBj(b)PY |Bj

(y|b) log2PBj |y(b|y)WBj |Y (b|y)

=

m∑

j=1

1∑

b=0

∑

y

Pj(y)PBj|Y (b|y) log2PBj |y(b|y)WBj |Y (b|y)

=∑

j=1

∑

y

Pj(y)1∑

b=0

PBj |Y (b|y) log2PBj |y(b|y)WBj |Y (b|y)

≥ 0 (A.13)

The last inequality comes from that Kullback-Leibler distance between two prob-

ability is always positive. And the equality holds when WBj |Y is given following

Eq. (A.11). Therefore,

maxPB

Ibicm0 (PB) ≡ maxPB

maxWBj |Y

I0(PB,WBj |Y ) (A.14)

And to find Cbicm0 and the corresponding input distribution PB is equivalent to

solve the optimization problem in Eq. (A.14) subject to the constraints that PB

and WBj |Y being valid input distribution and conditional distribution.

133

A.

A.3.1 Bit-by-bit Algorithm

The BA type of bit by bit algorithm works by alternatively optimizing over PB

and WBj |Y for each bit position.

Set n = 0 and start with an arbitrary initial distribution P 0,0B that satisfies the

probability constraint.

1. Set j = 1

2. Find W n,j

Bj |Y that maximize Ibicm(Pn,j−1B ,W

n,j

Bj |Y ) using the RHS of (A.11).

3. Find P n,jB to maximize Ibicm(P

n,jB ,W

n,j

Bj |Y ) over PBj.

4. if j + 1 ≤ m, j = j + 1, go to step 2, otherwise go to step 5.

5. n = n+ 1 let P n,0B = P

n−1,mB ,W n,0

Bj |Y = Wn−1,mBj |Y and go to step 1.

The maximization in step 3 is performed using Lagrange multipliers method,

since for fixed WBj |Y and PB1(b), . . . , PBj−1(b), PBj+1

(b), . . . , PBm(b), the function

I0(PB,WBj |Y ) is concave in PBj(b), since

∂I0(PB,WBj |Y )

∂PBj(b)

=

(log2

1

PBj(b)

+∑

y

PY |Bj(y|b) log2WBj |Y (b|y)

)− 1

+∑

k 6=j

1∑

b′=0

PBk(b′)

(log2

1

PBk(b′)

(A.15)

+∑

y

∑

x∈Xj,k

b,b′

PY |X(y|x)∏

l 6=k,j

PBl(bl(x)) log2WBk |Y (b

′|y))

(A.16)

∂2I0(PB,WBj |Y )

∂PBj(b)2

= − 1

PBj(b)

≤ 0, (A.17)

where Xj,kb,b′ denotes the set x : bj(x) = b, bk(x) = b′. Let

Fj(PB,WBj |Y , λj) =

m∑

j=1

1∑

b=0

PBj(b)

(log2

1

PBj(b)

+∑

y


)

+ λj

1∑

b=0

PBj(b). (A.18)

where λj is the Lagrange multiplier for Fj(·). Differentiating Fj(·) with respect toPBj

(b) and set the derivative to zero and noting the constraint∑1

b=0 PBj(b) = 1

134

A.3 Shaping Algorithm for BICM1 GMI over DMC

we have optimal PBj(b) satisfies

PBj(b) =

2Sj(b)

∑1b′=0 2

Sj(b′)(A.19)

where

Sj(b) =∑

y


+∑

k 6=j

1∑

b′=0

PBk(b′)

log2

1

PBk(b′)

+∑

y

∑

x∈Xj,k

b,b′

PY |X(y|x)∏

l 6=k,j

PBl(bl(x)) log2WBk |Y (b

′|y)

(A.20)

A.3.2 Convergence

In this section, we will show the convergence of the algorithm. In step 2, and 3,

the value of the function I0(PB,WBj |Y ) is always increasing, we have

Ibicm(Pi,jB ,W

i,j

Bj |Y ) ≤ Ibicm(Pi,jB ,W

i,j+1Bj |Y ) ≤ Ibicm(P

i,j+1B ,W

i,j+1Bj |Y )

≤ · · · ≤ Ibicm(Pi,mB ,W

i,m

Bj |Y ) ≤ Ibicm(Pi+1,0B ,W

i+1,1Bj |Y )

≤ Ibicm(Pi+1,1B ,W

i+1,1Bj |Y ) ≤ · · · ≤ log2M = m (A.21)

Eq. (A.21) shows that the value of Ibicm(·, ·) will keep increase following the

algorithm, and it is upperbounded by a finite constant. Therefore the convergence

of the algorithm is guaranteed, i.e., there exist a constant Ibicm such that

limi→∞

Ibicm(Pi,jB ,W

i,j

Bj |Y ) = Ibicm. (A.22)

According to Reference [87, 94], if I0(PB,WBj |Y ) is concave in PB and WBj |Y ,

then Ibicm = Cbicm0 surely. However, in our case, the concavity of I0(PB,WBj |Y ) is

not established in general.

To conclude, the algorithm works well in the sense that it always increases

I0(PB,WBj |Y ) through its optimization process, it might saturate at local opti-

mum. But one can initiate the algorithm with uniform distributions together

with several other random initial input distributions to increase the possibility

of finding global optimum or at least improve GMI for sure over uniform distri-

bution. Bocherer et al. [12] have also designed an algorithm that guarantees the

135

A.

local optimality using the convex-concave procedure [95].

A.4 Proof of Expression for c2

Ibicm0 (snr) = cρ+ c1ρ2 + c′ρ3 + c2ρ

4 + o (ρ4), (A.23)

To determine the exact expression for c2 as snr → 0 for arbitrary constellation,

mapping and symbol probabilities used for encoding and decoding, we start di-

rectly with the expression from (4.38). The Third-order derivative of fj(ρ, x, b, s)

with respect to ρ is given as

f(3)j (ρ, x, b, s) =

sλ′j(ρ, x, z, b)βj(ρ, x, z, b) + sλj(ρ, x, z, b)β′j(ρ, x, z, b)− 2sαj(ρ, x, z, b)α

′j(ρ, x, z, b)

βj(ρ, x, z, b)2

−(sλj(ρ, x, z, b)βj(ρ, x, z, b)− sαj(ρ, x, z, b)

2) 2β ′j(ρ, x, z, b)

βj(ρ, x, z, b)3

+ 2

(1∑

b′=0

PBj(b′)sβj(ρ, x, z, b

′)s−1αj(ρ, x, z, b′)

)·

∑1b′=0 PBj

(b′)(s2 − s)βj(ρ, x, z, b′)s−2β ′

j(ρ, x, z, b′)αj(ρ, x, z, b

′)(∑1

b′=0 PBj(b′)βj(ρ, x, z, b)s

)2

+

∑1b′=0 PBj

(b′)sβj(ρ, x, z, b′)s−1α′

j(ρ, x, z, b)(∑1b′=0 PBj

(b′)βj(ρ, x, z, b)s)2

− 2

(∑1b′=0 PBj


′))2

(∑1b′=0 PBj

(b′)βj(ρ, x, z, b′)s)3

×(

1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

)

−∑1

b′=0 PBj(b′)(s2 − s)(s− 2)βj(ρ, x, z, b

′)s−3β ′j(ρ, x, z, b

′)αj(ρ, x, z, b′)2

∑1b′=0 PBj

(b′)βj(ρ, x, z, b′)s

−2∑1

b′=0 PBj(b′)(s2 − s)βj(ρ, x, z, b

′)s−2αj(ρ, x, z, b′)α′

j(ρ, x, z, b′)

∑1b′=0 PBj


136


+

(∑1b′=0 PBj

(b′)(s2 − s)βj(ρ, x, z, b′)s−2αj(ρ, x, z, b

′)2)

(∑1b′=0 PBj

(b′)βj(ρ, x, z, b′)s)2

×(

1∑

b′=0

sβj(ρ, x, z, b′)s−1β ′

j(ρ, x, z, b′)

)

−∑1

b′=0 PBj(b′)(s2 − s)βj(ρ, x, z, b

′)s−2β ′j(ρ, x, z, b

′)λj(ρ, x, z, b′)

∑1b′=0 PBj


−∑1

b′=0 PBj(b′)sβj(ρ, x, z, b

′)s−1λ′j(ρ, x, z, b′)

∑1b′=0 PBj


+

(∑1b′=0 PBJ

(b′)sβj(ρ, x, z, b′)s−1λj(ρ, x, z, b

′))

(∑1b′=0 PBj

(b′)βj(ρ, x, z, b′)s)2

×(

1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

). (A.24)

The third-order coefficient c′ is expressed as

c′ =

∑m

j=1EB,XjB ,Z

[f(3)j (ρ, x, b, s)

]

3!

∣∣∣∣ρ=0

= 0. (A.25)

because in (A.24), there are only terms, z3r , z3i , z

2rzi and zrz

2i , which after the

integration over Z, are all zero.

We further derive (A.24) to the forth-order. Differentiate the first two terms

in (A.24), we have

g1(ρ) =(βj(ρ, x, z, b)

2(sλ

′′

j (ρ, x, z, b)βj(ρ, x, z, b) + sλ′j(ρ, x, z, b)β′j(ρ, x, z, b)

+ sλ′j(ρ, x, z, b)β′j(ρ, x, z, b) + sλj(ρ, x, z, b)β

′′

j (ρ, x, z, b)− 2sα′j(ρ, x, z, b)

2

− 2sαj(ρ, x, z, b)α′′

j (ρ, x, z, b))−(sλ′j(ρ, x, z, b)βj(ρ, x, z, b) + sλj(ρ, x, z, b)β

′j(ρ, x, z, b)

− 2sαj(ρ, x, z, b)α′j(ρ, x, z, b)

)2βj(ρ, x, z, b)β

′j(ρ, x, z, b)

)/βj(ρ, x, z, b)

4

−(βj(ρ, x, z, b)

3(sλ′j(ρ, x, z, b)βj(ρ, x, z, b) + sλj(ρ, x, z, b)β

′j(ρ, x, z, b)

− 2sαj(ρ, x, z, b)α′j(ρ, x, z, b)

)2β ′

j(ρ, x, z, b) + βj(ρ, x, z, b)3(sλj(ρ, x, z, b)βj(ρ, x, z, b)

137

A.

− sαj(ρ, x, z, b)2)2β

′′

j (ρ, x, z, b)−(sλj(ρ, x, z, b)βj(ρ, x, z, b)− sαj(ρ, x, z, b)

2)

· 2β ′j(ρ, x, z, b)3βj(ρ, x, z, b)

2β ′j(ρ, x, z, b)

)/βj(ρ, x, z, b)

6. (A.26)

Substitute ρ = 0 into the above equation and take into consideration AppendixA.4.1, we have

g1(0) =

((e−|z|2

)2(s(e−|z|2

)2T 1j (x, z, b) + s

(e−|z|2

)2T 2j (x, z, b) + s

(e−|z|2

)2T 2j (x, z, b)

+ s(e−|z|2

)2T 3j (x, z, b) − 2s

(e−|z|2

)2T 3j (x, z, b) − 2s

(e−|z|2

)2T 2j (x, z, b)

)

−(2s(e−|z|2

)4T 2j (x, z, b) + 2s

(e−|z|2

)4T 4j (x, z, b) − 4s

(e−|z|2

)4T 4j (x, z, b)

))/(e−|z|2

)4

−((

e−|z|2)3(

2s(e−|z|2

)3T 2j (x, z, b) + 2s

(e−|z|2

)3T 4j (x, z, b) − 4s

(e−|z|2

)3T 4j (x, z, b)

)

+(e−|z|2

)3(2s(e−|z|2

)3T 3j (x, z, b) − 4s

(e−|z|2

)3T 4j (x, z, b)

)− 6s

(e−|z|2

)6T 4j (x, z, b)

+ 6s(e−|z|2

)6T 5j (x, z, b)

)/(e−|z|2

)6(A.27)

Differentiate the 3rd term in (A.24), we have

g2(ρ) =

(1∑

b′=0

PBj(b′)βj(ρ, x, z, b

′)s

)2

2

(1∑

b′=0


′)s−1α′j(ρ, x, z, b

′)

+1∑

b′=0

PBj(b′)(s2 − s)βj(ρ, x, z, b

′)s−2β ′j(ρ, x, z, b

′)αj(ρ, x, z, b′)

)2

+

(1∑

b′=0


′)s

)2

21∑

b′=0


′)s−1α′j(ρ, x, z, b

′)

(

1∑

b′=0

PBj(b′)(s2 − s)(s− 2)βj(ρ, x, z, b

′)s−3β ′j(ρ, x, z, b

′)2αj(ρ, x, z, b′)

+1∑

b′=0


′)s−2β′′

j (ρ, x, z, b′)αj(ρ, x, z, b

′)

138


+

1∑

b′=0


′)s−2β ′j(ρ, x, z, b

′)α′j(ρ, x, z, b

′)

+

1∑

b′=0


′)s−2β ′j(ρ, x, z, b

′)α′j(ρ, x, z, b

′)

+1∑

b′=0


′)s−1α′′

j (ρ, x, z, b′)

)

− 2

1∑

b′=0


′)s−1αj(ρ, x, z, b′)

(1∑

b′=0


′)s−1αj(ρ, x, z, b′)

+1∑

b′=0


′)s−2β ′j(ρ, x, z, b

′)αj(ρ, x, z, b′)

)

· 21∑

b′=0


′)s ·1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

/(1∑

b′=0


′)s

)4

(A.28)

Substitute ρ = 0 into the above equation and take into consideration Appendix

A.4.1, we have

g2(0) =

(2(e−|z|2

)4s ((s2 − s)2T 8

j (x, z) + 2s(s2 − s)T 9j (x, z) + s2T 12

j (x, z))

+ 2(e−|z|2

)4ss(s2 − s)

((s− 2)T 6

j (x, z) + 3T 7j (x, z)

)+ 2

(e−|z|2

)4ss2T 11

j (x, z)

− 4(e−|z|2

)4ss2((s2 − s)T 10

j (x, z) + sT 13j (x, z)

))/(e−|z|2

)4s(A.29)

Differentiate the 4th term in (A.24), we have

g3(ρ) =

(1∑

b′=0


′)s)3

2

1∑

b′=0


′)s−1αj(ρ, x, z, b′)

139

A.

(1∑

b′=0


′)s−2β′j(ρ, x, z, b

′)αj(ρ, x, z, b′)

+1∑

b′=0


′)s−1α′j(ρ, x, z, b

′)

)2

1∑

b′=0


′)s−1β′j(ρ, x, z, b

′)

+

(1∑

b′=0


′)s)3( 1∑

b′=0


′)s−1αj(ρ, x, z, b′)

)2

2

(1∑

b′=0

PBj(b′)

· (s2 − s)βj(ρ, x, z, b′)s−2β′

j(ρ, x, z, b′)2 +

1∑

b′=0


′)s−1β′′

j (ρ, x, z, b′)

)

− 3

(1∑

b′=0


′)s)2( 1∑

b′=0


′)s−1β′j(ρ, x, z, b

′)

)2

(1∑

b′=0


′)s−1αj(ρ, x, z, b′)

)2

/(1∑

b′=0


′)s)6

(A.30)


A.4.1, we have

g3(0) =4s2((s2 − s)T 10

j (x, z) + sT 13j (x, z)

)+ 2s2

((s2 − s)T 10

j (x, z) + sT 13j (x, z)

)

− 6s4T 14j (x, z) (A.31)

Differentiate the 5th and 6th terms in (A.24), we have

g4(ρ) =

(1∑

b′=0


′)s

)(s2 − s)

(1∑

b′=0

PBj(b′)(s− 2)(s− 3)βj(ρ, x, z, b

′)s−4

· β ′j(ρ, x, z, b

′)2αj(ρ, x, z, b′)2 +

1∑

b′=0

PBj(b′)(s− 2)βj(ρ, x, z, b

′)s−3β′′

j (ρ, x, z, b′)

· αj(ρ, x, z, b′)2 +

1∑

b′=0


′)s−3β ′j(ρ, x, z, b

′)2αj(ρ, x, z, b′)

· α′j(ρ, x, z, b

′) +1∑

b′=0


′)s−3β ′j(ρ, x, z, b

′)2αj(ρ, x, z, b′)

140


· α′j(ρ, x, z, b

′) +

1∑

b′=0


′)s−22αj(ρ, x, z, b′)α′

j(ρ, x, z, b′)

+

1∑

b′=0


′)s−22αj(ρ, x, z, b′)α′

j(ρ, x, z, b′)

)−

1∑

b′=0


′)s−1

· β ′j(ρ, x, z, b

′) · (s2 − s)

(1∑

b′=0


′)s−3β ′j(ρ, x, z, b

′)αj(ρ, x, z, b′)2

+

1∑

b′=0


′)s−22αj(ρ, x, z, b′)α′

j(ρ, x, z, b′)

)

/(1∑

b′=0


′)s

)2

(A.32)


A.4.1, we have

g4(0) = (s2 − s)

((s− 2)(s− 3)

1∑

b′=0

PBj(b′)T 5

j (x, z, b′) + (s− 2)

1∑

b′=0

PBj(b′)T 4

j (x, z, b′)

+ 2(s− 2)

1∑

b′=0

PBj(b′)T 4

j (x, z, b′) + 2(s− 2)

1∑

b′=0

PBj(b′)T 4

j (x, z, b′) + 2

1∑

b′=0

PBj(b′)

·T 3j (x, z, b

′) +1∑

b′=0

PBj(b′)2T 2

j (x, z, b′)− s(s− 2)T 6

j (x, z) + 2sT 7j (x, z)

)

(A.33)

Differentiate the 7th term in (A.24), we have

g5(ρ) =

(1∑

b′=0


′)s

)2( 1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

)

· (s2 − s)

(1∑

b′=0


′)s−3β ′j(ρ, x, z, b

′)αj(ρ, x, z, b′)2

141

A.

+

1∑

b′=0

PBj(b′)2βj(ρ, x, z, b

′)s−2αj(ρ, x, z, b′)α′

j(ρ, x, z, b′)

)

+

(1∑

b′=0


′)s

)2( 1∑

b′=0


′)s−2αj(ρ, x, z, b′)2

)

·1∑

b′=0

PBj(b′)

((s2 − s)βj(ρ, x, z, b

′)s−2β ′j(ρ, x, z, b

′)2 + sβj(ρ, x, z, b′)s−1β

′′

j (ρ, x, z, b′))

−(

1∑

b′=0


′)s−2αj(ρ, x, z, b′)2

)2

(1∑

b′=0


′)s

)

·(

1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

)2

/(1∑

b′=0


′)s

)4

(A.34)


A.4.1, we have

g5(0) =s(s2 − s)

((s− 2)T 6

j (x, z) + 2T 7j (x, z)

)+ (s2 − s)

((s2 − s)T 8

j (x, z)

+sT 9j (x, z)

)− 2(s2 − s)s2T 10

j (x, z) (A.35)

Differentiate the 8th and 9th terms in (A.24), we have

g6(ρ) =

(1∑

b′=0


′)s

)(1∑

b′=0

PBj(b′)(s2 − s)(s− 2)βj(ρ, x, z, b

′)s−3

· β ′j(ρ, x, z, b

′)2λj(ρ, x, z, b′) +

1∑

b′=0


′)s−2β′′

j (ρ, x, z, b′)

· λj(ρ, x, z, b′) +1∑

b′=0


′)s−2β ′j(ρ, x, z, b

′)λ′j(ρ, x, z, b′)

+1∑

b′=0


′)s−2β ′j(ρ, x, z, b

′)λ′j(ρ, x, z, b′) +

1∑

b′=0

PBj(b′)s

142


· βj(ρ, x, z, b′)s−1λ′′

j (ρ, x, z, b′)

)−

1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

·(

1∑

b′=0


′)s−2β ′j(ρ, x, z, b

′)λj(ρ, x, z, b′)

+1∑

b′=0


′)s−1λ′j(ρ, x, z, b′)

)

/(1∑

b′=0


′)s

)2

(A.36)


A.4.1, we have

g6(0) =(s2 − s)

((s− 2)

1∑

b′=0

PBj(b′)T 4

j (x, z, b′) +

1∑

b′=0

PBj(b′)T 3

j (x, z, b′)

+1∑

b′=0

PBj(b′)T 2

j (x, z, b′) +

1∑

b′=0

PBj(b′)T 2

j (x, z, b′)

)+ s

1∑

b′=0

PBj(b′)T 1

j (x, z, b′)

− s2((s− 1)T 7

j (x, z) + T 11j (x, z)

)(A.37)

Differentiate the 10th terms in (A.24), we have

g7(ρ) =

(1∑

b′=0


′)s

)2( 1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

)

·(

1∑

b′=0


′)s−2β ′j(ρ, x, z, b

′)λj(ρ, x, z, b′)

+1∑

b′=0


′)s−1λ′j(ρ, x, z, b′)

)+

(1∑

b′=0


′)s

)2

·(

1∑

b′=0


′)s−1λj(ρ, x, z, b′)

)·(

1∑

b′=0

PBj(b′)(s2 − s)

·βj(ρ, x, z, b′)s−2β ′j(ρ, x, z, b

′)2 +1∑

b′=0


′)s−1β′′

j (ρ, x, z, b′)

)

143

A.

− 2

(1∑

b′=0


′)s−1λj

)(1∑

b′=0


′)s−1β ′j(ρ, x, z, b

′)

)2

·(

1∑

b′=0


′)

)

/(1∑

b′=0


′)s

)4

(A.38)


A.4.1, we have

g7(0) =s((s2 − s)T 7

j (x, z) + sT 11j (x, z) + (s2 − s)T 9

j (x, z) + sT 12j (x, z)

)

− 2s2T 13j (x, z) (A.39)

The 4th-order derivative of fj(ρ, x, b, s) is given by

f(4)j (ρ, x, b, s) =g1(ρ) + g2(ρ)− g3(ρ)− g4(ρ) + g5(ρ)− g6(ρ) + g7(ρ) (A.40)

Note that fj(ρ, x, b, s) does depends on Z, the dependency is removed to facili-

tate the presentation, hence according to Appendix A.4.1, when f(4)j (ρ, x, b, s) is

averaged over Z and letting ρ = 0, we have

EZ

[f(4)j (0, x, b, s)

]= −3s

(16EQX|Bj

[(xr −Xr)(xi −Xi)]2 + 8EQX|Bj

[(xr −Xr)

2]2

+8EQX|Bj

[(xi −Xi)

2]2)

+ 12s(8EQX|Bj

[(xr −Xr)

2]EQX|Bj

[(xr −Xr)]

+ 8EQX|Bj

[(xi −Xi)

2]EQX|Bj

[(xi −Xi)] + 16EQX|Bj[(xr −Xr)(xi −Xi)]

·EQX|Bj[(xr −Xr)]EQX|Bj

[(xi −Xi)])− 6s

(12EQX|Bj

[(xr −Xr)]4

+12EQX|Bj[(xi −Xi)]

4 + 24EQX|Bj[(xr −Xr)]

2EQX|Bj

[(xi −Xi)]2)− (s2 − s)

· (s− 2)(s− 3)(12EPBj

[EQX|Bj

[(xr −Xr)]4]+ 12EPBj

[EQX|Bj

[(xi −Xi)]4]

+24EPBj

[EQX|Bj

[(xr −Xr)]2EQX|Bj

[(xi −Xi)]2])

+ 4(s3 − s2)(s− 2)(12

· EPBj

[EQX|Bj

[(xr −Xr)]3]EPBj

[EQX|Bj

[(xr −Xr)]]+ 12EPBj

[EQX|Bj

[(xi −Xi)]3]

· EPBj

[EQX|Bj

[(xi −Xi)]]+ 12EPBj

[EQX|Bj

[(xr −Xr)]2EQX|Bj

[(xi −Xi)]]

· EPBj

[EQX|Bj


[EQX|Bj

[(xi −Xi)]2EQX|Bj

[(xr −Xr)]]

· EPBj

[EQX|Bj

[(xi −Xi)]] )

+ 12(s3 − s2)(8EPBj

QX|Bj[(xr −Xr)]

144


· EPBj

[EQX|Bj

[(xr −Xr)

2]EQX|Bj

[(xr −Xr)]]+ 8EPBj

QX|Bj[(xi −Xi)]

· EPBj

[EQX|Bj

[(xi −Xi)

2]EQX|Bj


QX|Bj[(xr −Xr)]

· EPBj

[EQX|Bj

[(xi −Xi)(xr −Xr)]EQX|Bj[(xi −Xi)]

]+ 8EPBj

QX|Bj[(xi −Xi)]

· EPBj

[EQX|Bj

[(xi −Xi)(xr −Xr)]EQX|Bj[(xr −Xr)]

])+ 3(s2 − s)2

(12

· EPBj

[EQX|Bj

[(xr −Xr)]2]2

+ 12EPBj

[EQX|Bj

[(xi −Xi)]2]2

+ 8EPBj

[EQX|Bj

[(xr −Xr)]2]EPBj

[EQX|Bj

[(xi −Xi)]2]

+ 16EPBj

[EQX|Bj

[(xr −Xr)]EQX|Bj[(xi −Xi)]

] )+ 6(s3 − s2)

(8EPBj

QX|Bj

[(xr −Xr)

2]

· EPBj

[EQX|Bj

[(xr −Xr)]2]+ 8EPBj

QX|Bj

[(xi −Xi)

2]EPBj

[EQX|Bj

[(xi −Xi)]2]

+ 16EPBjQX|Bj

[(xr −Xi)(xi −Xi)]EPBj

[EQX|Bj


] )

− 6(s2 − s)(s− 2)(8EPBj

[EQX|Bj

[(xr −Xr)

2]EQX|Bj

[(xr −Xr)]2]

+ 8EPBj

[EQX|Bj

[(xi −Xi)

2]EQX|Bj

[(xi −Xi)]2]

+ 16EPBj

[EQX|Bj

[(xr −Xr)(xi −Xi)]EQX|Bj[(xr −Xr)]EQX|Bj

[(xi −Xi)]])

− 3(s2 − s)(8EPBj

[EQX|Bj

[(xr −Xr)

2]2]

+ 8EPBj

[EQX|Bj

[(xi −Xi)

2]2]

+ 8EPBj

[EQX|Bj

[(xr −Xr)(xi −Xi)]2])

− 12(s4 − s3)(12EPBj

[EQX|Bj

[(xr −Xr)]2]

· EPBjQX|Bj

[(xr −Xr)]2 + 12EPBj

[EQX|Bj

[(xi −Xi)]2]EPBj

QX|Bj[(xi −Xi)]

2

+ 4EPBj

[EQX|Bj

[(xi −Xi)]2]EPBj

QX|Bj[(xr −Xr)]

2 + 4EPBj

[EQX|Bj

[(xr −Xr)]2]

· EPBjQX|Bj

[(xi −Xi)]2 + 16EPBj

QX|Bj[(xr −Xr)]EPBj

QX|Bj[(xi −Xi)]

· EPBj

[EQX|Bj


])+ 3s2

(8EPBj

[EQX|Bj

[(xr −Xr)

2]]

+ 8EPBj

[EQX|Bj

[(xi −Xi)

2]]

+ 16EPBj

[EQX|Bj

[(xr −Xr)(xi −Xi)]])

− 12s3(8EPBj

QX|Bj

[(xr −Xr)

2]EPBj

QX|Bj[(xr −Xr)]

2 + 8EPBjQX|Bj

[(xi −Xi)

2]

· EPBjQX|Bj

[(xi −Xi)]2 + 16EPBj

QX|Bj[(xr −Xr)(xi −Xi)]EPBj

QX|Bj[(xr −Xr)]

· EPBjQX|Bj

[(xi −Xi)])+ 6s4

(12EPBj

QX|Bj[(xr −Xr)]

4 + 12EPBjQX|Bj

[(xi −Xi)]4

+ 24EPBjQX|Bj

[(xr −Xr)]2EPBj

QX|Bj[(xi −Xi)]

2), (A.41)

145

A.

where xr and xi denote the real and imaginary part of x respectively. From (A.23)

we have

c2 =m∑

j=1

EBj ,QX|Bj

[EZ

[f(4)j (0, X,Bj, s)|X,Bj

]]

4!(A.42)

A.4.1 Functions

This section will serve as reference for the proofs in Section A.4, as the equations

in Section A.4 are shorten on purpose to facilitate presentation. We will mainly

define a few types of expression according to their forms and calculate their

expectations over Z. We first substitute ρ = 0 into the dummy functions and

their derivatives we have defined from Eqs. (4.34)-(4.39).

βj(0, x, z, b) = e−|z|2, (A.43)

β ′j(0, x, z, b) =

∑

x′∈Xjb

QX|Bj(x′|b)e−|z|2 [−2zr(xr − x′r)− 2zi(xi − x′i)] , (A.44)

β′′

j (0, x, z, b) =∑

x′

QX|Bj(x′|b)e−|z|2

((2zr(xr − x′r) + 2zi(xi − x′i)

)2 − 2(xr − x′r)2 − 2(xi − x′i)

2), (A.45)

αj(0, x, z, b) =∑

x′

QX|Bj(x′|b)e−|z|2(2zr(xr − x′r) + 2zi(xi − x′i)

), (A.46)

α′j(0, x, z, b) =

∑

x′

QX|Bj(x′|b)e−|z|2 (A.47)

(2(xr − x′r)

2 + 2(xi − x′i)2 −

(2zr(xr − x′r) + 2zi(xi − x′i)

)2), (A.48)

α′′

j (0, x, z, b) =∑

x′


((2zr(xr − x′r) + 2zi(xi − x′i)

)3

− 3(2zr(xr − x′r) + 2zi(xi − x′i)

)(2(xr − x′r)

2 + 2(xi − x′i)2)), (A.49)

λj(0, x, z, b) =∑

x′

QX|Bj(x′|b)e−|z|2 (A.50)

((2zr(xr − x′r) + 2zi(xi − x′i)

)2 − 2(xr − x′r)2 − 2(xi − x′i)

2), (A.51)

λ′j(0, x, z, b) =∑

x′

QX|Bj(x′|b)e−|z|2 (A.52)

(3(2(xr − x′r)

2 + 2(xi − x′i)2)(2zr(xr − x′r) + 2zi(xi − x′i)

)

146


−(2zr(xr − x′r) + 2zi(xi − x′i)

)3), (A.53)

λ′′

j (0, x, z, b) =∑

x′


((2zr(xr − x′r) + 2zi(xi − x′i)

)4

+ 3(2(xr − x′r)

2 + 2(xi − x′i)2)2

− 6(2zr(xr − x′r) + 2zi(xi − x′i)

)2(2(xr − x′r)

2 + 2(xi − x′i)2)). (A.54)

In the following, we define 14 terms to simplify the derivation of c1, c2.

A.4.1.1 E[T 1j (x, Z, b)]

First of all, we define,

T 1j (x, z, b)

=∑

x′

QX|Bj(x′|b)

((2zr(xr − x′r) + 2zi(xi − x′i)

)4+ 3(2(xr − x′r)

2 + 2(xi − x′i)2)2

− 6(2zr(xr − x′r) + 2zi(xi − x′i)

)2(2(xr − x′r)

2 + 2(xi − x′i)2)).

(A.55)

Note that Z is a complex Gaussian random variable with zero mean for both

real and imaginary parts, therefore, only the terms with z4r , z4i , z

2rz

2i will be non-

zero when averaging (A.55) over Z. So we expand (A.55) and only keep thecorresponding terms.

T 1j (x, z, b)

=∑

x′

QX|Bj(x′|b)

(16z4r (xr − x′r)

4 + 16z4i (xi − x′i)4 + 12(xr − x′r)

4 + 12(xi − x′i)4

+ 96z2r z2i (xr − x′r)

2(xi − x′i)2 − 48z2r (xr − x′r)

4 − 48z2i (xi − x′i)4 − 48z2r (xr − x′r)

2(xi − x′i)2

− 48z2i (xr − x′r)2(xi − x′i)

2 + 24(xr − x′r)2(xi − x′i)

2

). (A.56)

Note∫

zr

z2rfZr(zr) =1

2

∫

zr

z4rfZr(zr) =3

4(A.57)

∫

zi

z2i fZi(zi) =

1

2

∫

zi

z4i fZi(zi) =

3

4. (A.58)

147

A.

where fZr(zr) and fZi(zi) is the probability density function of the Gaussian

random variable zr and zi. We have

E[T 1j (x, Z, b)] = 0. (A.59)

A.4.1.2 E[T 2j (x, Z, b)]

We define,

T 2j (x, z, b) =

∑

x′

QX|Bj(x′|b)

(3(2(xr − x′r)

2 + 2(xi − x′i)2)(2zr(xr − x′r) + 2zi(xi − x′i)

)


)3)×(∑

x′

QX|Bj(x′|b)

(−2zr(xr − x′r)− 2zi(xi − x′i)

)).

(A.60)

Expand (A.60) and only keep the terms with z4r , z4i , z

2rz

2i ,

T 2j (x, z, b) = −24z2r

∑

x′

QX|Bj(x′|b)(xr − x′r)

3∑

x′


− 24z2i∑

x′

QX|Bj(x′|b)(xi − x′i)

3∑

x′


− 24z2r∑

x′


2(xr − x′r)∑

x′


− 24z2i∑

x′


2(xi − x′i)∑

x′


+ 16z4r∑

x′


3∑

x′


+ 16z4i∑

x′


3∑

x′


+ 48z2rz2i

∑

x′

QX|Bj(x′|b)(xr − x′r)(xi − x′i)

2∑

x′


+ 48z2rz2i

∑

x′

QX|Bj(x′|b)(xi − x′i)(xr − x′r)

2∑

x′

QX|Bj(x′|b)(xi − x′i).

(A.61)

Note (A.57) and (A.58), we have

E[T 2j (x, Z, b)] = 0. (A.62)

148


A.4.1.3 E[T 3j (x, Z, b)]

We define,

T 3j (x, z, b) =

(∑

x′

QX|Bj(x′|b)

((2zr(xr − x′r) + 2zi(xi − x′i)

)2 − 2(xr − x′r)2 − 2(xi − x′i)

2))2

.

(A.63)


2rz

2i ,

T 3j (x, z, b) = 4

(∑

x′


2

)2

+ 4

(∑

x′


2

)2

− 16z2r

(∑

x′


2

)2

− 16z2i

(∑

x′


2

)2

+ 16z4r

(∑

x′


2

)2

+ 16z4i

(∑

x′


2

)2

+ 8∑

x′


2∑

x′


2

− 16z2i∑

x′


2∑

x′


2

− 16z2r∑

x′


2∑

x′


2

+ 32z2rz2i

∑

x′


2∑

x′


2

+ 64z2rz2i

(∑

x′


)2

. (A.64)


E[T 3j (x, Z, b)] = 8

(∑

x′


2

)2

+ 8

(∑

x′


2

)2

+ 16

(∑

x′


)2

. (A.65)

149

A.

A.4.1.4 E[T 4j (x, Z, b)]

We define,

T 4j (x, z, b) =

∑

x′

QX|Bj(x′|b)

((2zr(xr − x′r) + 2zi(xi − x′i)

)2 − 2(xr − x′r)2 − 2(xi − x′i)

2)

×(∑

x′

QX|Bj(x′|b)


))2

. (A.66)


2rz

2i ,

T 4j (x, z, b) = 16z4r

∑

x′


2

(∑

x′


)2

+ 16z4i∑

x′


2

(∑

x′


)2

+ 16z2rz2i

∑

x′


2

(∑

x′


)2

+ 16z2rz2i

∑

x′


2

(∑

x′


)2

− 8z2r∑

x′


2

(∑

x′


)2

− 8z2i∑

x′


2

(∑

x′


)2

− 8z2r∑

x′


2

(∑

x′


)2

− 8z2i∑

x′


2

(∑

x′


)2

+ 64z2rz2i

∑

x′

QX|Bj(x′|b)(xr − x′r)(xi − x′i) ·

(∑

x′


)·

(∑

x′


). (A.67)

150



E[T 4j (x, Z, b)] = 8

∑

x′


2

(∑

x′


)2

+ 8∑

x′


2

(∑

x′


)2

+ 16∑

x′

QX|Bj(x′|b)(xr − x′r)(xi − x′i) ·

(∑

x′


)·

(∑

x′


). (A.68)

A.4.1.5 E[T 5j (x, Z, b)]

We define,

T 5j (x, z, b) =

(∑

x′

QX|Bj(x′|b)


))4

(A.69)


2rz

2i ,

T 5j (x, z, b) = 16z4r

(∑

x′


)4

+ 16z4r

(∑

x′


)4

+ 96z2rz2i

(∑

x′


∑

x′


)2

(A.70)


E[T 5j (x, Z, b)] = 12

(∑

x′


)4

+ 12

(∑

x′


)4

+ 24

(∑

x′


∑

x′


)2

(A.71)

151

A.

A.4.1.6 E[T 6j (x, Z)]

We define

T 6j (x, z) =

1∑

b′=0

(∑

x′

QX|Bj(x′|b′) (−2zr(xr − x′r)− 2zi(xi − x′i))

)3

×(

1∑

b′=0

PBj(b′)

∑

x′


)

(A.72)


2rz

2i ,

T 6j (x, z)

= 16z4r

1∑

b′=0

PBj(b′)

(∑

x′

QX|Bj(x′|b′)(xr − x′r)

)3

·(

1∑

b′=0

PBj(b′)∑

x′


)

+ 16z4i

1∑

b′=0

PBj(b′)

(∑

x′

QX|Bj(x′|b′)(xi − x′i)

)3

·(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 48z2rz2i

1∑

b′=0

PBj(b′)

(∑

x′


)(∑

x′


)2

×(

1∑

b′=0

PBj(b′)∑

x′


)

+ 48z2rz2i

1∑

b′=0

PBj(b′)

(∑

x′


)(∑

x′


)2

×(

1∑

b′=0

PBj(b′)∑

x′


)(A.73)


E[T 6j (x, Z)] =

121∑

b′=0

PBj(b′)

(∑

x′


)3( 1∑

b′=0

PBj(b′)

∑

x′


)

+121∑

b′=0

PBj(b′)

(∑

x′


)3( 1∑

b′=0

PBj(b′)

∑

x′


)

152


+121∑

b′=0

PBj(b′)

(∑

x′


)(∑

x′


)2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+12

1∑

b′=0

PBj(b′)

(∑

x′


)(∑

x′


)2

×(

1∑

b′=0

PBj(b′)

∑

x′


)(A.74)

A.4.1.7 E[T 7j (x, Z)]

We define

T 7j (x, z) =

1∑

b′=0

PBj(b′)

(∑

x′

QX|Bj(x′|b′)


))

×(∑

x′

QX|Bj(x′|b′)

(2(xr − x′r)

2 + 2(xi − x′i)2 −


)2))

(A.75)


2rz

2i ,

T 7j (x, z) = −8z2r

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

− 8z2i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

− 8z2r

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

153

A.

− 8z2i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 16z4r

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 16z4i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 16z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 16z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 32z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′

QX|Bj(x′|b′)(xr − x′r)(xi − x′i)

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 32z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


×(

1∑

b′=0

PBj(b′)

∑

x′


)(A.76)

154



E[T 7j (x, Z)] = 8

1∑

b′=0

PBj(b′)∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 8

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


2

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 8

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


×(

1∑

b′=0

PBj(b′)

∑

x′


)

+ 8

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


×(

1∑

b′=0

PBj(b′)

∑

x′


)(A.77)

A.4.1.8 E[T 8j (x, Z)]

We define

T 8j (x, z) =

1∑

b′=0

PBj(b′)

(∑

x′

QX|Bj(x′|b′) (2zr(xr − x′r) + 2zi(xi − x′i))

)2

2

(A.78)


2rz

2i ,

T 8j (x, z) = 16z4r

1∑

b′=0

PBj(b′)

(∑

x′


)2

2

155

A.

+ 16z4i

1∑

b′=0

PBj(b′)

(∑

x′


)2

2

+ 32z2rz2i

1∑

b′=0

PBj(b′)

(∑

x′


)2

×

1∑

b′=0

PBj(b′)

(∑

x′


)2

+ 64z2rz2i

(1∑

b′=0

PBj(b′)

∑

x′


∑

x′


)2

(A.79)


E[T 8j (x, Z)] = 12

1∑

b′=0

PBj(b′)

(∑

x′


)2

2

+ 12

1∑

b′=0

PBj(b′)

(∑

x′


)2

2

+ 8

1∑

b′=0

PBj(b′)

(∑

x′


)2

×

1∑

b′=0

PBj(b′)

(∑

x′


)2

+ 16

(1∑

b′=0

PBj(b′)∑

x′


∑

x′


)2

(A.80)

156


A.4.1.9 E[T 9j (x, Z)]

We define

T 9j (x, z) =

1∑

b′=0

PBj(b′)

(∑

x′

QX|Bj(x′|b′) (2zr(xr − x′r) + 2zi(xi − x′i))

2

)2

×1∑

b′=0

PBj(b′)

∑

x′

QX|Bj(x′|b′)

((2zr(xr − x′r) + 2zi(xi − x′i))

2 − 2(xr − x′r)2 − 2(xi − x′i)

2)

(A.81)


2rz

2i ,

T 9j (x, z)

= 16z4r

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

+16z4i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

+16z2rz2i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

+16z2rz2i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

+64z2rz2i

1∑

b′=0

PBj(b′)

(∑

x′


)(∑

x′


)

×(

1∑

b′=0

PBj(b′)

∑

x′


)

−8z2r

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

−8z2i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

−8z2i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

157

A.

−8z2r

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

(A.82)


E[T 9j (x,Z)] =

81∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

+8

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


2

)

+16

1∑

b′=0

PBj(b′)

(∑

x′


)(∑

x′


)

×(

1∑

b′=0

PBj(b′)

∑

x′


)(A.83)

A.4.1.10 E[T 10j (x, Z)]

We define

T 10j (x, z) =

1∑

b′=0

PBj(b′)

(∑

x′

QX|Bj(x′|b′)


))2

×(

1∑

b′=0

PBj(b′)

∑

x′

QX|Bj(x′|b′)


))2

(A.84)


2rz

2i ,

T 10j (x, z) =

16z4r

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

+16z4i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

+16z2r z2i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

158


+16z2rz2i

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

+64z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


∑

x′


×(

1∑

b′=0

PBj(b′)

∑

x′


)(1∑

b′=0

PBj(b′)

∑

x′


)

(A.85)


E[T 10j (x,Z)] =

121∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

+121∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

+4

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

+4

1∑

b′=0

PBj(b′)

(∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

+161∑

b′=0

PBj(b′)

∑

x′


∑

x′


×(

1∑

b′=0

PBj(b′)

∑

x′


)(1∑

b′=0

PBj(b′)

∑

x′


)

(A.86)

A.4.1.11 E[T 11j (x, Z)]

We define

T 11j (x, z) =

1∑

b′=0

PBj(b′)

∑

x′

QX|Bj(x′|b′)

(−(2zr(xr − x′r) + 2zi(xi − x′i)

)3

+ 3(2(xr − x′r)

2 + 2(xi − x′i)2) (

2zr(xr − x′r) + 2zi(xi − x′i)))

159

A.

×(

1∑

b′=0

PBj(b′)

∑

x′

QX|Bj(x′|b′)


))

(A.87)


2rz

2i ,

T 11j (x, z) =

−24z2r

1∑

b′=0

PBj(b′)∑

x′


3

(1∑

b′=0

PBj(b′)

∑

x′


)

−24z2i

1∑

b′=0

PBj(b′)∑

x′


3

(1∑

b′=0

PBj(b′)

∑

x′


)

−24z2r

1∑

b′=0

PBj(b′)∑

x′


2(xr − x′r)

×(

1∑

b′=0

PBj(b′)

∑

x′


)

−24z2i

1∑

b′=0

PBj(b′)∑

x′


2(xi − x′i)

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+16z4r

1∑

b′=0

PBj(b′)∑

x′


3

(1∑

b′=0

PBj(b′)

∑

x′


)

+16z4i

1∑

b′=0

PBj(b′)∑

x′


3

(1∑

b′=0

PBj(b′)

∑

x′


)

+48z2rz2i

1∑

b′=0

PBj(b′)∑

x′


2(xr − x′r)

×(

1∑

b′=0

PBj(b′)

∑

x′


)

+48z2rz2i

1∑

b′=0

PBj(b′)∑

x′


2(xi − x′i)

×(

1∑

b′=0

PBj(b′)

∑

x′


)(A.88)

160



E[T 11j (x, Z, b)] = 0. (A.89)

A.4.1.12 E[T 12j (x, Z)]

We define

T 12j (x, z) =(

1∑

b′=0

PBj(b′)

∑

x′

QX|Bj(x′|b′)

((2zr(xr − x′r) + 2zi(xi − x′i)

)2 − 2(xr − x′r)2 − 2(xi − x′i)

2))2

(A.90)


2rz

2i ,

T 12j (x, z) = 16z4r

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

+16z4r

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

+4

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

+4

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

+16z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)∑

x′


2

)

+16z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


2

)

−8z2r

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

−8z2i

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

161

A.

−16z2r

1∑

b′=0

PBj(b′)∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


2

)

−16z2i

1∑

b′=0

PBj(b′)∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


2

)

−8z2r

(1∑

b′=0

PBj(b′)∑

x′


2

)2

−8z2i

(1∑

b′=0

PBj(b′)∑

x′


2

)2

+41∑

b′=0

PBj(b′)∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


2

)

+4

1∑

b′=0

PBj(b′)∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


2

)

+64z2rz2i

(1∑

b′=0

PBj(b′)∑

x′


)2

(A.91)


E[T 12j (x, Z)] =

8

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

+8

(1∑

b′=0

PBj(b′)

∑

x′


2

)2

+16

(1∑

b′=0

PBj(b′)

∑

x′


)2

(A.92)

A.4.1.13 E[T 13j (x, Z)]

We define

T 13j (x, z) =1∑

b′=0

PBj(b′)

∑

x′

QX|Bj(x′|b′)

((2z2r (xr − x′r) + 2zi(xi − x′i)

)2 − 2(xr − x′r)2 − 2(xi − x′i)

2)

162


×(

1∑

b′=0

PBj(b′)

∑

x′


)2

(A.93)


2rz

2i ,

T 13j (x, z) =

16z4r

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)∑

x′


)2

+16z4i

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


)2

+16z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


)2

+16z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)∑

x′


)2

−8z2r

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)∑

x′


)2

−8z2i

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


)2

−8z2r

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


)2

−8z2i

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)∑

x′


)2

+64z2rz2i

1∑

b′=0

PBj(b′)

∑

x′


×(

1∑

b′=0

PBj(b′)

∑

x′


)(1∑

b′=0

PBj(b′)∑

x′


)

(A.94)

163

A.


E[T 13j (x, Z)] =

8

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


)2

+8

1∑

b′=0

PBj(b′)

∑

x′


2

(1∑

b′=0

PBj(b′)

∑

x′


)2

+161∑

b′=0

PBj(b′)∑

x′


×(

1∑

b′=0

PBj(b′)

∑

x′


)(1∑

b′=0

PBj(b′)

∑

x′


)

(A.95)

A.4.1.14 E[T 14j (x, Z)]

We define

T 14j (x, z) =

(1∑

b′=0

PBj(b′)∑

x′


)4

(A.96)


2rz

2i ,

T 14j (x, z) =

4z4r

(1∑

b′=0

PBj(b′)

∑

x′


)2

+4z4i

(1∑

b′=0

PBj(b′)

∑

x′


)2

+8z2rz2i

(1∑

b′=0

PBj(b′)

∑

x′


)

×(

1∑

b′=0

PBj(b′)

∑

x′


)

2

(A.97)

164

A.5 Proof of Proposition 4.2.1 and 4.2.2


E[T 14j (x, Z)] =

12

(1∑

b′=0

PBj(b′)

∑

x′


)4

+ 12

(1∑

b′=0

PBj(b′)

∑

x′


)4

+24

(1∑

b′=0

PBj(b′)

∑

x′


)2( 1∑

b′=0

PBj(b′)

∑

x′


)2

(A.98)

A.5 Proof of Proposition 4.2.1 and 4.2.2

A.5.1 Proof of Proposition 4.2.1

Without loss of generality, we choose 0 ≤ s1 ≤ s2, λ ∈ [0, 1], s′ = λs1+(1−λ)s2.

Then apply Holder’s inequality, we have that

1∑

b′=0

PBj(b′)QX|Bj

(xj,b′(x)|b′)s′

e−s′|x−xj,b′(x)|2 (A.99)

≤(

1∑

b′=0

PBj(b′)QX|Bj

(xj,b′(x)|b′)s1 e−s1|x−xj,b′(x)|2)λ

(A.100)

·(

1∑

b′=0

PBj(b′)QX|Bj

(xj,b′(x)|b′)s2 e−s2|x−xj,b′(x)|2)1−λ

. (A.101)

The proposition is then proved by substituting Eq. (4.49) into Eq. (A.101).

A.5.2 Proof of Proposition 4.2.2

For the BICM schemes we have considered in this disseratation, Ibicm0 (s, P bicmX ; snr)

is an weekly increasing function of snr, and this can be shown by substituting

the corresponding decoding metric into the expression of Ibicm0 (s, P bicmX ; snr) and

compute the first dirivative. We observe that

dIbicm0 (s, P bicmX ; snr)

dsnr≥ 0. (A.102)

165

A.

Let snri ≥ snrj , we have

Ibicm0 (s, P bicmX , snri) ≥ Ibicm0 (s, P bicm

X , snrj), (A.103)

for fixed s ≥ 0 and P bicmX ∈

P bicmX : EP bicm

X[|X|2] = 1

. Hence,

limsnri→∞

Ibicm0 (s, P bicmX , snri) ≥ Ibicm0 (s, P bicm

X , snrj). (A.104)

Take the supremum over all possible s, P bicmX and letting snrj → ∞ yields

sups,P bicm

X

limsnri→∞

Ibicm0 (s, P bicmX , snri) ≥ lim

snrj→∞sup

s,P bicmX

Ibicm0 (s, P bicmX , snrj) (A.105)

= limsnri→∞

sups,P bicm

X

Ibicm0 (s, P bicmX , snri). (A.106)

On the other hand, we have for all snr

sups,P bicm

X

Ibicm0 (s, P bicmX ; snr) ≥ Ibicm0 (s, P bicm

X ; snr). (A.107)

Taking the limit snr → ∞ and then the supremum over all possible s, P bicmX yields

limsnr→∞

sups,P bicm

X

Ibicm0 (s, P bicmX ; snr) ≥ sup

s,P bicmX

limsnr→∞

Ibicm0 (s, P bicmX ; snr), (A.108)

together with Eq. (A.106) imply Eq. (4.50).

166

Appendix B

B.1 Proof of Eqs. (5.8) to (5.12)

The GMI is the largest rate achievable for which Er,0(R) > 0, and hence we have

I0(PX) = sup0≤ρ≤1s≥0

E0

(ρ, s)

ρ. (B.1)

Before showing Eqs. (5.9)-(5.12) are true, we introduce the following lemmas.

Lemma B.1.1 Let f : A×B → R, and supa∈A,b∈B f(a, b) and supa∈A supb∈B f(a, b)

exist. Then

f ∗ = supa∈A,b∈B

f(a, b) = supa∈A

supb∈B

f(a, b) = supb∈B

supa∈A

f(a, b). (B.2)

Proof: Let x = supa∈A,b∈B f(a, b) and y = supa∈A supb∈B f(a, b). For each

a ∈ A, let ya = supb∈B f(a, b). Clearly, all ya exist, since y = supa∈A ya and of

course ya ≤ y for each a ∈ A.

Now we fix a0 ∈ A. Because f(a, b) ≤ x for all (a, b) ∈ A×B, so in particular

f(a0, b) ≤ x for all b ∈ B. Hence, ya0 = supb∈B f(a0, b) ≤ x). Thus, ya ≤ x for

all a ∈ A, and it follows that y = supa∈A ya ≤ x.

Suppose that y < x, and fix z ∈ (y, x). By the definition of x as a supremum,

there is some (a, b) ∈ A×B such that f(a, b) > z. This means that we must have

ya ≥ f(a, b) > z > y (B.3)

which is contradictory to ya ≤ y. Hence x = y. Similarly, we can also show that

x = w = supb∈B supa∈A f(a, b).

167

B.

Lemma B.1.2 [27, Appendix 5B] Let Q = [Q(0), . . . , Q(K − 1)] be a probability

vector and let a0, . . . , aK−1 be a set of nonnegative numbers. Then the function

f(s) = log

[K−1∑

k=0

Q(k)a1s

k

]s(B.4)

is nonincreasing and convex with s for s > 0. Moreover, f(s) is strictly decreasing

unless all ak for which Q(k) > 0 are equal. The convexity is strict unless all the

nonzero ak for which Q(k) > 0 are equal. The lemma is also valid when the

summation in Eq. (B.4) is replaced by integral.

For discrete channel input, we always have

I0(PX) ≤ I(X ; Y ) ≤ H(X), (B.5)

hence we let g(ρ, s) =E0

(ρ,s

)ρ

, and g(ρ, s) is bounded from above. Thus, applying

Lemma B.1.1 directly to the function g(ρ, s), we prove Eq. (5.9) is true.

Let g′(r, s) = g(1ρ, s), we have

supr≥1,s≥0

g′(r, s) = sup0≤ρ≤1s≥0

g(ρ, s). (B.6)

Apply Lemma B.1.2 to g′(r, s), we know that g′(r, s) is a nondecreasing and

concave function with r for r ≥ 1. Hence the supremum of g′(r, s) is given at

r → ∞ for fixed s. This is equivalent to say that the supremum of g(ρ, s) is given

at ρ→ 0 for fixed s, i.e., Eq. (5.10) is true.

For Eq. (5.11), it is equivalent to show

sups≥0

limr→∞

g′(r, s) = limr→∞

sups≥0

g′(r, s). (B.7)

To see this, we let ri ≥ rj, we have

g′(ri, s) ≥ g′(rj , s), (B.8)

for fixed s ≥ 0. Hence,

limri→∞

g′(ri, s) ≥ g′(rj , s). (B.9)

168

B.2 Concavity of E1′ and E1

Take the supremum over all possible s and letting rj → ∞ yields

sups≥0

limri→∞

g′(ri, s) ≥ limrj→∞

sups≥0

g′(rj , s) (B.10)

= limri→∞

sups≥0

g′(ri, s). (B.11)

On the other hand, we have for all r

sups≥0

g′(r, s) ≥ g′(r, s). (B.12)

Taking the limit r → ∞ and then the supremum over all possible s yields

limr→∞

sups≥0

g′(r, s) ≥ sups≥0

limr→∞

g′(r, s). (B.13)

Eqs. (B.11) and (B.13) together shows Eq. (B.7) is true, thus Eq. (5.11) is also

true.

Finally, Eq. (5.12) is true by applying L’Hospital’s rule to Eq. (5.11).

B.2 Concavity of E1′ and E1

B.2.1 Concavity of E1′(ρ, s, a(·)

)

Same as in Appendix A.1, without loss of generality, we choose 0 ≤ s1 ≤ s2,

λ ∈ [0, 1], s′ = λs1 + (1 − λ)s2, a1(x) ∈ R, a2(x) ∈ R for each x ∈ X and

a′(x) = λa1(x) + (1− λ)a2(x). Then apply Holder’s inequality, we have that

∑

x′

P bicmX (x′)

(q(x′, y)

q(x, y)

)s′ea

′(x′)

ea′(x)

≤(∑

x′

P bicmX (x′)

(q(x′, y)

q(x, y)

)s1 ea1(x′)

ea1(x)

)λ(∑

x′

P bicmX (x′)

(q(x′, y)

q(x, y)

)s2 ea2(x′)

ea2(x)

)1−λ

(B.14)

169

B.

So

E1′(ρ, s′, a′(·)

)

=− logE

[(∑

x′

P bicmX (x′)

(q(x′, Y )

q(X,Y )

)s′ea

′(x′)

ea′(X)

)ρ](B.15)

≥− logE

(∑

x′

P bicmX (x′)

(q(x′, Y )

q(X,Y )

)s1 ea1(x′)

ea1(X)

)λ(∑

x′

P bicmX (x′)

(q(x′, Y )

q(X,Y )

)s2 ea2(x′)

ea2(X)

)1−λ

(B.16)

Apply Holder’s inequality again, we have

E1′(ρ, s′, a′(·)

)≥ λE1′

(ρ, s1, a1(·)

)+ (1− λ)E1′

(ρ, s2, a2(·)

)(B.17)

which proves the joint concavity of E1′(ρ, s, a(·)

)for s ≥ 0 and any real function

a(·).Now we choose 0 ≤ ρ1 ≤ ρ2, λ ∈ [0, 1], ρ′ = λρ1 + (1 − λ)ρ2. Then apply

Holder’s inequality, we have that

E1′(ρ′, s, a(·)

)≥ λE1′

(ρ1, s, a(·)

)+ (1− λ)E1′

(ρ2, s, a(·)

). (B.18)

which proves the joint concavity of E1′(ρ, s, a(·)

)for 0 ≤ ρ ≤ 1.

B.2.2 Concavity of E1

(ρ, s, r, r, a(·)

)

All concavity can be shown in a similar way as in Appendix B.2.1.

B.3 Proof of Proposition 5.2.1

The optimality of s = 11+ρ

for Er,0(R) is shown in [67, Section IV-B]. For Er,1′(R)

we just need to show that s = 11+ρ

achieves the supremum/maximum in

sups≥0

E1′(ρ, s, a(·)

)= sup

s≥0− logE

[(∑

x′

PX(x′)q(x′, Y )sea(x

′)

q(X, Y )sea(X)

)ρ]. (B.19)

170


Function inside the logarithm of E1′(ρ, s, a) can be rewritten as

∫

y

(∑

x

PX(x)PY |X(y|x)e−a(x)q(x, y)−sρ

)×(∑

x′

PX(x′)ea(x

′)q(x′, y)s

)ρ

dy.

(B.20)

For a fixed channel observation y, the integrand is reminiscent of the RHS of the

following form of Holder’s inequality, with a slight abuse of notation,

(∑

i

aibi

)1+ρ

≤(∑

i

a1+ρi

)(∑

i

b1+ρρ

i

)ρ

. (B.21)

Therefore, if we let

ai = PX(x)1

1+ρPY |X(y|x)1

1+ρ e1

1+ρa(x)

q(x, y)−sρ1+ρ (B.22)

bi = PX(x′)

ρ1+ρ e

ρ1+ρ

a(x′)q(x′, y)

sρ1+ρ , (B.23)

we can lower bound Eq. (B.20) by

∫

y

(∑

x

PX(x)PY |X(y|x)1

1+ρ

)ρ

dy. (B.24)

This lower bound can be achieved by setting s = 11+ρ

and a(x) = 0 for all x. As

a result, these parameters also achieve the supremum in Eq. (B.19). Then it is

not difficult to see that when we substitute s = 11+ρ

and a(x) = 0 for all x in

Er,1′(R), we have exactly Eq. (5.25).

For Er,1(R), we rewrite the term inside the log as

∫

y

(∑

x

PX(x)PY |X(y|x)era(x)−raq(x, y)−sρ

)×(∑

x′

PX(x′)era(x

′)−raq(x′, y)s

)ρ

dy.

(B.25)

For a fixed channel observation y, the integrand is again reminiscent of the RHS

of the Holder’s inqueality in Eq. (B.21) with

ai = PX(x)1

1+ρPY |X(y|x)1

1+ρ e1

1+ρ(ra(x)−ra)q(x, y)

−sρ1+ρ (B.26)

bi = PX(x′)

ρ1+ρ e

ρ1+ρ

(ra(x′)−ra)q(x′, y)sρ1+ρ . (B.27)

171

B.

Thus, Eq. (B.25) can be lower-bounded by

∫

y

(∑

x

PX(x)er+ρr1+ρ

(a(x)−a)PY |X(y|x)1

1+ρ

)ρ

dy. (B.28)

Since r+ρr

1+ρ∈ (−∞,∞), if we let τ(x) = r+ρr

1+ρa(x) and τ = r+ρr

1+ρa, we have

sups≥0

r,r,a(·)

E1

(ρ, s, r, r, a(·)

)= sup

s≥0r,r,a(·)

− logE

[era(X)−ra

(∑

x′

PX(x′)era(x

′)−ra q(x′, Y )s

q(X, Y )s

)ρ]

(B.29)

≤ supr,r,a(·)

− log

∫

y

(∑

x

PX(x)er+ρr1+ρ

(a(x)−a)PY |X(y|x)1

1+ρ

)1+ρ

dy

(B.30)

= supτ(·)

− log

∫

y

(∑

x

PX(x)eτ(x)−τPY |X(y|x)

11+ρ

)1+ρ

dy.

(B.31)

Let τ ∗(x) achieves the supremum in Eq. (B.31), it is not difficult to observe that

E1(ρ, s, r, r, a) can achieve the same supremum by setting

r = r, s =1

1 + ρ, a(x) =

τ ∗(x)

r, a =

τ ∗

r, q(x, y) = PY |X(y|x), (B.32)

therefore, s = 11+ρ

and r = r is optimal for Er,1(R).

B.4 Proof of Lemma 5.2.1

When q(x, y) = PY |X(y|x) and τ(x) = a(x)− a for x ∈ X, we have

sups≥0

r,r,a(·)

E1

(ρ, s, r, r, a(·)

)= sup

s≥0,r,rτ(·):E[τ(X)]=0

− logE

[erτ(x)

(∑

x′

PX(x′)erτ(x

′) PY |X(Y |x′)sPY |X(Y |X)s

)ρ].

(B.33)

In Proposition 5.2.1, we have shown that when q(x, y) = PY |X(y|x), the max-

imizing r, r and s for E1

(ρ, s, r, r, a(·)

)satisfy Eq. (B.32) (for convenience of

presentation, we just substitute r = r and q(x, y) = PY |X(y|x) into Eq. (B.33),

172

B.5 Proof of Theorem 5.2.1

and we do not substitute the optiaml s = 11+ρ

yet). Thus,

sups≥0

r,r,a(·)

E1

(ρ, s, r, r, a(·)

)= sup

s≥0τ(·):E[τ(X)]=0

− logE

[eτ(x)

(∑

x′

PX(x′)eτ(x

′) PY |X(Y |x′)sPY |X(Y |X)s

)ρ]

(B.34)

= sups≥0

τ(·):E[τ(X)]=0

− logE[eτ(X)−ρis,τ (X;Y )

]. (B.35)

Lemma 5.2.1 is then proved.


For ML decoder, G(ρ, s, a(·)

)in Eq. (5.29) can be obtained from setting r = r and

E[a(X)] = 0 in E1

(ρ, s, r, r, a(·)

). According to Proposition 5.1.2, the function

E1

(ρ, s, r, r, a(·)

)is jointly concave in a(x) for all x ∈ X. G

(ρ, s, a(·)

)is then

also joinly concave in a(x) for all x ∈ X for fixed ρ and s, and E[a(X)] = 0 is a

linear/convex constraint on a(x) for x ∈ X.

Due to the above argument, the Karush-Kuhn-Tucker (KKT) conditions are

necessary and sufficient conditions for the optimal a∗. Moreover, according to

Proposition 5.2.1, the optimal s for E1

(ρ, s, r, r, a(·)

)satisfies s∗ = 1

1+ρ, therefore,

the optimal s for G(ρ, s, a(·)

)with E[a(X)] = 0 also satisfies s = 1

1+ρ. The KKT

conditions of a∗ for

G(ρ, a(·)

)= G

(ρ, s =

1

1 + ρ, a(·)

)= log

∫

y

(∑

x

PX(x)ea(x)PY |X(y|x)

11+ρ

)1+ρ

dy

(B.36)

are given as

∇G(ρ, a∗) + λ∇E[a∗(X)] = 0 (B.37)

E[a∗(X)] = 0. (B.38)

They are given by

−∫y(1 + ρ)

(∑x′ PX(x

′)ea(x′)PY |X(y|x′)

11+ρ

)ρPX(x)e

a(x)PY |X(y|x)1

1+ρdy

∫y

(∑x′ PX(x′)ea(x

′)PY |X(y|x′)1

1+ρ

)1+ρ

dy

+ λPX(x)

= 0 (B.39)

173

B.

for all x ∈ X. Substituting ρ = 0 and applying some algebra, we obtain

ea(x) = λ∑

x′

PX(x′)ea(x). (B.40)

Eq. (B.40) together with Eq. (B.38) suggest that a(x) = 0 for all x ∈ X at ρ→ 0.

We now move on to proving the second part of Theorem 5.2.1. From Eq.

(B.39), we have

∫

y

(1 + ρ)

(∑

x′

ea(x′)PY |X(y|x′)

11+ρ

)ρ

PX(x)ea(x)PY |X(y|x)

11+ρdy

=λ

∫

y

(∑

x′

PX(x′)ea(x

′)PY |X(y|x′)1

1+ρ

)1+ρ

dy. (B.41)

Since we are only interested in the neighborhood of ρ→ 0, we Taylor expand theLHS and RHS of Eq. (B.41). The first order derivative of the LHS of Eq. (B.41)is

∫

y

(∑

x′

PX(x′)ea(x′)PY |X(y|x′)

11+ρ

)ρ

+ (1 + ρ)

(∑

x′


11+ρ

)ρ

·

log

∑

x′


11+ρ + ρ·

∑x′ PX(x′)

(ea(x

′)a(x′)′PY |X(y|x′)1

1+ρ + ea(x′)PY |X(y|x′)

11+ρ logPY |X(y|x′)

(− 1

(1+ρ)2

))

∑x′ PX(x′)ea(x′)PY |X(y|x′)

11+ρ

· ea(x)PY |X(y|x)1

1+ρ + (1 + ρ)

(∑

x′


11+ρ

)ρ

·(ea(x)a(x)′PY |X(y|x)

11+ρ + ea(x)PY |X(y|x)

11+ρ log PY |X(y|x)

(− 1

(1 + ρ)2

))dy.

(B.42)

174


Substituting ρ = 0, a(x) = 0 into Eq. (B.42), Eq. (B.42) is equal to

1 + a(x)′ − i(x). (B.43)

Similarly, the first order derivative of the RHS of Eq. (B.41) is

λ

∫

y

(∑

x′


11+ρ

)1+ρ

log

∑

x′


11+ρ + (1 + ρ)·

∑x′ PX(x′)

(ea(x)a(x′)′PY |X(y|x′)

11+ρ + ea(x

′)PY |X(y|x′)1

1+ρ logPY |X(y|x′)(− 1

(1+ρ)2

))

∑x′ PX(x′)ea(x′)PY |X(y|x′)

11+ρ

dy

(B.44)

Substituting ρ = 0, a(x) = 0 into Eq. (B.44), Eq. (B.44) is equal to

−λI(PX). (B.45)

After a Taylor expansion, Eq. (B.41) becomes

1 + ρ(1 + a(x)′ − i(x)

)+ o(ρ) = λ+ λρ

(− I(PX)

)+ o(ρ). (B.46)

Ignoring the higher order terms, we solve a(x)′ as

a(x)′ =λ− 1− λρI(PX)

ρ− 1 + i(x). (B.47)

By construction, we have

dE[a(X)]

dρ= E[a(X)′] = 0 (B.48)

Substituting the value of a(x)′ in Eq. (B.47) into Eq. (B.48), we have

λ =1 + ρ− ρI(PX)

1− ρI(PX), (B.49)

which we substitute back into Eq. (B.47) to obtain

a(x)′ = i(x)− I(PX). (B.50)

175

B.

This concludes the proof.


Let s(ρ) achieve the supremum in

sups≥0

E0(ρ, s), (B.51)

for a given ρ. E0(ρ, s) can be rewritten as

E0(ρ, s) = − logE[e−ρis(X;Y )

], (B.52)

where

is(x; y) = logPY |X(y|x)s

EX′

[PY |X(y|X ′)s

] . (B.53)

The first two derivatives of E0(ρ, s) are then given by

d sups≥0E0(ρ, s)

dρ=dE0

(ρ, s(ρ)

)

dρ=

E [e−ρis(is + ρi′s)]

E [e−ρis](B.54)

d2 sups≥0E0(ρ, s)

dρ2=d2E0(ρ, s(ρ))

dρ2(B.55)

=E[e−ρis

(−(is + ρi′s)

2 + (2i′s + ρi′′

s ))]

E [e−ρis ]+

(dE0

(ρ, s(ρ)

)

dρ

)2

,

(B.56)

here we write is as a shorthand for is(x; y). Substituting ρ = 0 into Eqs. (B.54)

and (B.56), we have

E′′

0 (ρ)|ρ=0 = E[−i2s + 2i′s

]+ (E [is])

2. (B.57)

From Proposition 5.2.1 we have that for matched decoding,

s(ρ) =1

1 + ρ. (B.58)

Substituting Eq. (B.58) and ρ = 0 into Eq. (B.57) gives

E′′

0 (ρ)|ρ=0 = −var (i(X ; Y )) , (B.59)

176


where i(x; y) is the information density function defined by Eq. (5.33). We have

thus proved Eq. (5.56).

From Lemma 5.2.1, we have

sups≥0

r,r,a(·)

E1

(ρ, s, r, r, a(·)

)= sup

s≥0a(·):E[a(X)]=0

G(ρ, s, a(·)

), (B.60)

Let s(ρ) and a(ρ) be the maximizing s and a for RHS of Eq. (B.60), respec-

tively. The first and second order derivatives of sup s≥0r,r,a(·)

E1

(ρ, s, r, r, a(·)

)are

then given by

d sup s≥0r,r,a(·)

E1

(ρ, s, r, r, a(·)

)

dρ=dG (ρ, s(ρ), a(ρ))

dρ=

E[ea(X)−ρis,a

(is,a + ρi′s,a − a(X)′

)]

E[ea(X)−ρis,a ](B.61)

d2 sup s≥0r,r,a(·)

E1

(ρ, s, r, r, a(·)

)

dρ2=d2G (ρ, s(ρ), a(ρ))

dρ2(B.62)

=E

[ea(X)−ρis,a

(−(is,a + ρi′s,a − a(X)′

)2+ 2i′s,a + ρi

′′

s,a − a(X)′′)]

E[ea(X)−ρis,a ]

+

(dG (ρ, s(ρ), a(ρ))

dρ

)2

, (B.63)

where we have shortened is,a(x; y) to is,a. Substituting ρ = 0 into Eqs. (B.61)

and (B.63), we have

dG (ρ, s(ρ), a(ρ))

dρ

∣∣∣∣ρ=0

=E[ea(X) (is,a − a(X)′)

]

E[ea(X)](B.64)

d2G (ρ, s(ρ), a(ρ))

dρ2

∣∣∣∣ρ=0

=E[ea(X)

(− (is,a − a(X)′)2 + 2i′s,a − a(X)

′′)]

E[ea(X)]

+

(dG (ρ, s(ρ), a(ρ))

dρ

∣∣∣∣ρ=0

)2

(B.65)

177

B.

From the constraint E[a(X)] = 0, we obtain

dE[a(X)]

dρ= E[a(X)′] = 0 (B.66)

d2E[a(X)]

dρ= E[a(X)

′′

] = 0. (B.67)

From Theorem 5.2.1, we have the optimal a(x) for x ∈ X satisfies Eq. (5.30)

near ρ = 0. Substituting Eqs. (5.30) and (B.67) into Eqs. (B.64) and (B.65), we

obtain

dG (ρ, s(ρ), a(ρ))

dρ

∣∣∣∣ρ=0

= E[is,a] (B.68)

d2G (ρ, s(ρ), a(ρ))

dρ2

∣∣∣∣ρ=0

= −(var(is,a) + E

[a(X)

′2 − 2is,aa(X)′ − 2i′s,a

]).

(B.69)

Next, we show E[i′s,a] = 0 when ρ = 0.

E[i′s,a] = E

[d(s logPY |X(Y |X)− logEX′

[ea(X

′)PY |X(Y |X ′)s|Y])

dρ

](B.70)

= E

[s′ logPY |X(Y |X)− EX′

[PY |X(Y |X ′)sea(X

′)(a(X ′)′ + s′ logPY |X(Y |X ′)

)|Y]

EX′

[ea(X

′)PY |X(Y |X ′)s|Y]

]

(B.71)

= E[s′ logPY |X(Y |X)

]− E[a(X)′ + s′ logPY |X(Y |X)] (B.72)

= 0 (B.73)

The optimal value of a(x)′ is given in Eq. (5.31) in Theorem 5.2.1. Substituting

Eqs. (5.31), (B.73), s = 11+ρ

, ρ = 0 and a(x) = 0 into Eq. (B.69), we obtain

d2G (ρ, s(ρ), a(ρ))

dρ2

∣∣∣∣ρ=0

=−(var(i(X ; Y )

)+ E

[(i(X)− I(PX))

2 − 2i(X ; Y ) (i(X)− I(PX))])

(B.74)

=−(var(i(X ; Y )

)− var

(i(X)

))(B.75)

=E[var(i(X ; Y )|X

)], (B.76)

which conclude the proof.

178



Let Ri ≤ Rj , we have

Ex0 (ρ, s)− ρRi ≥ Ex

0 (ρ, s)− ρRj , (B.77)

for fixed ρ ≥ 1 and s ≥ 0. Hence,

limRi→0

Ex0 (ρ, s)− ρRi ≥ Ex

0 (ρ, s)− ρRj . (B.78)

Take the supremum over all possible s, ρ and letting Rj → 0 yields

sups≥0,ρ≥1

limRi→0

Ex0 (ρ, s)− ρRi ≥ lim

Rj→0sup

s≥0,ρ≥1Ex

0 (ρ, s)− ρRj (B.79)

= limRi→0

sups≥0,ρ≥1

Ex0 (ρ, s)− ρRi. (B.80)

On the other hand, we have for all R

sups≥0,ρ≥1

Ex0 (ρ, s)− ρR ≥ Ex

0 (ρ, s)− ρR. (B.81)

Take the limit R → 0 and then the supremum over all possible s, ρ yields

limR→0

sups≥0,ρ≥1

Ex0 (ρ, s)− ρR ≥ sup

s≥0,ρ≥1limR→0

Ex0 (ρ, s)− ρR, (B.82)

together with Eq. (B.80) imply Eq. (5.84). Eq. (5.85) can be shown similarly.

179

B.

180

References

[1] E. Agrell and A. Alvarado. Optimal Alphabets and binary labelings

for BICM at low SNR. IEEE Trans. Inf. Theory, 57[10]:6650–6672, Oct.2011.

[2] E. Agrell and A. Alvarado. Signal Shaping for BICM at Low SNR. to

appear in IEEE Trans. Inf. Theory, 2012.

[3] E. Agrell, J. Lassing, E. G. Strom, and T. Ottoson. On the

optimality of the binary reflected Gray code. IEEE Trans. Inf. Theory,

50[12]:3170–3182, 2004.

[4] S. M. Alamouti. A Simple Transmit Diversity Technique for Wireless

Communications. IEEE J. Select. Areas Commun., 16[8]:1451–1458, Octo-

ber 1998.

[5] S. Arimoto. An Algorithm for Computing the Capacity of Arbitrary Dis-

crete Memoryless Channels. IEEE Trans. Inf. Theory, 18:14–20, Jan 1972.

[6] A. T. Asyhari and A. Guillen i Fabregas. Nearest Neighbor Decoding

in MIMO Block-Fading Channels with Imperfect CSIR. IEEE Trans. Inf.Theory, 58[3]:1483–1517, 2012.

[7] V. B. Balakirsky. A Converse Coding Theorem for Mismatched Decoding

at the Output of Binary-Input Memeoryless Channels. IEEE Trans. Inf.

Theory, 41[6]:1889–1902, 1995.

[8] A. Bennatan and D. Burshtein. Design and analysis of nonbinary

LDPC codes for arbitrary discrete-memoryless channels. IEEE Trans. Inf.

Theory, 52[2]:549–583, 2006.

[9] E. Biglieri, J. Proakis, and S. Shamai. Fading channels: information-

theoretic and communications aspects. IEEE Transactions on Information

Theory, 44[6]:2619–2692, 1998.

181

REFERENCES

[10] Ezio Biglieri, Giuseppe Caire, and Giorgio Taricco. Error prob-

ability over fading channels: a unified approach. Europ. Trans. Commun.,

January 1998.

[11] R. E. Blahut. Computation of Channel Capacity and Rate-Distortion

Functions. IEEE Trans. Inf. Theory, 18:460–473, Jul 1972.

[12] G. Bocherer, F. Altenbach, A. Alvarado, S. Corroy, and

R. Mathar. An Efficient Algorithm to Calculate BICM Capacity. 2012

IEEE International Symposium on Information Theory Cambridge, MA,

USA, Jul. 2012.

[13] J. J. Boutros, A. Guillen i Fabregas, E. Biglieri, and G. Zemor.

Low-Density Parity-Check Codes for Nonergodic Block-Fading Channels.

IEEE Trans. Inf. Theory, 56[9], September 2010.

[14] R. W. Butler. Saddlepoint Approximations with Applications. Cambridge

University Press, 2007.

[15] G. Caire and K. Narayanan. On the distortion SNR exponent of hybrid

digital-analog space-time coding. IEEE Trans. Inf. Theory, 53[8]:2867–2878,

Aug. 2007.

[16] G. Caire, G. Taricco, and E. Biglieri. Bit-interleaved coded modu-

lation. IEEE Trans. Inf. Theory, 44[3]:927–946, 1998.

[17] A. R. Calderbank and L. H. Ozarow. Nonequiprobable Signaling on

the Gaussian Channel. IEEE Trans. Inf. Theory, 36[4]:726–740, July 1990.

[18] A. Tauste Campo, G. Vazquez-Vilar, and A. Guillen i Fabregas.

Converse Bounds for Finite-Length Joint Source-Channel Coding. In 50th

Annual Allerton Conference on Communications, Control and Computing,

Allerton, IL, 10/2012.

[19] A. Chuang, A. Guillen i Fabregas, L. K. Rasmussen, and I. B.

Collings. Optimal SNR exponent for discrete-input MIMO ARQ block-

fading channels. 2007 IEEE International Symposium on Information The-

ory Nice, France, Jun. 2007.

[20] T. M. Cover. Broadcast channels. IEEE Trans. Inf. Theory, IT-18[1]:2–

14, Jan. 1972.

182

REFERENCES

[21] T. M. Cover and J. A. Thomas. Elements of information theory. WileyNew York, 2nd edition, 2006.

[22] I. Csiszar. On the Error Exponent of Source-Channel Transmission with

a Distortion Threshold. IEEE Trans. Inf. Theory, 28[6]:823–828, November1982.

[23] I. Csiszar and J. Korner. Information Theory: Coding Theorems forDiscrete Memoryless Systems. New York-San Francisco-London: Academic

Press, Inc.,, 2008.

[24] I. Csiszar and P. Narayan. Channel Capacity for a Given DecodingMetric. IEEE Trans. Inf. Theory, 45[1]:35–43, January 1995.

[25] R. F. Fischer. Precoding and Signal Shaping for Digital Transmission.

John Wiley & Sons, Inc. New York, NY, USA, 2002.

[26] G. J. Foschini and M. J. Gans. On Limits of Wireless Communications

in a Fading Environment when Using Multiple Antennas. Wireless PersonalCommunications, 6[3]:311–335, 1998.

[27] R. G. Gallager. Information Theory and Reliable Communication. John

Wiley & Sons, Inc. New York, NY, USA, 1968.

[28] A. Ganti, A. Lapidoth, and I. E. Telatar. Mismatched decoding

revisited: general alphabets, channels with memory, and the wideband limit.IEEE Trans. Inf. Theory, 46[7]:2315–2328, 2000.

[29] M. Gastpar, B. Rimoldi, and M. Vetterli. To code or not to code:

Lossy source channel communication revisited. IEEE Trans. Inf. Theory,49[5]:1147–1158, 2003.

[30] S. Y. Le Goff, B. S. Sharif, and S. A. Jimaa. A New Bit-InterleavedCoded Modulation Scheme Using Shaping Coding. In IEEE Global Commun.

Conf, Dallas, Tx, USA, Nov.–Dec., 2004.

[31] A. Guillen i Fabregas and G. Caire. Coded modulation in the block-fading channel: coding theorems and code construction. IEEE Trans on Inf.

Theory, 52[1]:91–114, Jan. 2006.

[32] A. Guillen i Fabregas, A. Martinez, and G. Caire. Error Probabil-

ity of Bit-Interleaved Coded Modulation using the Gaussian Approximation.

In Proceedings of the Conf. on Inform. Sciences and Systems (CISS-04),Princeton, USA, March 2004.

183

REFERENCES

[33] A. Guillen i Fabregas, A. Martinez, and G. Caire. Bit-InterleavedCoded Modulation, 5. Foundations and Trends on Communications and

Information Theory, Now Publishers, 2008.

[34] D. Gunduz and E. Erkip. Source and channel coding for cooperativerelaying. IEEE Trans. Inf. Theory, 53[10]:3454–3475, Oct. 2007.

[35] D. Gunduz and E. Erkip. Joint source-channel codes for MIMO block-

fading channels. IEEE Trans. Inf. Theory, 54[1]:116–134, Jan. 2008.

[36] T. S. Han. Information-Spectrum Methods in Information Theory. Springer,

2003. Applications of Mathematics, Stochastic Modeling and Applied Prob-ability.

[37] F. B. Hildebrand. Handbook of applicable mathematics. Vol 3: Numerical

Methods. John Wiley & Sons, 1981.

[38] B. Hochwald. Tradeoff between Source and Channel Coding on a Gaussian

Channel. IEEE Trans. Inf. Theory, 44[7]:3044–3055, November 1998.

[39] B. Hochwald and K. Zeger. Tradeoff between source and channel cod-ing. IEEE Trans. Inf. Theory, 43[5]:1412–1424, Sep. 1997.

[40] T. Holliday, A. J. Goldsmith, and H. V. Poor. Joint Source and

Channel Coding for MIMO Systems: Is it Better to be Robust or Quick.IEEE Trans. Inf. Theory, 54[4]:1393–1405, Apr. 2008.

[41] M. J. Hossain, A. Alvarado, and L Szczecinski. BICM Transmissionusing Non-Uniform QAM Constellations: Performance Analysis and Design.

In IEEE Int. Conf. Commun., Cape Town, South Africa, 2010.

[42] M. J. Hossain, A. Alvarado, and L. Szczecinski. Towards FullyOptimized BICM Transceivers. IEEE Trans. on Commun., 59[11]:3027–

3039, nov 2011.

[43] J. Hui. Fundamental Issues of Multiple Accessing. PhD thesis, MIT, 1983.

[44] A. Guillen i Fabregas and A. Martinez. Bit-Interleaved Coded Mod-

ulation with Shaping. In 2010 IEEE Inf. Theory Workshop, Dublin, Ireland,Sep. 2010.

[45] S. Ihara, M. Ku, and and. Error Exponent for coding of Stationary

Memoryless Sources with Fidelity Criterion. IEICE Trans. Fundam., E88-A[5]:1339–1345, October 2005.

184

REFERENCES

[46] S. Ihara and M. Kubo. Error Exponent for Coding of Memoryless Gaus-

sian sources with a Fidelity Criterion. IEICE Trans. Fundam., 83[10]:1891–

1897, October 2000.

[47] H. Imai and S. Hirakawa. A new multilevel coding method using error-

correcting codes. IEEE Trans. Inf. Theory, 23[3]:371–377, May 1977.

[48] F. Jelinek. Probabilistic Information Theory. McGraw-Hill, Inc. New York,

NY, USA, 1968.

[49] J. L. Jensen. Saddlepoint Approximations. Oxford University Press, USA,

1995.

[50] G. Kaplan and S. Shamai. Information rates and error exponents of com-

pound channels with application to antipodal signaling in a fading environ-

ment. AEU. Archiv fur Elektronik und Ubertragungstechnik, 47[4]:228–239,

1993.

[51] A. Kavcic, X. Ma, and M. Mitzenmacher. Binary intersymbol inter-

ference channels: Gallager codes, density evolution, and code performance

bounds. IEEE Trans. Inf. Theory, 49[7]:1636–1652, 2003.

[52] J. C. Kieffer. Strong Converses in Source Coding Relative to a Fidelity

Criterion. IEEE Trans. Inf. Theory, 37[2]:257–262, March 1991.

[53] R. Knopp and P. A. Humblet. On coding for block fading channels.

IEEE Trans. Inf. Theory, 46[1]:189–205, 2000.

[54] V. Kostina and S. Verdu. Fixed-Length Lossy Compression in the Finite

Blocklength Regime. IEEE Trans. Inf. Theory, 58[6]:3309–3338, June 2012.

[55] F. R. Kschischang and Pasupathy S. Optimal Nonuniform Signaling

for Gaussian Channels. IEEE Trans. Inf. Theory, 39:913–929, 1993.

[56] J. N. Laneman, E. Martinian, and G. W. Wornell. Comparing

Application- and Physical-Layer Approaches to Diversity on Wireless Chan-

nels. Communications, 2003. ICC ’03. IEEE International Conference on,

4:2678–2682, May 2003.

[57] J. N. Laneman, E. Martinian, G. W. Wornell, and J. G. Apos-

tolopoulos. Source-Channel Diversity for Parallel Channels. IEEE Trans.

Inf. Theory, 51[10]:3518–3539, Oct. 2005.

185

REFERENCES

[58] A. Lapidoth. Mismatched Decoding and the Multiple-Access Channel.

IEEE Trans on Inf. Theory, 42[5]:1439–1452, September 1996.

[59] Y. Liang. Separation Optimality and Generalized Source-Channel Coding

for Time-varying Channels. PhD thesis, Stanford University, Standford,

USA, 2008.

[60] Y. Liang, A. Goldsmith, and M. Effros. Distortion Metrics of Com-

posite Channels with Receiver Side Information. In IEEE Inf. Theory Work-

shop, Lake Tahoe, CA, September 2–6, 2007.

[61] R. Lugannani and S. O. Rice. Saddle Point Approximation for the

Distribution of the Sum of Independent Random Variables. Adv. Appl. Prob.,

12:475–490, 1980.

[62] E. Malkamaki and H. Leib. Coded diversity on block-fading channels.

IEEE Trans. Inf. Theory, 45[2]:771–781, 1999.

[63] A. Martinez. On the Channel Dispersion for Memoryless Channels. Notes,

May 2012.

[64] A. Martinez, A. Guillen i Fabregas, and G. Caire. Error probabil-

ity analysis of bit-interleaved coded modulation. IEEE Trans. Inf. Theory,

52[1]:262–271, Jan. 2006.

[65] A. Martinez, A. Guillen i Fabregas, and G. Caire. A Closed-Form

Approximation for the Error Probability of BPSK Fading Channels. IEEE

Trans. Wireless Commun., 6[6]:2051–2056, Jun. 2007.

[66] A. Martinez, A. Guillen i Fabregas, G. Caire, and F. Willems.

Bit-interleaved coded modulation in the wideband regime. IEEE Trans. Inf.

Theory, 54[12]:5447–5455, Dec. 2008.

[67] A. Martinez, A. Guillen i Fabregas, G. Caire, and F. Willems.

Bit-Interleaved Coded Modulation Revisited: A Mismatched Decoding Per-

spective. IEEE Trans. Inf. Theory, 55[6]:2756–2765, Jun. 2009.

[68] A. Martinez and A. Guillen i Fabregas. Large-SNR Error Probability

Analysis of BICM with Uniform Interleaving in Fading Channels. IEEE

Trans. Wireless Commun., 8[8]:38–44, Jan. 2009.

[69] K. Marton. Error Exponent for Source Coding with a Fidelity Criterion.

IEEE Trans. Inf. Theory, 20[2]:197–199, March 1974.

186

REFERENCES

[70] N. Merhav, G. Kaplan, A. Lapidoth, and S. Shamai (Shitz).On information rates for mismatched decoders. IEEE Trans. Inf. Theory,

40[6]:1953–1967, Nov. 1994.

[71] U. Mittal and N. Phamdo. Hybrid digital-analog (HDA) joint source-channel codes for broadcasting and robust communications. IEEE Trans.

Inf. Theory, 48[5]:1082–1102, May. 2002.

[72] K. D. Nguyen, A. Guillen i Fabregas, and L. K. Rasmussen.Power allocation for block-fading channels with arbitrary input constella-

tions. IEEE Trans. Inf. Theory, 8[5]:2514–2523, May 2009.

[73] K. D. Nguyen, L. K. Rasmussen, A. Guillen i Fabregas, andN. Letzepis. MIMO ARQ with multilevel feedback: outage analysis. sub-

mitted to IEEE Trans. Inf. Theory, May 2009.

[74] K. D. Nguyen, L. K. Rasmussen, A. Guillen i Fabregas, andN. Letzepis. MIMO ARQ with Multibit Feedback: Outage Analysis. IEEE

Trans. Inf. Theory, 58[2]:765–779, February 2012.

[75] T. T. Nguyen and L. Lampe. Bit-Interleaved Coded Modulation withMismatched Decoding Metrics. IEEE Trans. Commun., 59[2]:437–447,

Feburary 2011.

[76] L. H. Ozarow, S. Shamai, and A. D. Wyner. Information theoreticconsiderations for cellular mobile radio. IEEE Transactions on Vehicular

Technology, 43[2]:359–378, 1994.

[77] Y. Polyanskiy, H. V. Poor, and S. Verdu. Channel Coding Rate inthe Finite Blocklength Regime. IEEE Trans. Inf. Theory, 56[5]:2307–2359,

May 2010.

[78] D. Raphaeli and A. Guervitz. Constellation Shaping for PragmaticTurbo-Coded Modulation with High Spectral Efficiency. IEEE Trans. on

Commun., 52[3]:341–345, mar 2004.

[79] M. Rezaeian and A. Grant. Computationo of Total Capacity forDiscrete Memoryless Multiple-Access Channels. IEEE Trans. Inf. Theory,

50[11]:2779–2784, nov 2004.

[80] J. Scarlett, A. Martinez, and A. Guillen i Fabregas. Ensemble-

Tight Error Exponents for Mismatched Decoders. In Fiftieth Annual Aller-

ton Conference on Communication, Control, and Computing, Allerton, IL,10/2012, 2012.

187

REFERENCES

[81] C. E. Shannon. A Mathematical Theory of Communication. Bell SystemTechnical Journal, 27:379–423 and 623–656, 1948.

[82] S. Shamai (Shitz). A Broadcast Stretegy for the Gaussian Slowly Fading

Channel. In IEEE Int. Symp. Inf. Theory, Ulm, Germany, June, 1997.

[83] S. Shamai (Shitz) and I. Sason. Variations on the Gallager Bounds,

Connections, and Applications. IEEE Trans. Inf. Theory, 48[12]:3029–3051,

Dec. 2002.

[84] V. Tarokh, N. Seshadri, and AR Calderbank. Space-time codes

for high data rate wireless communication: performance criterion and codeconstruction. IEEE Trans. Inf. Theory, 44[2]:744–765, 1998.

[85] I. E. Telatar. Capacity of multi-antenna Gaussian channels. European

Transactions on Telecommunications, 10[6]:585–595, 1999.

[86] G. Ungerbock. Channel Coding With Multilevel/Phase Signals. IEEE

Trans. Inf. Theory, 28[1]:55–66, 1982.

[87] N. Varnica, X. Ma, and A. Kavcic. Capacity of Power ConstrainedMemoryless AWGN Channels with Fixed Input Constellations. In Global

Telecommunications Conference, 2002.GLOBECOM, Taipei, Taiwan, Nov.2002.

[88] S. Vembu, S. Verdu, and Y. Steinberg. The source-channel separation

theorem revisited. IEEE Trans. Inf. Theory, 41[1]:44–54, Jan. 1995.

[89] S. Verdu. Spectral efficiency in the wideband regime. IEEE Trans. Inf.

Theory, 48[6]:1319–1343, Jun. 2002.

[90] S. Verdu and T. S. Han. A general formula for channel capacity. IEEETrans. Inf. Theory, 40[4]:1147–1157, 1994.

[91] A. J. Viterbi and J. K. Omura. Principles of Digital Communicationand Coding. McGraw-Hill, Inc. New York, NY, USA, 1979.

[92] U. Wachsmann, R. F. H. Fischer, and J. B. Huber. Multilevel codes:

theoretical concepts and practical design rules. IEEE Trans. Inf. Theory,45[5]:1361–1391, Jul. 1999.

[93] Y. Watanabe and K. Kamoi. A Formulation of the Channel Capacity of

Multiple-Access Channel. IEEE Trans. Inf. Theory, 55[5]:2083–2096, May2009.

188

REFERENCES

[94] R. W. Yeung. Information Theory and Network Coding. InformationTechnology: Transmission, Processing and Storage, Springer, 2008.

[95] A. L. Yuille and A. Rangarajan. The Concave-convex Procedure.

Neural Computation, 15:915–936, 2003.

[96] E. Zehavi. 8-PSK trellis codes for a Rayleigh channel. IEEE Trans. Com-

mun., 40[5]:873–884, 1992.

[97] L. Zheng and D. N. C. Tse. Diversity and multiplexing: a fundamental

tradeoff in multiple-antenna channels. IEEE Transactions on InformationTheory, 49[5]:1073–1096, 2003.

[98] Y. Zhong, F. Alajaji, and L. L. Campbell. A Type Covering Lemmaand the Excess Distortion Exponent for Coding Memoryless Laplacian

Sources. In 23rd Bienn. Symp. Communications, Kingston, ON, Cananda,Jun., 2006.

[99] Y. Zhong, F. Alajaji, and L. L. Campbell. Joint Source-Channel Cod-ing Excess Distortion Exponent for Some Memoryless Continous-Alphabet

Systems. IEEE Trans. Inf. Theory, 55[3]:1296–1319, March 2009.

189

Date post:	22-Jan-2021
Category:	Documents
Upload:	others
View:	7 times
Download:	0 times

Fundamentals of Bit-Interleaved Coded Modulation and Reliable Source ... - Home |...

Documents