© 1981 IEEE. Personal use of this material is permitted. However,
permission to reprint/republish this material for advertising or promotional
purposes or for creating new collective works for resale or redistribution to
servers or lists, or to reuse any copyrighted component of this work in
other works must be obtained from the IEEE.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-27, NO. 3, MAY 1981 299
Source Coding Theory for Cascade and Branching Communication Systems
HIROSUKE YAMAMOTO, MEMBER, IEEE
Alistrucr- A source coding problem is considered for cascade and branching communication systems. The achievable rate region is estab- lished for the cascade systems and bounds are obtained for the branching systems. Some examples are also included.
I. INTRODUCTIONAND INFORMALSTATEMENTOF THEPROBLEM
L ET US consider a cascade communication system (Fig. 1). The source output {(X,, Y,)}z=, is a se-
quence of independent drawings of a pair of random variables X E % and YE 3, where the source alphabets % and 9 are either discrete sets, the reals, or arbitrary mea- surable spaces. Receiver 0 is interested in obtaining the reproduction { gk} of the sequence {X,}, and receiver 1 is interested in obtaining the reproduction {$} of the se- quence {Y,}. The system has two channels. One is the “common trunk channel” channel 0, and the other is the “private branch channel” channel 1. Decoder 0 connects the former to the latter. The aim of this paper is to determine how much of the transmission rates in two channels, R, and R,, are required to produce { &} and {rk} within a prescribed distortion tolerance at each re- ceiver.
Obviously we can attain the prescribed distortion toler- ances d, and d-,,, provided R, 2 R,( d,) + R,( d,) and R, 2 R,( d,), where R(d) is the rate-distortion function for the communication system with one source, one channel, and one receiver [l]. However if X and Y are correlated, it may be possible to attain the distortion tolerances d, and dy with R, less than R,(d,) + RY(fy). On the other hand, if R, is equal to R&d,, d,), then it may be impossible to realize the distortion dy with R, = R,(d,). This fact can be explained schematically in Fig. 2. The ellipses in the figure represent conceptually the optimal enRxy(dx,dJ divisions of %” X 9” space. All sequences xy in each ellipse’ are mapped onto the same codeword j?jj that is recovered at the first decoder to attain the distortion tolerances d, and dy. The numbering represents the possibly optimal groupings of the ellipses to attain d, at the second decoder. All
Manuscript received October 25, 1979. This paper was presented at the 2nd Symposium on Information Theory and Its Applications, Kyoto, Japan, November 29-30, 1979.
The author was with the Institute of Space and Aeronautical Science, University of Tokyo, Tokyo, Japan. He is now with the Department of Electronic Engineering, Tokushima University, 2- 1, Minami-josanjimacho, Tokushima-shi, Japan-770.
‘Bold faced letters denote n-tuple vectors, and lower case letters repre- sent realizations of random variables (or vectors).
SO”ICC ‘(xksyk)’ Encoder iY,l
Receiver 1
Receiver 0
Fig. 1. Cascade communication system
n - / - % -
Fig. 2. Quantization of xn X “?i” space.
codewords @ in each group with the same number are now mapped onto another codeword 9’ which is recovered at the second decoder. Let the total number of the groups be M,*. This situation corresponds to transmitting at the rates R, = R,,(d,, d,) and R, = (log MT)/n as small as possi- ble to attain the distortion tolerance d,. On the other hand, the broken lines represent the optimal M,O = enRYcdy) divi- sion of 9” space that attain the distortion tolerance dy. Mf‘ should be greater than MF, which means R, > R,(d,).
The region of achievable rates for the system in Fig. 1 is obtained in the following sections. In Section II-A, a formal and precise statement of the problem and the results are given. Section II-B contains some examples. In Section III some extensions are investigated. Section IV contains the proofs of all the theorms and some discussions about the relation between the cascade communication system and the system studied by Gray and Wyner [2], [3].
II, FORMALSTATEMENTOFTHEPROBLEM,RESULTS, AND EXAMPLES-
A. Formal Statement of the Problem and Results
Let W,J’,J& b e a sequence of independent draw- ings of a pair of dependent random variables X and Y
00 18-9448/g l/0500-0299$00.75 0 198 I IEEE
300 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-27, NO. 3, MAY 1981
taking values in sets % and %l, respectively. It is assumed We also define two explicit bounds on %*(d,, d,), i.e. that we are given a probability law defining (X, Y). If % an innerbound si(d,, dy) and an outerbound %‘(d,, d,) and 9 are discrete sets, then we can write as follows:
Q(x,y)=Pr{X=x,Y=y}, xE%, YE%. ~(d,,d,)~co[{(R,,R,):R,~R,(d,)+R,(d,),
If % and %l are the reals, then (X, Y) may be defined by a w%4d,)~ probability density function Q(x, y), - cc < x, y -=c cc. For arbitrary measurable spaces % and 9, the pair (X, Y) is ” {owe &Jw,,(d,~d,)}]> (9) characterized by a probability measure Q on x.X 9.
A code (FE,F’) with parameters (n,M,,M,,A,,A,) is %“(d,,d,)& {(R,,R,): Ra)R,,(d,,d,), defined by a pair of mappings:
FE: %“X?l”-IMO, (14 ~,%4d,)j, (10)
where co[ & u $81 denotes the convex hull of sets @ and ‘%. FD : ho -{fi}~l~ {~j}~,&Wk (lb) We now state the main results.
where IMO= {O,l; * ., Ma - l} and % and ‘% are reproduc- Theorem 1: ing alphabets. Letting X=(X,;+.,X,) and Y= ~(d,,d,)=~*(d,,d,). (11) (Y,; . *> Y,), then (X,3) = Fo( F&X, Y)). The average dis- Theorem 2: tortions of the code, A\, and Ay, are given by
A,=$ i D&g%),
~(d,,d,) c~(d,,d,). (12)
(24 k=l
a) Equation (12) holds with equality either if dy = 0 or if X and Y are independent.
Ay=E;k$,nY(yk,‘k)~
b) Equation (12) does not hold with equality if for some (2b) p(x, y, R, p) E 9(d,, d,), I(XY; XY) is equal to
where 2, E $6 and Yk E % are the kih coordinates of X? Rx&d,, d,,) and at the same time2,3
and ,?, respectively, and D,: %X %* [0, cc) and D,: I(X;PlY)>O. (13)
‘%I X 9 + [0, cc) are single-letter distortion functions. Theorem 3: A rate-pair (R,, R,) is said to be (d,, d,)-achievable if,
for any E > 0 and n sufficiently large, there exists a code ~(d,,d,)2@(d,,d,). (14)
(FE, Fo) with parameters (n, M,, M,, A,, Av) such that Equation (14) holds with equality if and only if X and Y Mf < en(Rl+r), l=O,l, (3) are independent.3
Fig. 3 represents typical inclusion relations of %( d,, d,), A,<d,+e, AyGdy+e. (4) %“(d,, d,), and %(d,, d,,). In the figure, R,,fO(d,) where
Let us define %*(dh., d,) as the set of all (d,, d,)- the random variable Y0 is such that I( XY; YO) = I( Y; Y,) =
achievable rate-pairs. The main problem in this paper is to R,( dy ) holds, is the conditional rate-distortion function
determine ‘%*( d,, d,). [2], [4], and R,(d,, d,) is the infimum rate of R, required at
Now define C?(d,, dy) as the family of probability distc- R, = R&d,, d,). It is postponed to section IV to prove
butions p(x, y, Z,?) such that for x E xx, y E 3, R E xx, that the R, coordinate of point D is given by R,,;Jd,) +
JW-Q, RY( dy) as well as to prove the theorems stated above. In the next subsection, we calculate R&d, d), R,(d),
~(x,~,~,~)=p,(~,Plx,y>Q<x,r), (5) R,(d, 4, R,(d) + R,(d), and R,,fJd) + R,(d) for two special sources.
ED,(X,k)Gd,, ED,(Y,?)Gd,, (6)
where pt(Z, 91x, y) is a test channel and the random varia- B. Examples
bles X, Y, X, Y are characterized by p(x, y, 2, 9). I) Doubly Symmetric Binary Source: As the first exam- Define the subset of two dimensional real space corre- ple, let us consider the doubly symmetric bina? source
spending to p(x, y, 2, P> E %d,, d,), (DSBS) characterized by the alphabets %= 9 = %= 3 =
%,,,” {(R,,R,): R,>I(XY;@),R,>I(XY;f)}, {O,l} and
(7) Q(x,y)=~(l-~)~~,,+~~(l-6,,~), (15)
and where x, y = 0 or 1, and 0 up G l/2. Let the distortion
~(dx,dy)k 1
u qp( ) ‘3 I
(8) 2Ekcept for some peculiar cases, this equation may be satisfied whenever PC )E%d,,dy) X and Y are dependent and d, > 0.
where [ 1’ stands for the closure of the set [ 1. 3d, and d, are of course assumed such that R x( d,) > 0 and R y( dy) > 0,
respectively.
YAMAMOTO: CASCADE AND BRANCHING COh5MUNICATION SYSTEMS 301
RXy(dx,dr) -------A
Rl(dx,dy) -------g
Fig. 3. Typical inclusion relations of ai(d,, d,), %(d,, d,), and A”(d,, dy).
measure be
TABLE I MPlX,Y)l
O$dip 9
P <d<+-
Y 0 1
x
01+ +
1:
00 1 Bz 1+62 1+5’
11 -CL 1+g2 1+g2
lo+ +
and X - Y - Y0 forms a Markov chain. Hence from (15) and (21),
(16) Making use of [ 1, ex. 2.7.1 and 2.7.21, we obtain
Pr{X=x]Y0=y}=(1-d*p)8~,i+(d*p)(l-8~,r),
R,(d)=R,(d)=l-h(d), O&d+, (17) O<d<;, (22)
Rx&Cd)=
where
1 +h(p)-2h(d), Osdsp,,
L(1 -p)-;[L(Zd-p)
+L(2(1 -d) -p>],
where d*p = (1 - d)p + d(1 -p). Then from [4, ex. 11, we obtain
Rx,$dd)=h(d*d-h(d), O<d+ (23)
p,<dd 2’ (18) R.&d, 4, RAd), R,(d, d>, R,(d) + R,(d), and
R,,qdd) + R,(d) are depicted in Fig. 4 forp = 0.25. 2) Jointly Gaussian Source: Out second example is the
jointly Gaussian source where %, %, ‘%, ?4 are the reals and
PO=;(l-m), whose density function is given by
(194
Q(x,Y)= ’ (x2 +Y 2-2YxY)
h(a)= -uloga-(I-cr)log(l-a), (19b) L(a) = -crloga. (19c)
27r( 1 - y2y2 exp -
1 1 2(1-y2) ’
Now from [l, ex. 2.7.21, we can show that the random 0~y-c 1. (24)
variable Y that satisfies 1( Xr; XY) = R&d, d) is de- Let us employ the squared-error distortion measure scribed by p( 3) x, y) in Table I. Thus Dx(x,i)= (x-a)“,
I l+h(p)-h(d)+ &[L(-d) D,(Y, 9) = (Y -@ -w-cx,~,y,~<w. (25)
-L(p-d)-L(1 -p-d)], From [l, th. 4.3.21 we have
OgdGp,, R,(d> d) = <
~(1 -p) +(2d-p)
+I@(1 -d) -p)], 1
p,<d<-. 2 (20)
R,(d)=R,(d)= 2 i
L1og;) O<d< 1, (26)
0, 1 <d.
Similarly it can be shown that
l--y2 +og-
d2 ’ O<d<l-y,
On the other hand, the random variable Ya which satisfies R,,(W) = 1
I
l+Y I( XY, YO) = I(Y; pa) = R,(d), is such that [l, ex. 2.7.21 pg 2d- l+y’
1-y<d<l,
Pr{Y=y]Y0=~}=(1-d)6,,F+d(l-86,,E), 0, 1 Gd.
Osd+ (21) (27)
Furthermore, it can be shown that the random variable Y
302 IEEE TRANSACTIONS ON INFORhfATlON THEORY, VOL. IT-27, NO. 3, MAY 1981
Fig. 4. R,(d), R
2.0
I-
p=o.zs
1.6
0.6
0.4
0.2
0
Ry(dl k
0.1 p'o.2 0.3 0.4 0.5
d
xy(4 4, R&d, 4, R,(d) + R,(d), and R,(d) for the DSBS withp = 0.25.
Rx+,,4 + Fig. 5.
1.6
1.4
=‘ 0 1.0
“,
x 0.8 .:: P = 0.6
1 -Y 0.2 0.4 0.6 0.8 1.0
d
R,(d), R..(d,d), R,(d,d), R,(d) + R,(d), and R&d)+ R y(d) for the jointly Gaussian source with y = 0.5.
which satisfies I( XY, XY) = R&d, d) is characterized by On the other hand the random variable Ya is such that
& exp - 1
(PHP2x+&Y))2
P4 I
[l, th. 4.3.21
’ p(YlP)= l -exp l/a
-$(Y-9)’ , 1 OcdG 1,
O<d<l-y,
1 +Y> (31)
and X- Y - Y0 forms a Markov chain. Hence, from (24)
where
I { (P-tP3(X+Y))2 and (31) -
eexp - P&d- 1 +u> ’ 1
I-y<dGl, WE) =
(28) \/2?r{l -y2(1 -d)}
i 1
‘exp - 2{1 -y2(1 -d)} (x -h2
P,=2( +-4)~ (294 O<d<l.
p2=2yd w4 Then, from [4, ex. 21, we obtain
l-9 r
(32)
P3 = 2(1 -d)
1+y * (294
1 -y2(1 -d) +g d 3 O<dGl> (33)
0, l&d. Thus
In Fig. 5, Rdd, 4, R,(d), R,(d, d), R,(d) + R,(d),
O<d<l-y, and R,,gJd) + R,(d) are shown for y = 0.5. Fig. 4 and Fig. 5 are seen to have similar characteristics. Provided R,= R,,(d,d), R, must be as large as R,,(d,d) to
1-y<d<l, achieve the distortion d if d>p, for the DSBS or if
dal. d 2 1 - y for the joint Gaussian source. If d <p. or if d < 1 - y, R, need not be as large as R,,( d, d ), but must
(30) be larger than R,(d).
YAMAMOTO: CASCADE AND BRANCHING COMMUNICATION SYSTEMS 303
Receiver 0 Receiver 1 Receiver K-l
Fig. 6. Kth order cascade communication system.
III. EXTENSIONS
A. Multistage Cascade Communication System
As the first extension of Fig. 1 let us consider a Kth order cascade communication system (Fig. 6). The source generates K-dimensional vector X = (X0, X,, . . . , X,- ,) according to the probability distribution Q(x) indepen- dently every unit time. The k th receiver is interested in obtaining the reproduction of the kth component of X within a distortion tolerance d,.
The d= (d,; .., d,-,)-achievable region of this system, %*k(d), is similarly obtained as was done for the system of Fig. 1. Let 9k(d) be defined as the set of probability distributions ~k( x, 2) such that
PK(XO,...,XK-,,~O,...,RK-l)
=~~K(~~,..‘.,~~-,Ix~,...,x~-~I>Q(x~,...,x~-~~),
(344
W(X,,~+L OGk<K-- 1, (34b)
where D,: %k X %k * [0, co) is the per-letter distortion function for the k th component of X. Corresponding to p”( x, 2) E TK(d), we introduce a subset of K-dimensional real space defined by
and let
The following theorem is obtained.
Theorem 4:
cR*yd) = ev(d).
(35)
(36)
(37)
B. Bidirectional Branching Communication System
The second extension is a bidirectional branching com- munication system shown in Fig. 7. It is the same as the system of Fig. 1 except that the reproduction X of X is
~,~~‘T;~~~~ 1; Fig. 7. Bidirectional branching communication system.
obtained at decoder 2 which is connected with decoder 0 by way of the second private channel, channel 2.
Let the (d,, d,)-achievable region of this system be denoted by ‘%it*,(d,, d,). We can obtain two bounds on %z(dx, d,), namely the outerbound %~(d,, d,) and the innerbound %g(d,, dy). Corresponding to p( ) E ??(d,, d,), we define
““p( )A {(R,,R,,R,): R,qXY; H),
R,W(XY;P), (38)
R,W(XY; i)}
and
a;( )k co[{(R,,R,,R,):R,,R,rl(XY;RP),
R,r,I(XY;P)}
u {(Ro,R,,R,): R,,R,N(XY; H),
R,BI(XY; R)}].
Let
and the following theorem holds.
Theorem 5:
(39)
(40)
(41)
(42)
C. Multistage Branching Communication System
Finally, we consider the arbitrary multistage branching communication system illustrated in Fig. 8. Bounds on the achievable region C%&(d) for this system can be obtained in a similar way as for the system of Fig. 7. The outer-
304
Fig. 8.
bound obtained c,an be expressed as follows:
Multistage branching communication system.
(43)
l=O,l,***, L- l), W)
where L is the total number of the channels and X,,, . . . , X,, are the random variables corresponding to the outputs of all the decoders connected with the encoder through the 1 th channel.
The innerbound obtained can be expressed as follows:
%&Kc )4 co u ((R,,&-&-I): 8
where all X! represent the random variables corresponding to the outputs of the terminal decoders from which no more branching takes place. Xt is connected with the encoder via channels t,, t i,. . . , tJ, and decoders t,, t,; . * . , fJ,. (See Fig. 9.) If decoder tj has outgoing branches other than channel tj+ , that develop subtrees consisting of channels tj17 tj2,' ",tjsj9 and decoders tj,, tj2; . *, tjsj, then Xtj and
rit,, s= l,***, ,’ s. in (46) represent the random variables corresponding to the outputs of decoder tj and decoder tjs,
s=l,*** ,sj, respectively. If decoder tj has no branch other than tj+l, the inequality in (46) should be read as Rtj > 1(X,X, * * * XK- 1; xtj 1.
Theorem 6:
c?&Jd) c CR&(d) c CR&(d). (47)
Fig. 9. Part of a multistage branching communication system.
IV. PROOFS AND COMPARISON WITH THE GRAY-WYNER SYSTEM
A. Proof of Theorem I
Lemma 1 (Converse Part of Theorem 1):
~(d,,d,)z~*(d,J,). (48)
Proof: For any (R,, R,) E %*(d,, d,,) and t: > 0, there exists a code (FE, FD) with parameters (n, M,,, M,, An, A,,) which satisfy (3) and (4). If we can prove that
;W%,$O4 E%(A,,A,), 1 (49)
then from (3) and the monotonicity of ‘%(d,, d,), we obtain
(RO+~,R,+c)E~(A,,Ay)~%(dx+c,dy+c).
(50)
Letting E --) 0 and invoking the continuity of %(d,, d,) (Appendix I), we have (R,, R,) E %(d,, d,). This estab- lishes Lemma 1.
Now let us prove (49). For (2, Y) = I;b( F’( X, Y)), the following inequalities are easily proved in the same way as the proof of [l, th. 3.2.21:
i I( vi; x&J, (51) k=l
ilog& 21 n k&wk; fi). (52)
Let us define
h,k~ED,(Xk,~k), AykgE&-(y,,&). (53)
YAMAMOTO: CASCADE AND BRANCHING COMMUNICATION SYSTEMS
Then from (2),
Let the probability distribution of the k th coordinates of XYJ?f, i.e. XkYkgkYk, be denoted by pk(x, y, $9). Then obviously pk(x, y, 2’,9) E $?(Axk, Ayk). From a straightfor- ward extension of Lemma A of Appendix I, it is possible to define a set of random variables (X, Y, 2, Y) characterized by a probability distribution which belongs to $?(A,, A,) and satisfies the following inequalities:
; kilI(xkYk; ~krk)w(xY; if), (55)
~k~lr(x,r,;~k)rl(xY;~). (56)
From (51), (52), (55), and (56), equation (49) is established.
Lemma 2 (Direct Part of Theorem 1):
a*(d,,d,) 2 qdx,dy) (57)
Pcoot Let %,$l,%,% be finite. For the case where %,%,%,‘?J are the reals or other measurable spaces, the lemma can be proved by using the proof for the finite case and the technique of quantization of %, 9, %, ?!J under some appropriate assumptions on the source and the dis- tortion functions D,, D, as in [5]. Hence, in this paper we consider the case where %, 9, %, % are finite.
Let p(x, y, &9) E y(d,, d,) and define
forallxy@E%X%X%Xx , (58) I
where 6>0, J%[ is the cardinarity of % and n(z]z) stands for the number of indices k, 1 G k G n, such that the k th component of Z, i.e., zk, equals z. T,,;(6), T,,(6), etc. are similarly defined corresponding to the marginal distribu- tions p( x, y, p), Q(x, y), etc. Furthermore, we introduce
T,,;(S) k { 3: v.9 E T,,;(6)) (5%
and T xyif(8), etc., defined similarly. From the definition of T * * * (S), it is easy to see that if XJ@ E Txri;( S), then x2 E T&S), yjj E TY;( S), and so on.
For any xy@ E T,,,-;(6), the distortion is follows:
bounded as
305
where d,,, = rnaxxx^ D,( x, 2) < cc. Similarly
~k~~D~(Yk.?,)~d,+sd,,,,, w-w
where d,,, = max,$ D,( y, p) < co. Hence ihe ayerage distortions A, and A, of a code xx” X 3” --f %’ X ‘?J’ are bounded as follows:
A,=E; i D,(X,,ik) k=l
A,Gd,+ (s+Pr[x@$@ T,&-;(6)])d,,,. (61b)
Therefore, if we can show that for any 6, 6 > 0 there exists at least one code which satisfies
Pr[xy.?$ @ T,,,-;(6)] Gc, (62)
; log M, G(XY; SB)+q (63)
~logJ4*~I(xY;P)+c, (64)
the proof will be completed. Let {i;}:, and {.?{}F,,j= l;..,M,,M,$ =M,/M,, be
the set of codewords. We define the operation of the encoder and the decoders as follows. When the encoder has received xy, it first selects ij from { gj},2, such that gj E T,,;(6), next selects ii from {@}F, such that nij E T,,-&$S), and it sends (i, j) to decoder 0. Decoder 0 generates # and sends j to decoder 1. Decoder 1 generates 4. In the above coding process, the encoding failure event E,,: XY+~ 6i! Txui;( 8) occurs whenever any of the follow- ing events occur:
E,:x~Ww(~h .
E,:.?$T,,;(S), for all j,
E,: 2: @ T,,+.(6), for all i.
Hence the probability of the error event is
Pr[E,]=Pr I I
6 E, /=I
<Pr[E,]+Pr[E,IEE] +Pr[E,IEtE,‘], (65)
where EC represents the complement of the event E. We now consider a random code ensemble. The code in
this ensemble is such that each j$ is chosen equiprobably from the set T;(6) with replacement and i{ is also chosen equiprobably from the set Tig(6) with replacement.
For this ensemble, three terms of the right hand of (65) will be evaluated. From Lemma A in Appendix II,
Pr[E,] G o(n), (66)
where O(n) 3 0 as n -+ cc. From Lemma C of Appendix II,
pr[ E2) E;] < (1 - e --n[l(XY; f)+Q(s)l)Mt, (67)
where z2( S) > 0 and E*( 8) --) 0 as 6 --) 0. Using (1 - a)K G
306 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-27, NO. 3, MAY 1981
e-Ka 2
Pr[E,]E,‘] <e --M,exp(--n[l(XY;~)+Z*(b)]) (68)
If we define A
M, = e”wy;y)+2~2@)1 9 (6%
then
Pr[ E2) E,“] < e -exPnc2(S) -+ 0 asn-, co. (70)
From Lemma D of Appendix II,
p,[E,IE,‘E,‘] ~(1 -,-n[r(xr;~I~)+t~(S)l)~~
< e --MbexPc--n[eXY; ~16+~3(s)1~, (71)
where ~~(6) > 0 and es(S) -+ 0 as 6 --) 0. If we define ,. *
q = en[l(XY; xly)+2Q(Ql , (72)
then
Pr[E,] EfEt] <e-exp"Z3(*)+ 0, asn-, cc. (73)
Hence for any given e > 0, if we choose 6 > 0 sufficiently small and n sufficiently large, we obtain
Pr[xy.?/q?, @ Tx+(8)] GE, (74)
ilog Ma ? +ogiU; + ilog M,
qxY;RIP)+I(XY;P)+~
=I(XY; RP)+c, (75)
ilog M, qxY;P)+c. (76)
Then in this code ensemble there exists at least one code which satisfies (74)-(76).
B. Relation Between the Onestage Cascade Communication System and the Gray-Wyner System [2], [ 314
The proof of Lemma 2 reveals that the achievable rate region for the onestage cascade communication system can be attained by the configuration in Fig. 10 where Rb = (logMA)/n and R,= Rb + R,. The system without the broken-line square is a special case of the system studied by Gray and Wyner [2], [3], and the code used in the proof of Lemma 2 is essentially the same as the code used in [2, sec. 3.21. We can prove Lemma 2 briefly by using [2, th. 81. It is impossible, however, to prove the corresponding lemma for the multistage cascade communication system by direct extension of [2, th. 81. It is straightforward to extend our proof of Lemma 2 to the case of multistage system.
Let %‘(d,, d,) be the set’of all (d,, d,)-achiev_able rate- pairs (Rb, R,) for the system in Fig. 10 and let %(d,, d,) be the lower boundary of %*(d,, d,). Obviously, if (Rb, R,) E s’(d,, d,) then (Rb + R,, R,) E %*(d,, d,,). If CR,, RI) E %t(d,, d,), then (R, -RI, R,) E a’(&, d,,) be-
4A close relationship between the one-stage cascade communication system and the Gray-Wyner system has been pointed out by R. M. Gray.
Fig. IO. Gray-Wyner system.
cause the systemdn Fig. lo-can attain ‘$*(d,, d,). From [2, th. 81, we have (Rxl~,$d,),Ry(dyN E%‘(d,,d,J and (R,, R,(d,)) 4 %‘(d,, d,,) for R, < R,,g$d,). Hence, from the above mentioned relations, the R, coordinate of point D in Fig. 3 should be Rx,fJd,) + R,(d,,).
C. Proof of Theorem 2
For arbitrary (R,, R,) E %(d,, d,), there exists such p(x, y, &y^) E ‘??(d,, d,,) that satisfies R, 2 I( XY; RP), R, 2 I( XY, Y). Then
R,,I(XY; RP)>RXY(dx,d,,), (77)
R,>I(XY;?)=I(Y;P)+I(X;PIY)
>I(Y;I;)rR,(d,,). (78)
Hence we obtain %(d,, d,,) c %‘(d,, d4’). If X and Y are independent, rt is known that
R,,(d,, d,) = R,(d,) + R,(d,) holds [4]. Then since point C in Fig. 3 coincides with point E, equality holds in (12). On the other hand, if dy = 0, Y must be equal to Y. Then R y(O) = H(Y) and the R, -coordinate of point B in Fig. 3 is also R, = I( XY, Y) = H(Y). Since point B coin- cides with point C, equality holdssin (12).
Let d,,>O and R,=I(XY; XY)=R,,(d,,d,,). Then, from (78) R, is larger than R,(d,) if 1(X, YIY) >O. Namely the equality in (12) does not hold.
D. Proof of Theorem 3
Let p(x, y, R, p) E $?(d,, d,,) be such that it attains I( XY; XY) = R,,(d,, d,,). Then
I(XY;P)<I(XY;fP)=R,,(d,,d,). (79)
Hence, in Fig. 3 point A E %(d,, dy). From [4, th. 3.11, we have
Rx,&&) +R&) -x(4) +R&,h (80)
Hence, in Fig. 3 point E E %(d,? d,). From the convexity of %(d,, d,,) (Appendix I), %*(d,, d,) is included in a(d,> d,J.
Equality in (80) holds if and only if X and Ye are independent [4, th. 3.11. Then, if X and Y are dependent, the equality in (80) does not hold since X and Ye are also dependent. Thus the equality in (14) also does not hold.
E. Proofs of Theorems 4, 5, and 6
Theorem 4 can be proved in the same way as the proof of Theorem 1. A similar proof as the one for the converse part of Theorem 1 applies to the outerbounds of Theorem
YAMAMOTO: CASCADE AND BRANCHING COMMUNICATION SYSTEMS 307
5 and Theorem 6. The innerbound of Theorem 5 can be easily proved by the time sharing argument which uses two codes, one of which is the same as that for the system in Fig. 1 and the other is obtained by interchanging X and Y in the same system. Similarly the innerbound of Theorem 6 can be proved by the time sharing use of the codes em- ployed in the multistage cascade communication system. So all the proofs of Theorems-4, 5, and 6 are omitted.
V. CONCLUDINGREMARKS
We obtained the achievable rate region for the cascade communication systems and also obtained inner and outer bounds on the achievable rate region for the branching communication systems. The novelty of these systems, compared with the systems treated by the other authors [6], lies in the fact that these systems comprise the equipment that remaps the received words onto other words for further transmission, e.g., decoder 0 in Fig. 1 or Fig. 7 and decoder 0 and 2 in Fig. 8. While the achievable rate region of the onestage or multistage cascade communication sys- tem has been found to be attained by the system in Fig. 10 or its multistage version, it seems that the achievable rate region of the bidirectional or multistage branching com- munication system could not be attained by such a simple configuration. Hence, it remains an open question to find such codes that remap the codewords optimally.
We considered in this paper only cascade and branching communication systems. However, similar problems exist in the merging communication systems such as those shown in Fig. 11 or Fig. 12. If channel 0 is absent, Fig. 11 coincides with the system treated by Wyner and Ziv [7], and Fig. 12 coincides with that analyzed by Berger, Housewright, Omura, Tung, and Wolfowitz [8], [9], [12]. We can derive some bounds on the achievable rate regions for merging communication systems. However, it seems very difficult to obtain the exact achievable rate regions because, even for the system of Berger et al., it is the bounds and not the achievable rate region that have been found.
{Yl
1
IX1 Encoder Encoder
1 Channel 0 Channel Decoder IiU 1
1 0
Fig. 11. Merging communication system- I.
IX1 Encoder 1 Channel I
iYl-
Fig. 12. Merging communication system-II.
Proof: For p,(x, Y, 2, P) = P,,(%, 91x, Y)Q(x, Y> and p2(x,y,%9) =~,2(%9lx,y)Q(x,y), we define
P(x,ysd+ hP,( x,Y,p,g)+x,P,(x,Y,~,~)
= (X,p,,(~,9lx,y)+X,p,,(n,9Ix,y>>Q<x,~).
641) Let XY~,F,, XYk2p2, and XY2P be characterized by pl( ), p2( ), andp( ), respectively. Then
ED,(X,~)=X,ED,(X,~,)+A,ED,(X,~~)
= A,d,, + A,d,, = d,, 642)
EDy(Y,P)=A,EDy(Y,~,)+X,EDy(Y,~)
= X,d,, + A,d,, = d,. (A31
Hencep(x,y,x,y)E~(d,,d,). From the property that for a fixed input assignment an average
mutual information is a convex function of the set of transition probability [ 10, th. 4.4.31,
GX,R’,+X,R2,=R,, (A41
~(xu;~)s(x,~(xY;~,)+x,I(xY;~~)
GX,R’,+X,R;= R,. (A51
Thus (R,, R,) E%(d,,dJ. From Lemma A, ?I&( d,, dy) is a convex region and has con-
tinuity, i.e., ~(d,+~,d,+()-,~(d,,d,) as c-,0.
ACKNOWLEDGMENT APPENDIX II
The author would like to acknowledge the continuing guidance and encouragement of Dr. K. Itoh of the Univer- sity of Tokyo. The author would also like to thank Prof. R. M. Gray of Stanford University for his helpful comments.
APPENDIX I
Lemma A: Let pk(x, y, 2, j) E ??(d,., d,,), (Rk,, Rf) E %(d,,, d,,), and let A, + A, = I, A, > 0, k = 1,2. Then for
d,=h,d,,+~X,d,,, d,h,d,,+X,d,,,
ROh,R’,fX,R;, R,h,R;SX,Rf,
there existsp(x,y,Z,p) such thatp(x,y,8,j)E?i’(d,,d,) and (Rm RI) E ~GLd,).
Lemma A: For any 6 > 0, Pr{xy 65 T,,(6)} G O(n) where O(n)+Oasn-+co.
Lemma B: Let xy E T,,(S). Then for n sufficiently large,
where < ,( 6) > 0, E ,( 6) --) 0 as 6 -+ 0 and is independent of IZ. Lemma C: Fix xy E T,,(6). If $ is chosen at random from
r;(6), then, for n sufficiently large,
Pr{9ET,,c(~)) aexp{ -n[l(XY;P)+cz(S)]}
where c2( 8) > 0, E~( 8) -+ 0 as 8 --) 0 and is independent of n. Lemma D: Fix xy$ E T,,;(6). If f is chosen at random from
Tig(6), then for n sufficiently large,
WE T,,~#4 aexp{-n[l(XY;klli)+c,(S)]}
308
where ~~(8) > 0, ~~(8) + 0 as 6 --) 0 and is independent of il. Lemmas A-C are well known and can be proved in a similar
way as [6, lemma 1.1.1, 2.1.1-31. Lemma D also can be proved similarly except that j should be conditioned. Therefore we omit their proofs.
REFERENCES
111
PI
[31
141
T. Berger, Rate Distortion Theory, A Mathematical Basis for Data Compression. Englewood Cliffs, NJ: Prentice-Hall, 1971. R. M. Gray and A. D. Wyner, “Source coding for a simple network,” Bell System Tech. J., vol. 53, no. 9, pp. 1681- 1721, Nov. 1914. R. M. Gray, “Source coding a binary symmetric source over a simple network,” in Proc. Sixth Hawaii Znt. Conf. Syst. Sci., pp. 354-355, 1973. R. M. Gray, “A new class of lower bounds to information rates of stationary sources via conditional rate-distortion functions,” IEEE Trans. Inform. Theory, vol. IT-19, no. 4, pp. 480-489, July 1973.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-27, NO. 3, MAY 1981
[51
[61
171
PI
[91
WI
[Ill
[I21
A. D. Wyner, “The rate-distortion function for source coding with side information at the decoder-II: General sources,” Inform. Contr., vol. 38, pp. 60-80, 1978. T. Berger, “Multiterminal source coding,” CZSM Cources and Lec- tures, Springer, 1978. A. D. Wyner and J. Ziv, “The rate-distortion function for source coding with side information at the decoder,” IEEE Trans. Inform. Theory, vol. IT-22, no. 1, pp. I- 10, Jan. 1976. T. Berger and S. Y. Tung,.“Multiterminal source coding,” IEEE Int. Symp. Injorm. Theory, Oct. IO- 14, 1977, Ithaca, NY. J: K. Omura and K. B. Housewright, “Source coding studies for information networks.” in Proc. IEEE I977 Int. Conf. Comm.. June 13-15, Chicago, IL, pp. 237-240. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968. H. Yamamoto, “Source Coding Theory for Multiterminal Com- munication Systems,” (in Japanese), Dr. Eng. thesis, Dep. Elec. Eng., Univ. of Tokyo, Tokyo, Japan, 1980. T. Berger, K. B. Housewright, J. K. Omura, S. Tung, and J. Wolfowitz, “An upper bound on the rate distortion function with partial side information at the decoder,” IEEE Trans. Inform. Theory, vol. IT-25, 664-666, Nov. 1979.
Asymptotically Mean. Stationary Channels
Abstract-A necessary and sufficient condition for a source to satisfy the ergodic theorem and the Shannon- McMillan theorem- the two basic mathematical tools of the Shannon theory-is that it be asymptotically mean stationary (AMS). A channel is defined here to be AMS if whenever an AMS input source is connected to the channel, the resulting input/out. put process is AMS. We develop several characterizations and properties of AMS channels that resemble those of AMS sources. As an application we show that these ideas are useful in characterizing composite sources, and in particular that there exist sources that exhibit distinct short term and long term stationarity properties. Thus “locally stationary” or “quasi- stationary” processes such as those used to model speech waveforms may also be stationary. In addition, some preliminary results on coding for AMS channels are presented.
INTRODUCTION
L ET (A, aA) be a standard Bore1 space where A is called the alphabet; that is, A is a Bore1 subset of a
complete separable metric space, and $A consists of the Bore1 sets of A. Let X = { X,}T= _ m be a two-sided discrete-
Manuscript received April 9, 1979; revised July 23, 1980. This research was supported by National Science Foundation under Grant ENG 76 02276 Al.
R. J. Fontana is with the Department of Electrical Engineering, Carne- gie- Mellon University, Pittsburgh, PA 152 13.
R. M. Gray is with the Electrical Engineering Department and the Information Systems Laboratory, Stanford University, Stanford, CA 94305.
J. C. Kieffer is with the, Department of Mathematics, University of Missouri at Rolla, Rolla, MO 65401.
time source or random process described by a distribution or probability measure p on the sequence measurable space (Z,, S,), where Z, is the space of all doubly infinite sequences x = (. * *,x-i, x0,x,, . * a) and S, is the u-field of subsets of Z, generated by the rectangles {x: x, E B,; m~i~n},Bi~~~,allfinitem~n.Weshallalsoreferto a source as [A, ~1. Since (A, aA) is a standard Bore1 space, so is the Cartesian product (Z,, S,) [ 1, ch. 21.
The coordinate or sampling random variables X,: Z, + A are defined by X,(x) = x,. Define the shift transformation
: 2, + 2, by T( .a.,.~-,,x~,x,;..) = ;. ‘,~o,-q,~2,“‘), or equivalently, X,( TX) = X,, ,(x) = X n+l, all n.
A source p is said to be stationary (with respect to T) if
p(T-‘F) ‘P(F), all F E S, .
A source p is said to be N-stationary if it is stationary with respect to TN, that is,
PL(T-~F) =dF), all F E S, .
A source p is said to be asymptotically mean stationary (AMS) if
n-l
lim n-’ 2 p(T-‘F) exists, all FES,. n-a, i=o
If the above limit exists, then the limit, say ji, is itself a
ROBERT J. FONTANA, MEMBER, IEEE, ROBERT M. GRAY, FELLOW, IEEE, AND JOHN C. KIEFFER
001%9448/81/0500-0308$00.75 01981 IEEE
