Linear rank inequalities on five or more variables ∗ Randall Dougherty, Chris Freiling, and Kenneth Zeger August 13, 2010 Abstract Ranks of subspaces of vector spaces satisfy all linear inequalities satisfied by entropies (in- cluding the standard Shannon inequalities) and an additional inequality due to Ingleton. It is known that the Shannon and Ingleton inequalities generate all such linear rank inequalities on up to four variables, but it has been an open question whether additional inequalities hold for the case of five or more variables. Here we give a list of 24 inequalities which, together with the Shannon and Ingleton inequalities, generate all linear rank inequalities on five variables. We also give a partial list of linear rank inequalities on six variables and general results which produce such inequalities on an arbitrary number of variables; we prove that there are essen- tially new inequalities at each number of variables beyond four (a result also proved recently by Kinser). * This work was supported by the Institute for Defense Analyses and the National Science Foundation. R. Dougherty is with the Center for Communications Research, 4320 Westerra Court, San Diego, CA 92121-1969 ([email protected]). C. Freiling is with the Department of Mathematics, California State University, San Bernardino, 5500 University Parkway, San Bernardino, CA 92407-2397 ([email protected]). K. Zeger is with the Department of Electrical and Computer Engineering, University of California, San Diego, La Jolla, CA 92093-0407 ([email protected]).
Transcript
Linear rank inequalities on five or more variables∗
Randall Dougherty, Chris Freiling, and Kenneth Zeger
August 13, 2010
Abstract
Ranks of subspaces of vector spaces satisfy all linear inequalities satisfied by entropies (in-cluding the standard Shannon inequalities) and an additional inequality due to Ingleton. It isknown that the Shannon and Ingleton inequalities generate all such linear rank inequalities onup to four variables, but it has been an open question whetheradditional inequalities hold forthe case of five or more variables. Here we give a list of 24 inequalities which, together withthe Shannon and Ingleton inequalities, generate all linearrank inequalities on five variables.We also give a partial list of linear rank inequalities on sixvariables and general results whichproduce such inequalities on an arbitrary number of variables; we prove that there are essen-tially new inequalities at each number of variables beyond four (a result also proved recentlyby Kinser).
∗This work was supported by the Institute for Defense Analyses and the National Science Foundation.R. Dougherty is with the Center for Communications Research, 4320 Westerra Court, San Diego, CA 92121-1969
([email protected]).C. Freiling is with the Department of Mathematics, California State University, San Bernardino, 5500 University
Parkway, San Bernardino, CA 92407-2397 ([email protected]).K. Zeger is with the Department of Electrical and Computer Engineering, University of California, San Diego, La
It is well-known that the linear inequalities that are always satisfied by ranks of subspaces of avector space (referred to here aslinear rank inequalities) are closely related to the linear inequali-ties satisfied by entropies of jointly distributed random variables (often referred to asinformationinequalities). For background material on this relationship and other topics used here, a usefulsource is Hammer, Romashchenko, Shen, and Vereshchagin [10].
The present paper is about linear rank inequalities; nonetheless, the basic results from infor-mation theory will be useful enough that we choose to use the notation of information theory here.We use the following common definitions:
H(A|B) = H(A, B) − H(B)
I(A; B) = H(A) + H(B) − H(A, B)
I(A; B|C) = H(A, C) + H(B, C) − H(A, B, C) − H(C)
There are two interpretations of these equations. WhenA, B, andC are random variables,A, B denotes the joint random variable combiningA andB; H(A) is the entropy ofA; H(A|B)is the entropy ofA givenB; I(A; B) is the mutual information ofA andB; andI(A; B|C) is themutual information ofA andB givenC.
But whenA, B, andC denote subspaces of a vector space, thenA, B denotes the space spannedby A andB, which is〈A, B〉 or, sinceA andB are subspaces, justA + B; H(A) is the rank ofA; H(A|B) is the excess of the rank ofA over that ofA ∩ B; I(A; B) is the rank ofA ∩ B; andI(A; B|C) is the excess of the rank of(A + C) ∩ (B + C) over that ofC. In either interpretation,the equations above are valid.
The basic Shannon inequalities state thatI(A; B|C) (as well as the reduced formsI(A; B),H(A|B), andH(A)) is nonnegative for any random variablesA, B, C. Any nonnegative linearcombination of basic Shannon inequalities is called a Shannon inequality. We will use standardShannon computations such asI(A; B|C) = I(A; B, C) − I(A; C) (one can check this by ex-panding into basicH terms) andH(A|C) ≥ H(A|B, C) (because the difference isI(A; B|C)); anexcellent source for background material on this is Yeung [17].
A key well-known fact is that all information inequalities (and in particular the Shannon in-equalities) are also linear rank inequalities for finite-dimensional vector spaces. To see this, firstnote that in the case of afinitevector spaceV over a finite fieldF , each subspace can be turned intoa random variable so that the entropy of the random variable is the same (up to a constant factor)as the rank of the subspace: letX be a random variable uniformly distributed overV ∗ (the set oflinear functions fromV to F ), and to each subspaceA of V associate the random variableX ↾ A.The entropy of this random variable will be the rank ofA, if entropy logarithms are taken to base|F |. For the infinite case, one can use the theorem of Rado [16] that any representable matroid isrepresentable over a finite field, and hence any configurationof finite-rank vector spaces over anyfield has a corresponding configuration over some finite field.
The converse is not true; there are linear rank inequalitieswhich are not information inequali-ties. The first known such example is the Ingleton inequality, which in terms of basic ranks or joint
Page 1 of 32
Dougherty-Freiling-Zeger August 13, 2010
entropies is
H(A) + H(B) + H(C, D) + H(A, B, C) + H(A, B, D)
≤ H(A, B) + H(A, C) + H(B, C) + H(A, D) + H(B, D),
but which can be written more succinctly using theI notation as
I(A; B) ≤ I(A; B|C) + I(A; B|D) + I(C; D).
Ingleton [11] proved this inequality in 1971 and asked whether there are still further independentinequalities of this kind.
A key tool used by Hammer et al. [10] is the notion of common information. A random variableZ is a common informationof random variablesA andB if it satisfies the following conditions:H(Z|A) = 0, H(Z|B) = 0, andH(Z) = I(A; B). In other words,Z encapsulates the mutualinformation ofA andB. In general, two random variablesA andB might not have a commoninformation. But in the context of vector spaces (or the random variables coming from them),common informations always exist; ifA andB are subspaces of a vector space, then one can justlet Z be the intersection ofA andB, andZ will have the desired properties.
Hammer et al. [10] showed that the Ingleton inequality (and its permuted-variable forms) andthe Shannon inequalities fully characterize the cone of linearly representable entropy vectors onfour random variables (i.e., there are no more linear rank inequalities to be found on four variables).
2 New five-variable inequalities
We will answer Ingleton’s question here. Using the existence of common informations, one canprove the following twenty-four new linear rank inequalities on five variables (this is a completeand irreducible list, as will be explained below).
(Note that there is much more variety of form in these inequalities than there is in the four-variable non-Shannon-type inequalities from [5].)
Each of these inequalities is provable from the Shannon inequalities if we assume that eachmutual information on the left-hand side of the inequality is in fact realized by a common infor-mation. (Hence, since such common informations always exist in the linear case, the inequalitiesare all linear rank inequalities.) For instance, inequalities (1)–(10) all hold if we assume that thereis a random variableZ such thatH(Z|A) = H(Z|B) = 0 andH(Z) = I(A; B); inequality(23) holds if there exist random variablesZ andY such thatH(Z|A) = H(Z|B) = H(Y |A) =
Page 3 of 32
Dougherty-Freiling-Zeger August 13, 2010
H(Y |B, C) = 0, H(Z) = I(A; B), andH(Y ) = I(A; B, C); and so on. These assertions canall be verified using the information-inequality programITIP [18] (which runs underMATLAB;there is also a self-contained versionXitip [15]). In fact, all of these become Shannon inequal-ities if we replace the left-hand mutual information(s) with termsH(Z) or H(Y ) and add to theright-hand side appropriate terms likekH(Z|A) + kH(Z|B) for a sufficiently large coefficientk(k = 5 suffices for all of these inequalities). For example, for inequality (1), one can show that
is a Shannon inequality; if we setZ to be a common information forA andB, we get inequality (1).Again the verifications of these Shannon inequalities can beperformed usingITIP, or one canwork them out explicitly. In Section 3 we will present various alternate proof techniques.
These inequalities can be written in other equivalent forms.Obvious rewrites (move the first term on the right to the left):
I(A; B|C) ≤ I(A; B|D) + I(A; D|E) + I(B; E|C)
+ I(A; C|B, E) + I(C; E|B, D) (12a)
I(A, B; C|D) ≤ I(A; D|B, C) + I(B; D|A, C) + I(A; C|B, E)
Note that, for these variant forms, we donot make the claim that the inequality follows fromthe existence of common informations corresponding to the left-hand-side terms. For instance,inequality (19c) does not follow from the Shannon inequalities and the existence of a commoninformation forB andC. It turns out that inequality (24b) is provable from existence of a commoninformation for (A, B) and(C, D), and inequalities (19b), (21b), (22b), and (23b) are provablefrom existence of a common information forA and(B, C), but inequalities (18b) and (20b) arenot; in fact, no single common information (together with the Shannon inequalities) suffices toprove (18) or (20).
3 Alternate proofs and generalizations
In this section we will provide some alternate proof techniques for the inequalities. This will leadto natural generalizations.
Lemma 1. The inequalityH(Z|R) + I(R; S|T ) ≥ I(Z; S|T ) is a Shannon inequality.
Proof. Using Shannon inequalities, we see that
H(Z|R) + H(S|Z, T ) ≥ H(Z|R, T ) + H(S|Z, T )
≥ I(S; Z|R, T ) + H(S|Z, T )
≥ I(S; Z|R, T ) + H(S|R, Z, T )
= H(S|R, T ).
Page 5 of 32
Dougherty-Freiling-Zeger August 13, 2010
SoH(Z|R)−H(S|R, T ) ≥ −H(S|Z, T ); addH(S|T ) to both sides to get the desired result.�
The same pattern allows us to prove more general inequalities: if A0 andB0 have a commoninformation, then:
I(A0; B0) ≤ I(A0; B0|B1)
+ I(A0; B1|B2)
+ · · ·
+ I(A0; Bn−1|Bn)
+ I(B0; Bn) (25)
I(A0; B0) ≤ 2n−1I(A0; B0|A1) + 2n−1I(A0; B0|B1)
+ 2n−2I(A1; B1|A2) + 2n−2I(A1, ; B1|B2)
+ · · ·
+ I(An−1; Bn−1|An) + I(An−1; Bn−1|Bn)
+ I(An; Bn) (26)
(Note that (26) is related to results in Makarychev and Makarychev [13].) These can be generalizedfurther; for instance, in the right-hand side of (25) any number ofA0’s may be replaced byB0’sand/or vice versa.
In fact:
Theorem 3. Suppose we have a finite binary tree where the root is labeled with an informationterm of the formI(x; y) and each other node is labeled with a term of the formI(x; y|z). Theseterms may involve any variables. We single out two variablesor combinations of variables, calledA andB. Suppose that, for each node of the tree, if its label isI(x; y|z) [we allow z to be emptyat the root], then:
(a) x is A or B and there is no left child, or(b) there is a left child and it is labeledI(r; s|x) for somer ands;
and(a′) y is A or B and there is no right child, or(b′) there is a right child and it is labeledI(r′; s′|y) for somer′, s′.Then the inequality
I(A; B) ≤ sum of all the node labels in the tree (27)
is a linear rank inequality (in fact, it is true wheneverA andB have a common information).
Proof. Let Z be a new variable. We prove by induction in the tree (from the leaves toward theroot) that, for each noden, if Tn is the subtree rooted atn, and the node label atn is I(r; s|t), thenwe have as a Shannon inequality
H(Z|t) ≤ sum of node labels inTn + jnH(Z|A) + knH(Z|B) (28)
for somejn, kn ≥ 0. (The inductive step uses Lemma 1.) Applying this whenn is the root andZis a common information ofA andB gives the desired result. �
Page 7 of 32
Dougherty-Freiling-Zeger August 13, 2010
We get the Ingleton inequality and inequalities (1) and (2) by applying this to the trees:
Ingleton: I(C; D)
��
AA
I(A; B|C) I(A; B|D)
(1): I(A; E)
AA
I(C; D|E)
��
AA
I(A; B|C) I(A; B|D)
(2): I(B; E)
AA
I(A; D|E)
AA
I(A; C|D)
AA
I(A; B|C)
A longer ”linear” tree like the last one gives (25), while a complete binary tree of heightn gives(26).
Here is another version of Theorem 3:
Theorem 4. Let I(x1; y1|w1), I(x2; y2|w2), . . . , I(xm; ym|wm) be a list of information terms,where eachxi, yi, wi is chosen from the listA, B, r1, r2, . . . , rk with the exception thatw1 is empty(i.e., the first information term is justI(x1; y1)). Suppose that each of the variablesrj is usedexactly twice, once as awi and once as anxi or yi; while variablesA andB may be used as manytimes as desired as anxi or yi, but are not used as awi. Then the inequality
I(A; B) ≤
m∑
i=1
I(xi; yi|wi)
is a linear rank inequality (in fact, it is true wheneverA andB have a common information).
Proof. We build a tree for use in Theorem 3. Each node will be labeled with one of the termsI(xi; yi|wi). The root is labeledI(x1; y1). If we have a nodeI(xi; yi|wi) wherexi is notA or B,
Page 8 of 32
Dougherty-Freiling-Zeger August 13, 2010
then create a left child for this node and label itI(xj ; yj|wj) for the uniquej such thatwj = xi.Similarly, if yi is notA or B, then create a right child for this node and label itI(xj ; yj|wj) forthe uniquej such thatwj = yi. It is easy to show that no termI(xi; yi|wi) will be used more thanonce in this construction (look for the counterexample nearest the root). Hence, the constuctionwill terminate, and the sum of the labels used is less than or equal to
∑m
i=1I(xi; yi|wi) (it does not
matter if some of the termsI(xi; yi|wi) are not used as labels). Now Theorem 3 gives the desiredresult. �
Theorem 4 directly gives the Ingleton inequality and inequalities (1) and (2). It also gives anumber of the other listed inequalities once we write them inan equivalent form using equationssuch asI(A; B|C) = I(A; B, C|C):
I(A; B) ≤ I(A; C) + I(A; B|D) + I(B; C, E|C) + I(A; D|C, E) (3d)
I(A; B) ≤ I(A; C) + I(A; B|D, E) + I(B; C, D|C) + I(A; D, E|C, D) (4d)
For instance, inequality (5d) is obtained from Theorem 4 using the list of random variables
A, B, C, D, (C, E), (D, E).
Another approach is to prove the inequality
I(A; B) ≤ I(A; C) + I(B; D|C) + I(A; F |D) + I(A; B|E) + I(B; E|F )
Page 9 of 32
Dougherty-Freiling-Zeger August 13, 2010
directly from Theorem 4 and then apply the variable substitution
(A, B, C, D, E, F ) → (A, B, C, D, (C, E), (D, E))
to get (5d). Similarly, the other inequalities listed aboveare substitution instances of linear-variableinequalities on five to eight variables. (Note that (3d), (4d), and (11d) are substitution instances of(1c).)
We will now generalize Theorem 3 so as to generate additionalinequalities. One easy (butapparently useless) generalization is to replace the binary tree with a binary forest (a finite disjointunion of binary trees). Then the hypotheses of Theorem 3 can be stated just as before (with “theroot” replaced by “each root”); and the conclusion is the same except that the inequality becomes
mI(A; B) ≤ sum of all the node labels in the trees (29)
wherem is the number of trees (eqivalently, the number of root nodes).This modification alone is useless because the resulting inequality is just a sum of Theorem 3
inequalities, one for each tree. But it will become useful when combined with another modification.For this we need a tightening of Lemma 1:
Lemma 5. The inequalityH(Z|R) + I(R; S|T ) ≥ I(Z; S|T ) + H(Z|R, S, T ) is a Shannon in-equality.
Proof. The proof is just as for Lemma 1, with the slack made explicit in one step. Using Shannoninequalities, we see that
H(Z|R) + H(S|Z, T ) ≥ H(Z|R, T ) + H(S|Z, T )
= H(Z|R, S, T ) + I(S; Z|R, T ) + H(S|Z, T )
≥ H(Z|R, S, T ) + I(S; Z|R, T ) + H(S|R, Z, T )
= H(Z|R, S, T ) + H(S|R, T ).
So H(Z|R) − H(S|R, T ) ≥ H(Z|R, S, T ) − H(S|Z, T ); addH(S|T ) to both sides to get thedesired result. �
Using this twice (and noting thatI(Z; Z|T ) = H(Z|T ) andH(Z|Z, S, T ) = 0), we get
The expression in brackets is actually symmetric inC1, C2, . . . , Cn; it is equal to
H(C1) + H(C2) + · · ·+ H(Cn) − H(C1C2 . . . Cn).
One can use Lemma 5 to produce an extended form of Theorem 3 in which an additional optionis available: instead of having a left child, a node can have aleft pointerpointing to some othernode anywhere in the tree or forest, and similarly on the right side.
Theorem 6. Suppose we have a finite binary forest where each node is labeled with an informationterm of the formI(x; y|z), wherez is empty at each root node (i.e., the root labels are of the formI(x; y)). These terms may involve any variables. We single out two variables or combinations ofvariables, calledA andB. Suppose that, for each node of the forest, if its label isI(x; y|z) [withz possibly empty], then:
(a) x is A or B and there is no left child, or(b) there is a left child of this node and it is labeledI(r; s|x) for somer, s, or(c) there is a left pointer at this node pointing to some other node whose label isI(r′; s′|t′)
wherex = (r′, s′, t′);and
(a′) y is A or B and there is no right child, or(b′) there is a right child of this node and it is labeledI(r′; s′|y) for somer′, s′, or(c′) there is a right pointer at this node pointing to some other node whose label isI(r′; s′|t′)
wherey = (r′, s′, t′).Suppose further that no node is the destination of more than one pointer. Letm be the number oftrees in the forest (equivalently, the number of root nodes). Then the inequality
mI(A; B) ≤ sum of all the node labels in the trees (33)
is a linear rank inequality (in fact, it is true wheneverA andB have a common information).
Page 11 of 32
Dougherty-Freiling-Zeger August 13, 2010
Proof. As with Theorem 3, letZ be a new variable. For any left or right pointer, ifI(r; s|t)is the label at the destination of the pointer, we say that theterm associated with the pointerisH(Z|r, s, t). We prove by induction in the forest (upward from the leaves toward the roots) that,for each noden, if Tn is the subtree rooted atn, and the node label atn is I(r; s|t), then we haveas a Shannon inequality
H(Z|t) ≤ sum of node labels inTn + Outn − Inn + jnH(Z|A) + knH(Z|B) (34)
for somejn, kn ≥ 0, where Outn is the sum of the terms associated with pointersfromnodes inTn
and Inn is the sum of the terms asasociated with pointersto nodes inTn. (A pointer whose sourceand destination are both inTn will contribute to both sums, but these contributions will cancel eachother out.) The inductive step uses Lemma 5; the new term in that lemma is used to handle thecase where there is a pointer with destinationn (note that, by assumption, there is at most one suchpointer). Once (34) is proved, apply it to all of the root nodes and add the resulting inequalitiestogether to get
mH(Z) ≤ sum of all the node labels in the trees+ jH(Z|A) + kH(Z|B) (35)
for somej, k ≥ 0; the pointer sums cancel out because each pointer contributes to one Out sumand one In sum. Applying (35) whenZ is a common information ofA andB gives the desiredresult (33). �
Theorem 6 can be used to prove inequalities (8) and (9) using the following diagrams (pointersare represented as dashed curves):
(8): I(C, D; E)
AA
I(C; D)
I(A; B|E)��
AA
I(A; B|C) I(A; B|D)
(9): I(A; C)
AA
I(B; D, E|C)
I(D; E)
��
AA
I(A; B|D) I(A; B|E)
And by using equivalent forms of terms as was done in formulas(3d) through (17d), one canuse Theorem 6 to prove formulas (6), (10), (19b), and (21b)–(24b) via the following diagrams:
Page 12 of 32
Dougherty-Freiling-Zeger August 13, 2010
(6): I(A; C)
AA
I(C, D; E|C)
��
I(A; B|C, D)A
A
I(B; D, E|E)
AA
I(A; C, D, E|D, E)
(10): I(A; E)
AA
I(B; D, E|E)
AA
I(A; C, D|D, E)
I(C; D)
��
AA
I(A; B|C) I(A; B|D)
(19b): I(B; D)
��
I(A; B, C|B)A
A
I(A; C, D|D)
AA
I(B, C; D, E|C, D)
I(D; E)
��
AA
I(A; C|D)
AA
I(A; B, C|C)A
A
I(B, C; B, D|B, E)
I(A; B, E|E)
Page 13 of 32
Dougherty-Freiling-Zeger August 13, 2010
(21b): I(B; D)
��
I(A; B, C|B)A
A
I(A; C, D|D)
AA
I(B, C; C, E|C, D)
AA
I(A; B, C|C, E)
I(C; E)
��
AA
I(A; B, C|C)
AA
I(B, D; C, E|D, E)
I(A; D, E|E)
(22b):
I(B; E)
��
I(A; B, C|B)A
A
I(A; D, E|E)
AA
I(B, E; C, D|D, E)
I(C; D)
��
I(A; B, C|C)A
A
I(A; B, D|D)
AA
I(B, C; B, E|B, D)
I(B; E)
��
I(A; B, C|B)A
A
I(A; C|E)
AA
I(A; B, C|C)
(23b): I(B; E)
��
I(A; B, C|B)A
A
I(A; C, E|E)
AA
I(B, C; C, D|C, E)
AA
I(A; B, C|C, D)
I(D; E)
��
@@
I(A; B, D|D)
AA
I(B, C; B, E|B, D)A
A
I(B, C; D, E|C, E)
I(A; C, E|E)
Page 14 of 32
Dougherty-Freiling-Zeger August 13, 2010
(24b): I(A; E)
��
I(A, B; C, D|A)A
A
I(B, E; C|E)
@@
I(A, B; C, D|C)�
�
I(A, B; B, D|B, E)
AA
I(A, B; C, D|B, D)
I(B; D)
��
AA
I(C, D; D, E|D)
AA
I(A, E; B, D|D, E)
I(A, B; C, D|B)
One can also get a new extended version of Theorem 4 in the sameway, though it is harderto state precisely. It is also slightly less flexible becauseit disallows reuse of the same variable orcombination of variables; and the forest diagrams are easier to verify by inspection.
Here are two more explicit proofs.
Proof of inequality (18).Let Z be a common information ofA andB, and letY be a commoninformation ofA andC; note that we haveH(Y, Z|A) = 0. Then
I(B; C) + I(A; B|D) + I(A; C|D) + I(B; D|E)
+ I(C; D|E) + I(A; E)
≥ I(Z; Y ) + I(Y, Z; Z|D) + I(Y, Z; Y |D) + I(Z; D|E)
≥ I(Z; Y ) + I(Z; Z|E) + I(Y ; Y |E) + I(Y, Z; E) [from Lemma 1]
= I(Z; Y ) + H(Z|E) + H(Y |E) + I(Y, Z; E)
≥ I(Z; Y ) + H(Y, Z|E) + I(Y, Z; E)
= I(Z; Y ) + H(Y, Z)
= H(Z) + H(Y )
= I(A; B) + I(A; C).
�
Proof of inequality (20).Let Z be a common information ofA andB, and letY be a commoninformation ofA andC; note that we haveH(Y, Z|A) = 0 andH(C, Y |C) = H(C|C, Y ) = 0.
It is not yet clear how to generalize these two proofs.Since the inequalities in this paper have been proven using only common informations and the
Shannon inequalities, they apply not only to linear ranks but also in any other situation where wehave random variables which are known to have common informations. For instance, Chan notesin [1, Definition 4] that abelian group characterizable random variables always have common in-formations (which are still abelian group characterizablerandom variables); hence, the inequalitiesproven here hold for such variables.
Page 16 of 32
Dougherty-Freiling-Zeger August 13, 2010
4 Completeness
The complete (and verified nonredundant) list of linear-variable inequalities on five variables con-sists of:
• the elemental Shannon inequalities:
0 ≤ I(A; B)
0 ≤ I(A; B|C)
0 ≤ I(A; B|C, D)
0 ≤ I(A; B|C, D, E)
0 ≤ H(A|B, C, D, E)
and the inequalities obtained from these by permuting the five variablesA, B, C, D, E (seeYeung [17] for a proof that these imply all other 5-variable Shannon inequalities);
• the following instances of the Ingleton inequality:
I(A, B; A, C) ≤ I(A, B; A, C|A, D) + I(A, B; A, C|A, E) + I(A, D; A, E) (39)
and the ones obtained from these by permuting the five variablesA, B, C, D, E (see Guille,Chan, and Grant [9] for a proof that these imply all other 5-variable instances of the Ingletoninequality); and
• inequalities (1)–(24) and their permuted-variable forms.
To verify the completeness of this list, we consider the 31-dimensional real space whose coor-dinates are labeled by the nonempty subsets of{A, B, C, D, E} in the usual binary order:
defines a half-space of this space; the intersection of thesehalf-spaces is a polyhedral cone whichcan also be described as the convex hull of its extreme rays. If one of these extreme rays containsa nonzero pointv which is (linearly) representable(i.e., there exist a vector spaceU and sub-spacesUA, UB, UC , UD, UE of U such thatdim(UA) = v(A), dim(UB) = v(B), dim(〈UA, UB〉) =v(A, B), and so on), then this extreme ray can never be excluded by anyas-yet-unknown linearrank inequality. If we verify thatall of the extreme rays contain linearly representable points,thenthere can be no linear rank inequality which cuts down the polyhedral cone further, so the list ofinequalities must be complete.
Page 17 of 32
Dougherty-Freiling-Zeger August 13, 2010
The computations of the authors (as described in Section 5) show that there are 7943 extremerays inR
31 determined by the elemental Shannon inequalities and inequalities (1)–(24) and (36)–(39) (and permutations). If one considers two such rays to beessentially the same when onecan be obtained from the other by a permutation of the five variables, then there are 162 essentiallydifferent extreme rays. A full list of the vectors generating these rays is available electronically [6].
These computations also show that each of these vectors is representable over the field of realnumbers; in fact, up to a scalar multiple, this representation can be done using matrices with integerentries which actually represent the vector over any field (finite or infinite). For instance, considerthe extreme ray given by the vector
(a list of 31 ranks or entropies in the order given by (40)). Tothis we associate the five matrices:
MA =[
1 0 0]
MB =[
0 1 0]
MC =[
0 0 1]
MD =[
1 1 1]
ME =
[
1 1 00 0 1
]
The interpretation here is that we have a fixed fieldF , and the row space of each of these matricesspecifies a subspace ofF 3. The specified vector givesH(A) = 1, and the row space ofMA hasdimension 1; the vector givesH(B) = 1, and the row space ofMB has dimension 1; the vectorgivesH(A, B) = 2, and the vector sum of the row spaces ofMA andMB (i.e., the row space ofMA-on-top-of-MB) has dimension 2; and so on. Equivalently, if we take three random variablesx1, x2, x3 chosen uniformly and independently over the finite fieldF , and letA = x1, B = x2,C = x3, D = x1 + x2 + x3, andE = (x1 + x2, x3), then the entropies of all combinations ofA, B, C, D, E (with logarithms to base|F |) are as specified by the above vector.
The dimensions of the row spaces listed above are easily computed over the real field (as ranksof the corresponding matrices). In order to verify that the same dimensions would be obtainedover any field, one just has to note that, in each case where a matrix rank is computed to bek,there is actually ak × k submatrix whose determinant is±1, so the selectedk rows will still beindependent even after being reduced modulo any prime. (Actually, it would suffice to verify thatthe greatest common divisor of the determinants of allk × k submatrices is 1.)
All of the other listed vectors turn out to be representable in the same way, except that for afew of them a scalar multiplier must be applied. For instance, consider the vector
To represent this, we would normally takeMA to be a0 × 2 matrix andMB, MC , MD, ME tobe 1 × 2 matrices whose unique rows have the property that any two areindependent but anythree are dependent. (In other words, these row vectors are alinear representation for the uniform
Page 18 of 32
Dougherty-Freiling-Zeger August 13, 2010
matroidU2,4.) For example, we could take
MA = [ ]
MB =[
1 0]
MC =[
0 1]
MD =[
1 1]
ME =[
1 2]
over the real field, but these would not work over the field of two elements. In fact, no suchchoice of row vectors works over the field of two elements (thefirst two row vectors would beindependent, but then the only choice for the third vector would be the sum of the first two, andthat would also be the only choice for the fourth vector, contradicting the independence of the thirdand fourth vectors). But if we instead take the vector
corresponding to the uniform matroidU2,5, had to be tripled in order to get a matrix representationthat works over all fields.
5 Methodology; testing representability of polymatroids
The list of five-variable linear rank inequalities was produced by the following iterative process.Initially, we had the Shannon and Ingleton inequalities. Ateach stage, we took the current list ofinequalities and used Komei Fukuda’s polytope softwarecddlib [7] to get the corresponding list
Page 19 of 32
Dougherty-Freiling-Zeger August 13, 2010
of extreme rays. We then examined the vectors generating theextreme rays to see whether theywere representable (over the reals; we did not try to get representations working over all fields untilafter the iterative process was complete). When such a vector provably could not be represented,the proof (in each case we ran into here) yielded a new linear rank inequality provable via commoninformations; when we examined a vector where we had difficulty determining whether it wasrepresentable or not, we ran exhaustive tests on all ways of specifying a single common information(toward the end, we had to try a pair of common informations) to see whetherITIP [18] couldverify that the specified vector contradicted the Shannon inequalities together with the commoninformation specification. Again each such verification ledto a new linear rank inequality. (Ofcourse, this is a highly sanitized version of the process as it actually occurred.)
The testing of extreme rays for linear representability soon became a large task, so we graduallydeveloped software to automatically find such representations in a number of cases (and we addedmore cases when we found new ways to represent vectors). Thissoftware used combinatorialrather than linear-algebra methods; for instance, the output of the program for the sample vector
used above was a specification of five vector spacesA, B, C, D, E which could be paraphrased as:“A is generated by one vector,B is generated by one vector not inA, C is generated by one vectornot in A + B [the space spanned byA andB], D is generated by one vector inA + B + C, andE is generated by two vectors, one in(A + B) ∩ (C + D) and one inC.” The development of thesoftware involved recognizing as many cases as possible where one could find such a specificationwhich could be met over the reals (or over any sufficiently large finite field) and would yield thedesired rank vector.
The (attempted) construction of a representation is done one basic subspace at a time: first therepresentation ofA is constructed (this step is trivial), then the representation of B givenA, thenthe representation ofC givenA andB, and so on. And each of these subspace representations isconstructed one basis vector at a time. Given the representation of A, B, C, andD, the algorithmwill determine how many basis vectors are needed for subspace E and successively try to choosethem in suitable positions relative to the existing subspaces. At each step, a new vector will bechosen in general position in a subspace which is a sum of someof the already-handled subspacesA, B, C, D. (Here “general position” means in the selected subspace but not in any relevant propersubspace of it. Which subspaces are relevant depends on the current situation; we avoid having todetermine this explicitly by just saying that the underlying field is sufficiently large, or infinite.) Ifthere is a problem with specifying that the vector is in such asum of basic subspaces, then we mayhave to specify that the vector is in the intersection of two sums of basic subspaces.
Once the first vector is chosen, we take quotients of all of theexisting spaces by this vector toget the new situation in which the second vector needs to be chosen. This is all done by countingdimensions, not by constructing actual numerical vectors.For instance, suppose the first vector ischosen to be in general position in subspaceR which is a sum of basic subspaces fromA, B, C, D(e.g.,R = A+B). For each other sum subspace (i.e., sum of basic subspaces)T , if the new vectoris in T , then the quotient by the chosen vector will reduce the dimension ofT by 1; if the chosenvector is not inT , then the quotient will not change the dimension ofT . Since the vector is ingeneral position inR, the vector will be inT if and only if R ⊆ T , and to check whetherR ⊆ T
Page 20 of 32
Dougherty-Freiling-Zeger August 13, 2010
one simply has to see whetherdim(R +T ) = dim T . The case where the vector is chosen from anintersection of two sum subspacesR andS is more complicated; more on this below.
Consider the example (41). Suppose that we have already constructed the representations forsubspacesA, B, C, andD, and we are now ready to construct the representation for subspaceE.(In the following, we will not actually be using any information about the constructed representa-tions forA, B, C, andD other than the given ranks of their sums.) The current situation can besummarized by the following two-row array:
Here the first row displays the ranks of sums fromA, B, C, D in the order given by (40), butstarting with the empty space; this is just the first half of (41) with a 0 prepended. For each ofthese sums, the second row gives the amount by which adding the new subspaceE will increasethe dimension of the sum. (So the second entry in this row isH(A + E)−H(A) = 3− 1 = 2, thefourth entry isH(A + B + E) − H(A + B) = 3 − 2 = 1, and so on.; this row is the second halfof (41) minus the first half.)
From this array, we can see that, sinceE has dimension 2 but only increases the dimension ofA + B by 1, one of the nonzero vectors inE must be inA + B. So let us start by assuming thatone of the vectors inE is a vector chosen in general position inR = A + B. We can now checkfor all sums fromA, B, C, D whether the sum will contain this chosen vector; this information issummarized in the row
0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 1 (43)
where 1 means the chosen vector is in the corresponding sum. (For instance, the third-to-last entryin this row is for the sum subspaceA + C + D; this subspace contains the chosen vector becausedim((A+B)+(A+C+D)) = dim(A+C+D) = 3.) To get the result of taking a quotient by (thesubspace generated by) the chosen vector, we subtract (43) from the first row of (42) (because wehave used up one vector from each of the indicated subspaces)and subtract the one’s complementof (43) from the second row of (42) (because we have taken careof one of the new vectors forEbeyond each of the indicated subspaces; for instance, sincethe chosen vector is not inA, it givesone of the two vectors needed to account forH(A + E) − H(A) = 2). So the situation after thefirst vector is chosen is given by
Of course, the negative entry in this array means that a problem has occurred: we tried to takea new vector not inC + D, but the given ranks require all vectors inE to be inC + D. So we willtry again; instead of taking a vector in general position inR = A + B, we take a vector in generalposition inR ∩ S, whereS = C + D.
This leaves the problem of determining, for each sum subspaceT , whether the chosen vector isin T ; as before, this is equivalent to determining whetherR∩S ⊆ T . This is not as straightforwardas it was to determine whetherR ⊆ T ; in fact, there are situations where the given data on ranks
Page 21 of 32
Dougherty-Freiling-Zeger August 13, 2010
of sum subspaces simply do not determine whetherR ∩ S ⊆ T . But we have identified manysituations where the given data do allow this determinationto be made. Here is a list; note that(a) each such test can also be applied withR andS interchanged, and (b) reading this list is notnecessary for understanding the rest of the algorithm.
• If R ⊆ T , thenR ∩ S ⊆ T .
• If the dimensions ofR ∩ S, R ∩ T , andR ∩ (S + T ) are all equal, thenR ∩ S ⊆ T . [Iftwo subspaces have the same (finite) dimension and one is included in the other, then thetwo subspaces are equal. Hence, we getR ∩ S = R ∩ (S + T ) = R ∩ T , soR ∩ S =(R ∩ S) ∩ (R ∩ T ) = R ∩ S ∩ T , soR ∩ S ⊆ T . Also, recall that the dimension ofR ∩ Scan be determined from the given data; it is equal toI(R; S) = H(R) + H(S) − H(R, S).]
• If the dimensions ofR ∩ T , S ∩ T , (R + S) ∩ T , andR ∩ S are all equal, thenR ∩ S ⊆ T .[We haveR ∩ T = (R + S) ∩ T = S ∩ T , soR ∩ T = R ∩ S ∩ T . But nowdim(R ∩ S) =dim(R ∩ T ) = dim(R ∩ S ∩ T ), soR ∩ S = R ∩ S ∩ T , soR ∩ S ⊆ T .]
• If dim(R ∩ T ) < dim(R ∩ S), thenR ∩ S 6⊆ R ∩ T , so we must haveR ∩ S 6⊆ T .
• Let R ∩∗ S be the “nominal intersection” ofR andS (i.e., the sum of the basic subspaceslisted both in the sumR and the sumS; for example, ifR is A+B +C andS is A+B +D,thenR∩∗ S is A+B, regardless of whether the actual subspacesC andD intersect). ClearlyR ∩∗ S ⊆ R ∩ S, so, ifR ∩∗ S 6⊆ T , thenR ∩ S 6⊆ T .
• If dim(R ∩ T ) < dim(R ∩ ((R ∩∗ T ) + S)), thenR ∩ S 6⊆ T . [First note that, ifU, V, Ware subspaces such thatV ⊆ U , thenU ∩ (V + W ) = V + (U ∩ W ). (The right-to-leftinclusion is easy. For the left-to-right inclusion, ifu = v + w whereu ∈ U , v ∈ V , andw ∈ W , thenu − v = w ∈ U ∩ W , sov + w ∈ V + (U ∩ W ).) Hence, ifR ∩ S ⊆ T , thenR∩((R∩∗T )+S) = (R∩∗T )+(R∩S) ⊆ R∩T , sodim(R∩((R∩∗T )+S)) ≤ dim(R∩T ).]
• If T ′ ⊆ T andR ∩ S ⊆ T ′, thenR ∩ S ⊆ T . If T ⊆ T ′ andR ∩ S 6⊆ T ′, thenR ∩ S 6⊆ T .
• Let R\∗S be the “nominal difference” ofR andS (i.e., the sum of the basic subspaces listedin the sumR but not in the sumS; for example, ifR is A + B + C andS is A + B + D,thenR \∗ S is C), and letU = (R \∗ S) + (S \∗ R). If dim(U ∩ (R ∩∗ S)) = 0, then
R ∩ S = ((R \∗ S) ∩ (S \∗ R)) + (R ∩∗ S).
[The right-to-left inclusion is easy. For the left-to-right inclusion, note thatR = (R \∗ S) +(R∩∗S) andS = (S\∗R)+(R∩∗S). Hence, ifx ∈ R∩S, then we havex = y1+z1 = y2+z2
for somey1 ∈ R \∗ S, y2 ∈ S \∗ R, andz1, z2 ∈ R ∩∗ S. Theny2 − y1 = z1 − z2 is inU ∩ (R ∩∗ S), so we havey2 = y1 andz2 = z1; hence,y1 ∈ (R \∗ S) ∩ (S \∗ R) andx = y1 + z1 is in the desired form.] Hence, ifdim(U ∩ (R ∩∗ S)) = 0, R ∩∗ S ⊆ T , and((R \∗ S) ∩ (S \∗ R)) ⊆ T , thenR ∩ S ⊆ T .
Page 22 of 32
Dougherty-Freiling-Zeger August 13, 2010
These tests do suffice for the example here; the resulting membership vector for a chosen vectorin general position in(A + B) ∩ (C + D) is
0 0 0 1 0 0 0 1 0 0 0 1 1 1 1 1 (44)
and the new array after taking a quotient by the first chosen vector is
where again we obtain this new array by subtracting (44) fromthe first row of (42) and subtractingthe one’s complement of (44) from the second row of (42)
Let us call the new quotient spacesA′, B′, C ′, D′, E ′. The new ranks indicate that the remainingvector inE ′ must be chosen to be inC ′. If we take the new vector in general position inC ′, thenthe resulting membership vector is:
0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1
(Note that we needed the chosen vector to be inD′ as well as inC ′, but this turned out to beautomatic, because the given ranks impliedC ′ = C ′ + D′ = D′.) And the result of taking aquotient by the second chosen vector is:
The all-0 row means that the representation ofE has been successfully completed.The current algorithm does not try many possibilities for the next vector to choose; it simply
chooses one sum subspace (usually at the beginning of the list of available ones) to try to add avector to, and, if that yields an immediate contradiction, perhaps tries one intersection of two sumsubspaces. If any such step fails (either because of a contradiction, or because the algorithm cannotdetermine whetherR∩S ⊆ T in some case), the algorithm gives up. However, the algorithm doesgive itself up to 120 chances by trying all permutations of the 5 basic variables.
The above algorithm gives a positive test for representability; we also needed negative teststo verify that a vector wasnot linearly representable. To do this, we tested exhaustivelyagainstall possible ways of requiring that one or two specified common informations exist. For instance,suppose we have a vectorv assigning to each subsetS of {A, B, C, D, E, F} a rankvS, andwe want to test whether this vector is compatible with the assumption that there is a commoninformationZ for A andB. Then we can write down equations expressing that the ranksH(S) areproportional to the given valuesvS; we just say
v{A,B,C,D,E,F}H(S) = vSH(A, B, C, D, E, F ) (45)
for eachS ⊆ {A, B, C, D, E, F}. And the requirements onZ are just the equationsH(A, Z) =H(A), H(B, Z) = H(B), andH(Z) = H(A) + H(B) − H(A, B). We now want to test whetherthese equations plus the Shannon inequalities (on the sevenvariablesA, B, C, D, E, F, Z) imply
Page 23 of 32
Dougherty-Freiling-Zeger August 13, 2010
that all of the variables must collapse to nothing; such a collapse can be expressed as the inequalityH(A, B, C, D, E, F ) ≤ 0.
We have now expressed the desired test in exactly the form that can be performed byITIP [18].But since it is basically a linear programming problem, and since we had a huge number of suchtests to run, we found it useful to produce additional implementations, one usingcddlib [7] andone using the GNU Linear Programming Kit [8].
If such a test succeeds, then we can extract a new linear rank inequality from it as follows.Suppose that the collapse inequalityH(A, B, C, D, E, F ) ≤ 0 does follow from the stated condi-tions. Express each of the given equationsP = Q as two inequalitiesP ≤ Q andQ ≤ P ; then wehave determined that the inequalityH(A, B, C, D, E, F ) ≤ 0 follows from a given list of linearinequalities. Standard linear programming theory now tells us thatH(A, B, C, D, E, F ) ≤ 0 canbe expressed as a nonnegative linear combination of the given inequalities; say it can be writtenas a sum
∑
cII wherecI ≥ 0 andI varies over the given list of inequalities. (Linear program-ming software usually returns such coefficientscI as part of its output, but interior-point linearprogramming algorithms often yielded rather ugly floating-point coefficients, so we resorted totrial-and-error repeated computations to get more suitable coefficients, usually integers but some-times small-denominator rationals.)
Now we can split the list of given inequalities into two parts: the ones that come from (45)and the ones that do not (the Shannon inequalities and the inequalities coming from the com-mon information specification). LetIn be the inequality obtained from the collapse inequalityH(A, B, C, D, E, F ) ≤ 0 by subtracting offcII for those inequalities coming from (45). Sincethe vectorv satisfies the inequalities from (45) with equality and does not satisfy the collapse in-equality, it must not satisfyIn. But In is a linear rank inequality on the variablesA, B, C, D, E, F ,because it is also equal to the sum ofcII over all given inequalities not coming from (45), andthese inequalities are all true whenZ is a common information forA andB. Sincev satisfies allof the linear rank inequalities we previously had,In must be a new linear rank inequality, whichwe can add to our list.
So we performed an iteration of polytope computations (on the current list of known inequal-ities) and ray testing. Each time a new extreme ray was produced, the above representation algo-rithm was applied as a positive test for representability, while tests against common informationswere used as negative tests. If a negative test succeeded (showing that the ray was not repre-sentable), this yielded a new inequality as above. If both positive and negative tests failed, the raywas examined by hand. Sometimes this examination yielded a representation because we found anew way of determining whetherR∩S ⊆ T ; if so, this new test was added to the algorithm. At theend, the algorithm was able to verify representability of 152 of the final 162 extreme rays, leavingonly 10 to be done by hand (by methods which did not fit in the framework of this algorithm).
There are other possibilities for improving the representation algorithm that we have not yetimplemented. One is doing a backtrack search to consider more possibilities for choosing vectorsto add; another is to use the information on representation of previous subspaces in the constructionof the representation of the current subspace. (In the preceding example, we used only the dimen-sion data forA, B, C, D in the construction of the representation forE; we did not use the actualrepresentations constructed forA, B, C, D.) More ambitious would be to allow more options forchoosing new vectors in terms of the known relations betweenthe current subspaces.
Page 24 of 32
Dougherty-Freiling-Zeger August 13, 2010
6 Six-variable inequalities (ongoing work)
This iterative process for finding all linear rank inequalities is likely to be infeasible to completefor six or more variables. (Eachcddlib polytope computation in 31 dimensions took about 2–3days; although we now know more ways to speed this up, a full 63-dimension calculation wouldstill take far too long, as well as rapidly exceeding the memory available.) But we plan to continuethe study, because we expect to find new phenomena at higher levels, possibly including extremerays that are representable over some fields but not over others (hence yielding rank inequalitieswhich hold only over those other fields), and inequalities which hold for ranks of vector spacesbut are not provable via common informations. For instance,such situations could come from thevariables associated with the Fano and non-Fano networks in[4], or the network in [3]. (But thismight require going to seven or more variables, an even more infeasible task.)
In order to make any progress at all, we had to take some shortcuts (since, as noted above,63-dimensional polytope computations were out of the question). One of these was to reduce thedimension of the search by assuming equality for one or more of the inequalities found so far; ineffect, this is just concentrating on one face, corner, or intermediate-dimensional extreme part ofthe current region. Another was to work hard on trying to improve already-obtained inequalities,find additional instances of them, or strengthen them in multiple ways if they were not alreadyfaces of the region.
We will show here some of the 6-variable inequalities we havefound so far; a much longerlist is available on the Web [6]. All of these have been verified to be faces of the linear rankregion (so they cannot be improved). To do this, we used a stockpile of linearly representable6-variable vectors (the representability was proved by thealgorithm described in the precedingsection) encountered during the polytope computations. Ifa 6-variable linear rank inequality issatisfied with equality by 62 linearly independent vectors from the stockpile, then it must give aface of the linear rank region. (The stockpile currently contains 3280 polymatroids, or 1867049after one takes all instances obtained by permuting the six basic variables. It is also available at theabove website.)
First, there are the 6-variable elemental Shannon inequalities; there are 6 of these if one listsjust one of each form, but 246 of them if all of the permuted-variable versions are counted. Thenthere are 12 instances of the Ingleton inequality (1470 counting permuted forms). Again, seeYeung [17] and Guille, Chan, and Grant [9] for the proof thatthese inequalities imply all of theother Shannon and Ingleton inequalities.
Next come the instances of the 5-variable inequalities (1)–(24). The initial computation found183 of these instances that (with permuted forms) proved allof the others. However, 16 of theseinstances did not pass the face verification above and were later superseded by other 6-variableinequalities; this left 167 (61740 counting permuted forms) 5-variable instances which were facesof the 6-variable rank region.
Finally, there are the true 6-variable inequalities. We have found 3846 of these so far (2640975counting permuted forms) which pass the face verification, along with several hundred more whichdo not pass and which we expect to be superseded later (thoughthis is not guaranteed; perhapsour stockpile of representable polymatroids is insufficient, although the face test has been mostlyreliable so far). We give some examples of these here; see thewebsite mentioned above for the full
Page 25 of 32
Dougherty-Freiling-Zeger August 13, 2010
list.Some of these 6-variable inequalities follow directly fromTheorem 3, such as:
I(A; B) ≤ I(A; C) + I(B; D|C) + I(A; E|D) + I(B; F |E) + I(A; B|F ) (46)
I(A; B) ≤ I(A; C) + I(B; D|C) + I(A; E|D) + I(A; F |E) + I(A; B|F ) (47)
I(A; B) ≤ I(A; C) + I(B; D|C) + I(E; F |D) + I(A; B|E) + I(A; B|F ) (48)
I(A; B) ≤ I(A; C) + I(D; E|C) + I(A; B|D) + I(B; F |E) + I(A; B|F ) (49)
I(A; B) ≤ I(C; D) + I(A; B|C) + I(E; F |D) + I(A; B|E) + I(A; B|F ) (50)
And others follow directly from Theorem 6, such as:
2I(A; B) ≤ I(A; C) + I(D; E, F |C) + I(A; B|D)
+ I(E; F ) + I(A; B|E) + I(A; B|F ) (51)
2I(A; B) ≤ I(A; C) + I(B; D|C) + I(A; E, F |D)
+ I(E; F ) + I(A; B|E) + I(A; B|F ) (52)
2I(A; B) ≤ I(C; D) + I(A; B|C) + I(B; E, F |D)
+ I(E; F ) + I(A; B|E) + I(A; B|F ) (53)
2I(A; B) ≤ I(C, D; E) + I(C; D) + I(A; F |C)
+ I(A; B|F ) + I(A; B|D) + I(A; B|E) (54)
3I(A; B) ≤ I(C, D; E, F ) + I(C; D) + I(E; F ) + I(A; B|C)
+ I(A; B|D) + I(A; B|E) + I(A; B|F ) (55)
Then there are inequalities which follow from Theorem 3 or Theorem 6 using equivalent forms:
+ I(B; C|A, E, F ) + I(B; C|D, E) + I(A; D|B, C, F )
+ I(A; C|B, E, F ) + I(A; F |B, D, E) (57)
2I(A; B) ≤ I(D; F ) + I(A; C) + I(B; D|C) + I(A; B|F ) + I(A; E|D)
+ I(A; F |C, D) + I(A; B|E) (58)
I(A; B, C) ≤ I(A; C) + I(B; D|C) + I(A; F |D) + I(A; B|F ) + I(C; E|B, F )
+ I(A; C|B, E) (59)
3I(A, B; C, D, E) ≤ I(A; C, F ) + I(A, B; D) + I(A, B; E) + I(C; F |D) + I(D; F |E)
+ I(A; E|D, F ) + I(B; C|A, D, F ) + I(B; D|C, F ) + I(A; D, E|B, C)
+ I(A; D|B, C, E) + I(A; C|E, F ) + I(B; D|A, E, F ) + I(B; C, D|A)
+ I(A, B; E|C, D) + I(B; E|A, C, D) + I(B; D|C, E, F )
+ I(A, B; C|D, E) (60)
Page 26 of 32
Dougherty-Freiling-Zeger August 13, 2010
All of the sharp inequalities found so far using one common information have been verified tobe instances of Theorem 6. It seems quite possible that this theorem generates all one-common-information inequalities, but we have no proof of this.
There are also hundreds of inequalities that required two common informations to prove. (In-equalities requiring more than two common informations arebeyond the range of our software atpresent.) These are of two types. One type consists of inequalities like (18) and (20) which havetwo information terms on the left side and use the common informations corresponding to thoseterms:
I(A; B) + I(A; C) ≤ I(B; C) + I(A; D) + I(B; E|D) + I(C; F |D)
+ I(A; B|E) + I(A; C|F ) (61)
2I(A; B, C) + I(B; C, D) ≤ I(A; C, E) + I(A; F ) + I(A; C|D) + 2I(A; B|C, F )
+ I(B; C) + I(E; F |C) + 2I(B; D|C, E) + I(C; E|F )
+ I(A; D|E, F ) + I(D; E|A, C, F ) + 2I(A; F |C, D, E) (62)
The other type has just one information term on the left side but requires a second common infor-mation in addition to the one from the left term. Here are someexamples:
I(A; B) ≤ I(A; C) + I(B; D|C) + I(E; F |D) + I(A; B|E) + I(A; C|F )
+ I(B; E|C, F ) (63)
2I(A, B; C, D, E) ≤ I(A, B; D, E) + I(A, D, F ; C) + I(A, F ; D|C) + I(B; C|D, E)
+ I(A; C|B) + I(A; D|B, C, E) + 2I(A; C|D, E, F ) + I(B; C|A, D, E)
+ I(A; E|B, D, F ) + I(B; E|A, C, F ) + I(B; E|A, D, F ) + I(B; E|C, D)
+ I(B; D|A, E, F ) + I(A; F |B, D, E) + I(A; F |B, C, D) (64)
2I(A; B, C) ≤ I(A; B) + I(D; E) + I(A; B|C) + I(C; E|B) + I(D; F |B, E)
Inequality (63) is proved using a common information forA andB along with a common in-formation forE and (D, F ); inequality (64) is proved using a common information for(A, B)and(C, D, E) along with a common information for(B, F ) and(A, D, E); and inequality (65) isproved using a common informationZ for A and(B, C) along with a common information forFandZ. (The possible need for such iteration of common informations along with joining of vari-ables makes it conceivable that an unbounded number of common informations could be neededto prove linear rank inequalities even on a fixed number of initial variables such as 6.)
7 An infinite list of linear rank inequalities
The following theorem shows that there will be essentially new inequalities for each number ofvariables:
Theorem 7. For anyn ≥ 2, the inequality
(n − 1)I(A; B) + H(C1C2 · · ·Cn) ≤
n∑
i=1
I(A, Ci; B, Ci) (66)
Page 27 of 32
Dougherty-Freiling-Zeger August 13, 2010
is a linear rank inequality onn + 2 variables which is not a consequence of instances of linearrank inequalities on fewer thann + 2 variables.
Proof. First, it is not hard to show that (66) is equivalent to (32), and we have already seen that(32) is a linear rank inequality (this can also be proved using Theorem 6), so (66) is a linear rankinequality.
In the following, ifS = {i1, i2, . . . , ik} ⊆ {1, 2, . . . , n}, we will write CS for Ci1Ci2 · · ·Cik .Define a vectorv on the subsets of{A, B, C1, C2, . . . , Cn} as follows: for anyS ⊆ {1, 2, . . . , n},
v(CS) = 2|S|,
v(ACS) = n + |S|,
v(BCS) = min(2n − 2 + |S|, 2n),
v(ABCS) = min(2n − 1 + |S|, 2n).
One can easily check thatv does not satisfy (66). We will show thatv does satisfy all instances (us-ing the variablesA, B, C1, C2, . . . , Cn) of all linear rank inequalities on fewer thann+2 variables;this will imply that (66) is not a consequence of these instances, as desired.
For this purpose, we construct vectorswA, wB, w1, w2, . . . , wn, each of which is the same asvexcept for one value. The changed values are:
wA(A) = n − 1,
wB(B) = 2n − 3,
wi(BCi) = 2n.
We will show that each of thesew vectors is linearly representable over any infinite or suffi-ciently large finite fieldF . In each case, the representation will use a vector spaceV overF ofdimension2n, with a basisx1, x2, . . . , xn, y1, y2, . . . , yn, and the variableCj (1 ≤ j ≤ n) will berepresented by the two-dimensional subspace〈xj , yj〉.
For the representations ofA andB, instead of giving explicit formulas, it will be convenientto use the following concept. SupposeU is a nontrivial subspace ofV . A point u ∈ U is said tobe in general positionin U , relative to a given finite setS of points (if S is not specified, then welet S be the set of all points that have previously been mentioned explicitly), if u does not lie inany subspaceU ′ of V spanned by a subset ofS unlessU ′ includes all ofU . If the setS is of sizeat mostN , then the “in general position” condition excludes at most2N proper subspaces ofU(including the trivial subspace), so there is no problem finding points in general position as longas the field size|F | is greater than2N (because each excluded proper subspace contains at most1/|F | of the points inU). If we refer to multiple points being chosen in general position, then theyshould be considered as chosen successively, with later points being in general position relative toearlier points as well as the previous setS. This concept has been referred to by various terms; forinstance, in in [14] such points are referred to as “freely placed”. Points chosen in this way make iteasy to compute augmented subspace dimensions: ifu is in general position inU relative toS andU ′ is a subspace spanned by points inS, thendim(〈U ′, u〉) is equal todim(U ′)+1 unlessU ⊆ U ′,in which case it is equal todim(U ′).
Page 28 of 32
Dougherty-Freiling-Zeger August 13, 2010
For eachi ≤ n, a representation ofwi is obtained by assigning toA the space
X = 〈x1, x2, . . . , xn〉
and assigning toB the space spanned by all of thex’s exceptxi, together withn − 1 additionalpoints chosen in general position inV . (As stated before, in all cases we assign toCj the space〈xj , yj〉.)
For the representation ofwB, we again assign toA the spaceX; B is assigned a space spannedby n − 2 points in general position inX together withn − 1 additional points in general positionin V .
To representwA, choose pointsz1, z2, . . . , zn−1 in general position inX, and assign toA andB the spaces〈z1, z2, . . . , zn−1〉 and〈z1, z2, . . . , zn−2, y1, y2, . . . , yn〉, respectively.
It remains to show that, ifC(t1, . . . , tk) ≥ 0 is a linear rank inequality onk variables withk < n + 2, then no instance of this inequality fails forv. An instance of this inequality whichapplies tov is given by a mapf from {t1, . . . , tk} to the subsets of{A, B, C1, . . . , Cn}. (Thenthe definition off can be immediately extended to the subsets of{t1, . . . , tk} by the formulaf({tj1, . . . , tjm
}) = f(tj1) ∪ · · · ∪ f(tjm).) So suppose we have an instance, given byC andf as
above, which fails forv. SinceC(t1, . . . , tk) ≥ 0 is a linear rank inequality, the instance must notfail for the representable vectorwA. Therefore, the instance must use the value wherev disagreeswith wA. This means that there is a subset of{t1, . . . , tk} which is mapped byf to {A}; it followsthat there is some single valuejA ∈ {1, 2, . . . , k} such thatf(tjA
) = {A}. Similarly, since theinstance must not fail forwB, there is a subset of{t1, . . . , tk} which is mapped byf to {B}, sothere existsjB ∈ {1, 2, . . . , k} such thatf(tjB
) = {B}. And, for eachi ≤ n, the instance mustnot fail for wi, so there is a subset of{t1, . . . , tk} which is mapped byf to {B, Ci}; hence, thereexistsji ∈ {1, 2, . . . , k} such thatf(tji
) is either{Ci} or {B, Ci}. It is clear from thesef valuesthat the numbersjA, jB, j1, j2, . . . , jn are distinct; but this is impossible because{1, 2, . . . , k} hasfewer thann + 2 members. This contradiction completes the proof of the theorem. �
8 Concurrent work and open questions
During the preparation of this paper, the authors became aware of closely related concurrent work.Chan, Grant, and Kern [2] show nonconstructively that thereexist linear rank inequalities notfollowing from the Ingleton inequality. Kinser [12] presents a sequence of inequalities which canbe written in the form
I(A2; A3) ≤ I(A1; A2) + I(A3; An|A1) +n
∑
i=4
I(A2; Ai−1|Ai) (67)
for n ≥ 4. (This is a variant of (25) which follows from Theorem 4; the instance forn = 4 andn = 5 are permuted-variable forms of the Ingleton inequality andinequality (1c), respectively.)Kinser shows that (67) is a linear rank inequality for eachn ≥ 4 and uses a method similar tothe proof of Theorem 7 above to show that instancen of (67) is not a consequence of linear rankinequalities on fewer thann variables.
Here are some fundamental open questions that this researchhas not yet answered.
Page 29 of 32
Dougherty-Freiling-Zeger August 13, 2010
• For each fixedn, are there finitely many linear rank inequalities onn variables which implyall of the others?
• Is the method of using common informations incomplete? Thatis, are there linear rankinequalities that cannot be proved from the basic techniqueof assuming the existence ofcommon informations?
• Are there inequalities that are valid linear rank inequalities over some fields but not overothers? The authors expect to find such inequalities on sevenvariables by examining theFano and non-Fano matroids, which are representable only over fields of characteristic 2 andover fields not of characteristic 2, respectively. This willprobably yield examples of twolinearly representable (over different fields) vectors whose sum is not even a scalar multipleof a representable vector, just as the disjoint union of the Fano and non-Fano matroids is anon-representable matroid.
The authors would like to thank James Oxley for helpful discussions.
Page 30 of 32
Dougherty-Freiling-Zeger August 13, 2010
References
[1] T. Chan, “Group characterizable entropy functions”,2007 IEEE International Symposium onInformation Theory, 24–29 June 2007, pp. 506–510.
[2] T. Chan, A. Grant, and D. Kern, “Existence of new inequalities for representable polyma-troids”, arXiv 0907.5030 (2009).
[3] R. Dougherty, C. Freiling, and K. Zeger, “Insufficiency of linear coding in network informa-tion flow”, IEEE Transactions on Information Theory, vol. 51, no. 8, pp. 2745–2759, August2005.
[4] R. Dougherty, C. Freiling, and K. Zeger, “Networks, matroids, and non-Shannon informationinequalities”,IEEE Transactions on Information Theory, vol. 53, no. 6, pp. 1949–1969, June2007.
[5] R. Dougherty, C. Freiling, and K. Zeger, “Non-Shannon information inequalities”, in prepa-ration.
[8] GNU Project,GLPK (GNU Linear Programming Kit),
http://www.gnu.org/software/glpk
[9] L. Guille, T. H. Chan, and A. Grant, “The minimal set of Ingleton inequalities”,2008 IEEEInternational Symposium on Information Theory, 6–11 July 2008, pp. 2121–2125.
[10] D. Hammer, A.E. Romashchenko, A. Shen, and N.K. Vereshchagin, “Inequalities for Shan-non entropy and Kolmogorov complexity”,Journal of Computer and Systems Sciences, vol.60, pp. 442–464, 2000.
[11] A. W. Ingleton, “Representation of matroids”, inCombinatorial Mathematics and its Appli-cations, D. J. A. Welsh, ed., pp. 149–167, Academic Press, London, 1971.
[12] R. Kinser, “New inequalities for subspace arrangements”, arXiv 0905.1519 (2009); to appearin J. Combin. Theory Ser. A.
[13] K. Makarychev and Y. Makarychev, “Conditionally independent random variables”,arXiv:cs/0510029.
[14] D. Mayhew, M. Newman, and G. Whittle, “On excluded minors for real-representability”,J.Combin. Theory Ser. B, vol. 99, pp. 685–689, 2009.
