Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Quantifying information synergy

Using intermediate variables

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Information integration

or

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Setting

1X

Y2X

3X

p X p Y X

1 2, ,...X X X How much synergy in Y about X

…

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

INFORMATION THEORYBasics of

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Entropy of a coin flip

0 ( 0) 0.51 ( 1) 0.5

p Xp X

Carries 1 bit of information

0 ( 0) 01 ( 1) 1

p Xp X

Carries 0 bits of information

X

X

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Entropy of a coin flip

In general:

20,1

1( ) ( ) log( )x

H X p X xp X x

0 ( 0) 0.51 ( 1) 0.5

p Xp X

Carries 1 bit of information

0 ( 0) 01 ( 1) 1

p Xp X

Carries 0 bits of information

X

X

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Mutual information between coins

( 0) 0.5( 1) 0.5p Xp X

Transform

X Y

( | )p Y X ( | )p X Y ?

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

( 0) 0.5( 1) 0.5p Xp X

Transform

X Y

( | )p Y X ( | )p X Y

( | ) 1p Y x X x 1 bit transferred

( | ) 1/ 2p Y x X x 0 bits transferred

?

eq( | )p Y x X x p

,

( , ): ( , ) log( ) ( )x y

p x yI X Y p x yp x p y

Mutual information between coins

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Summary of information theory

( ) log ( )X x

H X p x p x

“Entropy”

“Mutual information”

,

( , ): ( , ) log ,( ) ( )

H .x y

p x yI X Y p x yp x p y

H X X Y

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

SYNERGISTIC INFORMATION

What is

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Individual versus synergistic

X1 X2 Y

0 0 0

0 1 0

1 0 1

1 1 1

X1 X2 Y

0 0 0

0 1 1

1 0 1

1 1 0

A

B

A

100% synergistic100% individualistic

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Individual versus synergistic

X1 X2 Y

0 0 0

0 1 0

1 0 1

1 1 1

X1 X2 Y

0 0 0

0 1 1

1 0 1

1 1 0

A

B

A

100% synergistic100% individualistic

?

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Naive attempt

1X

Y2X

3X

p X p Y X

“Whole-Minus-Sum” (WMS)

"synergy" : :WMS

ii

I X Y I X Y

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Naive attempt

1X

Y2X

3X

p X p Y X

“Whole-Minus-Sum” (WMS)

"synergy" : :WMS

ii

I X Y I X Y

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Previous attemptsCandidate redundancy

measureAchilles’ heel

WholeMinusSum (WMS) Redundant XOR

Imin Copy two bits

IΛ Noisy output

Iα Correlated inputs AND

… …

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Terminology

Stochastic variableiX

i iX X

X Y

Set of variables

Pr |Y X

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

HOW MUCH SYNERGISTIC INFORMATION DOES Y CONTAIN ABOUT X

New theory to quantify

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Definition: synergistic

is synergystic about iff

: 0,

: 0.

j

j

j i

S X

I S X

I S X

j jS S

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

First intuition

( : ) synergyI Y S

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Counterexample 1 2,X X X

0,1, 2iX

Pr( ) 1 9X

1 1 2 mod3S X

2 12 1 mod 3S X

1 1: 0I S X 2 1: 0I S X

1 2 1, : 1.22I S S X

Y=X1 would be synergistic?!

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Second intuition

( : ) synergyjj

I Y S

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Counterexample

1 2 3, ,X X X X

0,1iX

Pr X

1 1 2S X X

2 2 3S X X

3 1 3S X X

4 1 2 3S X X X

Choosing Y={S1,S2} would result in 3 bits of synergy

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Orthogonal decomposition

: , such that

, : ,

: 0,

: : .

D B B B

I B B B H B

I B A

I B A I B A

A B

BB ,

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Final result

*1 1: ; ,..., ,i i i i i

S S S D S S S

* *syn ordering for max : .

iiS S S

I X Y I Y S

For a given ordering of Si

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Proved consequences

syn, :X Y I X Y H S

synI X Y H S No ‘overcounting’

No ‘undercounting’ 3: not counted already countedI Y S

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Further proved consequences

syn 0.I X Y

syn : .I X Y I X Y

syn syn ' '' '.i i i ii j i i

I X Y I X Y

syn 1 0.I X Y

syn 1 0.I X X

syn max.i iSI X X H X H X

1( : ) ,..., max , where .N i iI X Y H X X H X Y X

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Prevailing framework

Partial Information Decomposition (PID)

indiv syn:I X Y I I

Williams, Paul L., and Randall D. Beer. "Nonnegative decomposition of multivariate information." arXiv preprint arXiv:1004.2515 (2010).

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Prevailing framework

Partial Information Decomposition (PID)

indiv syn:I X Y I I

Williams, Paul L., and Randall D. Beer. "Nonnegative decomposition of multivariate information." arXiv preprint arXiv:1004.2515 (2010).

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Operational meaning of synergy

Susc

eptib

ility

to p

ertu

rbin

g

Random Y Synergistic Y

†1X

Y2X

3X

†1 1X X

Rick Quax: Computational Science, University of Amsterdam, The Netherlands.

Conclusion• New framework to think about synergy• Start only from two simple ingredients

– Definition of ‘synergistic’– Correlate Y with intermediate S instead of X

• (=incompatible with PID)

• Limitation: computationally expensive

indiv syn:I X Y I I

https://bitbucket.org/rquax/info_metricsSoftware: