Post on 23-Jan-2018
transcript
Synergy and redundancy in dynamical systems:towards a practical and operative definition
Daniele Marinazzo1 Luca Faes 2 Sebastiano Stramaglia 3
1Ghent University, Belgium2Fondazione Bruno Kessler, Italy
3University of Bari and INFN, Italy
December 16, 2016
7 @dan marinazzohttp://users.ugent.be/~dmarinaz/
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Granger causality to recover dynamical networks
Context
Two time series X and Y
x , the future values of X
Operative definition, Wiener 1956, Granger 1969
Y is cause of X if the knowledge of Y allows to make more precisepredictions about x
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Granger causality to recover dynamical networks
Context
Two time series X and Y
x , the future values of X
Operative definition, Wiener 1956, Granger 1969
Y is cause of X if the knowledge of Y allows to make more precisepredictions about x
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Granger causality to recover dynamical networks
Context
Two time series X and Y
x , the future values of X
Operative definition, Wiener 1956, Granger 1969
Y is cause of X if the knowledge of Y allows to make more precisepredictions about x
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
GC in multivariate datasets: a well-known issue
Condition GC estimation to the effect of other variables, to avoidfalse positivesSeveral proposed approaches, starting from Geweke et al 1984
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
GC in multivariate datasets: a well-known issue
Condition GC estimation to the effect of other variables, to avoidfalse positives
Several proposed approaches, starting from Geweke et al 1984
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
GC in multivariate datasets: a well-known issue
Condition GC estimation to the effect of other variables, to avoidfalse positivesSeveral proposed approaches, starting from Geweke et al 1984
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Granger causality: definition
Predictive model of a multivariate system
n time series {xα(t)}α=1,...,n,state vectors
Xα(t) = (xα(t −m), . . . , xα(t − 1)) ,
m order of the model
Conditioned Granger Causality
δmv (β → α) = logε (xα|X \ Xβ)
ε (xα|X)
Pairwise Granger Causality
δbv (β → α) = logε (xα|Xα)
ε (xα|Xα,Xβ)
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Granger causality: definition
Predictive model of a multivariate system
n time series {xα(t)}α=1,...,n,state vectors
Xα(t) = (xα(t −m), . . . , xα(t − 1)) ,
m order of the model
Conditioned Granger Causality
δmv (β → α) = logε (xα|X \ Xβ)
ε (xα|X)
Pairwise Granger Causality
δbv (β → α) = logε (xα|Xα)
ε (xα|Xα,Xβ)
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Granger causality: definition
Predictive model of a multivariate system
n time series {xα(t)}α=1,...,n,state vectors
Xα(t) = (xα(t −m), . . . , xα(t − 1)) ,
m order of the model
Conditioned Granger Causality
δmv (β → α) = logε (xα|X \ Xβ)
ε (xα|X)
Pairwise Granger Causality
δbv (β → α) = logε (xα|Xα)
ε (xα|Xα,Xβ)
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Granger causality and Transfer entropy
GC and TE are equivalent for Gaussian variables and otherquasi-Gaussian distributions(Barnett et al 2009, Hlavackova-Schindler 2011, Barnett andBossomaier 2012)
In this case they both measure information transfer.
Unified approach (model based and model free)
Mathematically more treatable
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
False positives in pairwise GC
Ten unidirectionally coupled noisy logistic maps, withx1(t) = f (x1(t − 1)) + 0.01η1(t), andxi (t) = (1− ρ)f (xi (t − 1)) + ρf (xi−1(t − 1)) + 0.01ηi (t), withi = 2, . . . , 10, η Gaussian noise terms, coupling ρ, f (x) = 1− 1.8x2
Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
False negatives in pairwise GC due to synergy
Three unit variance iid Gaussian noise terms x1, x2 and x3. Let
x4(t) = 0.1(x1(t − 1) + x2(t − 1)) + ρx2(t − 1)x3(t − 1) + 0.1η(t)
.x2 is a suppressor variable for x3 w.r.t. the influence on x4
Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Redundancy due to a hidden source
h(t) hidden Gaussian variable, influencing n variablesxi (t) = h(t − 1) + sηi (t), and w(t) = h(t − 2) + sη0(t) influencedby h but with a larger delay, s is the noise level.
Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Redundancy due to synchronization
Multiplet of logistic maps {xi}, i = 1, . . . , 4,:xi (t) = (1− ρ)f (xi (t − 1)) + ρ
∑4j=1,j 6=i f (xj(t − 1)) + 0.01ηi (t),
and x5(t) =∑4
i=1xi (t−1)
8 + η5(t),where η are unit variance Gaussian noise terms, coupling ρ.
multiplet to x5 multiplet to multiplet
Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Partial conditioning
Conditioned Granger Causality (CGC)
δmv (β → α) = logε (xα|X \ Xβ)
ε (xα|X)
Pairwise Granger Causality (PWGC)
δbv (β → α) = logε (xα|Xα)
ε (xα|Xα,Xβ)
Partially conditioned Granger causality (PCGC)
δYc (β → α) = logε (xα|Xα,Y)
ε (xα|Xα,Xβ,Y)
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Partial conditioning
Conditioned Granger Causality (CGC)
δmv (β → α) = logε (xα|X \ Xβ)
ε (xα|X)
Pairwise Granger Causality (PWGC)
δbv (β → α) = logε (xα|Xα)
ε (xα|Xα,Xβ)
Partially conditioned Granger causality (PCGC)
δYc (β → α) = logε (xα|Xα,Y)
ε (xα|Xα,Xβ,Y)
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Partial conditioning
Fix a subset Y of the variables in X, excluding Xα and Xβ
Partially conditioned Granger causality
δYc (β → α) = logε (xα|Xα,Y)
ε (xα|Xα,Xβ,Y)
Strategy 1, Information-Based (IB)
Y maximizes the mutual information I{Xβ;Y} among all thesubsets of nd variables
Strategy 2, Pairwise-Based (PB)
Select Y = {Xγ}ndγ=1 as the nd variables with the maximal pairwiseGC δbv (γ → α) w.r.t. that target node, excluding Xβ
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Partial conditioning
Fix a subset Y of the variables in X, excluding Xα and Xβ
Partially conditioned Granger causality
δYc (β → α) = logε (xα|Xα,Y)
ε (xα|Xα,Xβ,Y)
Strategy 1, Information-Based (IB)
Y maximizes the mutual information I{Xβ;Y} among all thesubsets of nd variables
Strategy 2, Pairwise-Based (PB)
Select Y = {Xγ}ndγ=1 as the nd variables with the maximal pairwiseGC δbv (γ → α) w.r.t. that target node, excluding Xβ
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Partial conditioning
Fix a subset Y of the variables in X, excluding Xα and Xβ
Partially conditioned Granger causality
δYc (β → α) = logε (xα|Xα,Y)
ε (xα|Xα,Xβ,Y)
Strategy 1, Information-Based (IB)
Y maximizes the mutual information I{Xβ;Y} among all thesubsets of nd variables
Strategy 2, Pairwise-Based (PB)
Select Y = {Xγ}ndγ=1 as the nd variables with the maximal pairwiseGC δbv (γ → α) w.r.t. that target node, excluding Xβ
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Information-based partial conditioning
Given the previous Yk−1 , the set Yk is obtained adding thevariable with greatest information gain
This is repeated until nd variables are selected
Marinazzo et al. Comput. Mat. Methods Med. 2012, Wu et al. Brain Connectivity 2013
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Information-based partial conditioning
Given the previous Yk−1 , the set Yk is obtained adding thevariable with greatest information gain
This is repeated until nd variables are selected
Marinazzo et al. Comput. Mat. Methods Med. 2012, Wu et al. Brain Connectivity 2013
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Information-based partial conditioning
Given the previous Yk−1 , the set Yk is obtained adding thevariable with greatest information gain
This is repeated until nd variables are selected
Marinazzo et al. Comput. Mat. Methods Med. 2012, Wu et al. Brain Connectivity 2013
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Pairwise-based conditioned Granger causality
Facts
CGC performs poorly in presence of redundancy
Partial conditioning does not solve redundancy
Information about redundancy can be extracted from PWGC
Proposed approach
Some links inferred from PWGC are retained and added tothose obtained by CGC
The PWGC links that are discarded are those that can bederived as indirect links from the CGC pattern
Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Interim summary on partial conditioning
Synergy
The search for synergetic contributions in information flow isequivalent to the search for suppressors
PWGC bad, CGC ok, PCGC even better if the selectionstrategy succeeds in picking the suppressors
Information-based PCGC better with redundancy
Pruning-based PCGC better in tree-like structures
Redundancy
Bad for CGC, and not solvable
Indirect connections of CGC from PWGC links
Links not explained as indirect connections (redundant) aremerged into CGC
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Synergy and redundancy
Pairwise information measures are commonly agreed upon(e.g. mutual information)
Shannon’s information theory does not fit multivariateinformation measures dealing with the notions of synergy andredundancy (Williams, Beer, Lizier, Wibral, Faes, Barrett)
All the proposed partial information decompositions, in theGaussian case, lead to the following (undesirable) results: (i)redundancy is the minimum of MI between the target andeach source (ii) synergy is the extra information provided bythe weaker source when the stronger source is known (Barrett,PRE 2015)
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Joint information
Let’s go for an operative and practical definition
Relation (B and C) → A
synergy: (B and C) contributes to A with more informationthan the sum of its variables
redundancy: (B and C) contributes to A with less informationthan the sum of its variables
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Joint information
Let’s go for an operative and practical definition
Relation (B and C) → A
synergy: (B and C) contributes to A with more informationthan the sum of its variables
redundancy: (B and C) contributes to A with less informationthan the sum of its variables
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Joint information
Let’s go for an operative and practical definition
Relation (B and C) → A
synergy: (B and C) contributes to A with more informationthan the sum of its variables
redundancy: (B and C) contributes to A with less informationthan the sum of its variables
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Generalization of GC for sets of driving variables
Conditioned Granger Causality in a multivariate system
δX(B → α) = logε (xα|X \ B)
ε (xα|X)
Unnormalized version
δuX(B → α) = ε (xα|X \ B)− ε (xα|X)
An interesting property
If {Xβ}β∈B are statistically independent and their contributions in
the model for xα are additive, then δuX(B → α) =∑β∈B
δuX(β → α).
We remark that this property does not hold for the standarddefinition of Granger causality neither for entropy-rootedquantities, due to the presence of the logarithm.
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Generalization of GC for sets of driving variables
Conditioned Granger Causality in a multivariate system
δX(B → α) = logε (xα|X \ B)
ε (xα|X)
Unnormalized version
δuX(B → α) = ε (xα|X \ B)− ε (xα|X)
An interesting property
If {Xβ}β∈B are statistically independent and their contributions in
the model for xα are additive, then δuX(B → α) =∑β∈B
δuX(β → α).
We remark that this property does not hold for the standarddefinition of Granger causality neither for entropy-rootedquantities, due to the presence of the logarithm.
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Generalization of GC for sets of driving variables
Conditioned Granger Causality in a multivariate system
δX(B → α) = logε (xα|X \ B)
ε (xα|X)
Unnormalized version
δuX(B → α) = ε (xα|X \ B)− ε (xα|X)
An interesting property
If {Xβ}β∈B are statistically independent and their contributions in
the model for xα are additive, then δuX(B → α) =∑β∈B
δuX(β → α).
We remark that this property does not hold for the standarddefinition of Granger causality neither for entropy-rootedquantities, due to the presence of the logarithm.
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Question from the audience:
What does it ever mean to have an unnormalized measure ofGranger causality?
Don’t you lose any link with information?
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Question from the audience:
What does it ever mean to have an unnormalized measure ofGranger causality?
Don’t you lose any link with information?
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Define synergy and redundancy in this framework
Synergy
δuX(B → α) >∑β∈B δ
uX\B,β(β → α)
Redundancy
δuX(B → α) <∑β∈B δ
uX\B,β(β → α)
Pairwise syn/red index
ψα(i , j) = δuX\j(i → α)− δuX(i → α)
= δuX({i , j} → α)− δuX(i → α)− δuX(j → α)
Stramaglia et al. IEEE Trans Biomed. Eng. 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Define synergy and redundancy in this framework
Synergy
δuX(B → α) >∑β∈B δ
uX\B,β(β → α)
Redundancy
δuX(B → α) <∑β∈B δ
uX\B,β(β → α)
Pairwise syn/red index
ψα(i , j) = δuX\j(i → α)− δuX(i → α)
= δuX({i , j} → α)− δuX(i → α)− δuX(j → α)
Stramaglia et al. IEEE Trans Biomed. Eng. 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Define synergy and redundancy in this framework
Synergy
δuX(B → α) >∑β∈B δ
uX\B,β(β → α)
Redundancy
δuX(B → α) <∑β∈B δ
uX\B,β(β → α)
Pairwise syn/red index
ψα(i , j) = δuX\j(i → α)− δuX(i → α)
= δuX({i , j} → α)− δuX(i → α)− δuX(j → α)
Stramaglia et al. IEEE Trans Biomed. Eng. 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
ψ as cumulant expansion of the prediction error
ε (xα|Xα)− ε (xα|X) =∑B⊂X
S(B).
The Moebius inversion formula allows to reconstruct S(B). Calling |nB | and |nΓ| the number of variables in thesubsets B and Γ respectively, and exploiting also the relation:
∑Γ⊂B
(−1)|nΓ| = 0,
leads to the cumulant expansion:
S(B) =∑Γ⊂B
(−1)|nB |+|nΓ| δuB (Γ→ α).
The first order cumulant is thenS(i) = δ
ui (i → α),
the second cumulant isS(i, j) = δ
uij ({ij} → α)− δuij (i → α)− δuij (j → α) ,
the third cumulant is
S(i, j, k) = δuijk ({ijk} → α)− δuijk ({ij} → α)
−δuijk ({jk} → α)− δuijk ({ik} → α)
+δuijk (i → α) + δuijk (j → α) + δ
uijk (k → α) , (1)
and so on. The index ψ may then be seen as the order two cumulant of the expansion of the prediction error of thetarget variable;Stramaglia et al. IEEE Trans Biomed. Eng. 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Predictive multivariate models
Faes et al. Phil. Trans. A 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Variance decomposition
Faes et al. Phil. Trans. A 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Entropy decomposition
Faes et al. Phil. Trans. A 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Overview
Faes et al. Phil. Trans. A 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
fMRI data, N=90, Human Connectome Project
Regions forming redundant and synergetic multiplets with arepresentative region (black)
A
B
C
D
A B
DC
RED SYN
RED
SYN
Hierarchical structure of synergy and redundancy networksStramaglia et al. IEEE Trans Biomed. Eng. 2016
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Take-home message
With a wider applicability in sight, we advocate an intuitiverather than axiomatic view of partial informationdecomposition
We aim to detect the presence of redundant and synergeticmultiplets rather than precisely measure synergy andredundancy
Variance decomposition is a viable alternative to entropydecomposition
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
Thanks
7 @dan marinazzohttp://users.ugent.be/~dmarinaz/
Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems