Complex networks, synchronization and
cooperative behaviour
Johan Suykens
KU Leuven, ESAT-SCD/SISTAKasteelpark Arenberg 10
B-3001 Leuven (Heverlee), BelgiumEmail: [email protected]
http://www.esat.kuleuven.be/scd/
VUB Leerstoel 2012-2013 - Oct. 31 2012
Complex networks, synchronization and cooperative behaviour
Introduction
http://www.youtube.com”synchronization of metronomes”
(a modern version of the synchronization of two pendulum clocks observedby Christiaan Huygens, 1665)
Complex networks, synchronization and cooperative behaviour 1
Overview
• Chaotic systems synchronization, Lur’e systems
• Cluster synchronization and community detection in complex networks
• Optimization using coupled local minimizers, cooperative behaviour
Complex networks, synchronization and cooperative behaviour 1
Circuits and systems: Chua’s circuit
+ +
− −
vC2C2 C1
iL
LvC1
gNR(vC1
)
NR
G
Ga
Gb
Ga
−EE
gNR(vC1
)
vC1
Chua’s circuit [Chua et al., 1986]: in dimensionless form
x = α(y − x − f(x))y = x − y + z
z = −βy
where
f(x) = m1x +1
2(m0 − m1) (|x + 1| − |x − 1|)
(depending on α, β: bistability, limit cycles, chaos)
Complex networks, synchronization and cooperative behaviour 2
Bifurcation to Chaos
(vC1, vC2)-plane:
Power spectrum vC1:
−→ birth of the double scroll attractor −→
Complex networks, synchronization and cooperative behaviour 3
Lur’e system
L
L(s)
N
σ(·)
m(t) = 0
u
y(t)++
k1
k2
σ(y)
y
• Lur’e system:
x = Ax + Bu
y = Cx
u = σ(y)→ x = Ax + Bσ(Cx)
where x ∈ Rn and σ(·) : R
h → Rh satisfies a sector condition.
• Chua’s circuit: h = 1
Complex networks, synchronization and cooperative behaviour 4
More hidden units
• Multi-stability & Multi-scroll chaos:Extend the nonlinearity andcreate additional equilibrium points[Suykens & Vandewalle, 1991; Arena, 1996; Yalcin, 2001; Lu, 2006]
• Multilayer neural networks are universal approximators [Hornik, 1989]
(Chua’s circuit has 1 hidden unit (h = 1), more hidden units for multi-scrolls)
Complex networks, synchronization and cooperative behaviour 5
A gallery of multi-scroll attractors
[Suykens & Vandewalle, 1991; Yalcin et al., 2001]
Complex networks, synchronization and cooperative behaviour 6
Lur’e systems: examples
• Lur’e system:
x = Ax + Bu
y = Cx
u = σ(y)→ x = Ax + Bσ(Cx)
• Many examples of Lur’e systems in different areas:- Recurrent neural networks (Hopfield network: A = −I, C = I) [Hopfield, 1985]
- Cellular neural networks (sparse and structured matrices A, B, C) [Chua, 1988]
- Actuator saturation in control systems
- Chua’s circuit, multi-scroll circuits
- Arrays of coupled networks
- Genetic oscillator models
L
L(s)
N
σ(·)
m(t) = 0
u
y(t)++
k1
k2
σ(y)
y
Complex networks, synchronization and cooperative behaviour 7
Genetic oscillators
A general genetic oscillator form [Li, Chen, Aihara, 2006]:
x(t) = Ax(t) +
l∑
i=1
Bifi(x(t))
where
• x(t) ∈ Rn: concentrations of proteins, RNAs, chemical complexes
• fi(x(t)) = [fi1(x1(t)); ...; fin(xn(t))]: regulatory function (monotonicallyincreasing or decreasing: e.g. Michaelis-Menten or Hill form)
Examples: Goodwin model, repressilator, toggle switch, circadian oscillators
Complex networks, synchronization and cooperative behaviour 8
Stability analysis and LMIs (1)
• Linear system:x = Ax
Quadratic Lyapunov function:
V = xTPx, P = P T > 0
• Stability analysis:
V = xTPx + xTP x = xT (ATP + PA)x < 0
Global asymptotic stability for
ATP + PA < 0
Linear matrix inequality (LMI) for a given matrix A [Boyd et al., 1994]
Complex networks, synchronization and cooperative behaviour 9
Stability analysis and LMIs (2)
• Lur’e system:x = Ax + Bσ(Cx)
Try e.g. a quadratic Lyapunov function (leading to a sufficient stabilitycondition):
V = xTPx, P = P T > 0
• Stability analysis: exploit the fact that σ belongs to sector [0, k]
V = xTPx + xTP x
≤ xTPx + xTP x−∑
i 2λiσi(σi − kcTi x) = [xTσT ]Z
[
x
σ
]
If
Z =
[
ATP + PA PB + kCTΛBTP + kΛC −2Λ
]
< 0
then globally asymptotically stable (any initial state x(0) convergesto the origin), where Λ = diag{λi} with λi ≥ 0.
Complex networks, synchronization and cooperative behaviour 10
Synchronization of Lur’e systems
• Master-slave synchronization scheme (drive-response):
M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + K(x − z)
Master system M drives slave system S (follows behaviour imposed bythe master system): under which conditions do the systems M and Ssynchronize?
(studies in synchronization of chaotic systems, and applications to secure
communications [Pecora & Carroll, 1990; Chen & Dong, 1998; Yalcin et al., 2005])
• Mutual synchronization scheme:
M1 : x = Ax + Bσ(Cx) + K1(x − z)M2 : z = Az + Bσ(Cz) + K2(x − z)
Systems M1 and M2 mutually influence each other.
Complex networks, synchronization and cooperative behaviour 11
Synchronization example
Master system
No synchronization Synchronization
Complex networks, synchronization and cooperative behaviour 12
Error system
• Consider the error e = x − z relative between the master M and theslave S system:
e = (A − K)e + B[σ(C(e + z)) − σ(Cz)]
• Assume a sector condition on σ(C(e + z)) − σ(Cz)[Suykens & Vandewalle, IJBC 1997; Curran, Suykens, Chua, IJBC 1997]
• A sufficient condition for global asymptotic stability of the error systemcan be obtained by taking e.g. a quadratic Lyapunov function
V (e) = eTPe, P = P T > 0
and derive under which condition dVdt
< 0, ∀e ∈ Rn0 .
Complex networks, synchronization and cooperative behaviour 13
Interpretation as a control problem
• Master-slave synchronization scheme:
M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + u
C : u = K(x − z)
with control signal u.
• Control objective: for given matrices A,B,C design a controller C withmatrix K such that synchronization is achieved.
• For Lur’e systems synchronization can be characterized by LMIs.
• Synchronization can be achieved for any choice of initial states x(0), z(0):for all initial state choices the systems synchronize in the sense that‖x(t) − z(t)‖ → 0 when time t → ∞.
Complex networks, synchronization and cooperative behaviour 14
Different control problems and approaches
• Dynamic measurement feedback control instead of full state feedback:if one cannot measure complete state vectors x, z.
• Robust synchronization: A, B,C matrices non-identical for master andslave system: it is possible to synchronize two systems up to a smallsynchronization error (e.g. limit cycle versus chaos); control in thepresence of disturbances or noise (e.g. H∞ control)
• Control via impulses (sporadic coupling, only from time to time andnon-equidistantly in time) instead of continuously controlling
• Control in systems with time-delays
• Other forms of synchronization: partial synchronization, clustersynchronization, phase synchronization, connection with graph topology
[Chen et al.; Wu et al.; Suykens et al.; Nijmeijer et al.; Yalcin et al.]
Complex networks, synchronization and cooperative behaviour 15
Problems in synchronization theory
IMPULSIVECOUPLING
Robust
Impulsive
Time-delaySynchronization
Synchronization
Synchronization
Synchronization
Synchronization
Nonlinear H∞
Robust Nonlinear H∞
EXTERNALINPUT
MISMATCHPARAMETER
AutonomousNon-autonomous
Design Purposes
Master-slave Synchronization Schemes
DELAY
Chaotic Lur’e Systems
[Yalcin et al., 2005]
Complex networks, synchronization and cooperative behaviour 16
Overview
• Chaotic systems synchronization, Lur’e systems
• Cluster synchronization and community detection in complexnetworks
• Optimization using coupled local minimizers, cooperative behaviour
Complex networks, synchronization and cooperative behaviour 16
Complex networks
Random network Scale−free network
Number of links Number of linksNumber of links
Num
ber
of n
odes
Num
ber
of n
odes
[log scale]
[log
sca
le]
Num
ber
of n
odes
[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]
- Random networks: bell curve distribution- Scale-free networks: power law distribution
Robust against accidental failures, but vulnerable to coordinated attacks
Biological networks: growth (gene duplication) and preferential attachement
(rich-gets-richer mechanism: new nodes prefer to link to the more connected nodes)
Complex networks, synchronization and cooperative behaviour 17
Map of protein-protein interactions
[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]Highly linked proteins (network hubs) tend to be crucial for cell survival.
Only few proteins are able to physically attach to a huge number.
www.nd.edu/∼networks
Complex networks, synchronization and cooperative behaviour 18
Wave phenomena in neuronal networks
- Hodgkin-Huxley type model of oscillatory activity in the bursting neurons of a snail
- Burst waves of antiphase spiking excitation in a 200× 200 lattice of electrically coupled
nonidentical neurons (snapshots at different times)
[Komarov, Osipov, Suykens, Chaos 2008]
Complex networks, synchronization and cooperative behaviour 19
Synchronization in complex networks
• Synchronization of chaotic systems [Pecora & Carroll, 1990]:mainly low dimensional systems and regular network topologies
• Complex networks: larger networks, different network topologies
• Complex networks:- relation between network topology and synchronization into clusters?- how to design to achieve desired clusters?- how to cope with time delays or communication constraints?- how to enhance synchronizability of complex networks- how to rewire the network?- ...
[Suykens & Osipov, Focus issue, Chaos 2008; Arenas et al., PR 2008]
Complex networks, synchronization and cooperative behaviour 20
Link between synchronization and spectral clustering
• (generalized) Kuramoto model: N coupled phase oscillators
dθi
dt= ωi +
∑
j
Kij sin(θj − θi), i = 1, ..., N
Special case: ωi = ω, Kij = σaij with adjacency matrix [aij]
• Linearized dynamics (Laplacian matrix L)
dθi
dt= −σ
∑
j
Lijθj, i = 1, ..., N
• Relationship between topological scales and dynamic time scalesModular structures emerge at different time scales
[Arenas et al., PRL 2006, PR 2008]
Complex networks, synchronization and cooperative behaviour 21
Complex networks
Synchronization
Spectral clustering
Complex networks, synchronization and cooperative behaviour 22
Spectral clustering
SVM, kernel methods
Data
Complex networks, synchronization and cooperative behaviour 23
Complex networks
Synchronization
Spectral clustering
SVM, kernel methods
Data
Complex networks, synchronization and cooperative behaviour 24
Community detection from synchronization
• Kuramoto model: θi = ω + σ∑
j aij sin(θj − θi)
• Follow the evolution of
ρij(t) = 〈cos[θi(t) − θj(t)]〉
averaged over different initial conditions.
• Community detection based on a binary dynamic connectivity matrix
[Dt(T )]ij = 1 if ρij(t) > T, zero otherwise
T large enough: one finds set of disconnected clustersT smaller: inter-community connections become visible
• Other approach: matrix DT (t) unravels the topological structure of thenetwork at different time scales.
[Arenas et al., PRL 2006, PR 2008]
Complex networks, synchronization and cooperative behaviour 25
Finding communities in weighted networks (1)
0 0.2 0.4 0.60
0.5
1
1.5
2
2.5
3
time
Qw
0
0.5
1
1.5
2
2.5
3
Qw
=0.4947
Synthetic example [Lou & Suykens, Chaos 2011]: community detectionby considering [D]ij = tij if ρij(t) > T and zero otherwise, where tij isthe time needed for nodes i and j to synchronization in the sense thatρij(t) > T .
Complex networks, synchronization and cooperative behaviour 26
Finding communities in weighted networks (2)
1030
24
16
19
23
21
15
9
1312
20
17
11
22
4
7
5
6
28 25 32
2926
14
8 3
2
1
18
31
3334
27
C3
C2C
1
C4
(a)
6 7 17 5 11 1 12 18 2 8 14 20 3 4 13 22 9 10 31 15 33 34 19 21 16 24 28 27 30 23 25 26 32 290
0.5
1
1.5
2
2.5
3
3.5
time
(b)
Qw
=0.4439
on the Zachary’s karate club network [Lou & Suykens, Chaos 2011]
Complex networks, synchronization and cooperative behaviour 27
Finding communities in weighted networks (3)
on the American football team network [Lou & Suykens, Chaos 2011]
Complex networks, synchronization and cooperative behaviour 28
”Programming” clusters into complex networks
- cluster design on a 20 × 60 lattice of identical Rossler oscillators.- cluster ”CHAOS” obtained from randomly distributed initial conditions.
[Belykh, Osipov, Petrov, Suykens, Vandewalle, Chaos 2008]
Complex networks, synchronization and cooperative behaviour 29
Overview
• Chaotic systems synchronization, Lur’e systems
• Cluster synchronization and community detection in complex networks
• Optimization using coupled local minimizers,cooperative behaviour
Complex networks, synchronization and cooperative behaviour 29
Optimization
Local optimization
+ fast
- local optimum
Newton, QN, LM, CG
Complex networks, synchronization and cooperative behaviour 30
Optimization
Local optimization
+ fast
- local optimum
Newton, QN, LM, CG
Global optimization
- slow
+ global search
GA, SA, swarms
Complex networks, synchronization and cooperative behaviour 30
Optimization
Local optimization
+ fast
- local optimum
Newton, QN, LM, CG
???
+ fast
+ global search
???
Global optimization
- slow
+ global search
GA, SA, swarms
Complex networks, synchronization and cooperative behaviour 30
Local optimization
• Consider the unconstrained optimization problem:
minx∈Rn
U(x)
with cost function U(·) continuously differentiable.
• Simple continuous-time steepest descent algorithm:
x = −η∇xU(x)
converging to a local optimum.
• Better local optimization methods:momentum term, Newton method, conjugate gradients, ...
Complex networks, synchronization and cooperative behaviour 31
Coupled local minimizers
• Essential idea for Coupled Local Minimizers (CLM):
1. consider two (or more) local optimizers and let them interact2. enforce that the optimizers should reach the same final state,
i.e. require state synchronization
• Realizing cooperative behaviour for optimization: based on coupling ofoptimization processes and master-slave synchronization
• Hierarchical scheme: objectives (cost functions) at the individual leveland at the group level
[Suykens et al., IJBC 2001]
Complex networks, synchronization and cooperative behaviour 32
Coupled local minimizers
weight space
cost
Multi−start local optimization
No interaction
weight space
cost
Coupled Local Minimizers
Interaction and information exchange
[Suykens et al., IJBC 2001]
Complex networks, synchronization and cooperative behaviour 33
Array consisting of coupled local minimizers
space
space
cost
cost
Complex networks, synchronization and cooperative behaviour 34
CLM: a toy example
• Example: consider the following objective
minx,z
U(x) + U(z) subject to x = z
Lagrange programming network:
x = −∇xU(x) − (x − z) − λ
z = −∇zU(z) + (x − z) + λ
λ = x − z
Complex networks, synchronization and cooperative behaviour 35
Toy example: double potential well
−6 −4 −2 0 2 4 60
100
200
300
400
500
600
700
800
900
x
U(x
)
0 20 40 60 80 100 120 140 160 180 200−10
−5
0
5
10
15
20
25
30
t
x,z,λ
The initial states x(0), z(0) are chosen to be in the two different valleys.The states x(t), z(t) converge to the global solution at x = z = −2.9
(blue: x(t) - red: z(t) - green: λ(t))
Complex networks, synchronization and cooperative behaviour 36
Lagrange programming network
• Problem statement:
minx∈Rn
f(x) subject to h(x) = 0
• Lagrangian: L(x, λ) = f(x) + λTh(x)
• Lagrange programming network:
{
x = −∇xL(x, λ)
λ = ∇λL(x, λ)
This can be viewed as a continuous-time optimization algorithm.
[Zhang & Constantinides, 1992]
Complex networks, synchronization and cooperative behaviour 37
CLM: more general formulation (1)
• Consider a group consisting of q optimizers {x(i)}qi=1
• Minimize average energy cost subject to pairwise synchronization states
minx(i)∈Rn
1
q
q∑
i=1
U(x(i))
subject to x(i) − x
(i+1) = 0, i = 1, 2, ..., q
• Boundary conditions x(0) = x
(q), x(q+1) = x
(1)
[Suykens et al., IJBC 2001]
Complex networks, synchronization and cooperative behaviour 38
CLM: more general formulation (2)
• Augmented Lagrangian (synchronization as hard and soft constraint)
L(x(i)
, λ(i)
) =η
q
qX
i=1
U [x(i)
] +1
2
qX
i=1
γi ‖x(i) − x
(i+1)‖22+
qX
i=1
〈λ(i), [x
(i) − x(i+1)
]〉
• Lagrange programming network:
{
x(i) = −η
q∇
x(i)U [x(i)] + γi−1[x
(i−1) − x(i)] − γi[x
(i) − x(i+1)] + λ(i−1) − λ(i)
λ(i) = x(i) − x
(i+1) , i = 1, 2, ..., q
Complex networks, synchronization and cooperative behaviour 39
Optimal cooperation
• Decrease of ensemble energy cost:
d〈U〉
dt=
1
q
qX
i=1
〈∂U [x(i)]
∂x(i)
, x(i)〉
=1
q
qX
i=1
〈∂U [x(i)]
∂x(i)
,−η
q
∂U [x(i)]
∂x(i)
+ γi−1[x(i−1) − x
(i)]
−γi[x(i) − x
(i+1)] + λ
(i−1) − λ(i)〉
• Optimal cooperation: LP problem in γ (scheduling of γi values)
minγ∈Rq
d〈U〉
dt|x,λ such that γ < γi < γ, i = 1, 2, ..., q
This incorporates the principle of master-slave synchronization.
Complex networks, synchronization and cooperative behaviour 40
Example: optimization of Lennard-Jones clusters
• In predicting 3D structure of proteins from amino acid sequences,potential energy surface (PES) minimization is often related to thenative structure of the protein.Benchmark problem: optimization of Lennard-Jones (LJ) clusters [Sali,1994; Wales, 1997, 1999].
• Cost function:
ULJ = 4∑
i<j
(1
r12ij
−1
r6ij
)
with rij the Euclidean distance between atom i and j (j = 1, ..., N).
• (LJ)38 which possesses a double-funnel energy landscape and is knownto be an interesting test-case [Wales, 1997, 1999].
• Important role of p(x(0)) ∝ exp[− 12σ2x(0)T
x(0)] (similar to consideringa confining potential in effective potential minimization methods).
Complex networks, synchronization and cooperative behaviour 41
Case (LJ)38
0 0.5 1 1.5 2 2.5 3
x 10−7
100
102
104
106
108
1010
1012
1014
t
ULJde
lta
Evolution of the cost function for q = 50 coupled local minimizers, reachingthe global minimum configuration for (LJ)38 with double-funnel landscape.
Complex networks, synchronization and cooperative behaviour 42
Case (LJ)150
Potential for application to larger scale problems
Complex networks, synchronization and cooperative behaviour 43
Example: CLM training of MLP neural networks
• CLM with state vectors x(i) (i = 1, ..., q) equal to the unknown weight
vectors θ(i) of the MLP.
• CLM training process corresponds to coupled backpropagation processeswith weight vector synchronization.
• The initial distribution of p(x(i)(0)) (i = 1, ..., q) (at time 0) plays animportant role, similar to the choice of a regularization constant (inmethods of minimizing errors and keeping the weights small).
Complex networks, synchronization and cooperative behaviour 44
CLM training of neural networks (1)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
x
y
- MLP training (10 hidden units) of a sinusoidal function (green) given 20 noisy data
- Application of scaled CG without early stopping leading to overfitting (red) and best
result by Bayesian learning with regularization (blue).
Complex networks, synchronization and cooperative behaviour 45
CLM training of neural networks (2)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5
−1
−0.5
0
0.5
1
1.5
2
x
y
CLM result which optimizes a sum squared error on training data withoutregularization of the cost function.
Complex networks, synchronization and cooperative behaviour 46
CLM training of neural networks (3)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x 10−5
1
1.5
2
2.5
3
3.5
4
t
U
CLM evolution (group of q = 20 optimizers) of the sum squared error costfunction during optimization
Complex networks, synchronization and cooperative behaviour 47
Alternative Formulation to CLMs
• Capture a group of optimizers within a ball and shrink the ball
• Objective:
minx(i)∈Rn,r∈R
〈U〉 + 12 ν r2
subject to ‖x(i) − x(i+1)‖2
2 ≤ r2, i = 1, ..., q
where 〈U〉 = 1q
∑qi=1 U [x(i)].
• Advantage: always easy to find a feasible point to the constraints duringthe optimization process.
[Suykens & Vandewalle, 2002]
Complex networks, synchronization and cooperative behaviour 48
CLM: extensions
• Coupled Newton methods with applications in civil engineering[Teughels, De Roeck, Suykens, 2002]
• Additional noise can be injected into the system [Gunel et al., 2006]
• Extensions to coupled simulated annealing processes with cost functionevaluations only [Xavier-de-Souza, Suykens, Vandewalle, Bolle, IEEE-SMC-B 2010].
Successfully applied e.g. for tuning parameter selection in kernel methods,being more efficient than grid search, SA or GA [K. De Brabanter et al.,CSDA 2010].
• Stability analysis of CLMs [Lou & Suykens, IEEE-TCAS-I, in press].
• Hybrid CLMs: occasional impulsive coupling, suitable for parallelimplementations [Lou & Suykens, 2012].
Complex networks, synchronization and cooperative behaviour 49
Conclusions
• Synchronization phenomena: naturally happening in a wide range ofsystems and complex networks.
• Lur’e systems: broad class of nonlinear systems, conditions for globalstability and global synchronization can be obtained.
• Community detection in complex networks: obtainable also through asynchronization process
• Coupled local minimizers: aims for global search together with fasterconvergence.
Complex networks, synchronization and cooperative behaviour 50
Acknowledgements (1)
• Colleagues at ESAT-SCD (especially research units: systems, models,control - biomedical data processing - bioinformatics):
C. Alzate, A. Argyriou, J. De Brabanter, K. De Brabanter, L. De Lathauwer, B. De
Moor, M. Diehl, Ph. Dreesen, M. Espinoza, T. Falck, D. Geebelen, X. Huang, B.
Hunyadi, A. Installe, V. Jumutc, P. Karsmakers, R. Langone, J. Lopez, J. Luts, R.
Mall, S. Mehrkanoon, M. Moonen, Y. Moreau, K. Pelckmans, J. Puertas, L. Shi, M.
Signoretto, P. Tsiaflakis, V. Van Belle, R. Van de Plas, S. Van Huffel, J. Vandewalle,
T. van Waterschoot, C. Varon, S. Yu, and others
• L. Chua, P. Curran, A. Huang, T. Yang, A. Munuzuri, M. Yalcin, S.Gunel, S. Ozoguz, G. Osipov, M. Komarov, V. Belykh, V. Petrov, S.Xavier-de-Souza, X. Lou, A. Teughels, G. De Roeck, S. Arnout.
• Support from ERC AdG A-DATADRIVE-B, KU Leuven, GOA-MaNet,COE Optimization in Engineering OPTEC, IUAP DYSCO, FWO projects,IWT, IBBT eHealth, COST
Complex networks, synchronization and cooperative behaviour 51
Acknowledgements (2)
Complex networks, synchronization and cooperative behaviour 52
Thank you
Complex networks, synchronization and cooperative behaviour 53