Hypercubes and
Neural Networks
bill wolfe
10/23/2005
Modeling
Simple Neural Model
• ai Activation
• ei External input
• wij Connection Strength
Assume: wij = wji (“symmetric” network)
W = (wij) is a symmetric matrix
ai ajwij
ei ej
Net Input
eaWnet
i
j
jiji eawnet ai
aj
wij
Vector Format:
Dynamics
• Basic idea:
ai
neti > 0
ai
neti < 0
ii
ii
anet
anet
0
0
netdt
adnet
dt
dai
i eaW
Energy
aeaWaE TT 21
net
netnet
ewew
aEaE
E
n
j
nnj
j
j
n
,...,
,...,
/,....,/
1
11
1
netE
Lower Energy
• da/dt = net = -grad(E) seeks lower energy
net
Energy
a
Problem: Divergence
Energy
net a
A Fix: Saturation
))(1( iiii
aanetdt
da
corner-seeking
lower energy
10 ia
Keeps the activation vector inside the hypercube boundaries
a
Energy
0 1
))(1( iiii
aanetdt
da
corner-seeking
lower energy
Encourages convergence to corners
Summary: The Neural Model
))(1( iiii
aanetdt
da
i
j
jiji eawnet
ai Activation ei External Inputwij Connection StrengthW (wij = wji) Symmetric
10 ia
Example: Inhibitory Networks
• Completely inhibitory– wij = -1 for all i,j– k-winner
• Inhibitory Grid– neighborhood inhibition
Traveling Salesman Problem
• Classic combinatorial optimization problem
• Find the shortest “tour” through n cities
• n!/2n distinct tours
D
D
AE
B
C
AE
B
C
ABCED
ABECD
TSP solution for 15,000 cities in Germany
TSP
50 City Example
Random
Nearest-City
2-OPT
http://www.jstor.org/view/0030364x/ap010105/01a00060/0
An Effective Heuristic for the Traveling Salesman Problem
S. Lin and B. W. Kernighan
Operations Research, 1973
Centroid
Monotonic
Neural Network Approach
D
C
B
A1 2 3 4
time stops
cities neuron
Tours – Permutation Matrices
D
C
B
A
tour: CDBA
permutation matrices correspond to the “feasible” states.
Not Allowed
D
C
B
A
Only one city per time stopOnly one time stop per city
Inhibitory rows and columns
inhibitory
Distance Connections:
Inhibit the neighboring cities in proportion to their distances.
D
C
B
A-dAC
-dBC
-dDC
D
A
B
C
D
C
B
A-dAC
-dBC
-dDC
putting it all together:
Research Questions
• Which architecture is best?• Does the network produce:
– feasible solutions?– high quality solutions?– optimal solutions?
• How do the initial activations affect network performance?
• Is the network similar to “nearest city” or any other traditional heuristic?
• How does the particular city configuration affect network performance?
• Is there a better way to understand the nonlinear dynamics?
A
B
C
D
E
F
G
1 2 3 4 5 6 7
typical state of the network before convergence
“Fuzzy Readout”
A
B
C
D
E
F
G
1 2 3 4 5 6 7
à GAECBFD
A
B
C
D
E
F
G
Neural ActivationsFuzzy Tour
Initial Phase
Neural ActivationsFuzzy Tour
Monotonic Phase
Neural ActivationsFuzzy Tour
Nearest-City Phase
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
to
ur
len
gt
h
10009008007006005004003002001000iteration
Fuzzy Tour Lengths
centroidphase
monotonicphase
nearest-cityphase
monotonic (19.04)
centroid (9.76)nc-worst (9.13)
nc-best (7.66)2opt (6.94)
Fuzzy Tour Lengthstour length
iteration
12
11
10
9
8
7
6
5
4
3
2
tour length
70656055504540353025201510# cities
average of 50 runs per problem size
centroid
nc-w
nc-bneur
2-opt
Average Results for n=10 to n=70 cities
(50 random runs per n)
# cities
DEMO 2
Applet by Darrell Longhttp://hawk.cs.csuci.edu/william.wolfe/TSP001/TSP1.html
Conclusions
• Neurons stimulate intriguing computational models.
• The models are complex, nonlinear, and difficult to analyze.
• The interaction of many simple processing units is difficult to visualize.
• The Neural Model for the TSP mimics some of the properties of the nearest-city heuristic.
• Much work to be done to understand these models.
a1 a2
a3
w12
w23w13
3 Neuron Example
000
111
100110
010
001 011
a1
a2
a3
Brain State: <a1, a2, a3>
000
111
100110
010
001 011
a1
a2
a3
“Thinking”
Binary Model
aj = 0 or 1
Neurons are either “on” or “off”
Binary Stability
aj = 1 and Netj >=0
Or
aj = 0 and Netj <=0
Hypercubes
n=0
n=1
n=2
4-Cube
4-Cube
5-Cube
5-Cube
5-Cube
http://www1.tip.nl/~t515027/hypercube.html
Hypercube Computer Game
00
01 11
100
1
2
3
2-Cube
0123
0 1 2 3
0110
0110
1001
1001
Q2 =Adjacency Matrix:
Hypercube Graph
I
nn
1
1nQ
I
Recursive Definition
Theorem 1: If v is an eigenvector of Qn-1 with eigenvalue x then the concatenated vectors [v,v] and [v,-v] are eigenvectors of Qn with eigenvalues x+1 and x-1 respectively.
Eigenvectors of the Adjacency Matrix
v
vQn
I
Qn 1
1nQ
I
v
v
vv
vv
vQv
vvQ
v
v
n
n)1(
1
1
Proof
= +1
11
= -1
1-1
= +2
1111
= 0
11-1-1
= 0
1-11-1
= -2
1-1-11
= +3
11111111
= +1
1111-1-1-1-1
= +1
11-1-111-1-1
= -1
11-1-1-1-111
= +1
1-11-11-11-1
= -1
1-11-1-11-11
= -1
1-1-111-1-11
= -3
1-1-11-111-1
+1
+1
+1 +1 +1 +1
+1
-1
-1
-1-1
-1
-1 -1
n=0
n=1
n=2
n=3
Generating Eigenvectors and Eigenvalues
Walsh Functions for n=1, 2, 3
1
1
1
1
-1
-1
-1
-1
000
001
010
011
100
101
110
111
eigenvector binary number
000
100 110
010
111000
001 011
x
y
z
x
=-1k=1
=-1k=1
=+1k=2
=+1k=2
=+1k=2
=-3k=0
=-1k=1
y
z n=3
n=3
Theorem 3: Let k be the number of +1 choices in the recursive construction of the eigenvectors of the n-cube. Then for k not equal to n each Walsh state has 2n-k-1 non adjacent subcubes of dimension k that are labeled +1 on their vertices, and 2n-k-1 non adjacent subcubes of dimension k that are labeled -1 on their vertices. If k = n then all the vertices are labeled +1. (Note: Here, "non adjacent" means the subcubes do not share any edges or vertices and there are no edges between the subcubes).
n=5k=3
reduced graph
n=5k=2
reduced graph
Schamtice of the 5-cube Schamtice of the 5-cube
n=5, k= 3 n=5, k= 2
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1 -1
Inhibitory Hypercube
Theorem 5: Each Walsh state with positive, zero, or negative eigenvalue is an unstable, weakly stable, or strongly stable state of the inhibitory hypercube network, respectively.