8/10/2019 Biologically Inspired Massively-Parallel Computation
1/31
Biologically-InspiredMassively-Parallel
Computation
Steve FurberICL Professor of ComputerEngineering
The University of Manchester
1
8/10/2019 Biologically Inspired Massively-Parallel Computation
2/31
Turing Centenary
2
8/10/2019 Biologically Inspired Massively-Parallel Computation
3/31
Turing in Manchester
3
8/10/2019 Biologically Inspired Massively-Parallel Computation
4/31
Manchester Baby (1948)
4
8/10/2019 Biologically Inspired Massively-Parallel Computation
5/31
SpiNNakerCPU (2011)
ARM 968
5
8/10/2019 Biologically Inspired Massively-Parallel Computation
6/31
63 years of progress
Baby:
filled a medium-sized room
used 3.5 kW of electrical power executed 700 instructions per second
SpiNNaker ARM968 CPU node:
fills ~3.5mm2of silicon (130nm)
uses 40 mW of electrical power
executes 200,000,000 instructions
per second
6
8/10/2019 Biologically Inspired Massively-Parallel Computation
7/31
Energy efficiency
Baby:
5 Joules per instruction
SpiNNaker ARM968: 0.000 000 000 2 Joules per
instruction
25,000,000,000 times
better than Baby!
(James Prescott Jouleborn Salford, 1818)
7
8/10/2019 Biologically Inspired Massively-Parallel Computation
8/31
Bio-inspiration
Can massively-parallel computingresources accelerate our understanding of
brain function?
Can our growing understanding of brain
function point the way to more efficientparallel, fault-tolerant computation?
8
8/10/2019 Biologically Inspired Massively-Parallel Computation
9/31
Building brains
Brains demonstrate
massive parallelism (1011neurons)
massive connectivity (1015synapses)
excellent power-efficiency
much better than todays microchips
low-performance components (~ 100 Hz)
low-speed communication (~ metres/sec)
adaptivitytolerant of component failure
autonomous learning
9
8/10/2019 Biologically Inspired Massively-Parallel Computation
10/31
Neurons
multiple inputs, singleoutput (c.f. logic gate)
useful across multiple scales(102to 1011)
Brain structure
regularity e.g. 6-layer cortical
microarchitecture
Building brains
10
8/10/2019 Biologically Inspired Massively-Parallel Computation
11/31
Neural Computation
To compute we need: Processing
Communication
Storage Processing:
abstract model linear sum of
weighted inputs ignores non-linear
processes in dendrites
non-linear output function
learn by adjusting synaptic weights
w1
x1
w2x2
w3x3
w4
x4
yf
11
8/10/2019 Biologically Inspired Massively-Parallel Computation
12/31
0&
/
)(
Aset
f ireAif
IAA
xwI
ttx
A
A
i
ii
k
iki
Leaky integrate-and-firemodel
inputs are a series of spikes
total input is a weightedsum of the spikes
neuron activation is theinput with a leaky decay
when activation exceedsthreshold, output fires
habituation, refractoryperiod, ?
Processing
12
8/10/2019 Biologically Inspired Massively-Parallel Computation
13/31
Izhikevich model
two variables, one fast, one slow:
neuron fires when
v> 30; then:
a, b, c & d select behaviour
)(
140504.0 2
ubvau
Iuvvv
duu
cv
( www.izhikevich.com)
Processing
0 20 40 60 80 100 120 140 160 180 200-80
-60
-40
-20
0
20
40
0 20 40 60 80 100 120 140 160 180 200-14
-12
-10
-8
-6
-4
-2
0
2
v
u
13
http://www.izhikevich.com/http://www.izhikevich.com/8/10/2019 Biologically Inspired Massively-Parallel Computation
14/31
Communication
Spikes
biological neurons communicate principallyvia spike events
asynchronous
information is only:
which neuron fires, and
when it fires Address Event
Representation (AER)0 20 40 60 80 100 120 140 160 180 200
-80
-60
-40
-20
0
20
40
14
8/10/2019 Biologically Inspired Massively-Parallel Computation
15/31
Storage
Synaptic weights
stable over long periods of time
with diverse decay properties?
adaptive, with diverse rules
Hebbian, anti-Hebbian, LTP, LTD, ...
Axon delay lines
Neuron dynamics multiple time constants
Dynamic network states
15
8/10/2019 Biologically Inspired Massively-Parallel Computation
16/31
The Human Brain Project
An EU ICT Flagship project
headline 1B budget
54M initial funding 1stOctober 2013 to 31stMarch 2016
~900k to UoM
next 7.5 years funded under H2020 subject to review of ramp-up phase after 18 months
80 partner institutes, 150 PIs & Cis Open Call extended this
led by Henry Markram, EPFL
16
8/10/2019 Biologically Inspired Massively-Parallel Computation
17/31
The Human Brain Project
Research areas:
Neuroscience neuroinformatics
brain simulation Medicine
medical informatics
early diagnosis
personalized treatment
Future computing interactive supercomputing
neuromorphic computing
17
8/10/2019 Biologically Inspired Massively-Parallel Computation
18/31
SpiNNakerproject
A million mobile phoneprocessors in onecomputer
Able to model about 1%of the human brain
or 10 mice!
18
8/10/2019 Biologically Inspired Massively-Parallel Computation
19/31
Design principles
Virtualised topology
physical and logical connectivity are decoupled
Bounded asynchrony time models itself
Energy frugality
processors are free the real cost of computation is energy
19
8/10/2019 Biologically Inspired Massively-Parallel Computation
20/31
SpiNNakerchip
Multi-chip
packaging by
UNISEM Europe
20
8/10/2019 Biologically Inspired Massively-Parallel Computation
21/31
Chip resources
21
8/10/2019 Biologically Inspired Massively-Parallel Computation
22/31
48-node PCB
22
864 cores 2,592 cores
8/10/2019 Biologically Inspired Massively-Parallel Computation
23/31
SpiNNakermachines
23
20,000 cores 100,000 cores
8/10/2019 Biologically Inspired Massively-Parallel Computation
24/31
Building the 105 machine
24
8/10/2019 Biologically Inspired Massively-Parallel Computation
25/31
Networkpackets
Four packet types MC (multicast): source routed; carry events (spikes)
P2P (point-to-point): used for bootstrap, debug, monitoring, etc
NN (nearest neighbour): build address map, flood-fill code
FR (fixed route): carry 64-bit debug data to host Timestamp mechanism removes errant packets
which could otherwise circulate forever
Header (8 bits) Event ID (32 bits)
PER TST 0 -
Payload (32 bits)Header (8 bits) Address (16+16 bits)
PSQ TST 1 - SrceDest
25
8/10/2019 Biologically Inspired Massively-Parallel Computation
26/31
NetworkMC Router
All MC spike event packets are sent to a router
Ternary CAM keeps router size manageable at 1024 entries
(but careful network mapping also essential)
CAM hit yields a set of destinations for this spike event automatic multicasting
CAM miss routes event to a default output link
Inter-chip
0 0 1 00 X 1 1 X 000000010000010000 001001
On-chip
Event ID
26
8/10/2019 Biologically Inspired Massively-Parallel Computation
27/31
Topology mapping
06
07
03
09 01
07
01
94
Problem graph (circuit)
1
02
32
23
724
72
23
Node 94
14
15
Core 10
2
65
9
3
6
Synapse
10
2
7
11
1
8
12
0
1
3
0
1
2
0
1
2
0
22
3 3
23
72
94
0
-
0
23
72
94
3
2
3
23
72
94
3
2
2
23
72
94
0
1
0
23
72
94
0
2
-
23
72
94
-
1
2
Topology
Fragment of
MC table
27
8/10/2019 Biologically Inspired Massively-Parallel Computation
28/31
Problem mapping
SpiNNaker:
Problem: represented as a
network of nodes with a
certain behaviour...
...behaviour of each node
embodied as an interrupt
handler in code...
...compile, link...
...binary files loaded into core
instruction memory...
Our job is to makethe model
behaviour reflect
reality
...problem is
split into two
parts...
...problem topology loaded
into firmware routing
tables...
...abstract problem
topology...
The code says "send message" but has no
control where the output message goes
28
8/10/2019 Biologically Inspired Massively-Parallel Computation
29/31
Bisection performance
1,024 links
in each direction ~10 billion packets/s
10Hz mean firing rate
250 Gbps bisectionbandwidth
29
8/10/2019 Biologically Inspired Massively-Parallel Computation
30/31
SpiNNaker robot control
30
8/10/2019 Biologically Inspired Massively-Parallel Computation
31/31
Conclusions
We have come a long way in 60
years
x1010 improvement in efficiency We still dont have the computer
power to model the human brain
but we are getting there!
Manchester is still building
interesting machines