A Bird's-Eye View of PetaVision, theWorld's First Petaflop/s Neural Simulation*
Dan CoatesPortland State University,
Maseeh College of Engineering and Computer Science,
Portland OR
Parallel Implementations of Learning Algorithms:“What Have You Done For Me Lately?”
NIPS08Whistler, BC
December 13, 2008
Dan Coates Garrett Kenyon,Craig Rasmussen
Los Alamos National Laboratory, Los Alamos, NM
* The authors acknowledge the support of the National Science Foundation, under a grant administered by the New Mexico Consortium
12/16/08 Coates NIPS08: PetaVision 2
PetaVision Project at LANL
Goal: Achieve human-level performancein a “synthetic visual cognition” system
On: IBM/DOE Roadrunner petascalesupercomputer (or a multicore PC)
Running: A spiking LIF neural networkinspired by visual cortex.
12/16/08 Coates NIPS08: PetaVision 3
What level of abstraction?
Emulate the cortical circuits formid/low-level visual processing.
We model the gross architecture ofvisual cortex, trying not to violateproven neural science.
Binzegger, et. al.
12/16/08 Coates NIPS08: PetaVision 4
What are the crucial features of V1?
Retinotopic mapping.
Edge detectorsof Hubel & Wiesel
Distinct laminar neuralpopulations.
12/16/08 Coates NIPS08: PetaVision 5
What are the crucial features of V1?
Retinotopic mapping.
Edge detectorsof Hubel & Wiesel
Distinct laminar neuralpopulations.
12/16/08 Coates NIPS08: PetaVision 6
What are the crucial features of V1?
Bannister. Laminar circuit.
Retinotopic mapping.
Edge detectorsof Hubel & Wiesel
Distinct laminar neuralpopulations.
12/16/08 Coates NIPS08: PetaVision 7
Spiking neurons and specific connectivity
- efficient, possibly asynchronous operation- sparse inter-node communication
What are the elements, and how does that help us?
12/16/08 Coates NIPS08: PetaVision 8
Spiking neurons and specific connectivity
- connections are primarily local
- function inherent in wiring
What are the elements, and how does that help us?
Bosking, et al. “Patchy” connectivity expresses orientation preference ofhorizontal connections.
12/16/08 Coates NIPS08: PetaVision 9
Example: edge detection
V1 simple cells have been shown to respondlike a Gabor functions. We use 8 orientations.
12/16/08 Coates NIPS08: PetaVision 10
Beyond edges: long-range association field
Ben-Shahar and Zucker have proposedadditional connectivity patterns formalizedusing differential geometry. [Neural Computation, 2004]
Besides curve integration, such a schemecould also be used for shape-from-shadingand natural texture identification.
12/16/08 Coates NIPS08: PetaVision 11
Ben-Shahar and Zucker have proposedadditional connectivity patterns formalizedusing differential geometry. [Neural Computation, 2004]
Besides curve integration, such a schemecould also be used for shape-from-shadingand natural texture identification.
Beyond edges: long-range association field
12/16/08 Coates NIPS08: PetaVision 12
Beyond edges: long-range association field
12/16/08 Coates NIPS08: PetaVision 13
Summary of Biological Inspiration
Network structure for a computer visionsystem can be modeled after architectureof mammalian visual cortex.
There are analytic correlates of thesetechniques, although closed-formderivations are difficult.
Note: these connections have beenshown to be learnable, although wehard-code as mathematical functions.
12/16/08 Coates NIPS08: PetaVision 14
Implementation: software abstractions
PVLayer: Population of neurons. Retina, LIF.
PVConnection: Connectivity pattern,represented by a mathematical weight function.Anything-to-anything routing possible.
12/16/08 Coates NIPS08: PetaVision 15
Implementation: LIF
12/16/08 Coates NIPS08: PetaVision 16
Implementation: PVConnection
Connection kernels are translation-invariant.
12/16/08 Coates NIPS08: PetaVision 17
Implementation: Parallel Algorithm
Process each presynaptic event
Process each PVConnection:
Update effected postsynaptic neurons
Update each layer
Perform I/O
12/16/08 Coates NIPS08: PetaVision 18
How to interpret results?
Readout: Spike trains are post-processed forfiring rate. Temporal correlations such assynchrony and oscillatory power are alsomeasured.
12/16/08 Coates NIPS08: PetaVision 19
Roadrunner
12/16/08 Coates NIPS08: PetaVision 20
Roadrunner “core”
1 Opteron Core●1.8 Ghz● 4 GB DDR2
IBM PowerXCell 8i● 3 Ghz clock speed● 200 Gflops w/singleprecision & pipelining● 4 GB DDR2● SPEs = 256k
12/16/08 Coates NIPS08: PetaVision 21
Roadrunner node: triblade
12/16/08 Coates NIPS08: PetaVision 22
Roadrunner
3,240 nodes:- 2 Opteron dual-cores1.8Ghz,16 GB memory- 4 PowerXCell 8i
Infiniband connections
Peak system performance: ~1.7 Petaflop/s.
12/16/08 Coates NIPS08: PetaVision 23
Each node handles an image patch.
PetaVision SPMD on Roadrunner
12/16/08 Coates NIPS08: PetaVision 24
PetaVision SPMD on Roadrunner
Process local activity
Process remote activity
Update layer
Send output spikes
12/16/08 Coates NIPS08: PetaVision 25
Roadrunner SPMD Components
MPI
Coordination
Math: Euler IF & Connections
12/16/08 Coates NIPS08: PetaVision 26
Visual task
“Closed contour present?”
- No need for higher-levelknowledge
- Nontrivial
- Humans can solveeffortlessly. (psychophysics)
12/16/08 Coates NIPS08: PetaVision 27
Visual task - results
Prototypical network response.Color represents average firing rate.
12/16/08 Coates NIPS08: PetaVision 28
Thank you!
Questions?