Date post: | 19-Jan-2018 |
Category: |
Documents |
Upload: | elisabeth-hodges |
View: | 221 times |
Download: | 0 times |
Neural Network AlgorithmsReview, Quantum and Glial
Directions
Martin DimkovskiCSE 6111 Presentation
York UniversityMarch 31st, 2011
March 32st, 2011 Neural Network Algorithms 2
Presentation Goals Hope to increase awareness of NNs’ potential
Very general toolkit, applicable to many problems
(<10 min) Overview: Algorithm models Computational power Complexity, limitations
Interesting research directions (<5 min) Quantum (<5 min) Glia
March 32st, 2011 Neural Network Algorithms 3
Presentation Motivation Can’t explain
On a serious note…: ubiquitous (brains, computers, society, space-time fabric), accessible & good inspiration
Fight common over-simplification Backpropagation-only/mostly view Practical applications and solid theoretical
foundations exist for presented alternatives
0.5
March 32st, 2011 Neural Network Algorithms 4
Network Graphs Unidirectional: simple, feed-forward
Backpropagation is here
+ Bidirectional: recurrent, interactive Network dynamics comes up
++ lateral: resonance, competition, pattern completion
0.5
March 32st, 2011 Neural Network Algorithms 5
Learning Approaches Supervised vs Unsupervised Error-driven (ex: backprop)
Straightforward Mature Adaptive basis functions For well defined tasks input>output, functions
Hebbian-Style (more on next slide) More sophisticated, more bio-inspired, self-organizing But not as mature (still weak like error-driven in 1960s) Order and Chaos
Combinations (superior)
1.5
March 32st, 2011 Neural Network Algorithms 6
On Hebbian-Style Network dynamics, more complex graphs
Fire together – wire together Compete Resonate
Build internal models of environ. - Identify principal features
Constraint satisfaction Find energy function minima
Attractors = memorized patterns Deal with corrupt and partial memory
1.5
March 32st, 2011 Neural Network Algorithms 7
More Modeling Aspects Complex (superior) vs real valued
Temporal dynamics
Signal coding: Discreet, analog, pulse averaging, or Superior: detailed pulse/spike pattern modeling
Static vs dynamic weights (in between training) Stay the same for any input/output condition Superior: Adjust to input/output condition
Excitation - inhibition modeling Superior: not on the same weight
1.5
March 32st, 2011 Neural Network Algorithms 8
Use of Probability Bayesian: Solves a BIG problem
Over-fitting When noise and peculiarities become more attended-to than
the general features of interest Problem especially with error-driven, like backprop.
Solves it because it samples from whole posterior and does not depend on a single set of weights
To get a feeling, compare: MAP = argmax P( | data) E[] = P( | data) d
The benefit of noise To avoid local minima/maxima (ketchup)
1.5
March 32st, 2011 Neural Network Algorithms 9
Research Directions NNs have come a long way
Yet, still far below known upper bounds For precision, performance, and usability
What better place to turn for help, than back to our original inspiration? In green are my personal speculations
1.0
March 32st, 2011 Neural Network Algorithms 10
Cues to Quantum Classically unexplained brain features
Simultaneous synchronized stimulations in distant regions for same stimuli
highly structured in phase and amplitude Perception unification – global attractor states
Speculation brain has ingredients for macroscopic quantum state
High metabolic energy; extreme dielectric prop. Microtubules, superconducting waves, gap junctions (anaesthesia)
Interest in just plain quantum computing power
1.5
March 32st, 2011 Neural Network Algorithms 11
A (Qu)bit of Quantum Basics
If left alone – a linear ‘combination’ of basis states (in coherence)
| = ci|i Each |i is a single reality for us classical beings
(ex: |0 or |1) But in quantum world, they all exist at once
|ci|2 giving the probability
If ‘touched’ by anything – decoheres Into one of the basis states, as per probabilities
Entanglement Instantaneous sync link between remote qubits
1.5
March 32st, 2011 Neural Network Algorithms 12
How can Quantum Translate for Artificial
NNs?1. Run existing NN algorithms on quantum
computers …getting there, but will take a long time Extra slide in appendix
2. Could we simulate the quantum ‘spooky’ effects in new NN algorithms?
Using our classical computers
0.5
March 32st, 2011 Neural Network Algorithms 13
Simulating Quantum Effects
Maybe the brain uses certain quantum features for evolutionary reasons.
Could we program/simulate?: Synchronization and unification of distant physically
unconnected neurons?
Coherence and decoherence of macroscopic quantum states
even though we would have to use many more bits
Interference and quantum functions in discretized approximations?
1.0
March 32st, 2011 Neural Network Algorithms 14
Old View on Glia Myelinate for insulation only
Clean-up and recycle neurotransmitters
Feed and heal neurons
…But, Einstein’s brain Double the glia
0.5
March 32st, 2011 Neural Network Algorithms 15
Recent FindingsGlia-Neuron and Glia-Glia Information Processing
Listen to all neurotransmitters, and uses them to communicate with both glia and neurons
Control synapse formation and operations As many as 100,000 synapses per glia
Connect neurons which have no synapses between them, and correlate them
Run separate network in parallel to NNs
Control speed of neuron’s output (axon)
Most regulated genes during REM are in glia (integration/consolidation)
It’s a whole new brain out there… And there’s more:
2.0
March 32st, 2011 Neural Network Algorithms 16
Glia Quantum Correlates Brain-wide calcium broadcast network
Connect through gap junctions
Calcium messaging affects neural circuits Drive global broadcast waves
Glia Quantum Correlates Calcium stores related to microtubules Gap junctions as hypothesized
Quantum aspects might play a big function in glia networks
1.0
March 32st, 2011 Neural Network Algorithms 17
Glia as Biological Bayesian?
Accumulated effect from previous inputs (old posterior) serves as baseline for new input (new prior).
Glia excitations last second to minutes, compared to ms for neurons, and it span much wider
This could produce something alike cumulative data likelihood during the period t of glia excitation
Glia could then adjust/sample weights for neurons as per latest posterior (weight factors coupled to posterior)
Would need a mechanism for normalization to 1
)(*)1(*)(_)( tDatatPosteriortConstNormtPosterior
2.0
March 32st, 2011 Neural Network Algorithms 18
Conclusion Existing NN algorithms offer a rich toolkit for
computing Much more beyond plain backpropagation Take advantage of combinations and complex graphs Use as many of the superior modeling aspects as
affordable Use probability theory
Glia networks and interactions with neurons can be modeled in new algorithms
Might be possible to simulate quantum effects for more enhancements
1.0
March 32st, 2011 Neural Network Algorithms 19
The End Questions?
March 32st, 2011 Neural Network Algorithms 20
References O’Reilly, Y. Munakata, “Computational Explorations in Cognitive Neuroscience, The MIT
Press, 2000
V. Ivancevic, T. Ivancevic, “Quantum Neural Computation”, Springer, 2010
U. Ramacher, C. v.d. Malsburg, “On the Construction of Artificial Brains”, Springer, 2010
(Ed.) A. Volterra, P. Magistretti, P. Haydon, “The Tripartite Synapse”, Oxford University Press 2002
R. M. Neal, “Bayesian Learning for Neural Networks”, Springer, 1996
R. D. Fields, “The Other Brain”, Simon & Schuster 2009 S. Gupta, R. Zia, “Quantum Neural Networks”, Journal of Computer and Systems
Sciences 63, 355-383, 2001 A. A. Ezhov, D. Ventura, “Quantum Neural Networks”, Future Directions for Intelligent
Systems and Information Science. Physica-Verlang, 2000 J. J. Hopfield, “Neural networks and physical systems with emergent collective
computational abilities”, Proc. Natl. Acad. Sci. USA Vol 79, pp. 2554-2558, April 1982
Additional Slides
March 32st, 2011 Neural Network Algorithms 22
Existing NNs on Quantum Computers
Quantum Computing1. N-qubit register can contain all 2N
values at once2. You can have a quantum ‘circuit’
‘computing’ on all of them at once3. But when you ‘touch it’, you will get
one value only. 4. Goal – how to touch it, to get the
value you want, with high probability