From Neuroscience to Mechatronics

From Neuroscience to MechatronicsFrom Neuroscience to Mechatronics

Presentation by Fabian DiewaldJASS April 2006

OutlineOutline

The human brainThe vestibulo-ocular reflex (VOR)The VOR – technical applicationsCamera stabilizing system at TU MunichA possible algorithmTesting the camera stabilizing systemAnother field of neuroscience and technical application: face recognitionConclusion and outlook

Learning from nature – a justified Learning from nature – a justified strategystrategy

first vertebrates ~500 million years ago

civilization of man since ~10 000 years

Consequently nature has a great time advantage!

The human brainThe human brain

cerebellum = "little brain"responsible for accurate movementinstructions by the forebrain insufficientinstructions have to be translated into accurate commands by the cerebellumcerebellum essential part in learning motor skills

Neurons – elementary components of Neurons – elementary components of the central nerve systemthe central nerve system

dendrites: "input" of a neuronaxon: "output"axon terminals/boutons contact other neurons or muscles

Neurons – elementary components of Neurons – elementary components of the central nerve systemthe central nerve system

transmission works electricallyinformation through the axon is encoded in changes of electrical potentialshort impulses with fixed intensityinformation is contained in firing frequency("pulse rate modulation")

Synapses – link between neuronsSynapses – link between neurons

gap against electrical transmissiontransmission between neurons is controlled by chemicals called neurotransmitterscontrolled transmission is important in regard to adaptation and learning

The structure of the cerebellar cortexThe structure of the cerebellar cortex

aaa

Diagram from Max Westby

Inferior Olive

Purkinje cells

climbing fibre

+-

-+

mossyfibre

Granule cells

Deepcerebellarnuclei

parallel fibres

mossy fibres = information input,synapse on granule cells

granule cells, ~20 per mossy fibre,half of all neurons are granule cells

parallel fibres=axons of granule cells,synapse on Purkinje cells

Purkinje cells:~200 000 synapses per cell,only output for cerebellar cortex

climbing fibre: axon of inferior olive;one single climbing fibre for one Purkinje cellwraps around dendritic tree of Purkinje cell;essential for "learning" inferior olive

The influential theory of Marr-AlbusThe influential theory of Marr-Albus

twice independently proposed:

Marr, David: “A Theory of Cerebellar Cortex”, 1969, Journal of Physiology 202: 437-470

Albus, James S.: "A theory of cerebellar function", 1971, Mathematical Biosciences 10: 25-61

major aspects:

climbing fibre carries a teaching signal

signal influences the synapses of Purkinje cells

consequently the intensity of certain inputs to the Purkinje cells can be controlled

The influential theory of Marr-Albus – The influential theory of Marr-Albus – disappointment by Marr himselfdisappointment by Marr himself

Marr 1982:

"In my own case, the cerebellar study (...) disappointed me, because even if the theory was correct, it did not enlighten one about the motor system – it did not, for example, tell one how to go about programming a mechanical arm."

OutlineOutline


The vestibulo-ocular reflex (VOR)The vestibulo-ocular reflex (VOR)

aim: stable picture of our environment/object we are staring at in spite of moving headonly use of visual information would be too slow for stabilization (visual processing delay about 50-100ms)signals of the vestibular organ (motion of the head) are more or less directly transmitted to the extra-ocular eye muscles, leading to process times of 5-10msvestibular signals are also influenced by visual information

The vestibulo-ocular reflex (VOR) – The vestibulo-ocular reflex (VOR) – experimentation with monkeysexperimentation with monkeys

VOR does an excellent job in monkeys as well as in other vertebrates under normal conditionsmagnifying or miniaturizing glasses cause abnormal image motion speed perceived during head motionmonkeys with glasses show problems with head motion in the beginningafter several days: considerable improvement is completedthen: taking the glasses awaymonkeys show problems againafter some days: normal behaviour againunconscious use of VOR, motoric issue, calibration needed typical case concerning the cerebellum

The vestibulo-ocular reflex – The vestibulo-ocular reflex – paths of informationpaths of information

vestibular organ(head motion)

eye muscles

Purkinje cells

direct path: very quick not accurate

(not sensible to changes in the system, e.g. changes in eye muscle strength)

adaptation path: teaching signal transferred on

climbing fibres "strengthens" or

"weakens" the synapses

adaptive path over Purkinje cells: transferred

on parallel fibres

synapses ofPurkinje cells

inferior olive

eye(visual information)

cerebellum

OutlineOutline


The vestibulo-ocular reflex – The vestibulo-ocular reflex – possible technical applicationspossible technical applications

face-recognitionsystems

camera systems in general

The vestibulo-ocular reflex – The vestibulo-ocular reflex – also an important automotive applicationalso an important automotive application

Affordabledriver-assistance systems

wide angle cameras used to obtain an overview of the environmentfor a closer look (road signs, number plates etc.) a telephoto lens is needed, which is quite sensitive to motion

Requirements concerning driver-Requirements concerning driver-assistance systemsassistance systems

camera stabilization only by optical means is too slowinertial measurement of head angular velocity needed

affordable hardwarecheap inaccurate sensors must be allowed (multi-sensor fusion of translation/angular velocity and visual information)inaccuracies have to be compensated automatically

similar problems/requirements as in biologythe vestibulo-ocular reflex may solve our technical problem!

OutlineOutline


The VOR – biological and technical The VOR – biological and technical analogonsanalogons

the structure of the cerebellum is quite homogeneous – although different information appears in different regions, these regions are similaridea of "cerebellar chip"today: normally no need for cerebellar chip as DSPs are very multifunctional and cerebellar algorithms can be implemented software-based (e.g. in Matlab)

How can we implement cerebellar structures as computer hardware?

DSP dSPACE DS1103 PowerPC,used in camera stabilizing system atTU Munich

The VOR – biological and technical The VOR – biological and technical analogonsanalogons

vestibular organ in the inner ear 6-DOF inertial measurement unitdirect pathway vestibular to muscles CAN bus systemcerebellum DSP with specific algorithmextra-ocular eye muscles servo motors

neuroscience/biology: mechatronics:

main difficulties:commercially available inertial measurement units are too big for driver-assistance systemsthe algorithm biology uses in the cerebellum has to be detected and implemented, which is obviously difficult(remember Marr's quotation!)

serial gimballed configurationtwo perpendicular axespan actuator driven by:4.5 Watt motorinner frame driven by:11 Watt Maxon motor withbacklash free HarmonicDrivegearboxeach axis controlled by differential encoders with 512 stepscamera: CMOS sensor with effective resolution of 750x400 pixelsin the future: only a mirror is moved leading to further miniaturization

Camera stabilization system at TU Camera stabilization system at TU Munich, Institute of Applied MechanicsMunich, Institute of Applied Mechanics

Coping with difficulty 1:Coping with difficulty 1:size of inertial measurement unit (IMU)size of inertial measurement unit (IMU)

smallest available intelligent IMU 50x38x25 mm³too big for driver-assistance-systemsnew IMU was developed within the FORBIAS project:

accelerometer with 3 axes on one chip

(translation), bandwith up to 640Hz

3 gyroscopes(rotation), bandwith

up to 100Hz

edge length:15 mm

OutlineOutline


Coping with difficulty 2 (main difficulty):Coping with difficulty 2 (main difficulty):implementing an adequate algorithm (1)implementing an adequate algorithm (1)

central ideas/principles:

motor movements are calculated according to angular velocity of measurement unit ("direct path")theoretically (perfect sensors/perfect production) this is sufficient for camera stabilizationbut: due to inaccuracies in production and sensors themselves there is always relative movement of the picture of the environment ("optical flow")the relation between angular rates and this relative movement is calculatedthe result dynamically influences a matrix w

Coping with difficulty 2 (main difficulty):Coping with difficulty 2 (main difficulty):implementing an adequate algorithm (2)implementing an adequate algorithm (2)

central ideas/principles:

this matrix w calculates a certain output out of angular velocitythis output is used to "clean" the signal of the direct path (velocity) additivelyresult: after some time the matrix w is "perfect" (learning completed) and always knows what to add to the measured angular velocity so that there is no optical flow any more, i.e. optical flow and angular velocity are decorrelated

consequently further cost-reduction possible by teaching the system after assembling and storing the matrix w on a chipbut in this case inaccuracies because of e.g. plastic mechanical deformation during use cannot be compensated

The vestibulo-ocular reflex – The vestibulo-ocular reflex – paths of informationpaths of information

vestibular organ(head motion)

eye muscles

Purkinje cells

direct path: very quick not accurate

(not sensible to changes in the system, e.g. changes in eye muscle strength)

adaptation path: teaching signal transferred on

climbing fibres "strengthens" or

"weakens" the synapses

adaptive path over Purkinje cells: transferred

on parallel fibres

synapses ofPurkinje cells

inferior olive

eye(visual information)

cerebellum

from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive Applications AMAA, Berlin, April 25-27, 2006 (in press)

Coping with difficulty 2 (main difficulty):Coping with difficulty 2 (main difficulty):implementing an adequate algorithmimplementing an adequate algorithm

pf 0E

00 J d

3x33x3

3x33x3

dpww

brainstem B = direct path

Sdiagonal

sensitivity matrix

Ddecomposition to

"parallel fibre signals"

central idea: decorrelation of angular rates and optical flow,decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow, angular velocity and its derivative

Decorrelator with weight matrix ,equivalent to synapses of Purkinje cells

I P C+ + -

w f pfT

f xf yf z

x y z x y zp E

3 x30

3 x3

03x3

d E3 x3

f (optical flow = teaching signal)

pf 00

0E J p

3x33x3

3x33x3

dpww

3333

3333

0

0x

dx

xxp

zyxzyx

z

y

xT

E

E

f

f

f

pffw

pf

JSb

dp www

aaa

Diagram from Max Westby

Inferior Olive

Purkinje cells

climbing fibre

+-

-+

mossyfibre

Granule cells

Deepcerebellarnuclei

parallel fibres

Decomposition: splitting physical input signal in

adequate inputs for decorrelator (decorrelator should be able to learn by derivatives of velocity as well, not only by velocity!)

biological equivalent: parallel fibres



pf 0E

00 J d

3x33x3

3x33x3

dpww


Sdiagonal

sensitivity matrix

Ddecomposition to




I P C+ + -

w f pfT

f xf yf z

x y z x y zp E

3 x30

3 x3

03x3

d E3 x3


pf 00

0E J p

3x33x3

3x33x3

dpww

3333

3333

0

0x

dx

xxp

zyxzyx

z

y

xT

E

E

f

f

f

pffw

pf

JSb

dp www

output given by weighted sum:• proportional output p and• derivative path d (added after integrator I)• are calculated out of the "input" angular velocity (vector pf) by the dynamic matrix w• J: Jacobean projecting angular rates on degrees of freedom of motion device



pf 0E

00 J d

3x33x3

3x33x3

dpww


Sdiagonal

sensitivity matrix

Ddecomposition to




I P C+ + -

w f pfT

f xf yf z

x y z x y zp E

3 x30

3 x3

03x3

d E3 x3


pf 00

0E J p

3x33x3

3x33x3

dpww

3333

3333

0

0x

dx

xxp

zyxzyx

z

y

xT

E

E

f

f

f

pffw

pf

JSb

dp www

learning rule:• similar to LMS algorithm of adaptive control theory• here: teaching signal = optical flow f; in biology: teaching signal = climbing fibre signals• covariance learning rule (Sejnowski 1977):

–If parallel-fibre firing pf (angular velocity and derivative) is positively correlated with climbing-fibre firing f (optical flow), reduce the weight w,–if parallel-fibre firing pf (angular velocity and derivative) is negatively correlated with climbing-fibre firing f (optical flow), increase the weight w,–if parallel-fibre firing pf (angular velocity and derivative) is uncorrelated with climbing-fibre firing f (optical flow), no change

Possible variants of the basic systems Possible variants of the basic systems (1)(1)

pf 0E

00 J d

3x33x3

3x33x3

dpww


Sdiagonal

sensitivity matrix

Ddecomposition to




I P C+ + -

w f pfT

f xf yf z

x y z x y zp E

3 x30

3 x3

03x3

d E3 x3


pf 00

0E J p

3x33x3

3x33x3

dpww

3333

3333

0

0x

dx

xxp

zyxzyx

z

y

xT

E

E

f

f

f

pffw

pf

JSb

dp www

delay ~100ms

delay ~10ms

for frequencies above ~2.5Hz the phase difference can get quite big=> instability in the learning process possible=> delay of pf-signal (~100ms) and smoothing filter removing high frequencies("eligibility trace")


3333

3333

0

0x

dx

xxp

zyxzyx

z

y

xT

E

E

f

f

f

pffw

Possible variants of the basic systems Possible variants of the basic systems (2)(2)

pf 0E

00 J d

3x33x3

3x33x3

dpww


Sdiagonal

sensitivity matrix

Ddecomposition to




I P C+ + -

w f pfT

f xf yf z

x y z x y zp E

3 x30

3 x3

03x3

d E3 x3


pf 00

0E J p

3x33x3

3x33x3

dpww

3333

3333

0

0x

dx

xxp

zyxzyx

z

y

xT

E

E

f

f

f

pffw

3333

3333

0

0...

xd

x

xxp

zyxzyx

z

y

x

E

E

fsign

fsign

fsign

w

pf

JSb

dp www

capacity of climbing fibre pathway (teaching signal) is limited=> unimportant data flow can be avoidedby only evaluating the sign of the teaching signal,i.e. the direction of the optical flow=> learning was not significantly worse comparedto learning with exact values


OutlineOutline


Other visual technical problems to be Other visual technical problems to be solved for driver assistancesolved for driver assistance

optical flow caused by camera rotation can successfully be removed by camera stabilization as just seen, but...line of sight may have to be kept on the roadrapid changes of viewing direction ("saccades") have to be implementedfurthermore: closer examination of certain objects (signs etc.) need visual tracking

Testing of the camera stabilization Testing of the camera stabilization system under laboratory conditionssystem under laboratory conditions

testing in the laboratory by mounting the system on a hexapodcreated sensed angular rates of ~100°/soptical flow was reduced from 6 pix/frame to less than 1 pix/frame after 2-3 minutes of adaptationthe improvement within this time was also subjectively viewable

Testing of the camera stabilization Testing of the camera stabilization system "on the road" (1)system "on the road" (1)

system mounted near the rear-view mirroradditional camera installed as well to detect points of interest for the camera with telephoto lenssystem tested in association with saccade and visual trackingvehicle velocity and yaw rate added to the system via CAN bus to improve tracking of space fixed objects

Testing of the camera stabilization Testing of the camera stabilization system "on the road" (2)system "on the road" (2)

three modes were tested:1. lane marker was focused in a distance of 60m, followed up to a

distance of 20m, then the next was focused on2. lane separation was focused on in a constant distance of 40m3. nothing was focused on but the camera was stabilized around a

constant line of sight

in all modes: bumps could be compensated

static wide angle camera

stabilized camera with telephoto lens

OutlineOutline


Learning from the human brain – Learning from the human brain – another exampleanother example

a current example: face recognitionexcellent recognition abilities by humans under different circumstances, e.g.

illuminationviewing angleposefacial expression...

need for technical face recognition systems (e.g. war against terrorism)consequently need to continue exploring the neuroscience of face recognition

Face recognition – the problem with Face recognition – the problem with different angles of viewdifferent angles of view

How can we/a computer tell that each face belongs to the same person?

from: Vatentin, D.; Abdi, H..; Edelman, B.: What represents a face: A Computational Approach for the Integration of Physiological and Psychiological Data, 1997

Face recognition – adaption of Face recognition – adaption of weights and output as weighted sumweights and output as weighted sum

learning step: changing the weights within the network so that the output is "1"

for the viewed person

"hidden units": one for each angle of view – the more similar the input is to the "prototype" face of

one unit, the more "active" the unit gets

testing step: unfamiliar view of the

face, but nevertheless the highest output is

produced for "Betty"

OutlineOutline


Conclusion and outlookConclusion and outlook

technical use of neuroscience is the key togiving machines several typical human abilitiesoptimizing service intervals andminimizing complexity of installing systems by self-learning abilitiescost reduction...

consequently interesting to a large variety of technical fields

Thank you!Thank you!

Thank you for your attention!

Fabian [email protected]

Date post:	21-Jan-2016
Category:	Documents
Upload:	buck
View:	29 times
Download:	0 times

From Neuroscience to Mechatronics

Documents