IMage Processing Spiking Elements IMPULSEgerezsh/sendfile/impulse-b.pdf · IMPULSE IST-1999-10421...

IMPULSE IST-1999-10421

PART B – Page 1 of 23

RTD ProposalsDescription of scientific/technological objectives

and workplanCall part identifier: IST-99-1-1A

Key Action IV (Essential Technologies and Infrastructures)

Principal Action line IV.2.2 "Real Time Systems"

B1. Title page

IMage ProcessingUsing Large arrays of

Spiking Elements

IMPULSE14 th June 1999

IST-1999-10421



B2 Content list (Part B only)

B1. Title Page 1

B2. Content list 2

B3. Objectives 3

B4. Contribution to programme/key action objectives 5

B5. Innovation 6

Spiking neurons

New coding strategies

Computational efficiency

Low-cost parallel hardware

A Natural User-Friendly System

B6. Project workplan 8

WP1 8

WP2 9

WP3 12

WP4 15

Project planning and time table 16

Graphical presentation of the project's components 17

Detailed project description broken down into workpackages: 18

Form B1 18

Form B2 19

Form B3 WP1 20

Form B3 WP2 21

Form B3 WP3 22

Form B3 WP4 23



B3. Objectives

The main aim of this proposal is the development of innovative image processing systems basedon novel forms of neural networks that use the computational power inherent in networks ofspiking neurons. To achieve this aim over the three years of the project we will need to address thefollowing four objectives.

1. Theoretical analysis of computing with spiking neurons.

Conventional neural networks use networks of units with continuous output values (typically in therange 0 to 1), but real neurons transmit information in the form of spikes. Recent results suggestthat the use of spikes radically transforms the sorts of computations that can be performed. A firstobjective will therefore be to perform a comprehensive analysis of the computational implicationsof using networks with spiking units. This includes the following specific objectives.

• Develop efficient codes for transmitting information. In networks of spiking neurons, there area wide range of largely unexplored methods for encoding and transmitting information inaddition to the classic rate coding schemes on which virtually all conventional neural networksschemes are based. These include codes that use synchrony between neurons, patterns of firingin individual neurons, and the order in which different neurons fire. The relative advantagesand disadvantages of these different forms of coding will be analysed in detail.

• Develop novel learning algorithms for networks of spiking neurons. There are a number oflearning algorithms which have been developed for conventional neural networks, but acompletely new set of rules are required when spiking networks are used. We will devise andtest a range of algorithms for performing both supervised and unsupervised learning.

2. Develop efficient systems for simulating networks of spiking neurons.

Networks of spiking neurons can, in principle, be simulated very efficiently using conventionaldigital computing architectures. We will produce a documented simulation package that will runon a variety of platforms and which can be used for research into spiking neural networks. Inaddition, we will work towards real-time simulation of large networks in a number of ways:

• Produce an optimised version of the simulator. Because of way in which a spiking neuralnetwork simulator is written, it is possible to optimise the code using assembler coding and bytaking advantage of the multimedia instruction sets like MMX (on Pentium processors) andAltivec (on PowerPC chips). We will develop versions of the software that can take fulladvantage of these features.

• Develop versions that can run on workstation clusters. Many computer scientists are nowconvinced that the most efficient way to obtain the highest performance/price ratio is to linktogether clusters of relatively low cost commodity PC hardware. We will develop documentedversions of the spiking neuron simulation system that can be run on such clusters.

• Develop low-cost, low-power multiprocessor hardware to allow real-time processing withlarge networks of neurons. An even more promising approach is to use minimal computingnodes (each composed of just a CPU, some memory, and an interface). One of the industrialpartners will deliver such a system and a version of the simulation software will be optimised torun on such hardware. Such hardware will form the core of the real-time image processingsystems that are planned.



3. Develop specific visual processing architectures.

With the spiking neuron simulation system available, we will be able to continue work into thedevelopment of visual system architectures adapted to solving particular problems in visualprocessing. For this work, we will try to model the architecture as closely as possible to the primatevisual system. Target problems include:

• Analysis of complex motion patterns. We will develop architectures for determining localmotion within video sequences. A first stage of processing will detect motion in a variety ofdirections, whereas higher levels will detect important compound motion phenomena thatinclude expansion (useful for determining time-to-collision), rotation and deformation (usefulfor determining aspects of 3D-geometry).

• Analysis of stereoscopic information. We will develop biologically inspired architectures foranalysing the 3D-structure of visual scenes on the basis of stereoscopic information. Particularattention will be paid to integrating motion and stereo-based information in the same system.

• Development of optimal initial processing strategies. The human visual system can perform avery wide range of scene and object recognition tasks on the basis of a defined set of initialprocessing operations. We will compare a variety of different strategies in order to obtain thebest performance possible for a given level of computational power.

4. Develop commercial applications.

The potential commercial applications of a low-cost, low-power real-time visual recognitionsystem are immense. One of the aims for the next three years will be to seek suitable applicationareas and partners interested in exploiting the technology. These will include:

• Transport Control. We have already an industrial partner with whom we are developing asystem for monitoring road traffic. This system will be able to determine what types of roadvehicles are present (cars, lorries, busses, motorbikes, cycles etc.), their positions and speeds.Such an ability will no doubt find widespread applications with the area of traffic monitoringand control.

• Face localisation and recognition. We will extend previous work on face recognition andlocalisation with a view to producing viable real-time recognition systems that can be used insuch applications as security systems.

• Vigilance analysis. A further potential area will be the detection of driver fatigue. Bydetermining in real time the time that drivers spend with their eyes closed, we should be able toprovide a very useful signal to warn when fatigue is setting in.

• Medical diagnostics. Rapid visual categorisation systems could also prove very useful in theanalysis of medical images, including the detection of blood cell anomalies (leukaemia,malaria). We will evaluate the performance of spike-based visual processing in such areas.



B4. Contribution to programme/key action objectives

The main thrust of the present proposal is clearly aimed at Key Action IV (Essential Technologiesand Infrastructures) and more specifically action line IV.2.2 (Real Time Systems). Our main aimis "to develop technologies and tools supporting the design and implementation of data- and/orcompute-intensive real-time applications, such as online high-volume information handling,including data acquisition … and … image processing, including innovative approaches to signalrepresentation and coding". At the end of the three years, we firmly believe that we will havedeveloped technologies that will "cover both consumer and professional applications". Forexample, the system for detecting driver fatigue could well have a high consumer impact, and willuse "cost-effective, mass market embedded systems built by the integration of new technology off-the-shelf hardware and software components". On the other hand the proposed professionalsystems for traffic control, surveillance and medical diagnostics will focus on "performance andmastering complexity, including behaviour-based, modular and flexible systems with built-in self-learning and self-repairing capabilities".

In addition, the proposal will also address some of the aims of other objectives of Key Action IVsuch as action line IV.6.1 (Adaptable multi-sensory interfaces) which targets the "developmentand demonstration of integrated multi-sensor subsystems using advanced sensor … technologiesincluding image and … scene processing … aiming at low-cost, mass-market applications."

While the immediate aim of the proposal is to develop the underlying technology, it is clear thatthe applications that are planned will have significant impact in a number of the areas covered byKey Action I (Systems and Services for the Citizen). In particular, the medical diagnostic systemsenvisaged are relevant to Action Line I.2.2 (Clinical, biological, managerial and imaging systemsfor health professionals) which covers "innovative medical systems for screening [and] diagnosis".Similarly, the traffic control application that will be developed is directly relevant to Action LineI.6.1 (Intelligent transport infrastructure and mobility management). This is specifically aimed atdeveloping "intelligent infrastructures for data capture [and] processing… to support traffic anddemand management [for] collective and individual transport". This program specifically targets"image processing, monitoring technology and sensors" aimed at "new traffic control systemsintegrating processing, simulation, prediction and decision-support tools, including tools formanaging large-scale events and crises". Furthermore, applications such as the driver vigilancesystem are clearly directly relevant to Action Line I.6.2 (Systems for intelligent vehicles) whichincludes the development of "on-board systems to improve the safety of vehicles, in particular interms of … alertness enhancement and impairment watch."

It should be stressed that the small number of application areas to which the proposal is currentlycommitted constitutes just a small sample of the very large range of applications which we believewill become possible using this sort of technology. In this sense, the proposal also fits very wellinto the objectives of the Future and Emerging Technologies Initiative which aims to "nurtureinvention, creativity, and bright-spark ideas [that] require longer-term research [and] have thepotential of leading to significant breakthroughs in industrial or societal terms."



B5. Innovation

The IMPULSE project is highly innovative in a number of ways.

Spiking neurons

We will develop a novel type of neural network processing architecture that is based on the use ofspiking neurons. While there has been a great deal of interest in artificial neural networks in thelast 15 years or so, the present proposal is motivated by the belief that there is at least onefundamental aspect of biological neural networks that is lacking from virtually all artificial neuralnetworks – real neurons transmit information in the form of a series of pulses, or spikes. Standardneural network techniques involve large networks of simple computing units, linked byconnections which have weights. Each unit in the system has an activation level, typically a realvalue lying in the range 0...1 or in the range –1 to +1. At each step, the state of each unit in thesystem is updated by taking into account the activation levels of each of its inputs, together withthe connection strength of the input. In the neural networks that we will investigate, the outputs ofeach neuron will no longer be continuous values. Instead, the neurons in a spiking network onlyemit pulses and there is no information transmitted in between spikes.

New coding strategies

One of the reasons for wanting to use spiking neurons stems from the fact that the use of spikesopens up a wide range of largely unexplored coding strategies. The use of continuous activationlevels in conventional neural networks is based on the idea that this might be a reasonablerepresentation of the firing rate of biological neurons. Indeed, it is generally assumed that it is thefiring rate that is the critical code in biological systems since it is well known that a neuron's firingrate typically increases monotonically with increasing activation, at least within a certain range.However, the range of firing rates is relatively low for most neurons – typically in a range between0 and 100 Hz, and this fact, together with the fact that the firing patterns of real neurons tends tobe very stochastic (spikes are generated by a process which is close to being Poisson in nature),means that firing rate is not a very efficient way of transmitting information from one neuron toanother.

In recent years, a number of authors, included several involved in the present proposal have begunstudying the possibility that biological neural networks use spikes not because evolution never ledto the development of copper wires capable of transmitting analogue signals reliably (!), butbecause they allow radically different types of coding and computational algorithms to bedeveloped. Three of the groups involved in the present proposal are actively pursuing this threadby using both analytical mathematical approaches as well as simulation studies. The results of thepast few years have been very encouraging because they have shown that not only are networks ofspiking neurons theoretically capable of performing essentially any arbitrary input-outputmapping but that they may be able to do so in ways which are far more efficient thanconventional approaches. For example, one of the groups involved has shown that a simple feed-forward network of spiking neurons can be used to perform non-trivial visual processing taskssuch as identifying and localising faces in a crowd. One of the aims of the current proposal istherefore to begin to explore the remarkable rich computational properties of spiking neuralnetworks and to see how such networks could be used to perform challenging visual processingtasks which, in the past, have eluded implementation using more conventional approaches usingcomputer vision and classic neural network architectures.



Computational efficiency

There is a second reason for wanting to work with networks of spiking neurons that stems notfrom its biological plausibility but rather from the fact that such networks can be simulated veryefficiently with fairly standard digital computers. To simulate a conventional artificial neuralnetwork it is necessary to transmit the activation values of each neuronal element in the network toeach of its targets at every time step. With relatively small neural networks, this sort of algorithmcan be implemented fairly straightforwardly on a conventional computer, but when large amountsof input data need to be handled (as is the case in image processing for example), such anapproach becomes unwieldy. There have been a number of attempts in the past to producespecialised neural network hardware, typically involving very specialised (and expensive) arrayprocessors, such as the Synapse processing system developed with European funding by Siemens.The problem with such an approach is that the development costs are high, and by the time thesystem reaches the market place conventional microprocessor technology has improved so muchthat the relative advantage of dedicated neural network hardware has often evaporated.

In contrast, when a network of spiking neurons is being simulated, communications betweenprocessing elements are vastly reduced. The only information that needs to be transmitted is a listof all the neurons that fired in the previous time step, and given that the average firing rate ofneurons is likely to be low, the total amount of information the needs to be transmitted isminimised. Because of this, it becomes possible to use clusters of computing elements, linked byfast communications links (gigabit Ethernet, for example, or in the case of the multiprocessorboards that we will develop, PCI bus interfaces). For most problems, parallelisation only producesa cost-effective increase in performance for relatively small numbers of processors (4-8), but witha spiking neural network, it becomes perfectly feasible to imagine real-time simulation with verylarge numbers of low-cost processing elements (for example, microprocessors in the StrongARMfamily) linked together. Developing such architectures is one of the primary objectives of thepresent proposal.

Low-cost parallel hardware

The low-cost multiprocessor boards that are one of the main deliverables of the present proposalare an innovative solution to a fundamental problem – how to obtain the maximalperformance/price ratio. By reducing each computing node to the strict minimum (for example, aStrongARM CPU, some SDRAM memory, and a PCI bridge) and using open-source software(Linux) coupled with very rapid communications via industry-standard PCI interconnects, we willbe able to offer a system that is not only ideally suited for performing very large scale real-timesimulations of spike-based neural processing, but one which will also find a wide range ofapplications in other areas where low-power, low-cost and small volume are at a premium.

A Natural User-Friendly System

Unlike many projects on Image Processing, the present proposal is firmly based on a desire toimplement processing strategies that are directly inspired from biology, and in particular, thefunctioning of the human visual system. An image processing system based on large arrays ofspiking neurons will tend to respond much as a human will – when decisions are easy, the systemwill respond quickly, when the input data is noisy, the system will naturally take longer. This sortof behaviour is precisely what humans are used to dealing with – and will enhance the user-friendly nature of the system.



B6. Project workplanSeven partners are involved in the current project – 4 university research departments (Partners 1-4) and 3 industrial partners (Partners 5-7). The project is divided into four basic work-packages:

• WP1 (Theoretical analysis of computing with spiking neurons) involves partners 1-4.

• WP2 (Development of the spiking network simulation software and hardware) involvespartners 1, 4, 5, 6 and 7.

• WP3 (Development of specific visual processing architectures) involves partners 1, 2, 3 and 6.

• WP4 (Commercial application development) involves partners 1, 3, 6 and 7.

All four workpackages will run for the entire three-year period of the project, with three clearlydefined phases corresponding to the end of each 12-month period. Note that each one of the fourworkpackages can start immediately, without having to wait for the results from one of the otherworkpackages to materialise. On the other hand, by breaking the project into three phases andorganising meetings involving all the partners at regular intervals, the interactions between thedifferent workpackages will be optimised.

WP1 -Theoretical Analysis of Coding, Computing, and Learning with Spiking Neurons

WP1 will concentrate on three main topics (1) Coding, (2) Computation and (3) Learningalgorithms.

Coding.

The question of coding is fundamental to the IMPULSE project. The classic view is that allinformation transmitted between spiking neurons is encoded in the firing rate. However, in recentyears, it has become clear that there are many other possibilities when one takes into account thetemporal pattern of spikes, both at the level of a single neurons output and across a population ofcells. One particularly promising approach is based on the fact that sensory neurones will tend tofire at short latencies when they are strongly activated. As a result information can be encoded inthe order in which a population of cells fires.

A possible application of such a scheme can be seen in the visual system where the retina cantransmit enough information to the brain in a few tens of milliseconds to allow categorisation of aflashed image. How can this be done? There are roughly 1 million fibres in the optic nerve, andthese fibres originate from cells in the retina that can be divided into two main classes – ON-centrecells (which respond well to a bright spot on a dark background) and OFF-centre cells (thatrespond best to a dark spot on a bright background). The conventional view of image codingsupposes that information is sent to the brain in the form of a firing rate code – parts of the imagewhere the contrast is highest provoke the highest firing rates. However, a recent theoretical analysishas shown that because of the roughly Poissonian nature of spike generation, this would be anextremely inefficient way of transmitting information about the image. Fortunately, it appears thatby simply looking at the order in which the output fibres fire, one can reconstruct the imageremarkably accurately on the basis of only a few spikes. Indeed, initial tests indicate that evencomplex images can be easily recognised when as little as 1% of the cells in the retina have firedone spike. In some ways the scheme has considerable similarities to techniques such as waveletcompression – the main difference is that the order-coding scheme is one that may haveimplemented in biological visual systems for hundreds of millions of years! One of the key aimsof WP1 will thus be to explore these novel forms of coding based on spiking neuronal networks.

Computing.

In conventional artificial neural networks, each unit performs a weighted sum of all the inputs, inwhich the activation level (the equivalent of a firing rate) of each input is multiplied by thesynaptic weight. Although this sort of computation is straightforward, it becomes very expensivewhen one wants to implement networks with large numbers of neurones and connections. Therehave been attempts to produce specific neural network hardware to speed such computations (forexample, the Synapse system developed by Siemens), but it has never been practical to performtasks like real-time visual processing within such a framework.



In contrast, computing with networks of spiking neurons is very different. No computations arerequired as long as no neuron has spiked, and so it becomes possible to simulate very largenetworks of neurons even with very large numbers of connections – the limiting factor is just theproportion of active neurons. Recent neurophysiological data suggest average firing rates withinthe visual processing areas of the brain as low as 1-5 spikes/second, and this indicates that real-time simulation of very large networks of spiking units may be within reach.

In addition, the switch to spiking neuron networks opens up a wide range of new computationalstrategies that WP1 will explore methodically. One idea that has already been the subject of activeresearch by partner 1 is the idea that neurons can be made sensitive to the order in which theirinputs fire by using a desensitisation mechanism that makes them increasingly less sensitive duringthe course of a wave of processing. Such a mechanism results in neurons that can be extremelyselective for particular input patterns, implying that using networks of spiking neurons not onlyimproves information transmission (i.e. coding) but also transforms the fundamental mechanismsof computation. A further primary objective of WP1 will be to explore these different possibilities,and to allow interesting computational strategies to be incorporated into the Spiking NeuronSimulation Hardware and Software developed in parallel by WP2.

Learning Algorithms.

WP1 will also concentrate on developing learning algorithms that can be used in networks ofspiking neurons. Here again, we will draw inspiration from work in neuroscience that has begun tounravel the complex set of rules that determine how the strength of synapses in the nervous systemcan be modified by experience. There is a large literature on learning algorithms for artificialneural networks, but as noted previously, the vast majority of these are designed for networks ofneurons with continuous outputs.

Two particular types of learning will be analysed. One is supervised learning that adjusts synapticweights in such a way that a neuron will respond selectively to stimuli associated with a trainingsignal. We already have one such rule, but it may well be that other variants would be even moreeffective. The second type of learning is unsupervised learning where neurons in a particularlayer are in competition and where the pattern of connectivity is modified such that each neuronresponds to a different type of input pattern. Here again, we already have some putative learningrules, but we will want to explore and evaluate other variants during the course of the project.

WP2 - Development of a real-time spiking neuron simulation system

Since there are good theoretical reasons for believing that networks of spiking neurons may becapable of capturing some of the essential features of biological vision systems, can we implementsuch networks efficiently using currently available techniques? Recent work by members of theconsortium suggests that this is indeed the case. We already have a functional Spiking NeuralNetwork simulation program written in highly portable C++ code that has been successfully runon a range of computer platforms including the Power Macintosh, Intel-based computers runningWindows 95/NT, and machines using Linux.

The principles on which the simulator is based are relatively straightforward. It uses a very simplemodel neuron with only a very limited number of parameters. There is an activation value, whichis set at zero for a neuron at rest, and which increases by steps every time one of the neuronsinputs fires as a function of the connection strength. If the activation value exceeds a certainthreshold value (which can be varied separately for each neuron), the neuron's number is added tothe list of neurons that fired on the current time step, and the activation value is reset to some fixedvalue.

Neurons are typically arranged into two-dimensional arrays, and neurons in different arrayslinked by sets of connections that are often the same for each neuron in an array (shared weights).A simple case could have one array corresponding to the retina where cells fire as a function oflocal contrast in the image, and a second array corresponding to the visual cortex which receivesconnections from the retina. In response to an image, a simple convolution (using, for example, adifference of gaussians operator) is applied to the retinal image to decide the level of excitationfor each retinal cell. From this, we can generate a list containing, for each time step, the co-ordinates of all the cells in the retina that will fire, starting with those that receive the strongestinput.

At each step, the simulator will take the coordinates of the neurons in the retina that fired, andupdate the activity values of neurons in the projection area, using a pattern of connectivity thathas been defined previously. When it does so it checks to see if any of them has gone over



threshold. If so, it adds the co-ordinates of the target neuron to the Spike list for next step. Thissame process is repeated until all the incoming spikes have been processed, at which point thesimulator can start propagating the spikes for the next step.

Note that the number of connections that can be updated in a given time effectively limits thespeed at which such a simulator can function. Currently, and using a PowerPC 750 processorrunning at 400 MHz, we can update roughly 30 million connections per second. Now, this valuecan be used to calculate a value for the maximum number of neurons that can be simulated in realtime, where we define real-time as being as fast as the equivalent network of biological neurons.The actual number depends on the number of neurons, the average number of connections perneuron, and the average firing rate.

What sort of visual processing architecture can be simulated with such a system? We know that thehuman visual system can process roughly 10 independent images per second, so this is areasonable criterion of "real-time". We now need to choose a value for the percentage of neuronsthat need to fire one spike in order to allow recognition. From our analysis of the percentage ofneurons that need to discharge in order to allow recognition (see previous section), we estimatethat no more than 10% of cells need to fire in order to perform many visual tasks, and the realvalue may in fact be considerably less. This means that at 10 images per second, and with roughly10% of cells firing in response to each image, the overall average firing rate will be around 1spike/second (a value consistent with neurophysiological data on firing rates in the primate visualsystem).

Let us now take an extremely simple architecture in which we have separate ON-centre and OFF-centre arrays in the "retina" connected to a single cortical area. For such an architecture, we wouldneed three times as many neurons as pixels in the image, and if we suppose that each cell makesroughly 100 connections (corresponding to a connection matrix of roughly 10 x 10), we couldprocess an image with 100 000 pixels (roughly corresponding to an image 384 * 256 pixels insize) at 10 frames per second. Note that this calculation does not take into account the time takento perform the initial DOG convolution, but this operation can now be performed relativelyquickly with multimedia instruction sets such as MMX (on Pentium-class microprocessors) orAltivec (on the soon-to-be-released G4 series of PowerPC chips).

From this quick "back-of-an-envelope" calculation, it is clear that using this form of spikingnetwork simulation system, it is already possible to perform some very simple types of processingon realistically sized images at reasonably fast rates. However, as will be seen in the sectionconcerning our work on visual processing architectures (WP3), interesting image processingproblems will require far larger networks, with much greater levels of connectivity, and for thisreason, real-time applications will need far more computing power than can be provided by asingle workstation.

For example, we have recently looked at an architecture capable of identifying and localising thefaces of 40 different people in a crowd scene. This architecture involves some 3 000 000 neuronsand 40 billion connections, and although we are able to run the simulation on a desktop computer,the time required to propagate the spikes through the system was over 30 seconds, a time lag thatis clearly far too long for real-time applications.

Ultimately, we would like to be able to simulate systems that are close to the scale of the primatevisual system, but it is clear that to do this will require a great deal of effort. It is estimated that theprimate visual system contains 10 11 neurons, each of which makes between 10 3 and 10 4

connections. With an average firing rate of 1 spike/second this would require that the simulator becapable of around 1014 connection updates per second – more than 10 million times faster thancurrently possible. For this reason, one of the principle aims of WP2, and indeed the wholeIMPULSE program is to progressively increase the performance of the spiking neuron simulationsystem.

Improving real-time performance

There are a number of ways that we will use to tackle this problem. One is to improve theefficiency of the basic code using techniques such as assembly language coding, and this willcertainly be one of the early approaches used in WP2. But another is to make use of workstationclusters. In recent years, many computer scientists have concluded that the most cost-effective wayof obtaining high performance is to link together large numbers of relatively standardworkstations into a cluster using standard off-the-shelf interconnect techniques such as Fast-Ethernet or more recently Gigabit Ethernet. Such so-called "Beowulf-class" systems now rank withthe most expensive supercomputers (a cluster of alpha-powered workstations running Linux was



recently ranked 98 th in the list of the top 500 Supercomputers in the worldhttp://www.top500.org/). One of the primary aims of WP2 will therefore be to develop versions ofthe spiking neuron simulator that will run efficiently on a cluster of Linux-powered workstations.

There are various reasons for believing that such an approach will work very well. For manycompute-intensive tasks splitting a problem between several processing nodes only producessignificant improvements in performance to a limited extent because of commnication overheads.However, one of the very attractive features of software that simulates networks of spiking neuronsis that the communications bandwidth required can be remarkably low. Suppose we simulate anetwork with 5 million neurons using 5 workstations linked together in a cluster, and that real-timeperformance requires an average firing rate of 1 spike/second per neuron. Each second, eachworkstation would need to transmit the coordinates of 1 million units, which, using a code thatused 20 bits per neuron, would require a debit of 20 Mbits per second. By using multicasttechniques it would theoretically be possible to simulate such a 5 million neuron network in realtime without requiring more than standard 100 Mbit Fast Ethernet connection technology. Andwith techniques such as Gigabit Ethernet and the VIA protocol (Virtual Interface Architecture), itwill soon be possible to increase the network traffic by a factor of 10. It is therefore clear that amajor effort will be made by the participants in WP2 to take advantage of such techniques in aneffort to extend the range of real-time processing.

Multiprocessor Hardware.

However, while using workstation clusters is a reasonable way to start exploring multiprocessorimplementations of the spiking neuron simulation system, a major objective of IMPULSE is todevelop specific low-power embedded hardware. There are a number of reasons for this.

First, unlike many compute-intensive problems, simulating networks of spiking neurons does notrequire sophisticated floating-point hardware. Indeed, all the current code can operate with valuesencoded as short 16-bit integer arithmetic. Thus, much of the high technology used in many mainstream microprocessors (Pentium III, K7, Alpha, PowerPC) is not necessary. Instead, one candesign systems that use much simpler microprocessor designs, such as the StrongARM processorfamily.

One of the partners in the IMPULSE project has had extensive experience in designing andmanufacturing StrongARM based systems, and has already produced some prototypes of a smallCPU module which consists of a StrongARM SA-110 microprocessor with a clock frequency of233 MHz, a 21285 PCI bridge circuit, and 64 Mbytes of SDRAM. Tests have shown that such amodule is capable of executing the spiking neuron simulation software at a speed equivalent to aPentium class processor at the same clock speed. This is impressive, for a processor which costsonly $29 in quantity and which consumes less than 500 mW of power. Indeed the total powerconsumption of such a module is under 2 watts, which makes it possible to pack large numbers ofsuch CPU modules into a small volume without the need for additional ventilation. Indeed, duringthe first phase of the IMPULSE project we will have an 8 CPU module PCI board, four of whichcan be mounted in a standard PC. This would mean a total of 32 StrongARM CPUs each with 64Mbytes of fast local memory (a total of over 2 GBytes of Memory) in a standard PC. The lowproduction costs associated with such hardware mean that it would be possible to produce such asystem for a very competitive price.

In conventional workstation clusters, communications between processors are handled usingstandard Ethernet interconnects (typically limited to 100 Mbits per second). But in the case of themultiprocessor boards, communications between processors make use of local PCIinterconnections that allow transfers at up to 132 MBytes per second – more than 10 times thedebit of fast Ethernet. Furthermore, since each PCI board would effectively have its own local PCIbus, the total aggregate bandwidth within a PC equipped with 4 of the 8 CPU boards could be ashigh as 2 Gigabytes per second.

Hardware development will be divided into three phases as follows:

• Phase 1 (year 2000). (1) Develop 8 CPU module carrier board using existing CPU modules.This board will be based on existing technology that requires a re-design in order tomaximise the number of CPUs per board and reduce the cost. (2) Develop a new SA-1 basedCPU module with improved 64bit/66Mhz PCI interface. We would also use newer memorydevices to reduce the chip count and hence the size of the module. Alternative CPUs (forexample, PowerPC or MIPS-based devices) will be evaluated and used if they offer betteroverall performance. (3) Develop a new multi-processor carrier board, using the new CPUmodule. The processor interconnect topology is envisaged as being 64bit/66Mhz PCI,



although alternatives will be evaluated and implemented if they are more suitable e.g. aSebring ring architecture (www.sebringring.com).

• Phase 2 (year 2001). (1) Develop a new SA-2 based CPU module with an estimated600MHz CPU, using a 64bit/66MHz PCI interface. At this time it is not known whatimprovements there will be to the memory bus although it is reasonable to assume that it willhave improved from 60MHz to 100MHz and from 32bits wide to 64 bits wide. Again, newermemory devices will be used in order to reduce the overall size of the module. (2) Evaluatealternative processor interconnect topologies which are currently expected to include Intel'sNGIO, PCI-X or Future I/O.

• Phase 3 (year 2002). (1) Implement the alternative processor interconnect topologieschosen in phase 2. (2) Develop fourth generation CPU module using latest available CPU.The possible CPUs could include next generation StrongARM, ARM10, (ARM11?) or someof the proposed devices from Obsidian Technologies (http://24.1.163.6//ot.html) who areplanning to produce single chips with 16 or even 64 ARM processors cores each with its ownlocal memory.

Hardware developments on their own would be useless without the availability of a high-performance operating system environment. For this reason, one of the partners in the IMPULSEconsortium will develop an open-source Linux-based operating system to run on the hardware ateach stage of the project. This task will require considerable programming skills because of theneed to replace standard TCP and UDP based message passing mechanisms with routines thatmake optimal use of the available interconnect technology. Thus, in phase 1 of the project we willhave carrier boards with 8 CPU modules that can be fitted into a standard desktop computer. Onthe operating system front, we will provide a Linux implementation for the StrongARM boardsthat uses the PCI bus for message passing. Note that this software will be made public, whichmeans that the same hardware/software environment could be easily used by anyone wishing tobuild a low-cost high-performance computing cluster.

At the same time, we will produce optimised versions of the spiking neural network simulationsystem that will run efficiently on each of the different phases of hardware.

WP3 – Visual Architecture Development

The three phases of hardware design and development in WP2 will open increasingly high levelsof real-time performance, starting with 30 million connection updates per second in 1999, andincreasing progressively to an estimated performance of 100 billion connection updates/second in2002. Such a system would allow networks with as many as 100 million neurons to be simulatedin real-time, where real-time is defined as being "as fast as the equivalent biological network".

However, producing a system that is capable of real-time visual recognition in complex naturalscenes is not simply a question of being able to simulate very large networks of neurons. Anequally important task is to devise suitable visual processing architectures to solve the targetproblems. This is the goal of WP3.

WP3 will not be starting from scratch, since a number of relatively simple architectures havealready been developed. For example, one architecture which has proved remarkably successful invisual categorisation and recognition tasks takes standard ON- and OFF-centre maps (roughlyequivalent to the primate retina) and projects the outputs from these maps to 8 second layer maps.The connections produce second layer maps that are each selective to contours at a differentorientation. These orientation selective maps can then feed into third layer maps, which can betrained (using a training algorithm which has already been developed) to respond to key featuresin an image (e.g. eyes, mouth in the case of a system designed to detect and locate faces). Finally,the feature level maps project to a final layer containing units trained to respond to imagescontaining eyes and mouth in the appropriate positions.

Despite the simplicity of such an architecture, performance is remarkably good. In a recent test, anetwork was trained on a set of 400 face photographs, consisting of 10 different views of 40different people. The final layer contained 40 different maps, each trained to respond to aparticular person. Following training, the network was able to correctly identify all 400photographs. Furthermore, when presented with a montage containing the various photos inrandom positions, the network was able to correctly report the locations of the people present. It isdemonstrations of this type which convince us that the potential applications of such a processingsystem will be extremely large.



Note, however, that the architecture used is really very simple when compared with the complexityof the processing architectures used in the primate visual system. Thus a major objective of WP3will be to use knowledge about the physiological and anatomical organisation of the primatevisual system as a blueprint for designing new processing architectures. To achieve this aim,several members of the consortium will draw on their expertise and contacts in the area ofbiological vision. Specifically, they will develop specific architectures for a number of visualprocessing problems.

Low Level Form Vision

The first implementations developed for use with the spiking network simulator have used a verysimple architecture (ON- and OFF-centre cells in the first, retinal, level project to 8 orientation-tuned maps in the second level). This initial choice was fairly arbitrary, and one of the firstquestions to be addressed will look at the performance of systems with different numbers oforientation maps. With 8 orientation maps, the separation between the orientations is 45°, but wewill also explore variants with, for example, 6 maps separated by 60° or 12 maps separated by 30°.For each architecture, performance will be evaluated in order to obtain an optimal level ofperformance for a given level of hardware.

A similar evaluation will be performed in the spatial frequency domain. In currentimplementations, the incoming image is processed at a range of spatial scales. Thus a 240 by 240pixel image can be processed at full scale, at half scale (120 by 120 pixels), at quarter scale (60 by60 pixels), and at eighth scale (30 by 30 pixels) in order to be able to recognise the same object(such as a face) at different sizes. But there are numerous other possibilities for analysing atintermediate scales, including subsampling at 80 by 80 pixels, 40 by 40 and so on, and thesedifferent options will be considered with the specific aim of optimising the performance/speedratio for particular visual tasks.

The primate visual system has extensive horizontal connections between different orientationtuned columns that can be functionally both excitatory and inhibitory. Current theory suggeststhat excitatory connections between aligned units sharing compatible orientation selectivities canbe used to enhance weakly defined contours. This is believed to be a very important process incleaning up otherwise very noisy images. Inhibitory horizontal connections between units withsimilar properties could play a vital role is segregating regions with differing textures, byaugmenting the salience of regions in the image where orientation or spatial frequency change.Another of the aims of WP3 will therefore be to perform an extensive analysis of the benefits ofusing such horizontal connections, again with the aim of optimising performance on particularrecognition tasks.

Motion analysis

Work has already been initiated on the question of how networks of spiking neurons could beused to process motion in video sequences. First, we will want to devise an efficient scheme foranalysing the optic flow field - by implementing a series of direction selective maps in the secondand third stages of the visual processing hierarchy. Once these local motion maps have beenproduced, it is possible to combine their signals in order to detect more interesting higher-ordermotion patterns. For example, consider the following three patterns of motion:

↑← Ο → Implies local expansion.

↓

←↓ Ο ↑ Implies local rotation.

→

↑→ Ο ← Implies local shearing.

↓



A real-time image processing system that could detect the presence of such higher-order motionphenomena could prove extremely useful. For example, the rate of local image expansion can beused directly to calculate time-to-collision, since for geometrical reasons when the image of anobject doubles in size in T frames, we can predict that the object will collide with the camera in 2Tframes. Such a system could potentially have extremely important applications in areas such asdriver assistance since it could signal situations where braking is insufficient to avoid a collision.

Stereoscopic analysis

We will also investigate processing architectures that use the additional information gained fromhaving two cameras separated by a horizontal distance (just as the two eyes can be used by theprimate visual system). By feeding the outputs of the two separate "retinal" images to neural mapsat later stages, we can use the same strategies used by the primate visual system to detect thepresence of objects within the plane of fixation as well as behind and in front of the fixationplane. Such 3D information could greatly enhance the ability of a visual system to recogniseparticular objects on the basis of their shape.

Furthermore, coupling stereoscopic viewing with motion analysis could also be very useful. Forexample, when the image of an object seen through the left eye moves to the right but moves tothe left when seen through the right eye, the implication is very clear - the object will hit theobserver between the two eyes. Collision-detection neurons of this type have been reported in avariety of animals, and could be easily implemented in a spiking neuron based processing system.

Colour

Much information can be determined purely on the basis of luminance cues (i.e. from grey-scaleimages), but the availability of cheap colour video systems makes it useful to examine how colourprocessing could be implemented using networks of spiking neurons. Here again, we will drawinspiration from our knowledge of the primate visual system where it is known that the retina notonly has ON- and OFF-centre cells influenced by local luminance, but also wavelength sensitivecells which are influenced by changes in the ratio of activity in the different cone systems. Asimilar strategy could be used for the analysis of RGB colour signals by using, for example, onechannel that responds optimally to R/G contrast and another interested by contrast between theBlue channel and the average R and G signals.

The main aim would not be a simple analysis of the RGB characteristics of each point in theimage, but rather, as in the primate system, to determine the true colour of objects by eliminatingthe effects of background lighting. Thus, the banana in a photograph of a bowl of fruit looksyellow to a human observer, even when the lighting of the scene is such that the wavelengthcomposition of the light reflected by the banana is highly biased toward the red end of thespectrum. This ability, known as colour constancy, is an important feature of our own visualsystems, but one that has proved difficult to implement in machine vision systems. One of the aimsof WP3 will therefore be to attempt to produce a real-time visual processing system that canreproduce this property.

Position invariance

There is a simple way of producing a visual system that is capable of recognising a particularobject, irrespective of its position in the visual field, which involves having a specialised detectorfor each point in the image, but it is an approach that needs very large numbers of neurons. In theprimate visual system, natural selection appears to have favoured a solution where the identityand the position of objects in the visual field are analysed separately - the so-called ventral corticalvisual pathway appears specialised for determining object identity, whereas the so-called dorsalpathway is specialised for determining spatial information. We will therefore examine in detail theadvantages and disadvantages of such an arrangement. Note however, that it is not necessarilyclear that the solution adopted need follow that chosen by natural selection. In the primate visualsystem, space is at a premium, since each neuron requires not only space (around 100 000micron3) but also energy, and so the need to keep the total neuron count to an absolute minimumis clear. In contrast, in a spiking neuron simulation system, each neuron only requires around 20bytes of memory, and with current memory technology advancing at the rate that it is, the energyrequirements and space used by 20 bytes of RAM become almost negligible. A single chip willsoon be able to store 64 Mbytes of memory - enough to implement over 3 million neurons.Under such conditions, the real issue will be to optimise the performance of the simulation system- not necessarily to keep the neuron count low. Remember that what limits the speed of thespiking neuron simulation system is not the number of neurons so much as the average firing rateof the neurons in the system. Neurons that do not fire have no computational overhead - they just



need lots of RAM - but since RAM is becoming cheaper and cheaper, this may well becomelargely irrelevant.

Object recognition

In the end, the ultimate objective of work in WP3 is to devise architectures that can supportrecognition of objects. Our initial studies have been very encouraging, because we have alreadybeen able to produce systems that can identify faces, cars and other road vehicles. However, webelieve that one of the keys to producing a truly efficient object recognition system comes fromimplementing a very large number of object feature classes. This stems from the way in whichasynchronous spiking neural networks operate. Essentially, we use a first past the post strategy inwhich several different mechanisms operate in parallel, and the first one to receive sufficientsupport to reach its threshold will fire and, as a result of horizontal competitive connections,prevent other mechanisms from also firing. Thus we can have, for each point in the image a largenumber of possible features in competition, and we suspect that the overall performance of thesystem will improve with the number of candidate features.

WP4 – Application Development

In the end, it is by showing that large-scale arrays of spiking neurons can perform visual tasksmore efficiently than conventional approaches that we will be able to demonstrate the real value ofthe IMPULSE project. It is for this reason that we place a particularly high priority on developingreal-world applications. Here again, the state of advancement of the project is such that we will beable to start application development right from the word go since some preliminary ground workhas already been initiated.

Traffic surveillance system.

A collaboration already exists between the co-ordinating partner and an industrial company(partner 7) aimed at developing a traffic surveillance and control system using the spiking neuronsimulator. Tests have already been initiated using real video sequences of traffic in an urbanenvironment, and performance is encouraging. With a small 120 by 100 pixel window we are ableto detect the presence of a single class of road vehicle at roughly 10 frames per second, a level ofperformance sufficient for real-time operation. At the same time, since the system provides the XYco-ordinates of the vehicle within the window, it is a relatively straightforward question todetermine the speed of each vehicle. On the other hand, it is clear that we will need the higherlevels of performance provided by the hardware acceleration developed in WP2 if we are toprovide a more practical level of functionality. For example, we will want to increase the numberof classifications provided (car, van, lorry, bus, motorcycle, bicycle and so forth), but each addedcategory will slow down the system. Also, the value of 10 frames/second currently used was onlypossible because we used an extremely simple visual processing architecture. It is likely that theaccuracy and robustness of the system will need to be improved for a true commercial application.Finally, a practical system will need to have several analysis windows open within the scene, eachproviding information about a different part of the environment.

A vital component of the system is the built-in learning algorithms that allow the system to betrained "in-situ" for a particular traffic-monitoring task. Thus, in order to develop a functionalsystem, we would first connect a video camera to the system and then start a training phase inwhich the operator must (i) specify the part of the visual scene which is of interest, (ii) save animage of the Region of Interest every time one of the target vehicles passes through it, and (iii)manually click on an appropriate location on the test image (for example, the centre of the wind-screen, or the hub-cap of the front wheel). These training images, together with the XY co-ordinates of the targets are then used to set the connection strengths in the spiking networksimulation system.

It is important to realise that because of the very flexible nature of the system, several differentrecognition and categorisation tasks can be implemented at the same time. Thus one could (a)determine how many vehicles are waiting at a particular traffic light, (b) determine whether thetraffic lights are functioning correctly (by training a sub-network to respond every time the "go"light comes on, (c) determine whether a car has stopped in the centre of the junction, and so on.

The development of this traffic surveillance and control system will progress in three phases,coupled to the other WorkPackages in the IMPULSE program. Our aim during phase one is todevelop a fully functional test system using standard hardware components (commercial videocamera + frame grabber, desktop computer running Windows 98/NT). If possible we would like toinclude the first phase of multiprocessor hardware boards. During the second phase, we will



produce an autonomous system in which the camera, video capture system and processing are allintegrated such that the system could be mounted as an autonomous system in a real-worldsystem. Finally, in phase three, we will take advantage of the low-power compact hardwareproduced by WP2 to produce a high-performance system that should find widespread real-worldapplications.

Other application areas

Another of the partners in the IMPULSE program will have the specific task of developingapplications in other domains. There are a large number of potential target areas, and duringphase one, one of the specific aims will be to establish a list of possible partners. Thus forexample, visual processing systems using the technology developed by the IMPULSE projectcould be used in areas such as

• face identification systems

• security and surveillance

• quality control

• recycling and waste management

• agricultural applications

• satellite image analysis

For each of these areas, we will contact the major European companies to see if they would beinterested in using our hardware and software technology. We will produce simple demonstrationsystems to illustrate the sort of performance that might be possible in each application area. Ifthere is interest, there would be two main possibilites. One would be to provide the spiking neuronsimulation code in the form of a library or DLL so that the third-party company could integratethe technology into existing applications. Alternatively, we could develop a complete applicationincluding the user-interface. It may also be that this first year of analysis will reveal areas whichare currently undeveloped. In such a case, we may decide to develop a complete end-user systemourselves.

Note that there is at least one area outside the area of traffic control where we already haveinitiated an action. This is in the area of medical diagnostics where, in partnership with medicalresearchers, we are looking at the possibility of developing a system that could be used to analyseblood samples (detection of malarial infections and leukemia) and cervical smears, both areaswhere the current highly demanding manual analysis has proved inefficient.

Project planning and time table

Year 1 Year 2 Year 3Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

WP1: Theory

Phase 1

Phase 2

Phase 3

WP2: Hardware/Software

Phase 1

Phase 2

Phase 3

WP3: Visual Architecture

Phase 1

Phase 2Phase 3

WP4: Applications

Phase 1

Phase 2Phase 3

D1

D2D3

D4

D5

D6

D7D8

D9

D10

D11

D12D13

D14

D15D16



Graphical presentation of the project's components

IMPULSECo-ordination

WP1Theoretical analysis of

Coding, Computing andLearning with Spiking

Neurons

WP2Development of the

spiking network sim-ulation software and

hardware

WP3Development of visual

processingarchitectures

WP4Development of

Applications



B1. Work Package List

WorkPackage Number

WorkPackage TitleLead

Contractor Number

Person Months

Start Month

End Month

PhaseDeliverable Numbers

1

Theoretical Analysis of Coding, Computing, and Learning with Spiking

Neurones

2 123 1 36 1, 6, 11

2Development of Spiking

Neural Network Simulation system

1 153 1 362, 3, 7, 8,

12,13

3Development of Visual

Processing Architectures using Spiking Neurones

1 70,5 1 36 4, 9, 14

4Development of Applications

1 163,5 1 365, 10, 15,

16

TOTAL 510



B2. Deliverables List

Deliverable Number Deliverable title

Delivery Date Nature

DisseminatiLevel

1Report on Possible Strategies for Coding,

Computing and Learning in Spike Processing Systems

12 R PU

2 Phase 1 Multiprocessor Hardware & Software12 P PP

3 Phase1 Spiking Neuron Simulation System 12 P RE

4 Phase 1 Visual Architecture Implementation12 R PU

5 Assessment of Application Areas 12 R PP

6Report on Theoretical analysis of Coding, Computing and Learning in Spike Processing

Systems24 R PU

7 Phase 2 Multiprocessor Hardware & Software24 P PU

8 Phase 2 Spiking Neuron Simulation System24 P RE

9 Phase 2 Visual Architecture Implementation24 R PU

10 Vehicle Categorisation System - 1 24 P PU

11 Report on Practical Experience in applying strategies for Coding, Computing and Learning

36 R PU

12 Phase 3 Multiprocessor Hardware & Software36 P PU

13 Phase 3 Spiking Neuron Simulation System36 P RE

14 Phase 3 Visual Architecture Implementation36 R RE

15 Vehicle Categorisation System - 2 36 P RE

16 Medical Diagnostic System 36 P RE



B3. Workpackage Description

WorkPackage number: 1

Start date or starting event: 1

Participant number: 2 (Leader), 3, 1 & 4

Person-months per participant: 57, 57, 3, 6

Objectives

The goal of this WP is

• to consider various possible methods for coding information in spikes, methods for computing with spikes, and methods for the design of learning algorithms for spike-processing systems, • to provide theoretical tools for choosing specific methods that are optimal or close to optimal for the particular applications developed in WP 4 of this project, taking the specific constraints and capabilities of the hardware andsoftware developed in WP 3 into account.

Description of Work Qualitative notions on coding by spikes will be made more precise using information theory. Simplified scenarios will be studied analytically. Theoretical predictions regarding the performance of specific approaches will be tested for large size networks through computer simulations with the help of the simulation tools developed in WP 2. The largest portion of the work on this WP will be carried out by the partners B and C, who have extensive experience in the theoretical analysis and related computer simulations of coding, computing and learning in biologically motivated mathematical models for networks of spiking neurons. The more applications-oriented partners, especially partners 1 and 4, will help in focusing the models and problems considered on the specific applications and implementations that are developed in this project.

Deliverables

D1 : Written report to the commission listing a ranked list of the a-priori most reasonable appearing choices of strategies for coding, computing and learning with spike processing systems of the type developed in WP 3 for the tasks specified in WP 4, and formulation of theoretically tractable strategies for making optimal or close to optimal choices among these options. D6 : Written report to the commission about theoretical results that support particular choices regarding these options, and about results of computer simulations of the theoretically optimal and the theoretically next-best strategies for coding, computing and learning with spike processing systems. D11 : Written report to the commission about practical experiences with the choices reported in Deliverable 2, and modifications of the chosen strategies in the light of previously unmodeled finer details of the practical implementations.

Milestones and expected result Theoretical studies by members of the consortium have already demonstrated the remarkable computational power of networks of spiking neurones, and initial small-scale simulation studies have shown the potential of the approach for image processing and visual recognition. However, till now, we have only just begun to explore the potential of such networks and there is little doubt that many further insights will be obtained during the course of the project. At each of the three milestones during the project, the results of the theoretical studies will be examined to decide which features can be usefully incorporated into the simulation software developed in WP2. In this way, the performance of the final applications being produced in WP4 can be optimised.






Participant number: 1 (Leader), 4, 5, 6

Person-months per participant: 27, 48, 72, 6

Objectives

The aim of this workpackage is develop a low-cost system capable of allowing real-time simulations of very large networks of spiking neurones, where real-time is defined as being as fast as the equivalent biological system. Such technology is required in order to be able to reproduce some of the most interesting abilities of the human visual system in artificial systems. The final system should also have the following characteristics:• Large number of processing modules (to obtain the maximum real-time performance).• Low power consumption (to allow use in real-world situations).• Small size (again to allow use in real-world tasks) • Low cost (we will use components with an optimal price/performance ratio)• Industry standard operating system (to allow take-up by other users)

Description of WorkTo obtain the required speed increases we will (i) optimise the spiking neuron simulation code to use multimedia instructions and (ii) produce versions that can run on workstation clusters under Linux. In parallel we will design and manufacture embedded multiprocessor systems to allow practical simulations of very large networks. The operating system will be a version of Linux, allowing the same hardware to be used in a wide range of applications, well beyond the current project. The hardware will be developed in three phases :• Phase 1 : An 8 CPU processor board, using 32-bit 33 MHz PCI technology for message passing• Phase 2 : Second generation multiprocessor system using advanced PCI interconnect technology• Phase 3 : Third generation systems using next-generation interconnect technology and state-of-the art microprocessors

Deliverables

D2 : Phase 1 Multiprocessor Hardware & Software. This will be based on a multiprocessor version of a StrongARM-based CPU board of which prototype versions have already been produced.D3 : Phase1 Spiking Neuron Simulation System. This version of the software will be assembly-coded to optimise performance and will be designed to run on clusters of processors, including the multiprocessor boards of D2.D7 : Phase 2 Multiprocessor Hardware & Software. This generation of hardware generation will use the next generation of embedded processors running at clock speeds of 600 MHz or more. D8 : Phase2 Spiking Neuron Simulation System. Designed to take full advantage of phase 2 hardware.D12 : Phase 3 Multiprocessor Hardware & Software. The final generation of hardware based on advanced interconnect technologies that maximise the number of spiking neurones simulated in real-time.D13 : Phase1 Spiking Neuron Simulation System. Designed to take full advantage of phase 3 hardware.

Milestones and expected resultWe already have a simulation package that can implement 30 million connection updates per second on a desktop computer, which is enough to simulate a network of 300 000 neurones in real-time if each neurone makes roughly 100 connections, and the average firing rate is 1 spike per second. Over the three years of the project we aim to achieve the following levels of performance• Phase 1 : 1 billion connections/second i.e. 1 million neurones in real-time with 1000 connections/neurone• Phase 2 : 10 bilion connections/second i.e. 10 million neurones in real-time with 1000 connections/neurone• Phase 3 : 100 billion connections/second i.e. 100 million neurones in real-time with 1000 connections/neurone This final level of performance will be sufficient to allow real-time recognition of a large range of objects in natural environments






Participant number: 1 (Leader), 2, 3, 6

Person-months per participant: 54, 6, 4.5, 6

Objectives

The goal of this WP is to develop visual processing architectures that can exploit the hardware and software simulation systems developed by WP2. The architectures will take advantage of knowledge of biological implementations of specific visual processing functions including visual recognition, motion analysis, stereoscopic vision, and colour vision.To this end, WP3 will be conducted in close collaboration with a number of other researchers working on the brain mechanisms of visual processing in Europe, including a number of groups working on the processing of natural images.The overall objective is to try and reproduce at least some of the remarkable recognition capacities of the human visual system.

Description of WorkWe will design architectures that can run on the Spiking Neuron Simulation Software adapted to particular visual processing problems. These will include:• motion analysis (analysis of optic flow patterns in video sequences, detection of higher order motion patterns including expansion, rotation and shearing)• stereoscopic vision (analysis of 3D information based on the use of two camera views)• low-level form analysis (edge extraction, spatial frequency analysis, detection of more complex visual features)• higher order object recognition (faces, road vehicles, indoor objects....)

Deliverables

D4 : Phase 1 Visual Architecture Implementation. Will include architectures suitable for real-time analysis of simple motion, stereo and low-level form analysis. At this stage, real-time image processing will be limited to relatively small images (roughly 128 x 192 pixels).D9 : Phase 2 Visual Architecture Implementation. Will include architectures suitable for real-time analysis of more complex motion patterns including expansion, rotation and deformation. Phase 2 hardware should allow real-time analysis to be achieved using larger image sizes (up to 256 x 384 pixels)D14 : Phase 3 Visual Architecture Implementation. Will include architectures for real-time recognition of hundreds of different objects in natural environments. Phase 3 hardware should enable real-time analysis to be extended to full screen video (512 x 768 pixels).

Milestones and expected resultWe have already shown that visual processing using networks of spiking neurones is possible. Simple feed-forward architectures have already been used to perform tasks such as face localisation. The aim during the three years of the study is to expand on this potential by devising processing architectures that are particularly efficient and which are capable of supporting real-time visual processing using the available hardware.Our definition of real-time is processing that is about as quick as the human visual system which can process roughly 10 separate images/second. In response to the improvements in hardware resulting from WP2, this sort of real-time processing will be progressively increased from simple tasks with small images in phase 1, to complex recognition of multiple objects in large video images by the end of phase 3.






Participant number: 1 (Leader), 3, 4, 6, 7

Person-months per participant: 7.5, 6, 3, 72, 75

Objectives

The goal of this WP is • to develop useful applications of the visual processing technology developed in WP1, WP2 and WP3 • to demonstrate the viability of real-time visual processing using large arrays of spiking neurones. Specifically, we aim to produce systems in the following areas: • Road traffic analysis. We will develop a system capable of counting and categorising road vehicles, determining their speed and trajectory, and providing the information needed for real-time control of road traffic • Medical diagnotics. We will produce prototype systems for automatic screening of medical images including blood samples. We will also assess the possible applications in areas that include security systems and driver vigilance

Description of WorkFour partners will be involved in WP4. Two university reseach groups will each contribute 1 man-month per year, while the major contributors will be two industrial partners who will each contribute 24 man-months per year. One industrial partner will concentrate on the development of applications in the area of road traffic control, while the other will seek further industrial application areas for the technology. Both industrial partners will develop commercial applications that use the hardware and software developments developed in WP2, together with the visual processing architectures developed in WP3.

Deliverables

D5 : Assessment of Application Areas. Given the large number of potential application areas of the technology, a first step will be to report on the various possibilities, and establish an order of priorities that takes into account the EUs Social Objectives D10 : Vehicle Categorisation System - 1. We will produce a prototype vehicle categorisation system that uses spike based computing and which uses the multiprocessor hardware developed by WP2D15 : Vehicle Categorisation System - 2 . The second version of the vehicle categorisation system will aim at a compact, largely autonomous system that can be used in real-world traffic control situationsD16 : Medical Diagnostic System. We will produce a system for medical diagnostics in one of the areas defined by the evaluation in D5

Milestones and expected resultThe three main milestones of the project correspond to the following points in WP4 • At milestone 1, we will have assessed the best target areas for applications outside the area of traffic control and medical diagnotics. Progress will continue with specific visual architectures adapted to the chosen recognition tasks • At milestone 2, we will have an initial prototype traffic control system • At milestone 3, we will have a commercially viable traffic control system using embedded processor technology, together with a first prototype system in the area of medical diagnostics

Date post:	15-Jul-2018
Category:	Documents
Upload:	dinhngoc
View:	215 times
Download:	0 times

IMage Processing Spiking Elements IMPULSEgerezsh/sendfile/impulse-b.pdf · IMPULSE IST-1999-10421...

Documents