ARTIFICIAL RETINA - INESC-ID · ARTIFICIAL RETINA: Development of a Bio-Inspired Model with...

UNIVERSIDADE TECNICA DE LISBOAINSTITUTO SUPERIOR TECNICO

ARTIFICIAL RETINA:

Development of a Bio-Inspired Model

with Configurable Implementation

Jose Antonio Henriques Germano, n. 48014, AE: Sistemas ElectronicosRicardo Manuel Simoes Baptista, n. 48095, AE: Computadores

DEGREE IN ELECTRICAL AND COMPUTER ENGINEERINGGraduation Report

127/2003/M

Supervisor: Prof. Leonel Sousa

October 2004

Acknowledgements

Many people have been a part of our graduate education, as friends, teachers, and col-leagues. Prof. Leonel Sousa, first and foremost, has been all of these. The best advisorand teacher we could have wished for, he is actively involved in the work of all his stu-dents, and clearly always has their best interest in mind. Thank you for the guidance andconnivance throughout the project.

At the Signal Processing Systems Research Group (SiPS) of INESC-ID we were sur-rounded by knowledgeable and friendly people who helped us daily. A special thanks toEng. Pedro Tomas, who helped us so many times from research to implementation andeven in reviewing this report, to M.Sc. Tiago Dias and Eng. Ricardo Guapo for theirhelpful suggestions, discussions and ideas.

In addition to the people in INESC-ID, we have lucky enough to have the support ofmany good friends. This graduation project long working days and nights would not havebeen the same without our friends Eng. Tiago Rojao and (future Eng.) Pedro Pinho. Aspecial thanks also to our friend (future Eng.) Joao Pereira for reviewing this report insuch a short time.

Finally, we would like to thank those closest to us, whose presence helped make thecompletion of this graduate work possible. We would like to thank our families, especiallyour parents, and our friends and girlfriends, for their absolute confidence and to whom weare forever indebted for their understanding, endless patience and encouragement whenit was most required.

Jose Antonio Henriques Germano

Ricardo Manuel Simoes Baptista

i

ii

Abstract

Nowadays, in a society which relies heavily on sight, loss of vision poses extraordinarychallenges on individuals. The goal of the work described in this report is the developmentof a bio-inspired processing module to produce spike events capable of exciting the visualcortex cells and therefore provide some sense of visual rehabilitation to profound blindpeople.

Methods and technics are proposed for modelling the retina response to a visual sti-mulus and to generate neural impulses to stimulate the visual cortex. With the purposeof improving the resemblance of the visual neural code generated by the artificial systems,the modelling of the retina cell’s response was based on artificial neural networks in op-position to what the classic model previously offered. An architecture is proposed for theimplementation of both models.

To implement the complete system, a dedicated prototype board was designed. Thepurpose of this system is to achieve a small size prototype derived from a ”low power”design based on a Field-Programmable Gate Arrays (FPGA). Regarding to a previousprototype, the power consumption is reduced at about 50% and the available memoryresources increased five times. This allowed the simulation and validation of the fullmodel.

Keywords

Retina Model, Spike Sorting, Neural Networks, VLSI Architectures, Configurable Sys-tems.

iii

iv

Resumo

Nos dias de hoje, numa sociedade que depende fortemente da visao, a perda desta colocadesafios significativos a indivıduos invisuais. O trabalho descrito neste relatorio tem comoobjectivo o desenvolvimento de um modulo de processamento bio-inspirado, de forma aproduzir os impulsos electromagneticos capazes de excitar as celulas do cortex visual e,consequentemente, fornecer algum sentido da reabilitacao a cegos profundos.

Sao propostos metodos e tecnicas com vista a modelar a resposta da retina a umestımulo visual e gerar impulsos neuronais para estimular o cortex visual. Com a finalidadede melhorar o codigo neuronal gerado pelos sistemas artificiais, a modelacao da retina foibaseada na utilizacao de redes neuronais artificiais em oposicao a abordagem do modeloclassico. Sao propostas arquitecturas para a implementacao de ambos os modelos.

Para implementar o sistema completo, projectou-se um prototipo dedicado. A finali-dade deste prototipo e conseguir um sistema de dimensao reduzida e de ”baixo consumo”baseado numa FPGA. Comparando com um prototipo anteriormente desenvolvido, oconsumo de potencia e reduzido em aproximadamente 50% e os recursos de memoriadisponıveis aumentados em cinco vezes. Este novo prototipo permitiu a simulacao e avalidacao do modelo completo.

Palavras Chave

Modelo da Retina, Classificacao de Celulas por Potenciais de Accao, Redes Neuronais,Arquitecturas VLSI, Sistemas Configuraveis

v

vi

Contents

1 Introduction 1

1.1 Project description and main goals . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Contributions of this work . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Report organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Human Visual System 5

2.1 The human eye . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 An overview of the retina . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 Central visual pathways . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Retinal ganglion cell response to light . . . . . . . . . . . . . . . . . . . . . 10

2.5 Spike sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.5.1 Feature and principal components analysis . . . . . . . . . . . . . . 11

2.5.2 Cluster analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.5.3 Pre-processing and classification . . . . . . . . . . . . . . . . . . . . 13

3 Retina Neural Models 17

3.1 Defining the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2 Classic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3 Neural networks modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.3.1 Introducing neural networks . . . . . . . . . . . . . . . . . . . . . . 20

3.3.2 A neural networks approach . . . . . . . . . . . . . . . . . . . . . . 22

3.3.3 Training the neural network . . . . . . . . . . . . . . . . . . . . . . 23

3.3.4 Simulations and results . . . . . . . . . . . . . . . . . . . . . . . . . 24

4 Processing Architectures 27

4.1 Classic model architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.2 Neural networks architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 29

vii

viii CONTENTS

4.3 Spike multiplexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.4 Serial communication protocol . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.5 System architecture conclusions . . . . . . . . . . . . . . . . . . . . . . . . 34

5 Full System Prototype 35

5.1 Prototype processing core . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.2 Power distribution system design . . . . . . . . . . . . . . . . . . . . . . . 36

5.3 VGA display port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.4 Complete board and components placement . . . . . . . . . . . . . . . . . 42

5.5 Digital logic blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.5.1 Image capture and resize . . . . . . . . . . . . . . . . . . . . . . . . 44

5.5.2 Register configuration . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.5.3 Classic model implementation . . . . . . . . . . . . . . . . . . . . . 48

5.5.4 Neural networks implementation . . . . . . . . . . . . . . . . . . . . 49

5.5.5 Serial communication protocol . . . . . . . . . . . . . . . . . . . . . 50

5.5.6 Image display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.6 Conclusions and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

6 Conclusions 57

6.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

A Clustering Algorithms 59

A.1 K-MEANS algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

A.2 EM algorithm and bayesian classification . . . . . . . . . . . . . . . . . . . 60

B SPiKes Classifier - User Manual 63

C Neural Network Modelling Spike Trains Simulations 65

D FPL Filter Implementation 69

E Prototype Datasheet 71

F Prototype Board Schematics 77

List of Figures

1.1 CORTIVIS project main modules. . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Scheme of the human eye. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 The retina as a layered structure. . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 A rod-initiated pathway. B: Bipolar Cells, RB: Rod Bipolars, AII: Ama-crine cells, G: Ganglion Cells. (Source: [1]). . . . . . . . . . . . . . . . . . . 8

2.4 The human visual pathways (Source: [2]). . . . . . . . . . . . . . . . . . . . 9

2.5 Receptive field center-surround organization. . . . . . . . . . . . . . . . . . 10

2.6 Example of spike signal pre-processing and prototype traces after separation. 14

2.7 Retinal ganglion cells response to a fullfield flash stimulus. . . . . . . . . . 15

3.1 Model with space-time separability. . . . . . . . . . . . . . . . . . . . . . . 18

3.2 Model with space-time dependency as suggested in [1]. . . . . . . . . . . . 19

3.3 Proposed models with nonlinear processing. . . . . . . . . . . . . . . . . . 20

3.4 Neural network training by adjusting weight parameters. . . . . . . . . . . 21

3.5 Neural network processing unit. . . . . . . . . . . . . . . . . . . . . . . . . 21

3.6 A multi-layer network with two hidden layers of units. . . . . . . . . . . . . 22

3.7 Stimulus (upper panel) and response (lower panel). . . . . . . . . . . . . . 23

3.8 Organization of the training set data to feed the neural Network. . . . . . 24

3.9 Comparison of real and neural network modelled spike trains. . . . . . . . 26

4.1 Global architecture of the Artificial Retina. . . . . . . . . . . . . . . . . . . 27

4.2 Classic Bio-Inspired processing module. . . . . . . . . . . . . . . . . . . . . 28

4.3 Retina Early Layers diagram (adapted from [2]). . . . . . . . . . . . . . . . 28

4.4 Integrate-and-fire block diagram. . . . . . . . . . . . . . . . . . . . . . . . 29

4.5 Parallel architecture for a single perceptron. . . . . . . . . . . . . . . . . . 30

4.6 Perceptron data flow diagram. . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.7 MAC architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

ix

x LIST OF FIGURES

4.8 Serialization and data packing block. . . . . . . . . . . . . . . . . . . . . . 31

4.9 Implemented AER module. . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.10 Packet structure [3]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.11 New packet structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.12 Adopted packet structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.1 Block diagram of the prototype. . . . . . . . . . . . . . . . . . . . . . . . . 35

5.2 FPGA available I/O signals. . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.3 MAX1830/1831 adjustable configuration electrical diagram. . . . . . . . . 38

5.4 Equivalent circuit of a real capacitor. . . . . . . . . . . . . . . . . . . . . . 39

5.5 Power plane division, different colors identify split planes borders. Thedecoupling capacitors with the lower value are also represented. . . . . . . 41

5.6 DAC electrical diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.7 TPS78601 adjustable configuration electrical diagram. . . . . . . . . . . . . 42

5.8 Complete prototype board. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.9 Digital logic block diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.10 Camera module sync signals. . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.11 Implemented hardware for frame capture. . . . . . . . . . . . . . . . . . . . 45

5.12 Spatial low pass gaussian filter. . . . . . . . . . . . . . . . . . . . . . . . . 46

5.13 Block diagram for register configuration module. . . . . . . . . . . . . . . . 47

5.14 Register configuration block diagram. . . . . . . . . . . . . . . . . . . . . . 47

5.15 Block diagram of the program regs. . . . . . . . . . . . . . . . . . . . . . . 47

5.16 Write cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.17 Read cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.18 Early Layers FPL full architecture. . . . . . . . . . . . . . . . . . . . . . . 49

5.19 Integrate-and-fire adopted architecture. . . . . . . . . . . . . . . . . . . . . 49

5.20 Data packing block diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . 51

5.21 Data unpacking block diagram. . . . . . . . . . . . . . . . . . . . . . . . . 52

5.22 VGA timing diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.23 VGA monitor control circuit [2]. . . . . . . . . . . . . . . . . . . . . . . . . 54

5.24 Complete prototype system. . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.25 Photographs of the experimental results obtained with the artificial retinaprototype. The input image (after downsizing) is displayed on the top leftcorner and the output at the bottom right corner. . . . . . . . . . . . . . . 56

B.1 SPKC (SPiKes Classifier) user window. . . . . . . . . . . . . . . . . . . . . 63

LIST OF FIGURES xi

C.1 Comparison of real and modelled spike trains, neural network model withstimulus input only. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

C.2 Comparison of real and modelled spike trains, neural network model withstimulus and response feedback input. . . . . . . . . . . . . . . . . . . . . . 67

D.1 FPL implementation of the filters. . . . . . . . . . . . . . . . . . . . . . . . 70

E.1 Prototype block diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

E.2 JTAG connector pin order. . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

F.1 FPGA electrical diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

F.2 FPGA power connections and configuration. . . . . . . . . . . . . . . . . . 79

F.3 Power regulators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

F.4 Digital to analog converters. . . . . . . . . . . . . . . . . . . . . . . . . . . 81

F.5 Main schematic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

xii LIST OF FIGURES

List of Tables

5.1 Capacitor value percentages for a balanced decoupling network [4]. . . . . . 39

5.2 Decoupling capacitors quantities. . . . . . . . . . . . . . . . . . . . . . . . 40

5.3 Implementation costs for the Bio-inspired Processing Module. . . . . . . . 50

5.4 Neural networks implementation hardware cost for different topologies. . . 50

5.5 VGA Timings [5]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.6 Complete Artificial Retina system implemented on a Xilinx Spartan XC3S400FPGA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

E.1 VGA synchronization pins and DAC clock. . . . . . . . . . . . . . . . . . . 73

E.2 FPGA pins for the video DACs. . . . . . . . . . . . . . . . . . . . . . . . . 73

E.3 FPGA pins for the slide switches and the push button. . . . . . . . . . . . 74

E.4 FPGA pins for camera connector expansion slot. . . . . . . . . . . . . . . . 75

E.5 FPGA pins for the generic expansion slot. . . . . . . . . . . . . . . . . . . 75

xiii

xiv LIST OF TABLES

Acronyms

bit – Binary digit

CAD – Computer Aided Design

CGC – Contrast Gain Control

CMOS – Complementary Metal-Oxide Semiconductor

CORTIVIS – Cortical Visual Neuroprosthesis for the Blind

CRC – Cycle Redundancy Code

DAC – Digital to Analog Converter

DoG – Difference of Gaussians

EM – Expectation-Maximization

ESL – Equivalent Series Inductance

ESR – Equivalent Series Resistance

FIFO – First In First Out

FIR – Finite Impulsive Response

FPGA – Field-Programmable Gate Arrays

FPL – Field-Programmable Logic

GCLK – Global Clock

HCMOS – High Speed Complementary Metal-Oxide Semiconductor

I2C – Inter Integrated Circuit

IC – Integrated Circuit

IIR – Infinite Impulsive Response

JTAG – Joint Test Action Group-IEEE Standard 1149.1

LGN – Lateral Geniculate Nucleus

MAC – Multiply and ACcumulate

xv

xvi LIST OF TABLES

MOS – Metal-Oxide Semiconductor

MSE – Mean Squared Error

NTSC – National Television System Committee

PCA – Principal Components Analysis

PCB – Printed Circuit Board

PDS – Power Distribution System

PWM – Pulse Width Modulation

QVGA – Quarter Video Graphics Array

RAM – Random Access Memory

RF – Radio Frequency

RGB – Red Green Blue

ROM – Read-Only Memory

SCCB – Serial Camera Control Bus

VGA – Video Graphics Array

VHDL – VHSIC Hardware Description Language

VHSIC – Very High Speed Integrated Circuits

ZV – Zoomed Video format

Chapter 1

Introduction

The work presented in this report has been developed in the scope of the European projectCortical Visual Neuroprosthesis for the Blind (CORTIVIS) [6], conducted by a Europeanconsortium where INESC-ID is included. This project is financially supported by theCommission of the European Communities, specific RTD programme ”Quality of Lifeand Management of Living Resources”, QLK6-CT-2001-00279. The research work hasbeen carried out in narrow collaboration with the University Miguel Hernandez and theUniversity of Granada, Spain.

In this chapter, the project main goals are presented, as well as the contributions ofthis graduation project to achieve these goals.

1.1 Project description and main goals

Nowadays, in a society which relies heavily on sight, loss of vision poses extraordinarychallenges on individuals. Currently, there is no effective treatment for some patients whoare profoundly visually handicapped due to degeneration or damage in the retina, opticnerve or the brain.

The CORTIVIS European project aims to develop prototypes in the field of visualrehabilitation and to demonstrate the feasibility of a cortical neuroprosthesis, interfacedwith the visual cortex, as a means through which a limited but useful visual sense maybe restored to profoundly blind people. Even though the full restoration of vision seemsto be impossible, the discrimination of shape and location of objects could allow blindsubjects to ’navigate’ in a familiar environment and to read enlarged text, resulting in asubstantial improvement in the standard of living of the blind and the visually impaired.

A block diagram of the cortical visual neuroprosthesis is presented in Figure 1.1, wherethe grey blocks are the modules that will be addressed in this work. Following the sameapproach as our own visual system, the system will use a bio-inspired (retina-like) visualprocessing front-end. The visual stimulus is first captured by some sort of image capturedevice, like a digital video camera, and then sent to a visual encoding block. This block’sfunction is to map the visual stimulus in a sequence of action potentials, also called spike

1

2 CHAPTER 1. INTRODUCTION

VisualStimulus

VisualEncoding

RF Link Modulator

Electrode Stimulator

Inside Human Head

Visual Cortex Cells

Electrical stimulus for microelectrode array

Outside Human Head

Data Packing

Image capture

Data Unpacking

RF Link Demodulator

RF Channel

Figure 1.1: CORTIVIS project main modules.

trains, so that a blind individual is able to transform the visual world into electricalsignals that can be used to excite, in real time, the neurons at his visual cortex. Theoutput information from this bio-inspired peripheral device is then packed and sent via aRadio Frequency (RF) serial link to the inside of the human head. An electrode stimulatoris applied to a microelectrode penetrating array in order to stimulate the primary visualcortex cells. Such a visual neuroprosthesis is expected to recreate a limited, but usefulvisual sense in a blind individual who is using such a system.

1.2 Contributions of this work

The main goals of the CORTIVIS project were presented and now a brief notion of thiswork’s contribution is given.

The work focuses on the Visual Encoding block of Figure 1.1. Although there aremodels1 proposed for this functional block, they may not provide accurate results. Thepurpose of this work is to study another approach to model the response of retinal cellsresponsible for vision. This way, the core of the work will consist in the application andanalysis of supervised learning methods in order to determine the pattern of generationof action potentials matching the ones in the human retina.

Another part of this work consists on the development of a prototype board designedspecifically for this project. This new board will allow the implementation of the com-plete model, the highlighted blocks of Figure 1.1 outside the human head. Also, using adedicated design, it is possible to achieve a smaller board and a lower power consumption.The board will have a processing core based on a FPGA and will also provide a videoport.

Finally, it was designed and implemented a serial communication protocol to transmitthe spike information generated by the Visual Encoding processing module to the brainover the RF serial link.

1See Chapter 3 for a brief description of the retina neural models

1.3. REPORT ORGANIZATION 3

1.3 Report organization

This thesis is composed by six chapters, including those regarding to the introductionand conclusions – the first and sixth chapters, respectively. The introductory chapterdescribes the CORTIVIS project goals and the contributions of this graduation work tofulfill these goals.

The second chapter introduces the human biological visual system. The classificationof retinal ganglion cells is also analysed, for which clustering algorithms are discussed. Asa result, a spike-sorting software tool was developed.

Chapter 3 addresses the problem of modelling the neural code of the retina. In thischapter the Classic Model approach is presented and a new model based on Neural Net-works is proposed. This new approach provides a more precise information, spike trainsinstead of the instantaneous firing rate.

Chapter 4 presents a possible architecture for implementing these two retina models,the Classic Model and the Neural Networks. The architecture for a Spike Multiplexingmodule is also described. This module serializes the spike information generated by theretina processing module. To obtain a complete system, a serial communication protocolwas also designed.

Finally, Chapter 5 introduces a full prototype for the complete retina model. Thedigital logic blocks necessary to implement the retina model are described, includingthe retina Classic Model and the Neural Networks model. Other hardware modules,responsible for the interface with the digital camera and for the generation of an outputimage representing the processed stimulus, are also presented. The hardware design for thedata packing and unpacking module that implements the serial communication protocolis introduced. A dedicated prototype board is presented and the obtained experimentalresults are discussed.

4 CHAPTER 1. INTRODUCTION

Chapter 2

Human Visual System

Visual perception results from a series of optical and neural transformations [1]. Tounderstand how the models described in this report relate to biology, one must be atleast familiar with the present knowledge of the biological visual system. This section isintended to give the minimum background to someone who lacks this familiarity.

Light arriving at the eye is first transformed by the cornea and the lens, which focus itand create a retinal image. The retinal image is then transformed into neural responses bythe light-sensitive elements of the eye, the photoreceptors. The photoreceptors responsesare transformed into several neural representations within the eye, and these are transfor-med into a multiplicity of cortical representations. Many of these transformations occurin parallel streams within the visual pathways, and will be discussed in the next sections.

In addition, processing of neural spike activity is also addressed. The detection ofneural activity is an issue of great importance. This is because different retinal ganglioncells do not respond to light in the same way. To better study the visual system, onemust know which basic element carries information and how to group the different typesof cells that code this element in the same way. This subject is considered in the last twosections of the chapter.

2.1 The human eye

When arriving the eye, light is focused and inverted by the cornea and lens onto thephotoreceptors, a collection of light-sensitive neurons that are part of a thin layer ofneural tissue called the retina. Figure 2.1 shows a scheme of the human eye. The iris islike a shutter that enables the eye to regulate the amount of entering light. The lens is aspherical shape elastic structure, stretched out into a disc-like shape by the zonule fibersto allow far-focusing. When the ciliary muscle contracts, the zonule fibers go slack, thelens are released from their tension and free to round up. This change is necessary fornear-focusing and the adjustment process is called accommodation.

The photoreceptors can be fundamentally classified in two different types, the rods andthe cones. There are approximately 5 million cones and 100 million rods in each eye. The

5

6 CHAPTER 2. HUMAN VISUAL SYSTEM

Figure 2.1: Scheme of the human eye.

positions of these two types of photoreceptors differ in many ways across the retina. Forinstance, the central fovea contains no rods, but it does contain the highest concentrationof cones, which makes the fovea the region of highest visual acuity in the human retina.The photoreceptors transduce light into neural signals, which are transmitted through theseveral layers of retinal neurons to those neurons whose output fibers make up the opticnerve. The optic nerve fibers exit the eye at a location in the retina called the opticdisk1, and carry the neural signals to the brain for further processing.

2.2 An overview of the retina

The retina is a thin layer of neural tissue in the back of the eye. It can be decomposedin five layers: three layers of cell bodies and two layers of synaptic interconnectionsbetween the neurons. Yet, the fovea consists only in a single layer of neurons, the conephotoreceptors. This structural form is depicted in Figures 2.1 and 2.2.

Light enters from the ganglion cell layer side first, and must penetrate all cell typesbefore reaching the rods and cones. This is because the pigment-bearing membranes ofthe photoreceptors have to be in contact with the eye’s pigment epithelial layer, whichprovides vitamin A [7]. The photoreceptors’ cell bodies are located in the outer nuclearlayer of the retina. The synaptic terminals of the photoreceptors make contact withthe dendritic fields of the bipolar cells and horizontal cells in the outer plexiformlayer. The cell bodies of the bipolar and horizontal cells are located in the inner nuclearlayer. The horizontal cells make connections with the cells in the outer nuclear layer.The bipolar cells, however, make connections onto the dendrites of the ganglion cellswithin the inner plexiform layer. Since only the bipolar cells link the signals in theouter and inner plexiform layers, all the visual signals must go through the bipolar cells.

Another class of cells located in the inner nuclear layer are the amacrine cells. These

1The optic disk is also known as the blind spot

2.2. AN OVERVIEW OF THE RETINA 7

Figure 2.2: The retina as a layered structure.

cells have no identifiable axons, only dendrites. The dendritic fields of the amacrine andganglion cells connect in the inner plexiform layer. The retinal ganglion cell bodies arelocated in the ganglion cell layer, and their dendritic fields connect with the axon terminalsof the bipolars as well as with the dendritic fields of the amacrine cells.

The axons of the retinal ganglion cells provide the only retinal output signal. Theycomprise the optic nerve and exit from the retina at a single location, the optic disk.There are no photoreceptors at the optic disk so we do not perceive the light that fallsthere (blind spot). We are not aware of the blind spot in our eyes since the portion of thevisual field falling in the blind spot of one eye falls on the functional portion of the retinain the other eye, and the brain fills in the missing information with whatever patternsurrounds the hole.

The retina segregates visual information into parallel neural pathways specialized indifferent visual tasks. The signals initiated within the rods follow a separate rod pathwaywithin the retina until the signals arrive at the retinal ganglion cells, Figure 2.3. Therods make connections with a class of bipolar cells, the rod bipolars, which integratethe responses of many different rod photoreceptors and synapse directly on the amacrinecells. The amacrine cells synapse onto the ganglion cells which also receive signals via thecone pathway, merging both the rod and cone pathways. Due to the strong convergenceof the signals in the rod pathway, this makes it well-suited to capture information atlow light levels (night vision) while paying a penalty in terms of visual acuity. Thecone pathway, as opposite, is not so sensitive but is more acute and is used for colorvision. This way, specialized computational goals can be achieved as a result of specialanatomical connections. Rodents such as rats, which are nocturnal animals, have retinas


Figure 2.3: A rod-initiated pathway. B: Bipolar Cells, RB: Rod Bipolars, AII: Amacrinecells, G: Ganglion Cells. (Source: [1]).

overwhelmingly dominated by rods. Most fish, frog, turtle and bird retinas have threeto five types of cones and consequently very good color vision. It is presumed that eachvisual pathway carries an efficient retinal image representation that is most relevant forthe tasks carried out where the output is sent, the visual cortex. Finally, based on thesize and spread of the dendritic arborizations, the retinal ganglion cells can be mainlyclassified in two types: the midget cells, whose dendritic fields are more dense andcompact, and the parasol cells, with sparse dendritic fields and large to medium-sizecell bodies.

2.3 Central visual pathways

Once the ganglion cells axons leave the retina, they travel through the optic nerve to theoptic chiasm. The optic chiasm is a location where the optic nerve axons from the tworetinae join and are then reorganized into two separate groups that encode informationfrom the right and left visual fields. There, the fibers are sorted into two new groups,each connecting to only one side of the brain. Axons from ganglion cells whose receptivefields are located in the left visual field send their output toward the Lateral GeniculateNucleus (LGN) on the right side of the brain, and vice-versa. Thus, the LGN receives aretinal signal from both eyes, but only one half of the visual field. Figure 2.4 illustrates thepattern of connections schematically, from the retinas all the way to the visual cortex.

The LGN is a structure located in the thalamus which is a major recipient of axons fromthe retina. The primate LGN contains six different layers. The four superficial layers arecalled the parvocellular layers and the two deeper layers the magnocellular layers.The axons of the parasol and midget retinal ganglion cells connect with different layers:

2.3. CENTRAL VISUAL PATHWAYS 9

the axons of the midget retinal ganglion cells terminate in the parvocellular layers, whilethe axons of the parasol cells terminate in the magnocellular layers. This way, the midgetand parasol cells form separate visual pathways. The path from the midget ganglioncells to the parvocellular layers of the LGN is called the parvocellular pathway, whilethe pathway from the parasol cells to the magnocellular layers is called magnocellularpathway.

left hemifield right hemifield

LGN LGN

Meyer’s loop

optic radiations

Optic nerve

optic track

V1

occipital poles

Image Preview

Opticchiasm

Figure 2.4: The human visual pathways (Source: [2]).

There are several differences in the way neurons in the parvocellular and magnocellularpathways code information. Also, anatomical and physiological measurements suggestthat both pathways carry different types of information to the brain. Electrical signalstraveling from the retina to the LGN have longer conduction times to the parvocellularlayers than to the magnocellular layers [1]. When facing contrast patterns, the neuronsin these two pathways also respond differently. As the stimulus contrast increases, theresponse of neurons in the magnocellular pathway changes more rapidly than the responsein the parvocellular pathway. Thus, it appears that the magnocellular pathway exists as aspecialization that improves the ability to perform tasks requiring high temporal-frequencyinformation. Two examples of such tasks are motion detection and motion tracking.

The neurons in the LGN send their axons directly to area V1 (primary visual cortex)via the optic radiations. When entering the primary visual cortex, information fromleft and right visual fields is still separated. Here, this information will be mixed so thatbinocular vision is achieved. Cortical area V1 is the first point in the visual pathwayswhere individual neurons receive binocular input. These binocular neurons may play animportant role in our perception of stereo depth.


2.4 Retinal ganglion cell response to light

The retinal ganglion cells are part of the pathway that transforms light stimulus into atemporal series of discrete electrical pulses called action potentials or spikes. Withinthe field of electrophysiology2, the transformation associated with a neuron is called theneuron’s receptive field. In the vision domain, the receptive field of a neuron is definedas the retinal area in which light influences the neuron’s response. It is possible to obtainthis region by stimulating neurons with small flashes of light and simple moving barsand determining when the neuron responds and fails to respond. The responses of manyneurons in the visual pathway are influenced only by light falling in narrow regions of theretina, hence small regions of the visual field. Although one usually refers to the neuron’sreceptive field, in fact it depends on the properties of the entire visual pathway.

Most retinal ganglion cells respond with a random stream of action potentials whenthey are stimulated with a large uniform field. The average number of action potentialsper unit of time in the presence of a constant field is called the spontaneous firingrate. When the central region of the receptive field is flashed with a small spot of light,the firing rate increases when compared to the spontaneous activity. Conversely, if thespot is placed on a surrounding area, the cell’s activity decreases. The intermediateregion shows some excitation at the beginning of the stimulus and some inhibition atit’s extinction. This center-surround organization defines important properties ofour visual capacities, like edge-detection. The case just mentioned defines a on-center,off-surround receptive field and is illustrated in Figure 2.5(a). If the ganglion cell isinhibited by light falling in the center and excited by light falling in the surround theyare called off-center, on-surround cells, Figure 2.5(b).

(a) ON center, OFF surround.

(b) OFF center, ON surround.

Figure 2.5: Receptive field center-surround organization.

2The field that studies the electrical responses of neurons

2.5. SPIKE SORTING 11

2.5 Spike sorting

Retinal ganglion cells, like other neurons, communicate with the central nervous systemby firing action potentials. Therefore, the detection of retinal neural spike activity is atechnical challenge and a prerequisite for studying most retinal functions. These briefvoltage spikes can be recorded using a microelectrode, which often also picks up signalsfrom neurons in a local region. Due to the presence of large amounts of backgroundnoise and the difficulty to distinguish the action potentials of one neuron from thosenearby, measuring the activity of individual neurons accurately can be quite hard. Thus,experimental results can be greatly enhanced by the use of spike-sorting software.

This section reviews classification of action potentials, a problem commonly referredto as spike sorting, and presents the pre-processing of spike recorded signals necessaryfor neural network data preparation. Spike sorting revealed to be indispensable since mostexperimental retinal recording results were either not classified or the classification wasnot accurate.

2.5.1 Feature and principal components analysis

From the recorded neural activity it can be seen that there are several different types ofaction potentials. The spike shapes differ in many aspects: the sign of the principal peakdetermines whether the response is ON-type or OFF-type, the ratio of the two majorpeaks influences how transient the response is and the overall time scale of the curvesets the time scale of the response. Assuming that these different spike types identifydifferent neurons, this correspondence can be established. Also, recorded waveforms showa significant amount of background noise which could arise from the electronic amplifiersor from the biological complex system. Nevertheless, neurons classification should bereliable even in the presence of this background noise.

If shape could be characterized, this information could be used to classify each spike.One approach to characterize shape is to measure its features, such as: spike height,measured as the maximum spike amplitude, width, corresponding to the time lag betweenthe maximum and minimum, or peak-to-peak amplitude. This was one of the earliestapproaches to spike sorting [8]. In general, the more features considered, the better thedistinction of different spike shapes.

But choosing a feature based on an intuitive idea of what might be useful is an ad-hocapproach and, although simple, it can yield poor cluster separation. Therefore, featuresshould be chosen systematically and automatically. One way to do this is with PrincipalComponents Analysis (PCA). The idea behind PCA is to find an ordered set of orthogonalbasis vectors that capture, in the data, the directions of largest variation. In the presentcase, data is the spike recorded waveforms. Because the components are ordered in termsof how much variability they capture, adding together the first k components will describethe most variation in the data. Adding additional components yields progressively smallercorrections until the spike is described exactly. The principal component vectors areobtained by computing the eigenvectors of the covariance matrix of the data. A previousstudy on comparing clustering methods found that principal components as features yield


more accurate classification than other features [8].

2.5.2 Cluster analysis

Finding clusters in multidimensional data sets and classifying data based on those clustersis called cluster analysis. The subjacent assumption in clustering is that data resultsfrom several independent classes, each described by a relatively simple model. This isthe case of spike sorting, since each action potential is supposed to arise from a uniqueneuron.

There are many methods for clustering [9], but all share two common tasks. The firsttask is to describe both the cluster location and the variability of the data around thatlocation. The second task is, given a description of the clusters, to classify that data.

Perhaps the most simple approach to clustering is the nearest-neighbour or k-means3

clustering. K-means consists in defining the cluster locations as the mean of the datawithin that cluster. When using Euclidean distance, a spike is associated to whichevercluster that has the closest mean. This defines a set of implicit decision boundaries thatseparate the clusters while ignoring the distribution of data within the cluster. Thisapproach is adequate when the clusters are well separated, but fails when clusters signifi-cantly overlap or when the cluster shapes differ significantly from a spherical distribution.

Bayesian clustering is another approach to model data. In the most common approach,one tries to capture the statistical distribution of data modelling each cluster with a mul-tivariate Gaussian, centered on the cluster. This approach offers many advantages overthe one described above, and possibly the main one is that it quantifies the certainty ofclassification. Expectation-Maximization (EM)4 algorithm is a technique to this type ofclustering [10]. Note that the probabilistic nature of membership allows decision boun-daries to be computed as a function of confidence level. Also, an idea of the degree ofseparation between classes is possible by looking at the distribution of the probabilitiesfor a class.

A difficult task in clustering is to decide the number of classes to use. In spike sorting,the ideal number of classes corresponds to the number of neuron types being observed, butchoosing is not so simple. The distribution of spikes produced by individual neurons mightnot be well described by the model employed. When spikes reveal stationary shapes anduncorrelated noise, clusters appearance will be nearly spherical and conclusions drawnfrom nearest-neighbour or symmetric Gaussian models can be very accurate [11]. Inother situations, such as non-stationary spike shapes or correlated noise, Gaussian mixturemodels can still accurately model data. But complicated cases arise when neurons generatecomplex bursts or the background noise is non-stationary. Therefore, the structure of thedata can be very difficult to capture, and it will be difficult both to predict the numberof units and to make accurate classifications.

3See Appendix A.1: K-MEANS algorithm4See Appendix A.2: EM algorithm and bayesian classification


2.5.3 Pre-processing and classification

In order to model the spike firing events of retinal ganglion cells, in a way that a fullretinal model can be developed, cells were classified as part of the pre-processing of datato feed supervised classifiers such as the neural networks5. As already seen, Section 2.4,there are several types of retinal ganglion cells, mostly characterized by their ON-OFFresponse. A model of these must take this into account. For that reason, and beforeadvancing any further, experimental recorded action potentials were sorted into classes,a process which is described next.

The experimental results, unsorted spike data, were obtained as described in [12] andsupplied by the University Miguel Hernandez, Spain. Experiments were made mostly oncats and rabbit retinas.

The classification of this data was made with spike-sorting software specifically deve-loped for this task. The software was developed in MATLAB. An user manual for thissoftware can be found in Appendix B.

Typically, several spike waveforms occurred at one electrode (multi-unit activity).These spike ’prototypes’ at each electrode were separated using unsupervised classificationmethods: the k-means and EM algorithms. Each 4-ms waveform consisted of 120 equallyspaced sample voltage values. These formed one input vector, resulting a 120-dimensionalspace were the voltage traces cluster in a restricted volume. The two most importantdimensions for the cluster analysis were depicted graphically. The number of expectedclusters was adjusted and selected manually, after inspection of the raw data and it’s firsttwo principal components. After classification, a single cluster could be selected manuallyand all traces belonging to it were shown superimposed. Classification could always berepeated if the results were not satisfactory.

In the example of Figure 2.6(a) it was assumed that three prototypes were recor-ded in one electrode. The two most important principal components are shown in Fi-gure 2.6(b). Clustering was performed using k-means and results are shown in Figu-res 2.6(c) and 2.6(d). The three corresponding classified prototype traces are in Figu-res 2.6(e), 2.6(f) and 2.6(g). In general, multi-prototype signals were obtained and se-paration was difficult. Therefore, for future use were only selected those prototypes thathad waveforms typical of single unit activity, unequivocal in terms of both amplitudeand shape, such as those in Figures 2.6(e), 2.6(f) and 2.6(g). As suggested in [12], spikeprototypes of identical waveform at a given electrode are assumed to be originated froma single ganglion cell.

The sorting performed showed remarkable results. Before sorting the several electrodesinto their types, spike trains were as shown in Figure 2.7(a). As noted, there are spiketrains that do not appear to have correlation. Spike sorting grouped those that showedsimilarity in temporal firing. The results are patent in Figure 2.7(b).

5See Section 3.3


0 0.5 1 1.5 2 2.5 3 3.5 4-40

-30

-20

-10

0

10

20

30

40

50

60Spike electrode waveforms

Time (ms)

Am

plitu

de (

uV)

(a) Electrode multi-unit activity.

-150 -100 -50 0 50 100 150-150

-100

-50

0

50

100

150Principal Component Analysis

1st principal component

2nd

prin

cipa

l com

pone

nt

(b) Distribution of principal componentsof the response waveforms in Fi-gure 2.6(a).

0 0.5 1 1.5 2 2.5 3 3.5 4-40

-30

-20

-10

0

10

20

30

40

50


Time (ms)

Am

plitu

de (

uV)

(c) Electrode multi-unit activity sorted byclasses: unit 1 = blue, unit 2 = green,unit 3 = red.

-150 -100 -50 0 50 100 150-150

-100

-50

0

50

100

150Principal Component Analysis

1st principal component

2nd

prin

cipa

l com

pone

nt

(d) Distribution of principal components ofthe sorted response waveforms in Fi-gure 2.6(c).

0 0.5 1 1.5 2 2.5 3 3.5 4-40

-30

-20

-10

0

10

20

30

40


Time (ms)

Am

plitu

de (

uV)

(e) Responses classified tounit 1.

0 0.5 1 1.5 2 2.5 3 3.5 4-40

-30

-20

-10

0

10

20

30

40

50


Time (ms)

Am

plitu

de (

uV)

(f) Responses classified tounit 2.

0 0.5 1 1.5 2 2.5 3 3.5 4-40

-30

-20

-10

0

10

20


Time (ms)

Am

plitu

de (

uV)

(g) Responses classified tounit 3.

Figure 2.6: Example of spike signal pre-processing and prototype traces after separation.


Spike firing events

0 10 20 30 40 50 60

Time (s)

Retinal Ganglion

CellsResponse

Stimulus

Cellsof thesametype

(a) Before spike sorting.

Spike firing events

0 10 20 30 40 50 60

Time (s)

Retinal Ganglion

CellsResponse

Stimulus

(b) After spike sorting.

Figure 2.7: Retinal ganglion cells response to a fullfield flash stimulus.


Chapter 3

Retina Neural Models

When facing the problem of modelling the neural code of the retina one of the firstquestions we ought to ask is: How do action potentials represent information? [13]

Within the central nervous system, information is conveyed using action potentials asthe standard signal. Since our only source of visual experience results from the electricalspikes on the ganglion cell axons, it is important to understand how these electrical spikesshape our visual perception. The retina performs a significant amount of processing,compressing the visual signal from a neural population of 108 photoreceptors into 106

optic nerve fibers, and so one might even learn new principles for efficient coding.

3.1 Defining the problem

Since the structure of the retina is remarkably preserved from fish to primates, models oflight response that successfully predict a ganglion cell’s firing rate also share a commonstructure: differences among species affect the quantitative parameters of these models,but not their basic elements. Also, a useful description of the neural code should spe-cify [13]: (1) the relevant measure of neural activity in the ganglion cell population; (2)how this activity responds to any given visual stimulus; (3) the precision of this response;(4) and the variability of the stimulus response based on the recent history of the visualinput. Thus, throughout the stimulus presentation, the neuron’s instantaneous firingprobability at various times is generally taken to be the most important response feature.

As seen before, in Section 2.3, our visual perception is conveyed to the brain in parallelpathways, but most of what we know about retinal signalling results from recording singleretinal ganglion cells. The underlying assumption is that the retinal population consistsof groups of neurons with similar functional properties and that the firing of each neurondepends only on the stimulus presented and not on the activity of the surrounding neurons.Unfortunately, this last condition is not always met [14].

Due to the opposite action of the center and surround regions of the receptive field,ganglion cells respond to stimuli whose intensity varies in space, creating a responsesensible to object edges. Also, and because light step stimulus response lasts only a few

17

18 CHAPTER 3. RETINA NEURAL MODELS

milliseconds, ganglion cells seem to emphasize stimuli that vary in time over static ones.

In general, the stimulus is given by the intensity distribution I(x, t, λ) on the retina,as a function of the position x, time t, and wavelength λ. One thus seeks a mathematicalmodel that captures the most important aspects of the retinal processing, and whoseoutput is the temporal evolution of the cells firing rate.

3.2 Classic model

The basic model architecture is depicted in Figure 3.1. The input is a spatiotemporalstimulus contrast pattern I(x, t). The model consists of a set of band-pass spatial filters,S(x), and high-pass temporal filters, T (t), assumed to factorize into a spatial and temporalpart (space-time separability):

R(x, t) = S(x) · T (t) (3.1)

The spatial profile of the response amplitude, S(x), follows from a Difference of Gaus-sians (DoG) model. The DoG model assumes neural response as a combined signal of thecenter and surround as separate mechanisms. The center mechanism contains the inputonly from a small central region of the cell’s receptive field, whereas the surround mecha-nism includes the center and the surrounding region. Both mechanisms are assumed to

DoG High-Pass

StimulusI(x,t)

Rectification

Firing RateF(t)

S(x) T(t)

Figure 3.1: Model with space-time separability.

behave as space-time separable linear systems when responding to contrast stimuli. Todescribe the spatial sensitiveness of the center and surround, a curve with the shape ofa Gaussian distribution is used. The output of a neuron is obtained by the sum of thetemporal response of both mechanisms, although the temporal responses are different inthe center and surround, generally being slower in the latter, Figure 3.2. This spatialprofile can be formalized by:

S(x) =g+√2πσ2

+

e− x2

2σ2+ − g+√

2πσ2−e− x2

2σ2− (3.2)

where parameters g+ and g− determine the relative weights of the center and surround,respectively, σ+ and σ− their diameters.

The temporal profile can be approximated by a first-order high-pass filter:

T (t) = δ(t)− αH(t)e−αt (3.3)

where δ(t) denotes the Dirac delta function, H(t) is the Heaviside step function and α−1

is the decay time constant of the response.

3.2. CLASSIC MODEL 19

Delay

-+--

-

+

RectificationHigh-PassStimulusI(x,t) Firing Rate

F(t)

T(t)

Figure 3.2: Model with space-time dependency as suggested in [1].

As a result, R(x, t) can be seen as the change in the firing rate produced by flashing alight spot at time t = 0 and location x. Assuming linearity, the firing rate F (t) producedfollows from the sum on the entire visual field and becomes,

F (t) = f0 +∫ ∫

I(x, τ) · S(x) · T (t− τ)dxdτ (3.4)

where f0 is the spontaneous firing rate without stimulation, and negative values are trun-cated to zero. This expression can also be seen as a weighted version of the stimulusby S(x), summed over all space, and then passed through a temporal filter of impulseresponse T (t). It can be rewritten in a more compact form as

F (t) = ϕ[R(x, t) ∗ I(x, t) + f0]+ (3.5)

with the ∗ denoting the convolution operator, ϕ a scale factor and [x]+ := xH(x) the rec-tification operator. Although this model captures the main characteristics of the retinalstimulus-response, it is a coarse approach. Stimulus contrast exerts a strong modulatoryeffect on the responses of retinal ganglion cells. In humans, for example, visual stimula-tion with bright, full-field flicker of high frequency evokes impressive illusions of spatialpattern [12]. To improve this model, a nonlinear contrast gain control mechanismwas added and turned out to be crucial for motion processing. The model is depicted inFigure 3.31.

The contrast gain control mechanism consists on a low-pass first order temporal filte-ring followed by a nonlinear gain control function. The low-pass temporal filtering can berepresented by

v(x, t) = u(x, t) ∗ [H(t)Be−tβ ] (3.6)

where the B and β define the strength and constant time of the low-pass filter, u(x, t) isgiven by

u(x, t) = g(x, t)×R(x, t) (3.7)

and the nonlinear gain control function is

g(x, t) =1

1 + [v(x, t)]+4 (3.8)

Finally, the system’s output, the instantaneous firing rate, is obtained from the rectifica-tion of the result,

F (t) = ψ[u(x, t) + f0]+ (3.9)

1The following equations hold for the model presented in Figure 3.3(a)


DoG High-Pass

StimulusI(x,t)

Rectification

Firing RateF(t)

S(x) T(t)

Nonlinearity Low-Pass

x

(a) Nonlinear at the end [12].

DoG High-Pass

StimulusI(x,t)

Rectification

Firing RateF(t)

S(x) T(t)

Nonlinearity Low-Pass

x

(b) Feedback loop around the whole system [15].

Figure 3.3: Proposed models with nonlinear processing.

where ψ is once again a scale factor.

Functionally, this feedback loop generates a delayed suppression of high, sustainedretinal ganglion cell activation, and thus altering the temporal characteristics of the retinalganglion cells firing rates [12]. In other words, the net effect of the contrast gain control isthat, during large light fluctuations, the retinal response is less sensitive and faster [13].

3.3 Neural networks modelling

In this section, one presents a different approach to model retinal ganglion cell’s firing. Theidea is to train a neural network with experimental data of multi-electrode recordingsfrom the ganglion cell layer of a retina and attempt to catch the structure of the firedspike trains.

3.3.1 Introducing neural networks

Artificial neural networks can be most adequately characterized as computational mo-dels with particular properties such as the ability to adapt or learn, to generalise, clusteror organize data, and which operation is based on parallel processing [16]. These compu-tational models are inspired in biological nervous systems. As in nature, the connectionsbetween these elements determine the function of the network. For that reason, an arti-

3.3. NEURAL NETWORKS MODELLING 21

ficial neural network consists of a pool of simple processing units which communicate bysending signals to each other over a large number of weighted connections.

Neural networks are trained by adjusting the values of the connections (also calledweights) between elements so that a particular input leads to a specific target output, seeFigure 3.4. The ability to perform a particular function depends largely on the chosenweights. Typically the train is performed in a supervised manner. A training set isassumed to be available, containing both the input and the correspondent desired outputpatterns (also called target patterns). Training is based on the minimization of an errormeasure between the network outputs and the desired outputs.

NeuralNetwork

=?Input Output

Target

Adjustweights

Figure 3.4: Neural network training by adjusting weight parameters.

The most widely used kind of neural network is composed of simple units of the typeshown in Figure 3.5 [17]. The unit performs a weighted sum of it’s inputs, which is thenpassed through a nonlinear function known as the activation function. Units are arrangedin a feed-forward layered structure, as depicted in Figure 3.6. Each layer consists of unitsthat receive their inputs from the units in the previous layers and send their outputs tothe units in next layers. There are no connections within units in a layer. An example ofa network with two hidden layers is presented in Figure 3.6.

+

w0

wN

w2

w1

Inpu

ts

Activation functionx1

1

x2

xN

y

Output

Figure 3.5: Neural network processing unit.

The basic algorithm for training a neural network is based on the gradient method.This method consists in iteratively adapting the weights, w, by taking steps proportionalto the negative of the gradient of the function to be minimized, E,

wn+1 = wn − η · ∇En (3.10)

where η is the step size. In the case of feed-forward networks, the gradient componentsare computed by the backpropagation rule [16], which can be derived by the applicationof the chain rule of differenciation.


+

Inputs Hidden Layers Outputs

Figure 3.6: A multi-layer network with two hidden layers of units.

3.3.2 A neural networks approach

The structure of firing of retinal ganglion cells to a stimulus can be seen as the probabilityof some sequence ti, i = 1, 2, . . . , n, where ti are the times of fired spikes, given that thestimulus I(t) is presented. As a result, given a particular time-dependent stimulus I(t),one aims to find a complete description of the neural response based on the conditionaldistribution P [ti|I(t)], which measures the relative likelihood that spikes will arriveat the set of times t1, t2, . . . , tN. However, in practice, experiments usually seek acharacterization through the first few moments of such a distribution, like the mean, thevariance, and so on. It can be shown that the mean of this conditional distribution,P [ti|I(t)], matches the firing rate attained by the model of the previous section [18].Therefore, the model presented in Section 3.2 suffers from limitations by only consideringfiring rates and neglecting the other moments of the distribution. It thus neglects thefine temporal structure of retinal spike trains and correlations between retinal ganglioncells [14, 19].

The presented approach tries to overwhelm some of these deficiencies. The idea is totrain a neural network using experimental spike trains to simple types of stimuli. Dueto the strong ability of adaptation of these systems they would ideally learn the featuresets that produce the spikes occurrence. At each moment in time, the information feedto the network should allow it to predict the occurrence of a spike. When looking to atime interval, the spike train produced would be identical to a spike train produced bythe retinal ganglion cells themselves.

It has been assumed that spatial correlations between retinal ganglion cells do notexist, and that cells of the same type fire approximately the same way. This means thatthe firing of a cell is not influenced by its neighbouring cells, which is not true in realbiological systems. The similarity between groups of cells, ON/OFF , might help oneto find patterns in the way neurons represent information conveyed to the brain. Afterclassifying cells using the procedure described in Section 2.5, they were grouped by typeof response. Each group represented a type of firing that should be modelled.

The stimulus consisted in full-field flashes i.e., stimulations where the whole of thestimulated area is illuminated with a spatial color constant stimulus for a given timeperiod (ON), followed by a dark (no light stimulation) period (OFF). This stimulus and


the response of a cell in a given electrode is depicted in Figure 3.7. Notice that most ofthe time there is no spiking activity.

Spike firing events

0 2 4 6 8 10

Time (s)

Stimulus

Cellspikes

Figure 3.7: Stimulus (upper panel) and response (lower panel).

3.3.3 Training the neural network

Artificial neural networks were constructed and trained using the Neural Network Toolboxfrom MATLAB. Up to four-layer, fully connected, feed-forward networks with twentyhidden units in each hidden layer were used, as illustrated in Figure 3.6.

To model retinal ganglion cells firing, a set of different experiences were performed.These experiences aimed at finding the network topology that most accurately achievesthe best classification of the firing events.

The first subset of experiences had the purpose of searching the faster and mostaccurate training method. Activation functions were also tested and established. Afterthis, networks were trained by backpropagation to minimize the Mean Square Error (MSE)between the spike prediction and the true spike. The direction of the change to each weightwas computed using Powell-Beale restarts conjugate gradient descent [20, 21] (traincgbin MATLAB), and the size of the step was computed by Charalambous line search [22](srchcha in MATLAB), which provides an adaptive learning rate with cubic interpolationand sectioning. The activation sigmoid function was given by (3.11).

f(s) =2

1 + e−2s− 1 (3.11)

The search for the best network topology ended in a second group of simulations. Here,the number of hidden layers and units was varied so that modelling was done withoutunderfitting or overfitting data.

The input layer of the network was presented with a window of spike trains 0.96 s long(sampling window width used for rabbit retinas [23]), binned at 10 ms resolution. In thefirst group of experiences, each input unit represented the stimulus in each time bin and,a second group of experiences also added the existence of spikes in time bins. The valueof the output unit represented the spike prediction. In other words, the network takes thestimulus sequence I(t1), I(t2), . . . , I(tN) and the correspondent value of the spike trainat times ti, i = 1, 2, . . . , N , F (t1), F (t2), . . . , F (tN−1) (1 if spike, 0 if not) as input


and is trained in order to reproduce a s ingle spike firing event at time tN , F (tN). Twodifferent types of trainings were carried out. The difference resided in the nature of theinput used, just stimulus as seen in Figure 3.8(a) or stimulus and past spikes occurrence,see Figure 3.8(b). In both cases there is only a single output value, the predicted spikeevent at time tN .

I(tN)I(tN-1)I(tN-2)

...I(t2)I(t1)

F(tN)

(a) Training set with stimulus input, fi-ring event output at time t.

F(tN)

I(tN)I(tN-1)I(tN-2)

...I(t2)I(t1)

F(tN-1)F(tN-2)

...F(t2)F(t1)

(b) Training set with stimulus and firingevents input, firing event output attime t.

Figure 3.8: Organization of the training set data to feed the neural Network.

These experimental results allowed the convergence to the topology that producedthe best results. In the following subsection a comparison of the performed trainings isreported.

3.3.4 Simulations and results

To illustrate the performance of the model, raster plots of real spike trains are presented,along with the corresponding predicted spike trains from the neural networks modelling(see Appendix C).

The inputs consisted in pattern vectors of dimension N = 96 when using only stimulusand N = 96 + 95 = 191 when using stimulus and feedback of past spikes. Networks weretrained with a training set of approximately 60 000 patterns in the first case and 80 000 inthe later. From these, 75% was used as training and 25% as validation. Finally, networkperformance was assessed using a third data set reserved for testing, about the same sizeas the validation set.

Modelling with stimulus input

The first type of training used only stimulus as input. The assumption is straightforward:there is no temporal correlation between spikes.

The result is patent in Figure C.1. As noted, the system was not able to recreate thespike train reproduced by the cell. The adaline2 responded with a periodic output, as

2This is a particular case in which the unit activation function is equivalent to f(x) = 1


expected since the input is totally periodic. This is a property of linear systems. As to thenonlinear networks with one and two hidden layers, no matter the number of units used,they also generated almost periodic responses. This can be explained by considering thatsigmoid functions are approximately linear close to the origin and that activation entries3

are falling in that region.

When looking to the cell response it is observable that the assumption was incorrect.Temporal correlation is one of the reasons by which the visual system has such goodcoding properties.

Modelling with stimulus and past spikes input

The introduction of new information in the learning system by adding past spikes revealedvital. Now, the input and the output are no longer periodic. As a result, neural networksprovided a much higher accuracy in the prediction of the spiking events. An example of theresults is depicted in Figure C.2 with the same topologies trained earlier. The inclusion ofpast information transformed this into a system with memory. For that reason, temporalcorrelation between successful spikes was exploited and the results are at sight. Dependingon the network topology, the structure of the spike train was somehow captured.

In order to choose the topology that best classified spike events, zooming was perfor-med. Figure 3.9 is a combination of the best results obtained (in this example) in theprevious trainings. As in other experiments performed, the best results are associatedwith nonlinear topologies, generally using one hidden layer with 20 hidden units or twohidden layers with 20 plus 10 or 10 plus 5 units. In this case, the best topology is a twohidden layer with 10 and 5 units in each layer.

3Activation function input values


0 10 20 30 40 50 60

Time (s)

Cell

ADALINE

Stimulus

Model 10-5-1

Model 20-1

15 16 17 18 19 20 21

Time (s)

Cell

ADALINE

Stimulus

Model 10-5-1

Model 20-1

Figure 3.9: Comparison of real and neural network modelled spike trains.

Chapter 4

Processing Architectures

In this chapter, architectures are proposed for implementing the two retina models, ClassicModel and Neural Networks. Also, in order to create a complete system, it is requiredto design an architecture for the serial communication sub-system. The serial protocolallows the spike information to be transmitted into the brain over a serial asynchronousRF link. The full diagram of the system can be seen in Figure 4.1, in which the grey colorhighlights the blocks that are addressed in this chapter.

Serialization and Data Packing

SpikesVisual

StimulusDigital Camera

Bio-Inspired processing module of the Artificial Retina

RF Link Data

RF Link Modulator

Data Packets

RF Link Demodulator

Data Unpacking

Data PacketsElectrode

Stimulator

SpikesBit Vector

Bit

Inside Human Head

Figure 4.1: Global architecture of the Artificial Retina.

The input is generated by an image capture device, in this case a digital camera, whoseoutput is in Red Green Blue (RGB) format, and connects to the processing modules. Thespike information is then wrapped into packets by a data formatting block. These datapackets are transmitted over a RF channel. At the receiver end, a RF demodulatorreceives the serial data and feeds it serially to a block responsible for data unpacking.Finally the spike information is sent to the electrode stimulator.

4.1 Classic model architecture

The developed processing modules for the bio-inspired processing module of the artificialretina model are divided in two distinct parts: Retina Early Layers and Neuromorphic

27

28 CHAPTER 4. PROCESSING ARCHITECTURES

Pulse Coding. Figure 4.2 represents the block diagram of this bio-inspired processingmodule.

Retina Early Layers

Neuromorphic Pulse Coding

Visual Stimulus Spikes

Figure 4.2: Classic Bio-Inspired processing module.

Here, the Retina Early Layers is the model introduced in Section 3.2 and NeuromorphicPulse Coding generates the action potentials using the instantaneous firing rate. This lastblock will be described in the present chapter.

It is specified that the Retina Early Layers module should be able to process fra-mes at a maximum rate of 100 Hz and that the maximum spike rate generation for theNeuromorphic Pulse Coding must, at most, be 1 kHz per cell [2].

The retina early layers

The full processing mechanism for the Retina Early Layers is presented in Figure 4.3.

kS2(r)

kS2(r)

Visual Stimulus

Space filtering

Delay

Delay

Delay +

+

+

+ + x

F(x,t)

Low PassNon LinearHigh Pass

Time filtering

Contrast G

ain Control

R(x,t)

u(x,t)

IR(x,t) IG(x,t) IB(x,t)

Figure 4.3: Retina Early Layers diagram (adapted from [2]).

The spatial calculus is done using one DoG1 filter, equation (D.2), if space-time in-dependency is considered. Nevertheless, the spatial filtering should be done using twogaussians in order to independently convolve the center and surround since several au-thors have suggested that space-time filtering is not completely separable [1, 13]. Thusthis implies the use of a low pass filter, equation (D.8), after each gaussian. With sucharchitecture it is not only possible to implement space-time dependent models, but alsothe one with space-time separability.

As different colors are treated differently by the retina, the spatial filter can be ex-panded into three separate filters, one per each RGB channel. As depicted in Figure 4.3,considering space-time dependency there will be six receptive fields, two per RGB chan-nel. The time filtering is done using the high-pass filter presented in (D.6). The Contrast

1See Appendix D for an overview of filters implementation

4.2. NEURAL NETWORKS ARCHITECTURE 29

Gain Control (CGC) non-linear function was implemented using a look-up table and thelow-pass corresponds to equation (D.8).

Neuromorphic pulse coding

As the human brain responds to action potentials, the instantaneous firing rate at theoutput of the Early Layers must be modified to meet this new representation.

This task is carried out by the Neuromorfic Pulse Coding block. A first approach tofulfill this goal is to use the standard integrate-and-fire mechanism. To improve this model,some modifications were suggested: (1) a leakage factor in the integrator [23]; (2) a spikeheight modulation factor [24]. Integrate-and-fire was implemented using a simplification:the spike amplitude remains constant, only the leakage factor was included. This solutiondecreased the needed hardware as it requires less memory. Figure 4.4 shows the schematicrepresentation of this block.

x γ+-

+

d

-

pulse[q,n]+Pacc[q,n]

φ

F[q,n]

Z-1

Z-1

Figure 4.4: Integrate-and-fire block diagram.

The module can be described by (4.1) and (4.2)

Pacc[q, n] = F [q, n] + γ · Pacc[q, n− 1]− pulse[q, n− 1]− d (4.1)

pulse[q, n] = H(Pacc[q, n]− φ) (4.2)

where γ is the feedback loop gain, d is a leakage factor, φ is the firing threshold and qrepresents the discrete spatial dimentions.

4.2 Neural networks architecture

The architecture for the bio-inspired processing module of the artificial retina model thatis presented does not include the learning mechanism of the neural network. It is assumedthat a similar net has been previously trained and thereby all the necessary coefficientsare known.

The basic block for implementing the neural networks model is the perceptron. Asthe polarization input requires one adder but does not require a multiplier, a perceptronwith N inputs implies that the system requires N multipliers and an adder tree withN adders. Figure 4.5 represents the parallel architecture for a single perceptron. Thisparallel solution requires to many hardware resources. Since a parallel solution is not


coef0x

coef1 I1

x

+

coef2 I2

+

+

nl_input

x

coef IN N

x

+

IN-1coef N-1

x

+

IN-2coef N-2

++

Non Linear

spike

Figure 4.5: Parallel architecture for a single perceptron.

reasonable to implement, the proposed architecture was achieved by using folding. Thebasic block is composed by a multiplier and an adder, for which a data flow diagram tocalculate a single output of a perceptron is represented in Figure 4.6.

x

coef1 I1

x

coef2 I2

x

+

coef3 I3

x

+

coef4 I4

x

+

coef I

+

N N

x

+

coef5 I5

x

+

IN-1

...

coef N-1

+coef0 +

t1 t2 t3 t4 t5 t t tN-1 N N+ 1

nl _input

Figure 4.6: Perceptron data flow diagram.

The necessary functionality can be performed by a simple Multiply And Accumulate(MAC) unit (Figure 4.7) considering that the initial value of the feedback register is coef0.

x +coef i

Ii

Figure 4.7: MAC architecture.

To estimate the cost of this solution it is necessary to determine how many of thesebasic blocks are required. It was specified a maximum spike frequency (fspike = 1/Tspike)for each cell. By considering a micro-electrode array of size D, the available time forprocessing the output of one cell (Telectrode) will be

Telectrode =Tspike

D(4.3)

Considering N as the number of inputs of a percetron, using one MAC will require N + 1

4.3. SPIKE MULTIPLEXING 31

clock cycles to compute the output

Tpercept(N) = (N + 1) · TMAC (4.4)

where TMAC is the delay time of the MAC block2. Denoting Ni, i = 1, . . . , k, as thenumber of inputs of the ith layer, the necessary time to compute the output of a networkwith k layers is given by (4.5).

T ′spike = N2 · Tpercept(N1) + ... + Nk · Tpercept(Nk−1) + Tpercept(Nk) ⇐⇒

⇐⇒ T ′spike =

k∑

j=2

Nj · Tpercept(Nj−1) + Tpercept(Nk) (4.5)

In order to achieve the desired spike frequency, the following equation must be respected:

Telectrode >T ′

spike

M(4.6)

where M is the number of the necessary MACs. The coefficients, since their value isknown, can be stored into a Random Access Memory (RAM) or Read-Only Memory(ROM) block. The amount of required memory is given by (4.7) where b is the numberof bits used to represent a coefficient.

Memcoef =

k∑

j=2

Nj · (Nj−1 + 1) + Nk + 1

· b (4.7)

Finally, the output of the adder tree must suffer a nonlinear transformation. Thiscan be achieved by using the adder result to index a look-up table where samples of thenonlinear function are stored.

4.3 Spike multiplexing

This module serializes the spike information generated by the artificial retina processingmodule module in order to create a single data flow that can be transmitted to the brainover a serial link. Figure 4.8 represents the block diagram of the serialization and datapacking block. The data flow generated by the spike multiplexing module is packed

Data PacketsSpike

MultiplexingData Packing

Spikes

Figure 4.8: Serialization and data packing block.

accordingly to the serial communication protocol defined in the next section.

2The multiplier introduces the higher delay


The serialization module is based on the Address Event Representation (AER) [25]protocol which is designed to be a neuromorphic representation of signals. The protocoldefines a way to asynchronously transmit information without timestamps. When a spikeis generated, it is placed onto the bus as quickly as possible in order to minimize thelatency. No spikes are lost due to collisions, if two spikes occur simultaneously they aretransmitted as quickly as possible in sequence. A possible implementation for this moduleis the use of an AER tree [26]: it consists on multiple stages of arbiters which multiplexthe spike requests in their inputs to the output. However, this solution requires too muchhardware and is not scalable with the number of electrodes.

To overcome these drawbacks, a new solution was proposed using a First In FirstOut (FIFO) memory [2]. Since the spike generation circuit is intrinsically sequential forboth the classic and neural networks models, a FIFO memory can be used to registerthe generated spikes. This memory stores the information if the line if busy. In thisimplementation, the needed hardware does not increase with the number of electrodes,although the performance is more dependent on channel characteristics, it implies incre-asing the latency. Figure 4.9 depicts the architecture adopted for implementing the spikemultiplexing module.

Dual Port RAMDIAAddA

WEACLKA

DOA

DIBAddBWEBCLKB

DOB

Counter B

CLK Q

COUNT/STOP

CLK

pulse addr

pulse

eventBit VectorBit

Figure 4.9: Implemented AER module.

The pulse signal is only asserted by the retina processing module when a new spikeoccurs and the spike address is given by pulse addr. When a new spike occurs pulse isactive, enabling the write of the spike address in the RAM block. This signal also enablesthe counter that generates the write address of FIFO memory.

4.4 Serial communication protocol

In order to transmit spike information to the brain it is necessary to design a serial commu-nication protocol. The protocol frame format and main characteristics were described bythe CORTIVIS project specifications [3]. The emitter module receives event informationfrom the spike multiplexing module and sends events as soon as possible.

The RF link is a serial transmission channel that may distort signals and introduceerrors. A simple error detection mechanism, without error correction capabilities is used.The requirements of low power consumption at the receiver module was determinant foradopting this approach. There are no retransmissions, so, if the receiver detects errors, thecircuit discards all data corresponding to the packet. The transmission is asynchronousand a continuous flow of data maintains the channel busy. When there are no new events,

4.4. SERIAL COMMUNICATION PROTOCOL 33

the packet is filled with dummy bits, ensuring that the receiver can always extract powerfrom the received data. To simplify the protocol no kind of handshake is used. The packetstructure is presented in Figure 4.10.

Type Event 1

22 x 8 (or 4 ) = 176 ( or 88) bit

CRCHeader

LSB MSB LSB LSBMSB MSB

Event 8 ( or 4)Event 2 ...

DurationAmplitudeAddress

10 bit 7 bit 5 bit

22 bit

Valid

Reserved ( 1 bit)

16 bit4 bit3 bit10 bit

Figure 4.10: Packet structure [3].

The packet structure is as follows:

• Header, delimits a packet and synchronizes the communication;

• Type, this field is constant in this implementation and set to ”Stimulation”;

• Reserved, unused, equal to zero; added to allow future expansions to the protocol;

• Valid, represents the number of valid events in the packet;

• Event, carries the electrode address and data. Figure 4.10 highlights the structureof an event. The adopted retina model does not provide information about spikeamplitude or duration; in this implementation, these fields are constant and cannotbe adjusted dynamically; the number of events in a packet must also be adjusted incompile time: only the values 4 and 8 are accepted at the moment;

• CRC, this system uses a 16 bits Cycle Redundancy Code (CRC) (CRC-CCITT)with a polynomial of the type x16 + x12 + x5 + 1.

Although the CRC field was already defined in [3], it was necessary to change itsstructure. The packet header was smaller than the CRC, therefore the receiver coulddetect an invalid header. A way found to avoid this misinterpretation was to change thepacket structure and to include some bits delimiting the CRC field. Figure 4.11 shownsa possible packet structure, that would avoid the header erroneous detection.

Type Event 1CRC

(16 downto 8)Header Event 8 (or 4)...

22 bit

Valid

Reserved (1 bit)

4 bit3 bit10 bit

22 x 8 ( or 4 ) = 176 ( or 88) bit

CRC(7 downto 0)0 0 0

1 bit 1 bit 1 bit8 bit 8 bit

CRC

Figure 4.11: New packet structure.

The bits function remains the same, but the CRC field has a different structure. Usingzeros to delimit the CRC field and putting a zero in the middle, the header sequence will


never occur in the payload of the packet. This solution requires a more complex receiver,increasing its power consumption, which is a critical aspect since the receiver module willoperate inside the human head and retrieves energy from the serial link. As there are noretransmissions, the effect of an erroneous decoding is not much different from the lost ofspike information. If a packet is corrupted the data will be lost. The simple solution ofeliminating the CRC field was chosen and the architecture in Figure 4.12 is the adoptedpacket structure.

Type Event 1Header Event 8 (or 4)...

22 bit

Valid

Reserved (1 bit)

4 bit3 bit10 bit

22 x 8 ( or 4 ) = 176 ( or 88) bit

0...0

16 bit

Figure 4.12: Adopted packet structure.

The CRC field was replaced by a sequence of zeros. This solution requires less hardwareand thus reduces the power consumption.

4.5 System architecture conclusions

This chapter presented two different architectures for the visual encoding processing mo-dule. For the Classic Model architecture suggested in [2] the hardware requirements arescalable, allowing to process visual input to a larger target micro electrode array withthe same computational resources and only the necessary memory increases. The archi-tecture required to implement a neural network in Field-Programmable Logic (FPL) wasalso developed. Considering the data path only, the developed architecture can be usedfor different electrode matrix sizes and different neural network topologies.

A Spike Multiplexing module was created to achieve a single data flow for the spikeinformation generated by the processing module. Although this module was developedby [2] in a different model, as Neural Networks also present a sequential nature it can stillbe used with the proposed new model. To deal with a bigger electrode matrix, only thememory size needs to be changed.

To enable the transmission of the spike information generated by the processing mo-dule over a RF serial link channel a Serial Communication protocol was developed. Thisprotocol was specified in [3] although changes were required to achieve the correct func-tioning of the system. The chosen specification was the simplest one in order to ensurethat the unpacking module does not require much power. This is a critical aspect sincethe unpacking module will operate inside the human head.

Chapter 5

Full System Prototype

In Chapter 4, the system architecture was introduced considering a FPL implementation.This chapter presents a full prototype for the complete retina model.

Analyzing the hardware requirements of the bio-inspired processing module, the pre-vious prototype [2] was based on a FPGA with a small amount of memory. In order tomake the validation of the complete model and to allow future expansions a prototypeboard with more resources was developed. The main goal of this system is to achieve asmall size prototype derived from a low power design based on a FPGA with more com-putational resources than the previous solution. The low power objective was the mostdifficult to achieve, considering that the main component is a FPGA. This board willalso have a Video Graphics Array (VGA) port, in order to simulate and to validate themodel implementation. The digital output image of the processing module is convertedto analog level using three video Digital to Analog Converter (DAC)s. Figure 5.1 showsa block diagram of the intended prototype.

RF Link

Processed Image

C3188A Camera Module

Standard Monitor

VG

A

RF Link ModulatorC

amera C

onn

ector

Ge

neric C

onne

ctor

8-bit DACs

Artificial Retina Prototype

Input Image

Data PacketsXILINX

SPARTAN XC3S400

Figure 5.1: Block diagram of the prototype.

The prototype board was developed using Altium Protel 2004, a complete board-leveldesign system Computer Aided Design (CAD) software. A datasheet for this board ispresented in Appendix E. The developed prototype was implemented on a four layerPrinted Circuit Board (PCB) in which the two inner planes are dedicated to VCC andGND and the outside layers are used for rooting.

35

36 CHAPTER 5. FULL SYSTEM PROTOTYPE

This chapter also describes the digital logic blocks necessary to implement the retinamodel. The two considered architectures for the main processing module were presentedin the previous chapter. To create a complete system, it was necessary to develop otherhardware modules. These modules, addressed in this chapter, are responsible for theinterface with the digital camera and for generating the output image representing theprocessed stimulus. Finally, the hardware design for the data packing and unpackingmodule, that implements the serial communication protocol, is also presented.

5.1 Prototype processing core

The developed board uses the XILINX SPARTAN-3 XC3S400 [27] FPGA as the proces-sing core. This represents a low cost solution but also ensures a significant increase inhardware resources when compared to the FPGA employed in the previous developedprototype [2], the XILINX SPARTAN-2 XC2S200 [28] FPGA. The XC3S400 has 400 ksystem gates, a total memory of 288 kb distributed in 16 RAM blocks, 16 dedicated 18-bitmultipliers and fast look-ahead carry logic. The main difficulty with the previous proto-type was the amount of memory available. The current solution is more adequate, sincethe available memory in the RAM blocks is more than five times greater. The dedicatedmultipliers allow to save system gates and to increase the operating frequency. This is animportant feature because a higher operation frequency allows to process data for a biggerelectrode matrix. However, there is a main drawback in using this FPGA, it requires threeindependent supply voltages: VCCINT internal supply voltage (1.2 V ); VCCAUX auxiliarysupply voltage (2.5 V ); VCCO output driver supply voltage (1.2 V to 3.3 V ).

Although XC3S400 has a maximum of 264 I/O ports, this design does not use themall. Figure 5.2 represents the FPGA I/O available signals. Two external connectorsare provided: i) Cam is dedicated to the interface with the digital camera and uses 25pins including two Global Clock (GCLK)1 inputs; ii) Gen is a generic connector thatprovides 36 pins, four of them are GCLK inputs, this connector can be used to debug,to output results or to connect expansion modules. It also provides three dedicatedbuses that connect directly to the video DACs, a clock net to the video DACs and thesynchronization signals necessary to the VGA port. The connection diagrams for theFPGA can be found on Appendix F.1 and Appendix F.2.

The pins corresponding to the Joint Test Action Group-IEEE Standard 1149.1 (JTAG)interface used to configure the FPGA are not represented in Figure 5.2, but can be foundin Appendix F.2. The configuration pins are only available through a six pin header, inorder to reduce the PCB size.

5.2 Power distribution system design

Since FPGAs can implement an almost infinite number of applications at undeterminedfrequencies and in multiple clock domains, it can be very complicated to predict its cur-

1Global Clock, low-capacitance, low-skew network well-suited to high-frequency signals

5.2. POWER DISTRIBUTION SYSTEM DESIGN 37

IO_L01P_7/VRN_72

IO_L01N_7/VRP_73

IO_L16P_7/VREF_74

IO_L16N_75

IO_L19P_77

IO_L19N_7/VREF_79

IO_L20P_710

IO_L20N_711

IO_L21P_712

IO_L21N_713

IO_L22P_715

IO_L22N_716

IO_L23P_718

IO_L23N_719

IO_L24P_720

IO_L24N_721

IO_L39P_722

IO_L39N_724

IO_L40P_726

IO_L40N_7/VREF_727

IO_L40P_6/VREF_628

IO_L40N_629

IO_L39P_631

IO_L39N_633

IO_L24P_634

IO_L24N_6/VREF_635

IO_L23P_636

IO_L23N_637

IO_L22P_639

IO_L22N_640

IO_L21P_642

IO_L21N_643

IO_L20P_644

IO_L20N_645

IO_L19P_646

IO_L19N_648

IO/VREF_650

IO_L01P_6/VRN_651

IO_L01N_6/VRP_652

IO_

L01

P_5/

CS

_B57

IO_

L01

N_

5/R

DW

R_

B58

IO_

L10

P_5/

VR

N_5

61

IO_

L10

N_

5/V

RP_

562

IO63

IO_

L27

P_5

64

IO_

L27

N_

5/V

RE

F_5

65

IO_

L28

P_5/

D7

67

IO_

L28

N_

5/D

668

IO71

IO_

L31

P_5/

D5

72

IO_

L31

N_

5/D

474

IO_

L32

P_5/

GC

LK

276

IO_

L32

N_

5/G

CL

K3

77

IO/V

RE

F_5

78

IO_

L32

P_4/

GC

LK

079

IO_

L32

N_

4/G

CL

K1

80

IO_

L31

P_4/

DO

UT

/BU

SY

81

IO_

L31

N_

4/IN

IT_B

83

IO/V

RE

F_4

85

IO_

L30

P_4/

D3

86

IO_

L30

N_

4/D

287

IO_

L27

P_4/

D1

90

IO_

L27

N_

4/D

IN/D

092

IO93

IO_

L25

P_4

94

IO_

L25

N_

495

IO/V

RE

F_4

96

IO97

IO_

L01

P_4/

VR

N_4

100

IO_

L01

N_

4/V

RP_

410

1

IO/V

RE

F_4

102

IO_L01P_3/VRN_3 106IO_L01N_3/VRP_3 107IO_L17P_3/VREF_3 108IO_L17N_3 109

IO_L19P_3 111IO_L19N_3 113IO_L20P_3114IO_L20N_3 115IO_L21P_3 116

IO_L21N_3 117IO_L22P_3 119IO_L22N_3 120IO_L23P_3/VREF_3 122IO_L23N_3

123IO_L24P_3 124IO_L24N_3125IO_L39P_3 126IO_L39N_3 128

IO_L40P_3 130IO_L40N_3/VREF_3 131

IO_L40P_2/VREF_2132IO_L40N_2 133IO_L39P_2 135

IO_L39N_2 137IO_L24P_2 138IO_L24N_2 139IO_L23P_2 140IO_L23N_2/VREF_2

141IO_L22P_2 143IO_L22N_2144IO_L21P_2 146IO_L21N_2 147

IO_L20P_2 148IO_L20N_2 149IO_L19P_2 150IO_L19N_2 152IO/VREF_2

154IO_L01P_2/VRN_2 155IO_L01N_2/VRP_2 156

IO_L

01P

_1/V

RN

_116

1IO

_L0

1N_1

/VR

P_1

162

IO_L

10P

_116

5IO

_L10

N_1

/VR

EF_

116

6IO

167

IO_L

27P

_116

8IO

_L

27N

_116

9IO

_L2

8P_1

171

IO_

L28

N_1

172

IO17

5IO

_L3

1P_1

176

IO_L

31N

_1/V

RE

F_1

178

IO_

L32

P_1/

GC

LK

418

0IO

_L32

N_1

/GC

LK

518

1IO

182

IO_

L32

P_0/

GC

LK

618

3IO

_L32

N_0

/GC

LK

718

4IO

_L3

1P_0

/VR

EF_

018

5IO

_L

31N

_018

7IO

189

IO_L

30P

_019

0IO

_L

30N

_019

1IO

_L2

7P_0

194

IO_

L27

N_0

196

IO19

7IO

_L2

5P_0

198

IO_

L25

N_0

199

IO/V

RE

F_0

200

IO_L

01P

_0/V

RN

_020

3IO

_L0

1N_0

/VR

P_0

204

IO/V

RE

F_0

205

BANK 0 BANK 1

BA

NK

3B

AN

K 2

BANK 4BANK 5

BA

NK

7B

AN

K 6

U6AXC3S400-5PQ208C

Gen0

Gen1

Gen2

Gen3

Gen4

Gen5

Gen6

Gen7

Gen8

Gen9

Gen10

Gen11

Gen12

Gen13

Gen14

Gen15

Gen16

Gen17

Gen18

Gen19

Gen20

Gen21

Gen22

Gen23

Gen24

Gen25

Gen26

Gen27

Gen28

Gen29

Gen30

Gen31

Gen32Gen33

Gen34Gen35

Gen[0..35]

DG0DG1DG2DG3DG4DG5DG6DG7 DB[0..7]

DG[0..7]

DR[0..7]

Cam

0C

am1

Cam

2C

am3

Cam

4C

am5

Cam

6C

am7

Cam

8C

am9

Cam

10C

am11

Cam

12

Cam[0..24]

BTN0BTN1BTN2BTN3

BTN4

BTN[0..4]

Gen[0..35]

DB[0..7]

DG[0..7]

DR[0..7]

BTN[0..4]

CL

K_d

ac

VGA_HSVGA_VS

CLK_dacCLK_dac

CL

K_5

0Mhz

CLK_50MhzCLK_50Mhz

Cam17Cam18Cam19Cam20Cam21Cam22Cam23Cam24

Cam

[0..2

4]

Cam16

Cam

14C

am13

Cam15

DR0DR1DR2DR3DR4DR5DR6DR7

DB4DB5DB6DB7

DB0DB1DB2DB3

Figure 5.2: FPGA available I/O signals.

rent demands. These transient currents are the main cause of ground bounce thus makingthe decision on the regulators to use on the Power Distribution System (PDS) very im-portant. Two choices were considered to generate the required voltage levels: linear andPulse Width Modulation (PWM) regulators. While the first one requires less externalcomponents, its efficiency is highly dependent on the dropout2. In opposition, the PWMregulators are able to achieve high efficiency levels even with high voltage dropouts, thussaving thermal dissipation area. Based on such arguments, the choice was to use linearregulators for the 5V power grid and PWM regulators for all other cases. This allows toguarantee a highly efficient PDS with a minimum number of external components.

For all the supply voltages of the FPGA (1.2 V , 2.5 V and 3.4 V ) it was used theMAX1830/MAX1831 [29] a low voltage PWM, step-down regulator, which delivers acurrent of up to 3 A with a peak efficiency of 94%. This PWM architecture regulatesthe output voltage by changing a PMOS switch on-time relative to the constant off-time.Increasing the on-time increases the peak inductor current and the amount of energytransferred to the load per pulse. This device also has another mode of operation, savingpower when the load current is low. Considering the device datasheet, it is possible

2Difference between input and output voltage levels


to obtain all of the desired voltages. The adjustable configuration for this regulator ispresented in Figure 5.3.

10

2.2uF

470pF

130K

Rtoff

22uF

Cin 1uF

R2

R1

2.2uH

L1

Inductor Isolated

150uF

Cout

Vout

Vcc

VCC12

n_SHDN5

LX 1

LX3

LX 14

PGND 13GND 9

IN4 IN2

COMP6 FB 8LX 16

FBSEL 11

REF10

TOFF7

PGND 15

MAX1830

Figure 5.3: MAX1830/1831 adjustable configuration electrical diagram.

This regulator has an internal voltage reference source. The internal voltage is com-pared to the feedback voltage, provided by a resistive divider placed between the outputvoltage and the ground. The output of this comparator along with a current sense signalis used as input to the PWM logic that controls the output MOS transistors. In the ad-justable mode, the output voltage is determined by an equation presented in the regulatordatasheet. Applying that equation to the circuit in Figure 5.3 yields

R2 = R1

(Vout

VREF

− 1)

, (5.1)

where VREF is the regulator internal reference voltage (typically 1.1 V ) and R1 = 30 kΩ.Hence, for VCCINT = 1.2 V , R2 must be set to 2.7 kΩ. The values for RTOFF and for theinductor L were obtained considering the values proposed in the datasheet. The othervoltage levels were achieved using the preset output available voltages. Table 2 of theregulator datasheet shows how the output voltage programming can be done. Consideringthis table, VCCAUX is obtained by using a MAX1830 and connecting the FBSEL to VCC

and FB to the output voltage. For the VCCO, voltage FBSEL must be connected to theREF pin and FB to the output voltage. The inductor and RTOFF are the same as forthe adjustable mode.

The second component of the PDS is the bypass or decoupling capacitors networkwhich act as a local energy storage. Every capacitor has a narrow frequency band whereit is most effective as a decoupling capacitor. The Equivalent Series Resistance (ESR)of the capacitor determines the quality factor (Q), which determines the width of theeffective frequency band. Real capacitors also have a parasitic inductance3; these twoparasite components form an RLC circuit, Figure 5.4, and the resonant frequency of thecapacitor will be the frequency associated with the RLC circuit.

In order to determine a proper placement, it is necessary to consider the capacitor’sparasite inductance, LSELF , previous designated as ESL, and the mounting parasitic

3Also known as Equivalent Series Inductance (ESL)

5.2. POWER DISTRIBUTION SYSTEM DESIGN 39

C

ESLESR

Figure 5.4: Equivalent circuit of a real capacitor.

inductance, LMOUNT . This LMOUNT is within the range of 300 pH to 1500 pH [4].Considering that the resonant frequency (FR) for this RLC circuit is given by (5.2),

FR =1

2π√

(LSELF + LMOUNT )C(5.2)

it is now possible to determine the effective resonant frequency associated with a capacitorplaced on the board. Noise from the FPGA falls in certain frequency bands, and differentsizes of decoupling capacitors will operate in those different bands. For this reason,capacitor placement is determined based on the effective frequency of each capacitor.Capacitors need to be close to the device to perform the decoupling function. There aretwo basic reasons for this requirement. First, shortening the distance between the deviceand the decoupling capacitors reduces the inductance resulting in a less impeded transientflow. The second reason deals with the phase relationship between the FPGA noise sourceand the mounted capacitor. The placement of the capacitor determines the length of thetransmission line that connects the capacitor and the FPGA being the propagation delaythe relevant factor. For any transient current demanded by the FPGA, there is a round-trip delay to the capacitor before any relief is seen. As result, if the distance is greaterthan a quarter of wavelength, the transferred energy is negligible. The energy transferredto the FPGA increases to approximately 100% at zero distance. In practical applications,one tenth of a quarter wavelength is a good target [4]. The wavelength corresponds tothe capacitor mounted resonant frequency.

To achieve a balanced decoupling network, it is desirable to use the capacitor valuespresented in Table 5.1. The exact value of these capacitors is not critical, but it is

Table 5.1: Capacitor value percentages for a balanced decoupling network [4].

Capacitor Value Quantity Percentage Capacitor Type

470 µF to 1000 µF 4% Tantalum

1.0 µF to 4.7 µF 14% X7R 0805

0.1 µF to 0.47 µF 27% X7R 0603

0.01 µF to 0.047 µF 55% X7R 0402

necessary to have some capacitors in every order of magnitude. According to Table 5.1,the smaller capacity will be 0.01 µF , considering (5.2) the resonant frequency is

FR =1

2π√

(0.9× 10−9 + 1.0× 10−9)0.01× 10−6= 36.5 MHz (5.3)


assuming a LSELF = 0.9 nH [4] and a LMOUNT = 1.0 nH. With a propagation speedof about 1.54 × 108 m/s [4] for the FR4 dielectric, the wavelength associated with thiscapacitor is

λ =1.54× 108

36.5× 106= 4.22 m (5.4)

Therefore the target radius, rPLACE, will be one tenth of a quarter wavelength, rPLACE =4.22/40 = 0.11 m . Since all the other capacities are larger then this one, this will bethe smallest necessary radius. Considering the board expected size, this radius will beachievable.

It is only necessary to provide one capacitor per VCC pin, if all pins are used. HenceVCCINT and VCCAUX must always be decoupled, but VCCO can be prorated according tothe I/O utilization. As in this design not all FPGA input/output pins are connectedthe decoupling of VCCO will only use a ratio of one capacitor per two pins. The adopteddesign uses the capacitors present in Table 5.2.

Table 5.2: Decoupling capacitors quantities.

Quantity

Capacitor Value VCCINT VCCAUX VCCO

470 µF 1 1 1

1.0 µF 1 1 2

0.1 µF 2 2 4

0.01 µF 4 4 8

Only the VCCINT has one capacitor per pin, as it is the most critical supply. TheVCCAUX and VCCO have a rate of one capacitor per two supply pins, having the quantitiesfor each capacitor value been calculated accordingly to Table 5.1.

Also, in order to achieve a more effective decoupling network, the prototype uses afour layers PCB in which the two internal layers are power planes and the outside layersare used for rooting. Figure 5.5 represents the arrangement of the power planes.

Different planes are identified with different colors. The split planes utilized for theDACs are also represented. The ground plane is also divided, but in a more simplemanner. All the digital circuitry shares the same plane, only the analog part of the DACshas a separate ground. The arrangement of the decoupling capacitors with the lower valueis also represented in the figure since their placement is more critical. The usage of thisarrangement for the split planes lead to a small distance between the capacitors and thepower pins and therefore increases the decoupling efficiency.

In Appendix F.3, the electrical diagram for all the regulators is presented and in theAppendix F.2 is the decoupling network can be found.

5.3. VGA DISPLAY PORT 41

Generic Connector

Camera Connector VGA

8-bit DACXILINX SPARTAN

XC3S400

Vccaux Plane

Vcco Plane

RegVccaux

RegVccint

RegVcco

Vccint Plane

DAC Plane B

DAC Plane G

DAC Plane R

Figure 5.5: Power plane division, different colors identify split planes borders. The de-coupling capacitors with the lower value are also represented.

5.3 VGA display port

In order to test the model implementation a VGA display port and 3 video DACs wereadded to the prototype. The choice of 3 distinct DACs allows future expansions. Ifthe model output is extended for separate processing for each color channel this sameprototype board can be used for validation. This prototype is intended to display animage with color information, if required. Also, in order to achieve a better image quality,the simple resistor DAC used in the previous prototype had to be put aside. The displayport is based in three separate 8-bit Video Digital-to-Analog converters, TLC5602C [30],one per RGB channel. This DAC presents a low power consumption (typically 80 mW ),20 MHz conversion rate, single 5 V power supply and TTL digital input voltage. Inorder to ensure the correctness of the digital output logic levels from the FPGA, theVCCO voltage must be set to 3.3 V . Another necessary input for this DAC is a referencevoltage of 4 V . This voltage is generated with a MAX6004 [31], a low power and lowdropout voltage reference Integrated Circuit (IC). The electrical diagram for the DACresponsible for one of the VGA channels is presented in Figure 5.6.

600z

L12

Inductor

600z

L13

Inductor1uF

C25Cap Semi

600z

L14

Inductor

600z

L15

Inductor0.01uF

C26Cap Semi

AGNDR

AGNDR

AGNDR

AVDDR

DGNDR

DVDDR

COMP_RCLK_dac

DR0DR1DR2DR3DR4DR5DR6DR7

DR[0..7]DR[0..7]

1uF

C27Cap Semi

COMP_R

AoutR

+5

+5

Vref_dac

D514 D415 D316 D217

ANLGGND10

D118

COMP3

D0(LSB)19

D7(MSB)12

CLK11

REF4

AOUT 6

DGTLGND1

NC20 20

NC77

DGTLVDD29

ANLGVDD28ANLGVDD15

DGTLVDD12

D613

U9

TLC5602CDW

VIN1

3

VOUT 2

GND

U5

MAX6004

+5 Vref_dac

Figure 5.6: DAC electrical diagram.


According to the device datasheet, it is necessary to decouple the power pins, whichis accomplish by placing a ferrite bead in series with the power and ground pins and byputting a capacitor between these pins, close to the device. Also, in order to decreasethe circuit noise, each DAC has a separate analog power and analog ground plane, beingisolated from the digital circuitry and from the other DACs. The electrical diagram forall DACs is presented in Appendix F.4.

5.4 Complete board and components placement

While in the preceding sections the individual parts of this prototype were characterized,this section integrates the several blocks into a complete board.

To design an autonomous system, it is necessary to include an additional power sup-plier for the digital camera. As the required voltage is the same for the DACs, these deviceswill be powered by the same source. This voltage level is generated by a TPS78601 [32],a ultra low-noise, low-dropout linear regulator with a maximum output current of 1.5 A.This regulator has an adjustable output voltage. Figure 5.7 presents diagram of the circuitfor the adjustable voltage mode.

C1R2 R1

1uF2.2uF

VoutVcc

OUT 4IN2

GN

D3

TA

B6

EN1 FB5

U10

TPS78601

Figure 5.7: TPS78601 adjustable configuration electrical diagram.

To control the output, the IC uses an internal reference voltage. This reference iscompared to the feedback voltage provided by a resistive divider between the outputvoltage and the ground. The output of the comparator along with a current sense signalis used to control the gate voltage of Metal-Oxide Semiconductor (MOS) transistor placedbetween the unregulated input and the regulated output voltage. By changing the gatevoltage the on-resistance of the transistor is altered and thereby the output voltage iscontrolled. Hence, since the transistor is placed in series, the power dissipation will bedirectly proportional to the voltage difference between the input and the output. Asstated in Section 5.2, the linear regulator has a much simpler circuit, although it onlypresents a good efficiency if the input voltage is close to the output voltage. However,since the maximum input voltage for this regulator is 6 V , the maximum dropout voltageis 1 V leading to a minimum efficiency of over 80%. Moreover, as it requires less exteriorcomponents then the PWM regulator, the electrical diagram is simpler. Thus, the usageof this device allows the PCB size to be reduced. To obtain the desired voltage the R2

5.4. COMPLETE BOARD AND COMPONENTS PLACEMENT 43

resistor must be set according to equation (5.5),

R2 = R1

(Vout

VREF

− 1)

, (5.5)

where R1 = 30 kΩ and VREF is typically 1.2246 V . Hence, to achieve the desired voltage,R2 must be set to 92.4 kΩ. Also, according to equation 3 of the regulator datasheet, toimprove the stability of the output voltage, C1 must follow the next equation

C1 =(3× 10−7)× (R1 + R2)

(R1 ×R2)(5.6)

and thereby C1 will be equal to 15 pF .

The PCB has a generic input/output connector to enable debugging, the output ofresults and/or future expansions. It also has a connector to deliver power and receive theinput image from the digital camera. Another necessary component is a clock generator,for this design it was employed the CFPS-73 [33], a 50 MHz, High Speed ComplementaryMetal-Oxide Semiconductor (HCMOS) 3.3 V oscillator. Finally, to create simple controlsignals, a push button and a four way slide switch were included. Figure 5.8 shows apicture of the actual developed PCB board, were the main components are highlighted.

Camera Connector

Generic Connector

JTA

G

XILINX SPARTAN3

XC3S400Video DACs

VGA

VTORegulator

Vcco

RegulatorVccint

RegulatorVccaux

Regulator5 V

ControlSwitches

Power Jack

Oscilator

Figure 5.8: Complete prototype board.

Appendix E provides a short datasheet of this prototype. It includes the connectorpinouts and all the pin connections to onboard input and output devices. The electricaldiagram for the full prototype can be found on Appendix F.5.

When assembling the prototype some minor errors were detected, none of which com-promise the correct operation of the board. The pin order in the JTAG interface wasincorrect and the footprint size of some components was not adequate. There are certainaspects that could be improved. The usage capacitors with a smaller package outlinewould allow the placement of more decoupling capacitors on the VCCAUX source, there-fore creating a better decoupling network. Moreover, by placing more components on theback side of the PCB, a decrease in the board size would be achieved.


5.5 Digital logic blocks

In order to perform a validation of the retina processing module, the architecture wasimplemented in VHDL and tested on the developed prototype board. Only the ClassicModel was implemented but an analysis of the Neural Networks was made towards de-termining the required hardware resources. The tests were made using only one colorchannel and an input represented in 8-bit greyscale, however a 12-bit representation wasused internally to reduce computational and filter discretization errors. The considereddimension of the electrode array was 32 × 32. It was also necessary to develop othermodules, Figure 5.9 represents the developed logic circuits.

Register Configuration

Serialization and Data Packing

SDA

SCL

Input Image

Data Packets

HS

VS

R,G,B

Image Capture and Resize

Image Display

Resized Image

Visual EncodingSpikes

Processed Image

Figure 5.9: Digital logic block diagram.

Two modules were created for image acquisition, one to capture the input image andresize it and the other to make the power-on configuration of the digital camera internalregisters. The resized image is then feed to the visual encoding module. This moduleproduces two outputs, one is the train of action potentials that will be sent to the brainand the other is the firing rate that results from the retina model filtering. The datapacking module prepares the spike information to be transmitted over a serial link. Atlast, the display module produces a screen visual image to allow the validation of theprocessing module. This module displays the input image, the spike information and theprocessed image before the spikes are generated.

The synthesis results presented in the following subsections where achieved using theXILINX ISE WebPACK 6.2i synthesis tool with a XILINX SPARTAN-3 XC3S400-4 astarget device.

5.5.1 Image capture and resize

The input image was obtained using a digital camera module from Quasar Electronics [34]that uses Omnivision OV7620 [35] Complementary Metal-Oxide Semiconductor (CMOS)image sensor. All camera functions can be configured using a serial data transmissionprotocol, Serial Camera Control Bus (SCCB) [36], which is a simplified version of PhilipsInter Integrated Circuit (I2C) [37] protocol. This camera is able to capture a window sizeof 4.84 mm × 3.64 mm generating an array of 644×492 with a pixel size of 7.6 µm ×7.6 µm. The digital video port supports 60 Hz YCrCb 4:2:2 16-bit/8-bit format, ZoomedVideo Format (ZV) Port output format, RGB raw data 16-bit/8-bit output format andCCIR601/CCIR656 format, with progressive or interlaced scan. It also has a black and

5.5. DIGITAL LOGIC BLOCKS 45

white composite video signal output (VTO) in National Television System Committee(NTSC) format that can be used for test purposes. The sensor has a signal to noise ratiogreater then 48 dB and a dynamic range of more than 72 db. The supply voltage for thissensor is 5 V and it requires less then 120 mW when it is active and only 50 µW while instandby. This camera module includes lens with a focal length and an aperture of 6 mmand F1.6, respectively. The current retina model version only uses greyscale data and thecamera is programmed to capture frames in Quarter Video Graphics Array (QVGA), ata frame rate of 20 Hz in non-interlaced mode.

The camera module supplies three synchronization signals: vertical sync (VSYN), ho-rizontal window reference (HREF) and pixel clock (PCLK). Frame synchronization is doneby detecting a high pulse on VSYN and a new line occurs when HREF has a low pulse,Figure 5.10 shows the temporal diagrams for this two signals.

HREF

VSYN

Tdisp

Tpw

Tbp

Tfp

TLINE

TSWL

HREF

PCLK

Figure 5.10: Camera module sync signals.

The PCLK is used to identify pixels on a line, the pixels information is updated everyfalling edge of pixel clock. This implies that pixel information can be read on the risingedge of PCLK. Another signal, odd field flag (FODD), is also required in order to only assertWE when FODD is high.

A simple hardware can be used in order to capture the frames using these four synchro-nization signals. In Figure 5.11, it is shown the implementation used for this module [2].

Counter

CLK Q

COUNT/STOP

RESET

Counter

CLK Q

COUNT/STOP

RESET

FFD FFD

D D

FFD FFD

D D

Falling edge detection

Rising edge detection

CLK

HREF

PXL_CLK

VSYNC

FODD WE

Columncount

Linecount

Bit Vector

Bit

D

FFD

Figure 5.11: Implemented hardware for frame capture.

The hardware costs of this synchronization device are 15 slices (1%) with a maximumoperating frequency of 25 MHz.


The developed processing module for this prototype was configured to have an inputof only one color channel, in this case greyscale, and a frame window of 32×32. In afirst approach, the desired window was obtained using only register configuration, but theimage frames for this solution where very affected by noise. Another solution was to makean image resize: the visual frames were captured with a 128×128 window size and, usingfiltering followed by decimation, the image was reduced to 32×32 pixels. The filter mainpurpose is to avoid the aliasing generated by the decimation process, but it is also used toreduce some of the vertical noise. Three types of filters were considered: bicubic, gaussianand bilinear. The chosen filter was a gaussian low pass type with a standard deviationof two, since it provides good results and does not require much logic. The filtering wasdone line by line delivering an output in the same way and can be described by equation(D.1). The schematic representation can be seen in Figure 5.12, where the symmetry ofthe gaussian was used to reduce the necessary logic, it uses less multipliers and less delaylines.

x

x

x

x

x

Register

++ + +

Delay line

a1

a2

a3

a4

a0

+ ++

+

Inputpixel

Output pixel

Decimaion

a 1x[

n+1]

a 2x[

n+2

]

a3x[

n+3

]

a4x[n+4]

a0x[

n]

a 1x[

n]a 2

x[n

]

a3x[

n-3

]

a4x[

n]

a 2x[

n+3

]

a 1x[

n+2]

a 1x[

n+3]

a0x[

n+1]

a0x[

n+2]

a0x[

n+3]

a 2x[

n+1

]

a3x[

n-2

]

a3x[

n-1

]

a 4x[n

+2]

a 4x[

n+3]

a 4x[n

+1]

a3x[n+4]

a2x[n+4]

a1x[n+4]

a0x[n+4]p[n]

p[n-1]=

x[n+4]

Y[n]

a 2x[

n-1

]a 1x

[n-1

]

a 2x[

n-2

]

a 4x[

n-4]

a 4x[

n-2]

a 4x[

n-1]

a 4x[

n-3]

a3x

[n]

a3x[

n+1

]

a3x[

n+2

]

Figure 5.12: Spatial low pass gaussian filter.

The hardware costs of this filter are 130 slices (3%) and 1 RAM block and the maximumoperating frequency is 179 MHz.

5.5.2 Register configuration

A new hardware module was developed to improve the output image of the digital camera.This independent system was only designed to experiment new register settings, althoughthe VHDL core had to be included in the main design in order to make the registerconfiguration at setup time.

To allow a more flexible operation, the VHDL core receives data with the registersettings from a PC through the Digilab 2 [38] parallel port. The data is sent by a simpleprogram written in C language. Figure 5.13 shows the connections diagram for thismodule.


C3188ACameraModule

Parallel port

connector

DIGILAB D2 Board

8-bit DACStandartMonitor

HSYNC

VSYNC

R,G,B

XILINXSPARTANXC2S200

BoardPowerSupply

Serial portconnector

PC

DB25

SDA

SCL

Figure 5.13: Block diagram for register configuration module.

Initially, the camera composite video output, VTO, was used to evaluate the resultingimage. However, this more simple solution had to be put aside as it was necessary toadjust the frame rate to achieve better results. The block diagram of the developedsystem is shown in Figure 5.14.

C3188A Camera Module

Dual Port RAM

DB25 program_ regs

SCL

SDA

epp2PCQA

DB

Figure 5.14: Register configuration block diagram.

The control program supplies the register address and value to the VHDL core andstores the register configuration in a file. A first VHDL block, epp2, only reads theprogramming data from the parallel port and then stores it in a dual port RAM; thisblock has a counter that is incremented whenever a new byte is received. The dual portRAM is also connected to a second block, program_regs, responsible for the interface withthe camera using the SCCB protocol. The protocol was implemented using a hierarchicalthree level state machine as shown in Figure 5.15.

camera_config

read_cycle write_cycle

gen_startread_data gen_stop write_data

Figure 5.15: Block diagram of the program regs.

The first layer, camera_config, sends commands to enable a read or a write cycleand supplies the register address and the value, which were stored in the dual port RAM.


The cycle is initiated when a button on Digilab 2 is pressed. The next layer consistsof two state machines: one to implement the write cycle (write_cycle), and the otherthe read cycle (read_cycle), only one is active at a time. The third layer does theinterface with the camera module. The last layer is responsible for generating all the bitsequences necessary to implement the SCCB protocol. A write and a read cycle is shownin Figure 5.16 and 5.17, respectively.

ID6ID7

.. .

.. . ID0 X ADD7 ADD6

.. .

.. . ADD0 X D7 D6

.. .

.. . D0 X

gen_start write_data write_data write_data gen_stop

Figure 5.16: Write cycle.

ID6ID7

.. .

.. . ID0 X ADD7 ADD6

.. .

.. . ADD0 X

gen _start write_data write_data

ID6ID7

.. .

.. . ID0 X D7 D6

.. .

.. . D0 X

gen _start write_data read_data

gen_stop

gen_stop

Figure 5.17: Read cycle.

After achieving a suitable register configuration, the PC interface is no longer ne-cessary. A modified version of this system was included in the main design, in order tomake the power-on configuration. In this solution, the configuration values are storedin a ROM memory block and only write operations are necessary. This register configu-ration device requires a total of 114 slices (3%), operating at a maximum frequency of191 MHz.

5.5.3 Classic model implementation

The processing model was described in Section 4.1. The employed implementation onlyprocesses one of the RGB channels and considers space-time separability. The global FPLarchitecture for one RGB channel is shown in Figure 5.18.

Figure 5.19 shows the adopted FPL architecture for the Spike Generation and theimplementation of the several filters can be found in Appendix D. This model was usedin order to test the developed prototype board and also the other developed hardwaremodules. In Table 5.3 it is shown the needed hardware resources for the processing modulewithout the CGC block.

The complete model, including the CGC block, occupies a total of 365 slices (10%)and 7 RAM blocks (43%), and has a maximum operating frequency of 86MHz.


DoG x

Non Linear

Low pass V

isual stimulus Rectif.

QA

DB

Early Layers Neuromorphic Pulse Coding

QA

DB

Dual port RAM

Spike Generation

QA

DB

Dual port RAM

Spike Multiplexing

High pass

Spikes to send

Figure 5.18: Early Layers FPL full architecture.

Signal Processing >=

+-

+

x

'0'

Spike Amplitude

dRegister A

++

-

+

MUX B

0

1

I[q,n] pulse[q,n]

Bit Vector

Bit

Pacc[q,n]

FR[q,n]

γ

Figure 5.19: Integrate-and-fire adopted architecture.

5.5.4 Neural networks implementation

For the architecture presented in Section 4.2, it is now calculated the hardware resourcesneeded to implement the neural networks approach. It is important to notice that thiscalculus does not consider the unpredictable extra hardware costs due to routing neitherthe delays generated by this same routing. This estimation will assume the usage of a18-bit representation for the internal calculations.

The maximum spike rate generation must is 1 kHz per cell [2]. Considering an array32 × 32, electrodes, the available time for calculating the output of one cell accordinglyto (4.3) is

Telectrode =1

1024× 1000= 977 ns (5.7)

One of the employed neural networks has two hidden layers. The first hidden layer receives100 inputs, the second 20 and the output layer only has 10 inputs. Using (4.5) and (4.4),it is possible to calculate the necessary time for calculating one output using a single MAC

T ′spike = 20 · Tpercept(100) + 10 · Tpercept(20) + Tpercept(10) = 2241 · TMAC (5.8)

The implementation costs of a single MAC using the developed prototype are, consideringthe usage of embedded multipliers, 19 slices and the maximum operating frequency is190 MHz. The minimum required number of MACs that are necessary to respect thetemporal restrictions can be calculated using (4.6),

M =

⌈2241× 5, 26× 10−9

977× 10−9

⌉=⇒ M = 13 (5.9)


Table 5.3: Implementation costs for the Bio-inspired Processing Module.FPGA Block Slice Operating RAM Blocks

Occupation Frequency (MHz) (Total 16)

Retina 21% 47∗ 5

Xilinx Spartan Early Layers

XC2S200 Neuromorphic 2% 51 3

Pulse Coding

Retina 5% 85∗ 1

Xilinx Spartan Early Layers

XC3S400 Neuromorphic 0% 186 1

Pulse Coding (24 slices)

*49 clock cycles are required to process each pixel

Also, to store all the coefficients, the total amount of required memory considering a18-bit fixed point representation can be calculated using (4.7), resulting in

Memcoef = (101 · 20 + 21 · 10 + 11) · 18 = 22.16 kb (5.10)

Assuming that each MAC requires an activation function, the implementation of thisfunctions will require a total of approximately 30 kb of memory, using 128 samples with18-bit each. Following the same procedure for the different networks topologies consideredin Section 3.3 the results represented in Table 5.4 were achieved.

Table 5.4: Neural networks implementation hardware cost for different topologies.

Network Number of Number Number of Coefficients Look-up Table

Topology inputs MAC Coefficients Memory (kb) Memory (kb)

Adaline 191 2 192 3.46 -

20-1 191 21 3861 69.50 48.38

10-5-1 191 11 1981 35.66 25.34

20-10-1 100 13 1231 22.16 29.95

Analyzing the hardware requirements presented in table 5.4 it is concluded that aprocessing system based on the neural networks can be implemented in the developedprototype. The only limitation is the available memory. Although, since the prototypeoffers one expansion connector, this drawback can be surpassed using an external memorymodule.

5.5.5 Serial communication protocol

The function of this hardware module is to create the packet structure presented inSection 4.4. Figure 5.20 represents the adopted hardware implementation.

The function of each block can be described as follows:


Data Packets

Dual Port RAM

DIAAddAWEACLKA

DOA

DIBAddBWEBCLKB

DOB

counter_5

clk Cbitrstclear Cevent

out_buffer

clk

s_outrstload

p_data

MU

X

State Machine

clk Cbitrst

Cevent

RAMAddA cnt_clearloadsel_data-

“CONST_HEAD”

clkrst

RAMAddB SUB new_event_cnt

event

‘1’

‘0’

x“0000”

Bit VectorBit

Figure 5.20: Data packing block diagram.

• Dual Port RAM, storage element of the FIFO memory introduced in Section 4.3;

• counter_5, bit counter, also generates the number of sent events;

• SUB, generates the number of events waiting to be sent;

• MUX, selects the data to be loaded into the output buffer;

• out_buffer, sends the data to the RF modulator; with a parallel load, it stores thedata corresponding to a given chunk of the packet;

• State Machine, this Mealy4 state machine generates all the control signals ne-cessary to perform the packet formatting; it also manages the read pointer of theFIFO memory; the output from the counter_5 and result of SUB are input signalsto the state machine.

This module requires 148 slices (4%) and 1 RAM block (6%), operating at maximumfrequency of 121 MHz.

It was also necessary to develop a hardware module capable of receiving and unpackingthe output data of the RF link demodulator. The implementation of this hardware deviceis shown in Figure 5.21.

The function of each block can be described as follows:

• in_buffer, shift register that stores incoming data;

• counter_5, bit counter, also generates the number of send events;

• event_data, stores the recovered spike information that will be written in the me-mory;

• num_events, stores the number of events from the current packet;

• comp1, active when a new packet is detected;

4Finite state machine where the outputs are determined by the current state and input


Data Packets

Dual Port RAM

DIAAddAWEACLKA

DOA

DIBAddBWEBCLKB

DOB‘0’

counter_5

clk Cbitrst

clearCevent

State Machine

clk

rst Cev

ent

RAMAddAnew_event

in_buffer

clk

s_inrst

clear

p_data

cnt_

clea

r

en_datanum_events

en_num

num_events

clk

rst

DQ

en

event_data

clk

rstD

Qen

Cbi

tbu

ff_cl

ear

new

_pac

k

=?

“CONST_HEAD”

/=?

Counter

CLK Q

COUNT/STOP

data_valid

Sclk

rst

Spikes

Bit Vector

Bit

comp1

comp2

Figure 5.21: Data unpacking block diagram.

• State Machine, generates all the control signals necessary for the data unpacking;it also controls the write signal and the write pointer of the Dual Port RAM; theoutput from counter_5, the num_events signal and the result of comp1 are theinput signals for this state machine;

• Dual Port RAM, implements a spike buffer,

• Counter, uses the signal data_valid from the State Machine and the output ofcomp2 to manage the read pointer of the Dual Port RAM.

This unpacking module was implemented on a Xilinx Spartan XC2S200 requiring 72 slices(3%) and 3 RAM blocks; the maximum operating frequency is 57 MHz.

5.5.6 Image display

To verify the proper functioning of the artificial retina processing module, an outputstimulus display module was included in the design. This module should generate all syn-chronization signals necessary for the standard VGA monitor. The adopted configurationuses a resolution of 640×480 and a refresh rate of 60 Hz. Figure 5.22 shows the temporaldiagram of the VGA signals and Table 5.5 the necessary timings for the signals.

The circuit of Figure 5.23 fulfills this task. As the captured image is in greyscale, sois the output. Also, since an array of 32×32 pixels if very small for visual analysis, theoutput image was enlarged four times by repeating the same pixel of the original image16 times in the output image resulting in a window of 256×256 pixels. Although thismodule is relatively simple, namely required additional RAM.

The signals generated by this circuit only control the image display. This devicerequires 37 slices (1%) and uses 2 RAM blocks; the maximum operating frequency is204 MHz.

5.6. CONCLUSIONS AND RESULTS 53

TPW Tbp TfpTdisplay

TS

H/V SYNC

R/G/Bsignals

Figure 5.22: VGA timing diagram.

Table 5.5: VGA Timings [5].

Symbol Parameter Vertical Sync Horizontal Sync

TS Period 16.7 ms 32 µs

Tdisp Display time 15.36 ms 25.6 µs

Tpw Sync pulse width 64 µs 3.84 µs

Tfp Sync front porch 320 µs 640 ns

Tbp Sync back porch 928 µs 1.92 µs

In order to validate the actual spike generation it was developed a module that recoversthe image using the spike information [2]. This recovery system was implemented by usinga low-pass filter with a very low cutoff frequency. To decrease the filter size it was usedan Infinite Impulsive Response (IIR) filter, with the following transfer function

H(s) =c1

(s + a)· c2

(s + a)=

c1 · c2

(s + a)2(5.11)

where c1 and c2 are gain coefficients. This filter can be implemented using a chain oftwo low-pass IIR filters, equation (D.7). The discrete filter chain requires coefficientswith an higher value than the second order filter. Hence, the filter chain makes theimplementation less sensible to discretization errors. The adopted design implements thefilter chain requiring 98 slices (2%), 2 RAM blocks and 2 multipliers. The maximumoperating frequency is 91 MHz.

5.6 Conclusions and results

Table 5.6 presents the implementation costs of the developed prototype. This table inclu-des the hardware costs of all the developed digital circuits. Considering the results shownin Table 5.6, this new platform achieves the proposed objectives. It implementes thecomplete system with a high operating frequency and using a small amount of hardwareresources. To achieve an independent spacial processing, only the space calculus wouldbe replicated. As the hardware resources for the classic model are only 10% of the totalslices, this system allows the implementation of the classic model considering indepen-dent spacial processing for the three RGB channels. The power requirements also followthe expected results: using the full implementation of the digital blocks on the FPGA


CounterMOD 800

CLK Q

COUNT/STOP

CounterMOD 521

CLK Q

COUNT/STOP

>=656

<752

<640

<480

>=490

<492

CLK

‘1’

‘1’

HorizontalSync Pulse

VerticalSync Pulse

Blank

Line count

Column count

Bit Vector

Bit

DIV2

Columnaddressing

section

Lineaddressing

section

Figure 5.23: VGA monitor control circuit [2].

the board requires 450 mW , value that increases to 950 mW when the digital camera isconnected. The usage of the VGA output, with a conversion rate of 12.5 MHz for theDACs, requires 1.2 W . Regarding the to the previous prototype [2] the relative powerconsumption is about 50% less (1.8 W ). Figure 5.24 presents a picture of the completeprototype system.

Figure 5.24: Complete prototype system.

In order to test the processing model on the new board, a switch was used to controlthe information shown in the display device. By using this switch, the output image canbe set to display the output of the Early Layers or the recovered image from the spikeoutput. Also, using another switch, it is possible to remove the temporal high-pass filter.Figure 5.25 shows the output images obtained from the prototype. The input image

5.6. CONCLUSIONS AND RESULTS 55

Table 5.6: Complete Artificial Retina system implemented on a Xilinx Spartan XC3S400FPGA.

Block Slice Operating RAM Blocks Multipliers

Occupation Frequency (MHz) (Total 16) (Total 16)

Image Capture 3% 178 1 0

and Resize

Register 3% 190 0 0

Configuration

Classic Model∗ 10% 85∗∗ 7 3

Serialization and 4% 121 1 0

Data Packing

Image Display 3% 91 4 2

Data 3% 57 3 0

Unpacking∗∗∗

Complete 20% 85 12 5

System

* includes Image Capture and Resize** 49 clock cycles are required to process each pixel*** synthesized on a Xilinx Spartan XC2S200 FPGA

resized is on the upper left corner and the processed image is on the lower right corner.

The output image from the Early Layers when the high-pass filter is included wasobtained by moving the camera. This resulted, as expected, in an image were onlythe moving edges are displayed. When the filter is removed, the resulting image onlypresents the shapes of the objects. In order to test the spike generation the input of theNeuromorphic Pulse Coding was set to be the input resized image. The results followedthe expected model behavior. The developed board also operates as expected and theoutput image is slightly better when compared with the previous developed board.

The developed board can also be employed as a development platform for a differentsystem, since it offers a total of 61 I/O pins in which 6 are global clock pins. Theseconnectors also provide a power source to a possible expansion module. The cameraconnector provides a 5 V supply and the generic connector provides 5 V , 3.3 V and theunregulated input voltage. The video port uses 8-bit per RGB channel, generating atrue color video format, and the video DAC has a maximum conversion rate of 20 MHz.This video port makes this board suitable for image processing, although for bigger imageframes it might be necessary to introduce a memory expansion module.


(a) Early Layers experimental results. (b) Early Layers without time filtering.

(c) Neuromorphic Pulse Coding experimental re-sults after image recovery.

Figure 5.25: Photographs of the experimental results obtained with the artificial retinaprototype. The input image (after downsizing) is displayed on the top left corner and theoutput at the bottom right corner.

Chapter 6

Conclusions

Modelling retinal ganglion cells response to a visual stimulus is the key to create an arti-ficial system for visual rehabilitation. Visual information is conveyed to the visual cortexin the form of spike trains and the detection of spikes poses a wide number of challenges.After capturing neural activity with the proper devices, the application of Spike-Sortingtechniques is a fundamental task as pre-processing of neural recorded data to any modelrepresentation. Although simple, the algorithms applied to cluster data revealed goodresults, allowing different responses in the same electrode to be split and filtered to indi-vidual cells. Aggregation of individual cells by type maintained the structure of the firedspike trains.

The networks trained achieved their primary objective by showing that, like othernervous systems, the process of seeing can be explained by a learning rule. Neural net-works performed quite well in retaining the structure of the spike trains. The topologiesthat revealed the best results used a nonlinear activation function and at least twentyhidden units to achieve the proposed results, mostly ever the number of units was su-perior and distributed among two hidden layers. Model results comparison cannot bedirectly accomplished since the output variables of the models are not comparable. TheClassic Model outputs variable is the instantaneous firing rate whereas Neural Networks’modelling predicts the individual spikes.

The hardware requirements for the Classic Model are scalable with the number ofconsidered electrodes. The architecture required to implement a neural network in FPLwas also developed. Considering the data path, the developed architecture can be usedfor different electrode matrix sizes and different neural network topologies. The SpikeMultiplexing generates a single data flow for the spike information generated by the pro-cessing module. This architecture can also be employed for the Neural Networks. Thetransmission of the spike information over a RF serial link channel was done accordinglyto a protocol previously specified in [3], even though changes were required to achieve thecorrect functioning of the system. The chosen specification was the simplest, thus ensu-ring that the unpacking module does not require much power. This is a critical, aspectas the unpacking module operates inside the human head.

This new prototype achieves the expected goals. It implements the complete systemwith a higher operating frequency and using a small amount of hardware resources, when

57

58 CHAPTER 6. CONCLUSIONS

compared to previous solutions. The results followed the expected model behavior. Thesystem allows the implementation of the classic model, considering independent spacialprocessing for the three RGB channels. The power requirements also follow the expectedresults: using the full implementation of the digital blocks on the FPGA the board requires450 mW , value that increases to 950 mW when the digital camera is connected. Thedeveloped board also operates as expected and the output image is slightly better whencompared with one obtained with the previous board. The developed board can also beemployed as a development platform for a different system. It offers I/O pins through twoconnectors, providing also a power source. The video port uses 8-bit per RGB channeland high speed DACs generating a true color video. This video port makes this boardsuitable to image processing.

6.1 Future work

Neural Networks modelling showed that the use of learning systems as predictors to modelspike trains is a hypothesis that can be much more explored. As for now, the results areencouraging. There are also other learning methods that could be used instead of neuralnetworks to perform this task, such as decision learning trees or even unsupervised learningmethods. The use of information theory could also be an approach in coding spike trains,as a measure of spike added information transmission.

There are several models of retinal responses in the literature and, as a means tocompare the different models, an almost universal error measure should be established andimplemented. Also, Neural Networks modelling only approached the temporal dimensionof the response and the spatial dimension is yet to be addressed.

Finally, using the new prototype board it is now possible to implement the completeClassic Model with independent spacial processing. The Neural Networks model wasnot completely implemented, an additional effort should made to develop the controlunit. Also, since the coefficients require a great amount of memory, implementing largerneural networks would require the development of an expansion board to provide externalmemory.

Appendix A

Clustering Algorithms

A.1 K-MEANS algorithm

The k-means algorithm [9, 39] is one of the simplest unsupervised learning algorithms thatsolve the well known clustering problem. The procedure follows a simple and easy way toclassify a given data set through a certain number of clusters (assume k clusters) fixed apriori. The main idea is to define k centroids, one for each cluster. These centroids shouldbe placed in a cunning way, because different location causes different results. The firststep is to take each point belonging to a given data set and associate it with the nearestcentroid. When no point is pending, an early groupage is done. At this point we needto re-calculate k new centroids as barycenters of the clusters resulting from the previousstep. After we have these k new centroids, a new binding has to be done between thesame data set points and the nearest new centroid. A loop has been generated and, asa result, we may notice that the k centroids change their location step by step until nomore changes are made. In other words, centroids do not move any more. Finally, thisalgorithm aims at minimizing an objective function, in this case a squared error function.The objective function (A.1) is an indicator of the distance of the n data points fromtheir respective cluster centres, where ‖xk

i − cj‖ is a chosen distance measure between adata point xk

i and the cluster centre cj.

J =K∑

j=1

n∑

i=1

‖xki − cj‖2 (A.1)

The algorithm is composed of the following steps:

1. Place K points into the space represented by the objects that are being clustered.These points represent initial group centroids.

2. Assign each object to the group that has the closest centroid.

3. When all objects have been assigned, re-calculate the positions of the K centroids.

4. Repeat Steps 2 and 3 until the centroids no longer move. This produces a separationof the objects into groups from which the metric to be minimized can be calculated.

59

60 APPENDIX A. CLUSTERING ALGORITHMS

Although it can be proved that the procedure will always terminate, the k-meansalgorithm does not necessarily find the most optimal configuration. K-means can convergeto a local optimum, which in this case is a partition of points in which moving anysingle point to a different cluster increases the total sum of distances. The algorithm isalso significantly sensitive to the initial randomly selected cluster centres. The k-meansalgorithm can be run multiple times to reduce this effect.

A.2 EM algorithm and bayesian classification

The EM algorithm is a general method of finding the maximum-likelihood estimate ofthe parameters of an underlying distribution from a given data set when the data isincomplete or has missing values [10].

Consider a data set X of size N taken from a distribution with density functionp(x|Θ) characterized by the set of parameters Θ (e.g. in the current case p is a Gaussiandistribution and Θ are the mean and covariance of p). If assumed that the observed datapoints X = x1,x2, . . . ,xN are independent and identically distributed (i.i.d.), then, theresulting density for the samples is

p(X|Θ) =N∏

i=1

p(xi|Θ) = L(Θ|X) (A.2)

where the function L(Θ|X) is called the likelihood function. The likelihood is considereda function of the parameters Θ where the data X is fixed. The maximum likelihoodproblem consists on finding the parameters Θ that maximize L, that is

Θ∗ = arg maxΘ

L(Θ|X) (A.3)

It is usual to maximize the log-likelihood function, log(L(Θ|X)), for analytical simplifi-cation.

Frequently, the probability density function (pdf) results of a mixture of M classes ofsome kind. In this case the pdf is defined by

p(x|Θ) =M∑

i=1

πipi(x|θi) (A.4)

where the parameters are Θ = (π1, . . . , πM , θ1, . . . , θp), such that each πi is the ith classpriori probability,

∑Mi=1 πi = 1 and each pi is a density function parameterized by θi.

The log-likelihood expression for this density function is given by

log(L(Θ|X)) = logN∏

i=1

p(xi|Θ) =N∑

i=1

log

M∑

j=1

πjpj(xi|θj)

(A.5)

and it is not easy to optimize since it consists of the log of the sum. However, consideringX incomplete and the existence of unobserved data items Y = y1,y2, . . . ,yM whose

A.2. EM ALGORITHM AND BAYESIAN CLASSIFICATION 61

values represent the component density that generated each data observation, the log-likelihood expression (A.5) becomes substantially simplified. This way, it is assumed thatyi = k if the ith sample was generated by the kth mixture component, the likelihoodbecomes

log(L(Θ|X,Y)) = log(P(X,Y|Θ)) =N∑

i=1

log(P (xi|yi)P (yi)) =N∑

i=1

log(πyipyi(xi|θyi)) (A.6)

and can be computed using a variety of techniques, like the EM algorithm.

The algorithm is composed of the following steps:

1. Start (t=0) with an initial arbitrary set of parameters Θ0 = (π01, θ

01), (π

02, θ

02),

. . . , (π0M , θ0

M), and initial mixture estimation p(x|Θ0);

2. Compute posteriors

P t(k|xi, θt) =

πtkp

tk(xi|θt

k)

pt(xi|Θt)(A.7)

3. Estimate the parameters Θt+1 by maximizing the likelihood function with fixedP t(k|xi, θ

t). Assuming a d -dimensional Gaussian with mean µ and covariance matrixΣ, i.e., θi = (µi, Σi) then

pi(x|µi, Σi) =1

(2π)d/2|Σi|1/2e−

12(x−µ)Σ−1

i (x−µ)T

(A.8)

and the maximization at each iteration of leads to

µt+1k =

∑Ni=1 xiP

t(k|xi, θtk)∑N

i=1 P t(k|xi, Θt)(A.9)

Σt+1k =

(xi − µtk)(xi − µt

k)T ∑N

i=1 P t(k|xi, Θt)

∑Ni=1 P t(k|xi, θt

k)(A.10)

πt+1k =

1

N

N∑

i=1

P t(k|xi, Θt) (A.11)

4. Set t = t + 1 and repeat steps 2 and 3 while the likelihood increases.

In the case of unsupervised learning, after estimating the parameters of a probabi-lity density functions mixture it is possible to compute the posteriors P (k|x), which areobtained with Bayes’ rule,

P (k|x, θk) =πkpk(x|θk)∑M

k=1 πkpk(x|θk)(A.12)

If it is possible to assume that the each mixture component matches a class, the EMalgorithm provides an estimate of the a posteriori probability of each class given anobservation, without explicitly indicating the observation classes.

The natural classification is: x ∈ Classk → k = arg maxj P (j|x)

62 APPENDIX A. CLUSTERING ALGORITHMS

Appendix B

SPiKes Classifier - User Manual

SPKC (SPiKes Classifier) is a Matlab-based spike-sorting program. It implements twoalgorithms for classifying spikes based on the principal component analysis of the spikeswaveform. The two algorithms for sorting are the k-means clustering and the expectation-maximization EM. This software was developed by Jose Germano and Ricardo Baptistafor their graduation project (IST / SIPS, INESC-ID).

Graph 1 Graph 2

Figure B.1: SPKC (SPiKes Classifier) user window.

63

64 APPENDIX B. SPIKES CLASSIFIER - USER MANUAL

Description of buttons and functions:

• File - Open Select one or more similarly configured *.nev data files for analysis.

• Electrode# Select channel to analyze. This box lists all available channels.units,classified or not.

• Get Waveforms Loads a sub-sample of the total number of available waveformevents, displays the waveforms in window 1 and displays the waveforms projectedonto their first two principal components in window 2.

• Clustering Algorithm combo box selection defines the clustering algorithm forsorting spikes.

• Classes combo box selection defines the number of classes to use in classification.

• Princ.Comp. combo box selection determines the number of principal componentsused to define each waveform in the principal component analysis. This determinesthe size of the feature set used in determining the clusters. However, only the firsttwo leading principal components are used for display in graph 2.

• Sort Classifies the sub-samples into several units. Additional options for the clus-tering process can be set in the ClusteringAlgorithm box, the Classes box and thePrinc. Comp. box.

• Remove Clears the classification on a given channel.

• Save Saves classification generated by ’Sort’ button.

Description of Graphs:

• Graph 1 displays the voltage-time waveforms of the neural events. Waveforms arecolor coded to indicate their defined units.

• Graph 2 displays the neural waveforms projected onto their principal components.

Appendix C

Neural Network Modelling SpikeTrains Simulations

65

66APPENDIX C. NEURAL NETWORK MODELLING SPIKE TRAINS SIMULATIONS

15 16 17 18 19 20 21

Time (s)

0 10 20 30 40 50 60

Time (s)

Cell

Model

Stimulus

(a) ADALINE.

0 10 20 30 40 50 60

Time (s)

CellModel 5-1

Stimulus

Model 20-1Model 10-1

15 16 17 18 19 20 21

Time (s)

(b) Modelling with one hidden layer.

0 10 20 30 40 50 60

Time (s)

Cell

Model 10-5-1

Stimulus

Model 20-10-1

Model 10-10-1

Model 5-5-1

15 16 17 18 19 20 21

Time (s)

(c) Modelling with two hidden layers.

Figure C.1: Comparison of real and modelled spike trains, neural network model withstimulus input only.

67

0 10 20 30 40 50 60

Time (s)

Cell

Model

Stimulus

15 16 17 18 19 20 21

Ti me ( s)

(a) ADALINE.

0 10 20 30 40 50 60

Time (s)

Cell

Model 5-1

Stimulus

Model 20-1

Model 10-1

15 16 17 18 19 20 21

Time (s)

(b) Modelling with one hidden layer.

15 16 17 18 19 20 21

Time (s)

0 10 20 30 40 50 60

Time (s)

Cell

Model 10-5-1

Stimulus

Model 20-10-1

Model 10-10-1

Model 5-5-1

(c) Modelling with two hidden layers.

Figure C.2: Comparison of real and modelled spike trains, neural network model withstimulus and response feedback input.

68APPENDIX C. NEURAL NETWORK MODELLING SPIKE TRAINS SIMULATIONS

Appendix D

FPL Filter Implementation

To implement the desired modules several types of filters are required. This sectionproposes a possible FPL implementation for these filters.

The DoG space filter was discretized using a square window to limit the order of theresulting Finite Impulsive Response (FIR) filter. In discrete time

m[n] =l∑

k=−l

CDoG(k) · I[n− k] (D.1)

where I represents the input, 2 · l + 1 is the size of the window and CDoG are the filtercoefficients. Applying the Z transform to (D.1) the discrete transfer function is obtained

M(z)

I(z)=

l∑

k=−l

CDoG(k) · z−k (D.2)

Since the response was truncated it is necessary to correct the filter coefficients inorder to guarantee that the static gain is zero, this results in

ak = CDoG(k)− 1

2l + 1(

l∑

k=−l

CDoG(k)− A0) (D.3)

where A0 represents the static gain and ak are the new filter coefficients.

This DoG filter was implemented using one multiplier by folding its architecture N2 =(2l + 1)2 times, therefore taking N2 cycles to compute the convolution. Figure D.1(a)represents the FPL implementation adopted for this filter where the ROM stores the filtercoefficients. Using this solution the working frequency will be N2 times smaller than theparallel solution1.

The discrete high-pass filter, which was obtained applying the bilinear transformationto (D.4), which is the laplace transform for this filter.

GHP (s) =s

s + α(D.4)

1Not implemented as it needed to much resources

69

70 APPENDIX D. FPL FILTER IMPLEMENTATION

As the bilinear transformation is given by s = 2Ts

1−z−1

1+z−1 the resulting IIR filter is:

GHP (z) = CHP1− z−1

1−BHP z−1(D.5)

where CHP and BHP are the filter coefficients. Resulting in discrete time:

r[n] = BHP · r[n− 1] + CHP · (m[n]−m[n− 1]) (D.6)

in which m represents the input and r the output.

The discrete IIR low-pass filter, which has a frequency response given by

HLP (s) =B

s + 1τ

(D.7)

was also discretized using the bilinear transformation resulting in

v[n] = BLP · v[n− 1] + CLP (u[n] + u[n− 1]) (D.8)

where u represents the input and v the output.

These two IIR filters can be implemented in FPL. In Figure D.1(b) a possible archi-tecture is presented.

Output

x

Q

ROM

++

+

Input

(a) Gaussian filter.

+

Q D

RAM

x

+

Coef.+/-

+

+

+

Output

Input

(b) IIR filter

Figure D.1: FPL implementation of the filters.

The only difference from a low-pass to a high-pass filter is the coefficient, coef, andthe feedback signal. A high-pass has coef = −BHP

CHPand a negative feedback; the low-pass

has coef = BLP

CLPand the feedback is positive.

Appendix E

Prototype Datasheet

OVERVIEW

This prototype board was developed to allow the validation of the complete processingmodule developed in the scope of the European project CORTIVIS - Cortical VisualNeuroprosthesis for the Blind. Figure E.1 presents the block diagram of this board.

VG

A

Cam

era Co

nnector

Ge

neric C

onne

ctor

RGB 8-bit DACs

XILINX SPARTAN XC3S400

24

3625

ControlSwitches

FPGA Power Regulators

5V Power Regulator

Power Jack

Artificial Retina Prototype

JTAG

1239

40

213

23

1

Figure E.1: Prototype block diagram.

The board can also be employed as development platform for a different system. Itoffers a total of 61 I/O pins in which 6 are global clock pins on two expansion connectors,these connectors also provide a power source to a possible expansion module. The videoport uses 8-bit per RGB channel generating a true color video format making this boardsuitable for image processing.

71

72 APPENDIX E. PROTOTYPE DATASHEET

FEATURES

This developed board presents the following main characteristics:

• Xilinx Spartan3 XC3S400 FPGA as processing core. This device as 400 K systemgates, a total memory of 288 kbits distributed in 16 RAM blocks and 16 dedicated18 bits multipliers;

• three high efficiency, 3 A maximum output current power regulators (1.2 V , 2.5 Vand 3.3 V ) and a 1.5 A maximum output current 5 V linear power regulator. The3.3 V and 5 V are available in the expansion connectors;

• 50 MHz HCMOS 3.3 V oscillator;

• VGA true color display port, uses three onboard low power consumption 20 MHzconversion rate video DACs;

• it offers four slide switches and one push button;

• FPGA programming through a six pin JTAG header;

• two expansion slots with a total of 61 I/O pins in which 6 are GCLK inputs. Bothslots can provide supply power;

POWER SUPPLY

This board requires power supply with an output voltage within the range of 5 V to 6 VDC. The power connector must be connected to a 2.1 mm female center-positive plugand the supply must be capable of delivering at least 1 A. The board generates a total offour supply voltages, 1.2 V , 2.5 V , 3.3 V and 5 V . To save power all the supply voltages,except the 5 V , are generated using high efficiency regulators. Some of these voltages,5 V and 3.3 V , as well as the unregulated input, are available through the expansionconnectors.

OSCILLATOR

The employed oscillator has an output frequency of 50 MHz with a 3.3 V maximumamplitude. The oscillator is connected to a GCLK FPGA pin (P181) and is placed closeto the FPGA. Oscillators for different frequencies can be employed.

VGA DISPLAY PORT

The display port is based on three separate 8-bit Video Digital-to-Analog converters,TLC5602C, one per RGB channel. The TLC5602C DAC as a low power consumption(80 mW typ), 20 MHz conversion rate, single 5 V power supply and TTL digital inputvoltage. Each of these video DACs are connected by a dedicated 8-bit wide data bus to

73

the FPGA and the clock net is the same for all the devices. The VGA synchronizationpins are connected directly to the FPGA. Table E.1 shows the FPGA pins that providethose signals and also the output clock net for the DACs.

Table E.1: VGA synchronization pins and DAC clock.

FPGA PIN Signal

139 HS

138 VS

180 DAC clock

The pin mapping between the FPGA I/O and the DACs is summarized in Table E.2.

Table E.2: FPGA pins for the video DACs.

Data Red Data Green Data Blue

FPGA PIN BUS num FPGA PIN BUS num FPGA PIN BUS num

106 0 (LSB) 116 0 (LSB) 126 0 (LSB)

107 1 117 1 128 1

108 2 119 2 130 2

109 3 120 3 131 3

111 4 122 4 132 4

113 5 123 5 133 5

114 6 124 6 135 6

115 7 (MSB) 125 7 (MSB) 137 7 (MSB)

When using this video port ensure that the clock input for the video DACs is lowerthan 20 MHz. Also, only connect the VGA plug when the board is connected to thepower supply.

SLIDE SWITCHES AND PUSH BUTTON

When in the ON position the FPGA pin is pulled to ground; when in OFF the pin willgo high. The push button will also be pulled to ground when pressed, otherwise the pinis pulled to VCC . Table E.3 presents the pin mapping for the slide swiches and the pushbutton.

FPGA CONFIGURATION

The FPGA is configured using a 6 pin JTAG header. Figure E.2 shows the function ofeach pin in the connector.


Table E.3: FPGA pins for the slide switches and the push button.

FPGA PIN Name PIN num

156 SW(0) 1

155 SW(1) 2

154 SW(2) 3

152 SW(4) 4

150 PB -

VGA Camera Connector

RG

B

25

Co

ntro

lS

witc

hes

. . . . . .VCC

GND

TCK

TDO

TDI

TMS

Figure E.2: JTAG connector pin order.

The header can be connected to a standard JTAG programming cable (Digilent JTAG3),the Vcc pin is connected to the 3.3 V supply voltage.

EXPANSION CONNECTORS

This board provides two expansion connectors, both can provide supply power to anexpansion module. The pin arrangement is not equal for the two connectors. As thecamera connector must connect a digital camera, the pin arrangement must be the sameas the camera module. Table E.4 showns the pin mapping between the FPGA I/O and thecamera connector. Table E.5 shows the FPGA pins for the generic connector expansionslot.

75

Table E.4: FPGA pins for camera connectorexpansion slot.FPGA PIN Name Connect num

189 1

187 2

191 3

190 4

196 5

194 6

198 7

197 8

200 9

199 10

204 11

203 12

205 13

183∗ 14

- GND 15

184∗ 16

- GND 17

2 18

3 19

- 5 V 20

- GND 21

- 5 V 22

4 23

5 24

7 25

9 26

10 27

11 28

12 29

13 30

- GND 31

- VTO 32

* CCLK pins

Table E.5: FPGA pins for the generic ex-pansion slot.FPGA PIN Name Connect num

48 1

46 2

52 3

51 4

58 5

57 6

62 7

61 8

64 9

63 10

67 11

65 12

71 13

68 14

74 15

72 16

77∗ 17

76∗ 18

79∗ 19

78∗ 20

81 21

80 22

85 23

83 24

87 25

86 26

92 27

90 28

94 29

93 30

96 31

95 32

100 33

97 34

102 35

101 36

- 5 V 37

- 3.3 V 38

- Vccunreg 39

- GND 40

* CCLK pins


Appendix F

Prototype Board Schematics

77

78 APPENDIX F. PROTOTYPE BOARD SCHEMATICS

11

22

33

44

DD

CC

BB

AA

Tit

le

Nu

mb

erR

evis

ion

Siz

e

A4

Dat

e:13

-09

-200

4S

heet

o

fF

ile:

D:\

Tra

bal

ho\..

\sp

arta

n3.S

chD

oc

Dra

wn

By:

IO_

L01

P_7

/VR

N_

72

IO_

L01

N_

7/V

RP

_7

3

IO_

L16

P_7

/VR

EF

_7

4

IO_

L16

N_

75

IO_

L19

P_7

7

IO_

L19

N_

7/V

RE

F_7

9

IO_

L20

P_7

10

IO_

L20

N_

711

IO_

L21

P_7

12

IO_

L21

N_

713

IO_

L22

P_7

15

IO_

L22

N_

716

IO_

L23

P_7

18

IO_

L23

N_

719

IO_

L24

P_7

20

IO_

L24

N_

721

IO_

L39

P_7

22

IO_

L39

N_

724

IO_

L40

P_7

26

IO_

L40

N_

7/V

RE

F_7

27

IO_

L40

P_6

/VR

EF

_6

28

IO_

L40

N_

629

IO_

L39

P_6

31

IO_

L39

N_

633

IO_

L24

P_6

34

IO_

L24

N_

6/V

RE

F_6

35

IO_

L23

P_6

36

IO_

L23

N_

637

IO_

L22

P_6

39

IO_

L22

N_

640

IO_

L21

P_6

42

IO_

L21

N_

643

IO_

L20

P_6

44

IO_

L20

N_

645

IO_

L19

P_6

46

IO_

L19

N_

648

IO/V

RE

F_6

50

IO_

L01

P_6

/VR

N_

651

IO_

L01

N_

6/V

RP

_6

52

IO_L01P_5/CS_B57

IO_L01N_5/RDWR_B58

IO_L10P_5/VRN_561

IO_L10N_5/VRP_562

IO63

IO_L27P_564

IO_L27N_5/VREF_565

IO_L28P_5/D767

IO_L28N_5/D668

IO71

IO_L31P_5/D572

IO_L31N_5/D474

IO_L32P_5/GCLK276

IO_L32N_5/GCLK377

IO/VREF_578

IO_L32P_4/GCLK079

IO_L32N_4/GCLK180

IO_L31P_4/DOUT/BUSY81

IO_L31N_4/INIT_B83

IO/VREF_485

IO_L30P_4/D386

IO_L30N_4/D287

IO_L27P_4/D190

IO_L27N_4/DIN/D092

IO93

IO_L25P_494

IO_L25N_495

IO/VREF_496

IO97

IO_L01P_4/VRN_4100

IO_L01N_4/VRP_4101

IO/VREF_4102IO

_L

01P

_3/V

RN

_3

106

IO_

L01

N_

3/V

RP

_3

107

IO_

L17

P_3

/VR

EF

_3

108

IO_

L17

N_

310

9IO

_L

19P

_311

1IO

_L

19N

_3

113

IO_

L20

P_3

114

IO_

L20

N_

311

5IO

_L

21P

_311

6IO

_L

21N

_3

117

IO_

L22

P_3

119

IO_

L22

N_

312

0IO

_L

23P

_3/V

RE

F_

312

2IO

_L

23N

_3

123

IO_

L24

P_3

124

IO_

L24

N_

312

5IO

_L

39P

_312

6IO

_L

39N

_3

128

IO_

L40

P_3

130

IO_

L40

N_

3/V

RE

F_3

131

IO_

L40

P_2

/VR

EF

_2

132

IO_

L40

N_

213

3IO

_L

39P

_213

5IO

_L

39N

_2

137

IO_

L24

P_2

138

IO_

L24

N_

213

9IO

_L

23P

_214

0IO

_L

23N

_2/

VR

EF

_214

1IO

_L

22P

_214

3IO

_L

22N

_2

144

IO_

L21

P_2

146

IO_

L21

N_

214

7IO

_L

20P

_214

8IO

_L

20N

_2

149

IO_

L19

P_2

150

IO_

L19

N_

215

2IO

/VR

EF

_215

4IO

_L

01P

_2/V

RN

_2

155

IO_

L01

N_

2/V

RP

_2

156

IO_L01P_1/VRN_1161

IO_L01N_1/VRP_1162

IO_L10P_1165

IO_L10N_1/VREF_1166

IO167

IO_L27P_1168

IO_L27N_1169

IO_L28P_1171

IO_L28N_1172

IO175

IO_L31P_1176

IO_L31N_1/VREF_1178

IO_L32P_1/GCLK4180

IO_L32N_1/GCLK5181

IO182

IO_L32P_0/GCLK6183

IO_L32N_0/GCLK7184

IO_L31P_0/VREF_0185

IO_L31N_0187

IO189

IO_L30P_0190

IO_L30N_0191

IO_L27P_0194

IO_L27N_0196

IO197

IO_L25P_0198

IO_L25N_0199

IO/VREF_0200

IO_L01P_0/VRN_0203

IO_L01N_0/VRP_0204

IO/VREF_0205

BA

NK

0B

AN

K 1

BANK 3BANK 2

BA

NK

4B

AN

K 5

BANK 7BANK 6

U6

AX

C3

S400

-5P

Q20

8C

Gen0Gen1Gen2Gen3Gen4Gen5Gen6Gen7Gen8Gen9Gen10Gen11Gen12Gen13Gen14Gen15Gen16

Gen17Gen18Gen19Gen20Gen21Gen22Gen23Gen24Gen25Gen26Gen27Gen28Gen29Gen30Gen31

Gen

32G

en33

Gen

34G

en35

Gen

[0..3

5]

DG

0D

G1

DG

2D

G3

DG

4D

G5

DG

6D

G7

DB

[0..

7]

DG

[0..7

]

DR

[0..

7]

Cam0Cam1

Cam2Cam3Cam4Cam5Cam6Cam7Cam8Cam9Cam10Cam11Cam12

Cam

[0..2

4]

BT

N0

BT

N1

BT

N2

BT

N3

BT

N4

BT

N[0

..4]

Gen

[0..3

5]

DB

[0..

7]

DG

[0..7

]

DR

[0..

7]

BT

N[0

..4]

CLK_dac

VG

A_H

SV

GA

_VS

CL

K_

dac

CL

K_

dac

CLK_50Mhz

CL

K_

50M

hz

CL

K_

50M

hz

Cam

17

Cam

18

Cam

19

Cam

20

Cam

21

Cam

22

Cam

23

Cam

24

Cam[0..24]

Cam

16

Cam14Cam13

Cam

15

DR

0D

R1

DR

2D

R3

DR

4D

R5

DR

6D

R7

DB

4D

B5

DB

6D

B7

DB

0D

B1

DB

2D

B3

Figure F.1: FPGA electrical diagram.

79

11

22

33

44

DD

CC

BB

AA

Tit

le

Nu

mb

erR

evis

ion

Siz

e

A4

Dat

e:13

-09

-200

4S

heet

o

fF

ile:

D:\

Tra

bal

ho\..

\sp

arta

n3_p

ower

.Sch

Doc

Dra

wn

By:

M1

54M

055

M2

56

DO

NE

103

CC

LK

104

TD

O15

8

TC

K15

9

TM

S16

0

HS

WA

P_E

N20

6

PR

OG

_B

207

TD

I20

8

VC

CIN

T70

VC

CIN

T88

VC

CIN

T17

4V

CC

INT

192

VC

CA

UX

17V

CC

AU

X38

VC

CA

UX

69V

CC

AU

X89

VC

CA

UX

121

VC

CA

UX

142

VC

CA

UX

173

VC

CA

UX

193

GN

D1

GN

D8

GN

D14

GN

D25

GN

D30

GN

D41

GN

D47

GN

D53

GN

D59

GN

D66

GN

D75

GN

D82

GN

D91

GN

D99

GN

D10

5

GN

D11

2

GN

D11

8

GN

D12

9

GN

D13

4

GN

D14

5

GN

D15

1

GN

D15

7

GN

D16

3

GN

D17

0

GN

D17

9

GN

D18

6

GN

D19

5

GN

D20

2

VC

CO

_7

6

VC

CO

_7

23

VC

CO

_6

32

VC

CO

_6

49

VC

CO

_5

60

VC

CO

_5

73

VC

CO

_4

84

VC

CO

_4

98V

CC

O_

311

0V

CC

O_

312

7

VC

CO

_2

136

VC

CO

_2

153

VC

CO

_1

164

VC

CO

_1

177

VC

CO

_0

188

VC

CO

_0

201

U6

B

XC

3S4

00-5

PQ

208C

Vcc

int

Vcc

aux Vcc

o

4.7K

R11

Res

3

123456

J5 Hea

der

6

Vcc

o

4.7K

R14

Res

3

Vcc

aux

4.7K

Rc1

3

Res

3

4.7K

R12

Res

3

0.01

uF

C28

Cap

Sem

i0.

01u

F

C29

Cap

Sem

i0.

01u

F

C30

Cap

Sem

i0.

01u

F

C31

Cap

Sem

i

Vcc

int

0.1u

F

C32

Cap

Sem

i0.

1uF

C33

Cap

Sem

i

Vcc

int

1uF

C34

Cap

Sem

i

Vcc

int

Vcc

int

0.01

uF

C36

Cap

Sem

i0.

01u

F

C37

Cap

Sem

i0.

01u

F

C38

Cap

Sem

i0.

01u

F

C39

Cap

Sem

i0.

01u

F

C40

Cap

Sem

i0.

01u

F

C41

Cap

Sem

i0.

01u

F

C42

Cap

Sem

i0.

01u

F

C43

Cap

Sem

i

0.1u

F

C44

Cap

Sem

i0.

1uF

C45

Cap

Sem

i0.

1uF

C46

Cap

Sem

i0.

1uF

C47

Cap

Sem

i1u

F

C48

Cap

Sem

i

Vcc

o

Vcc

oV

cco

Vcc

o

0.01

uF

C51

Cap

Sem

i0.

01u

F

C52

Cap

Sem

i0.

01u

F

C53

Cap

Sem

i0.

01u

F

C54

Cap

Sem

i0.

1uF

C55

Cap

Sem

i

Vcc

aux

Vcc

aux

Vcc

aux

1uF

C57

Cap

Sem

i

Vcc

aux

470u

F

C35

Cap

Po

l3

470u

F

C50

Cap

Po

l3

470u

F

C58

Cap

Po

l3

1uF

C63

Cap

Sem

i

Figure F.2: FPGA power connections and configuration.


11

22

33

44

DD

CC

BB

AA

Tit

le

Nu

mb

erR

evis

ion

Siz

e

A4

Dat

e:13

-09

-200

4S

heet

o

fF

ile:

D:\

Tra

bal

ho\..

\po

wer

.Sch

Doc

Dra

wn

By:

12 3

J1 PW

R2.

5

15pF

C1

Cap

Sem

i91

K

R1

Res

330

K

R2

Res

3

1uF

Cap

Sem

i2.

2uF

Cap

Sem

i

10R3

Res

3

2.2u

F

C4

Cap

Sem

i

470p

F

C6

Cap

Sem

i

130K

R4

Res

3

22uF

C5

Cap

Po

l31u

F

C8

Cap

Sem

i

2.7K

R5

Res

3

30K

R6

Res

3

2.2u

H

L1

Ind

ucto

r Is

ola

ted

150u

F

C7

Cap

Po

l3

Vcc

int

+5

Vcc

_un

reg

Vcc

_un

reg

Vcc

_un

reg

10R9

Res

3

2.2u

F

C15

Cap

Sem

i

470p

F

C16

Cap

Sem

i

56K

R10

Res

3

22uF

C14

Cap

Po

l31u

F

C17

Cap

Sem

i

2.2u

H

L3

Ind

ucto

r Is

ola

ted

150u

F

C18

Cap

Po

l3

Vcc

_un

reg

Vcc

in_2

5

Vcc

in_2

5

10R7

Res

3

2.2u

F

C10

Cap

Sem

i

470p

F

C11

Cap

Sem

i

39K

R8

Res

3

22uF

C9

Cap

Po

l31u

F

C12

Cap

Sem

i

2.2u

H

L2

Ind

ucto

r Is

ola

ted

150u

F

C13

Cap

Po

l3

Vcc

_un

reg

Vcc

aux

Vcc

o

OU

T4

IN2

GND3

TAB6

EN

1F

B5

U1

0

TP

S78

601

10uF

C60

Cap

Po

l368

0uF

C61

Cap

Po

l3

VC

C12

n_S

HD

N5

LX

1

LX

3

LX

14

PG

ND

13G

ND

9

IN4

IN2

CO

MP

6F

B8

LX

16

FB

SEL

11

RE

F10

TO

FF7

PG

ND

15

U3

MA

X1

831

VC

C12

n_S

HD

N5

LX

1

LX

3

LX

14

PG

ND

13G

ND

9

IN4

IN2

CO

MP

6F

B8

LX

16

FB

SEL

11

RE

F10

TO

FF7

PG

ND

15

U4

MA

X1

830

VC

C12

n_S

HD

N5

LX

1

LX

3

LX

14

PG

ND

13G

ND

9

IN4

IN2

CO

MP

6F

B8

LX

16

FB

SEL

11

RE

F10

TO

FF7

PG

ND

15

U2

MA

X1

830

100n

F

C62

Cap

Sem

i

Figure F.3: Power regulators.

81

11

22

33

44

DD

CC

BB

AA

Tit

le

Nu

mb

erR

evis

ion

Siz

e

A4

Dat

e:13

-09

-200

4S

heet

o

fF

ile:

D:\

Tra

bal

ho\..

\DA

C.S

chD

ocD

raw

n B

y:

600z

L1

2

Ind

ucto

r

600z

L1

3

Ind

ucto

r

1uF

C25

Cap

Sem

i

600z

L1

4

Ind

ucto

r

600z

L1

5

Ind

ucto

r

0.01

uF

C26

Cap

Sem

i

AG

ND

R

AG

ND

R

AG

ND

R

AV

DD

R

DG

ND

R

DV

DD

R

CO

MP

_RC

LK

_da

c

DR

0D

R1

DR

2D

R3

DR

4D

R5

DR

6D

R7

DR

[0..

7]D

R[0

..7]

CL

K_

dac

1uF

C27

Cap

Sem

iC

OM

P_R

CL

K_

dac

Ao

utR

600z

L8

Ind

ucto

r

600z

L9

Ind

ucto

r

1uF

C22

Cap

Sem

i

600z

L1

0

Ind

ucto

r

600z

L1

1

Ind

ucto

r

0.01

uF

C23

Cap

Sem

i

AG

ND

G

AG

ND

G

AG

ND

G

AV

DD

G

DG

ND

G

DV

DD

G

CO

MP

_GC

LK

_da

c

DG

0D

G1

DG

2D

G3

DG

4D

G5

DG

6D

G7

DG

[0..7

]D

G[0

..7]

1uF

C24

Cap

Sem

iC

OM

P_G

Ao

utG

600z

L4

Ind

ucto

r

600z

L5

Ind

ucto

r

1uF

C19

Cap

Sem

i

600z

L6

Ind

ucto

r

600z

L7

Ind

ucto

r

0.01

uF

C20

Cap

Sem

i

AG

ND

B

AG

ND

B

AG

ND

B

AV

DD

B

DG

ND

B

DV

DD

B

CO

MP

_BC

LK

_da

c

DB

0D

B1

DB

2D

B3

DB

4D

B5

DB

6D

B7

DB

[0..

7]D

B[0

..7]

1uF

C21

Cap

Sem

iC

OM

P_B

Ao

utB

Ao

utG

Ao

utB

Ao

utR

+5

+5

+5

+5

+5

+5

Vre

f_d

ac

Vre

f_d

ac

Vre

f_d

ac

11 12 13 14 15

1 2 3 4 56 7 8 9

0

10

J8 Con

nect

or

15

VG

A_H

S

VG

A_V

S

D5

14D

415

D3

16D

217

AN

LG

GN

D10

D1

18

CO

MP

3

D0

(LS

B)

19

D7

(MS

B)

12

CL

K11

RE

F4

AO

UT

6

DG

TL

GN

D1

NC

20

20

NC

77

DG

TL

VD

D2

9

AN

LG

VD

D2

8A

NL

GV

DD

15

DG

TL

VD

D1

2

D6

13

U9

TL

C5

602C

DW

D5

14D

415

D3

16D

217

AN

LG

GN

D10

D1

18

CO

MP

3

D0

(LS

B)

19

D7

(MS

B)

12

CL

K11

RE

F4

AO

UT

6

DG

TL

GN

D1

NC

20

20

NC

77

DG

TL

VD

D2

9

AN

LG

VD

D2

8A

NL

GV

DD

15

DG

TL

VD

D1

2

D6

13

U7

TL

C5

602C

DW

D5

14D

415

D3

16D

217

AN

LG

GN

D10

D1

18

CO

MP

3

D0

(LS

B)

19

D7

(MS

B)

12

CL

K11

RE

F4

AO

UT

6

DG

TL

GN

D1

NC

20

20

NC

77

DG

TL

VD

D2

9

AN

LG

VD

D2

8A

NL

GV

DD

15

DG

TL

VD

D1

2

D6

13

U8

TL

C5

602C

DW

VIN

1

3VO

UT

2

GN

D

U5

MA

X6

004

+5

Vre

f_d

ac

Figure F.4: Digital to analog converters.


11

22

33

44

DD

CC

BB

AA

Tit

le

Nu

mb

erR

evis

ion

Siz

e

A4

Dat

e:13

-09

-200

4S

heet

o

fF

ile:

D:\

Tra

bal

ho\..

\mai

n.Sc

hD

ocD

raw

n B

y:

DR

[0..

7]C

LK

_da

c

DG

[0..7

]D

B[0

..7]

VG

A_H

SV

GA

_VS

U_

DA

CD

AC

.Sch

Doc

U_

pow

erpo

wer

.Sch

Doc

U_

spar

tan3

_po

wer

spar

tan3

_pow

er.S

chD

oc

Cam

[0..2

4]

Gen

[0..3

5]

DB

[0..

7]D

G[0

..7]

DR

[0..

7]B

TN

[0..4

]

VG

A_H

SV

GA

_VS

CL

K_

dac

CL

K_

50M

hz

U_

spar

tan3

spar

tan3

.Sch

Do

c

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

J3 Hea

der

16X

2H

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

3334

3536

3738

3940

J4 Hea

der

20X

2H

Cam

14

Cam

15

Cam

16

Cam

17

Cam

18

Cam

19

Cam

20

Cam

21

Cam

22

Cam

23

Cam

24

Cam

0C

am1

Cam

2C

am3

Cam

4C

am5

Cam

6C

am7

Cam

8C

am9

Cam

10

Cam

11

Cam

12

Cam

13

BT

N0

BT

N1

BT

N2

BT

N3

BT

N4

Gen

33

Gen

17G

en18

Gen

19G

en20

Gen

21G

en22

Gen

23G

en24

Gen

25G

en26

Gen

27G

en28

Gen

29G

en30

Gen

31

Gen

0G

en1

Gen

2G

en3

Gen

4G

en5

Gen

6G

en7

Gen

8G

en9

Gen

10G

en11

Gen

12G

en13

Gen

14G

en15

Gen

16

Gen

34G

en35

Gen

32

BT

N[0

..4]

Gen[0..35]

Cam[0..24]

DR

[0..

7]D

G[0

..7]

DB

[0..

7]

1 2 3 4

8 7 6 5

S1

SW

DIP

-4

S2

SW

-PB

BT

N0V

cco 4.

7K

R15

Res

3

BT

N1V

cco 4.

7K

R16

Res

3

BT

N2V

cco 4.

7K

R17

Res

3

BT

N3V

cco 4.

7K

R18

Res

3

BT

N4V

cco 4.

7K

R19

Res

3

50MHz

Vcc

o

Vcc

_un

reg+5

Vcc

o

VT

O

1 2

J2 Hea

der

2

VT

O

+5

100p

F

C59

Cap

Sem

i

VC

C4

GN

D2

OU

T3

E/D

1

U1

CF

PS-7

3

Figure F.5: Main schematic.

Bibliography

[1] Brian A. Wandell. Foundations of Vision. Sinauer Associates, Inc., 1995.

[2] Pedro F. Z. Tomas. Bio-inpired processing module for the development of an artificialretina. Graduation Report, Instituto Superior Tecnico, Lisbon, 2003.

[3] CORTIVIS Project. Communication: Protocol, data formats and digital circuits.Technical report, INESC-ID, June 2003.

[4] Xilinx. Power Distribution System (PDS) Design: Using Bypass/Decoupling Capa-citors.http://www.xilinx.com/bvdocs/appnotes/xapp623.pdf.

[5] Digilent Inc. Digilab DIO2 Reference Manual.http://www.digilentinc.com/Data/Products/DIO2/DIO2-rm.pdf.

[6] Cortical Visual Neuroprosthesis for the Blind - CORTIVIS.http://cortivis.umh.es.

[7] Helga Kolb. How the retina works. American Scientist, 91, 2003.

[8] Heetderks W. J. Wheeler B. C. A comparison of techniques for classification ofmultiple neural signals. IEEE Trans. Biomed. Eng., 29:752–759, 1982.

[9] Everitt B. S. Cluster Analysis. New York: Wiley, 1993.

[10] T. Krishnan G.J. McLachlan. The EM Algorithm and Extentions. New York: Wiley,1997.

[11] Michael S. Lewicki. A review of methods for spike sorting: the detection and clas-sification of neural action potentials. Network: Comput. Neural Syst., 9:R53–R78,1998.

[12] Stefan D. Wilke, Andreas Thiel, Christian W. Eurich, Martin Greschner, MarkusBongard, Joseff Ammermuller and Helmut Schwegler. Population coding of motionpatterns in the early visual system. Journal of Comparative Physiology A, 187(7):549–558, March 2001.

[13] Markus Meister and Michael J. Berry II. The Neural Code of the Retina. Neuron,22:435–450, March 1999.

83

84 BIBLIOGRAPHY

[14] Iman H. Brivanlou, David K. Warland and Markus Meister. Mechanisms of ConcertedFiring among Retinal Ganglion Cells. Neuron, 20:527–539, March 1998.

[15] Michael J. Berry II, Iman H. Brivanlou, Thomas A. Jordan and Markus Meister.Anticipation of moving stimuli by the retina. Nature, 398:334–338, March 1999.

[16] Ben Krose, Patric van der Smagt. An Introduction to Neural Networks. 1996.

[17] E. Fiesler and R. Beale, editors. Handbook of Neural Computation. Institute ofPhysics and Oxford University Press, 1996.

[18] Fred Rieke, David K. Warland, Rob de Ruyter van Steveninck and William Bialek.Spikes: Exploring the Neural Code. Cambridge, MA: MIT Press, 1997.

[19] Michael J. Berry, David K. Warland and Markus Meister. The structure and precisionof retinal spike trains. Proc. Natl. Acad. Sci. USA, 94:5411–5416, May 1997.

[20] E. M. L. Beale. A derivation of conjugate gradients. In F. A. Lootsma, editor,Numerical methods for nonlinear optimization. Academic Press, London, 1972.

[21] M. J. D. Powell. Restart procedures for the conjugate gradient method. MathematicalProgramming, 12:241–254, 1977.

[22] C. Charalambous. Conjugate gradient algorithm for efficient training of artificialneural networks. IEEE Proceedings, 139(3):301–310, 1992.

[23] Justin Keat, Pamela Reinagel, R. Clay Reid and Markus Meister. Predicting EverySpike: A Model for the Responses of Visual Neurons. Neuron, 30:803–817, June2001.

[24] Ruben Moreno and Nestor Parga. Firing rate for a generic integrate-and-fire neuronwith exponentially correlated input. Lecture Notes in Computer Science of Springer-Verlag Heidelberg, 2415, 2002.

[25] Kwabena A. Boahen. Point-to-point connectivity between neuromorphic chips usingaddress-events. IEEE Transactions on Circuits and Systems, pages 100–117, 1999.

[26] Lazzaro J. and Wawrzynek J. A multi-sender asynchronous extension to the addressevent protocol. In Proc. of 16th Conference on Advanced Research in VLSI, pages158–169, 1995.

[27] Xilinx. Spartan-3 FPGA Family: Complete Data Sheet.http://direct.xilinx.com/bvdocs/publications/ds099.pdf.

[28] Xilinx Inc. Spartan-2 Complete Data Sheet.http://direct.xilinx.com/bvdocs/publications/ds077.pdf.

[29] MAXIM. MAX1830/1831, 3A, 1MHz, Low-Voltage, Step-Down Regulators withSynchronous Rectification and Internal Switches.http://pdfserv.maxim-ic.com/en/ds/MAX1830-MAX1831.pdf.

BIBLIOGRAPHY 85

[30] Texas Instruments. TLC5602C, VIDEO 8-BIT DIGITAL-TO-ANALOG CONVER-TERS.http://www-s.ti.com/sc/ds/tlc5602.pdf.

[31] MAXIM. MAX6004, Low-Cost, Low-Power, Low-Dropout,SOT23-3 Voltage Refe-rences.http://pdfserv.maxim-ic.com/en/ds/MAX6001-MAX6005.pdf.

[32] Texas Instruments. TPS78601, ULTRALOW-NOISE, HIGH PSRR, FAST RF 1.5A LOW-DROPOUT LINEAR REGULATORS.http://www-s.ti.com/sc/ds/tps78601.pdf.

[33] C-MAC. CFPS-73, 50 MHz, Tri-state HCMOS (3.3V) oscilator.http://www.cmac.com/mt/databook/oscillators/zarlink/72-73/CFPS-72,

[34] Quasar Electronics Ltd. C3188A - 1/3” Color Camera Module With Digital Output.http://www.electronic-kits-and-projects.com/kit-files/cameras/d-c3188a.pdf.

[35] OmniVision Technologies Inc. OV7620 - Single-Chip Cmos Color Digital Camera.http://mxhaard.free.fr/spca50x/Doc/Omnivision/OV7620.pdf.

[36] OmniVision Technologies Inc. The Serial Camera Control Bus Functional Specifica-tions.http://www.ovt.com/pdfs/ds note.pdf.

[37] Philips Semiconductors. THE I2C-BUS SPECIFICATION.http://www.semiconductors.philips.com/acrobat/literature/9398/39340011.pdf.

[38] Digilent Inc. Digilab 2 Reference Manual.http://www.digilentinc.com/Data/Products/D2/D2-rm.PDF.

[39] MacQueen, J. B. Some Methods for classification and Analysis of Multivariate Ob-servations, Proceedings of 5-th Berkeley Symposium on Mathematical Statistics andProbability. Berkeley, University of California Press, 1:281–297, 1967.

Date post:	14-Jun-2020
Category:	Documents
Upload:	others
View:	7 times
Download:	0 times

ARTIFICIAL RETINA - INESC-ID · ARTIFICIAL RETINA: Development of a Bio-Inspired Model with...

Documents