Physics of Neural Networks

Series Editors: J.D. Cowan E.Domany J.L. van Hemmen

Springer Science+Business Media, LLC

Advisory Board: l A. Hertz U . Hopfield M. Kawato TJ. Sejnowski D. Sheninglon

Physics of Neural Networks

Models of Neural Networks E. Domany, J.L. van Hemmen, K. Schulten (Eds.)

Models of Neural Networks II: Temporal Aspects of Coding and Information Processing in Biological Systems

E. Domany, IL. van Hemmen, K. Schulten (Eds.)

Models of Neural Networks III: Association, Generalization, and Representation E. Domany, IL. van Hemmen, K. Schulten (Eds.)

Models of Neural Networks IV: Early Vision and Attention IL. van Hemmen, ID. Cowan, E. Domany (Eds.)

Neural Networks: An Introduction B. Muller, J. Reinhart

J. Leo van Hemmen Jack D. Cowan Eytan Domany (Eds.)

Models of Neural Networks IV Early Vision and Attention

With 139 Figures

, Springer

Series and Volume Editors:

J. Leo van Hemmen Institut fUr Theoretische Physik Technische UniversiUiI MUnchen D·85747 Garehing bei MUnchen Gennany lvh@physik.tu-muenchen.de

Eytan Domany Depanment of Electronics Weizmann Institute of Science 76100 Rehovot

Jack D. Cowan Oepanment of Mathematics University of Chicago Chicago, IL 60637 USA cowan@synapse.uchicago.edu

Israel fedomany@weizmann.ac.il

Library of Congress Cataloging-in.Publication Data Models of neural networks IV I 1.L. van Hemmen, 1.D. Cowan, E. Domany,

editors. p. cm. - (Physics of neural networks)

Includes bibliographieal rd erenees and index.

1. Neural networks (Computer sdenee) - Mathematieal model!. 1. Cowan, lO. (1ack D.). II. Domany, E. (Eytan). m. Hemmen, I .L van (Jan Leonard). IV. Series. QA76.87.M59 2001 006.3-de20 95-14288

Prinled on acid-free paper.

o 2002 Springer Science+Business Media New YOl:K

Originally publishcd by Springer-Verlag New Yorl<, Inc in 2002

Softcover reprint of the hardcover lst edition 2002 AII righls reserwd. This work ma)" noi be translaled or copicd in wholc or in pari wilhoul Ihc wrinel! IIHmisslon of Ihe publlsher (Springer Seicnec+Bu siness Media, LLe), e.'u.'epl for brlef e:r.:eerpls In conncelion wilh rcvicws or scholarly analysis. Use in conneelion wilh any form of Information storage a nd rclrie"al, elec tronic adalJlalion, computer software, or by slmtlar or dlsslmUar mNhodology now known or hereaftcr de"cl0lled is forbiddcn . The use ofgenenl descriptive names, Irade namcs, Irademarks, ctc .. in Ihis Ilublicalion, c"cn if Ihc formcr arc not eSllccially Idenlilicd, is 1101 to be laken as a slgn Ihal such namcs, as unders lood by Ihe Tradc Marks and Mcrchandisc Marks Aci, llIay ac'-'ordin gly bc used frecl y byanyone,

Production managed by Alian Abrams; manufacturing supervised by leffrey Taub. Pho1OComposed copy prepared from the editors' I61'EX files.

9 8 7 6 5 4 321

ISBN 978-1-4419-2875-7 ISBN 978-0-387-21703-1 (eBook) SPIN 10774300

DOI 10.1007/ 978-0-387-21703-1

Preface Close this book for a moment and look around you. You scan the scene by directing your attention, and gaze, at certain specific objects. Despite the background, you discern them. The process is partially intentional and partially preattentive. How all this can be done is described in the fourth volume of Models of Neural Networks devoted to Early Vision and Atten­tion that you are holding in your hands. Early vision comprises the first stages of visual information processing. It is as such a scientific challenge whose clarification calls for a penetrating review. Here you see the result. The Heraeus Foundation (Hanau) is to be thanked for its support during the initial phase of this project.

John Hertz, who has extensive experience in both computational and ex­perimental neuroscience, provides in "Neurons, Networks, and Cognition" a theoretical introduction to neural modeling. John Van Opstal explains in "The Gaze Control System" how the eye's gaze control is performed and presents a novel theoretical description incorporating recent experimental results. We then turn to the relay stations thereafter, the lateral genicu­late nucleus (LGN) and the primary visual cortex. Their anatomy, phys­iology, functional relations, and ensuing response properties are carefully analyzed by Klaus Funke et al. in "Integrating Anatomy and Physiology of the Primary Visual Pathway: From LGN to Cortex", one of the most comprehensive reviews that is available at the moment.

How do we discern patterns? That is to say, how do we perform scene segmentation? It has been shown that this process is partially preattentive and, so to speak, done on the spot in the primary visual cortex. Reinhard Eckhorn explains the underlying "Neural Principles of Preattentive Scene Segmentation" while Esther Peterhans et al. analyze a neuronal model of "Figure-Ground Segregation and Brightness Perception at Illusory Con­tours" .

Scene segmentation can also be performed by a feedback process that is called 'attention'. A glance suffices to convince every beholder that the eye catches megabytes of data. Through attention we reduce this data flood by singling out specific objects. Ernst Niebur et al. indicate how this can be done by "Controlling the Focus of Visual Selective Attention" while Julian Eggert and Leo van Hemmen elucidate the feedback mechanism proper in "Activity-Gating Attentional Networks".

Ever tried to smash a busy buzzing fly against the wall? Then you know how good it is in avoiding you. That is to say, you realize that also insects such as flies may perform highly efficient visual-information processing. In

vi Preface

their essay "Timing and Counting Precision in the Blowfly Visual System" Rob de Ruyter van Steveninck and Bill Bialek explain how this is done in early vision and show what key role is played by spikes. Finally, Wolfgang Maass approaches "Paradigms for Computing with Spiking Neurons" from the point of view of a computer scientist who is concerned with biological information processing. Enjoy!

The Editors

Contents

Preface v

Contributors xiii

1 Neurons, Networks, and Cognition: An Introduction to Neural Modeling 1 J. A. Hertz 1.1 Introduction............... 1

1.1.1 A few neuroanatomical facts . 2 1.1.2 A few neurophysiological facts 4

1.2 Neurons . . . . . . . . . . . . . . 4 1.2.1 Hodgkin-Huxley neurons. 5 1.2.2 Integrate-and-fire neurons 10 1.2.3 Binary (Ising) neurons . . 12

1.3 Local Cortical Network Dynamics. 13 1.3.1 Mean field theory. . . . . . 14 1.3.2 Simulations with spiking neurons 18

1.4 Collective Computation: Associative Memory 22 1.4.1 Hopfield model . . . . . . . . . . . . . 22 1.4.2 Sparse-pattern model ......... 27 1.4.3 Memory with time-dependent patterns . 30

1.5 Concluding Remarks 43 1.6 Acknowledgments. 44 1. 7 References...... 44

2 The Gaze Control System 47 John van Opstal 2.1 Introduction......................... 47 2.2 The Gaze Control System in One and Two Dimensions. 48 2.3 New Aspects for Eye Rotations in 3D ... 56 2.4 Mathematics of 3D Rotational Kinematics. 59

2.4.1 Finite rotations. 59 2.4.2 Quaternions........ 61 2.4.3 Rotation vectors. . . . . . 64

2.5 Donders' Law and Listing's Law 64 2.5.1 Listing's law for head-fixed saccades 68 2.5.2 Spontaneous violations of Listing's law. 70 2.5.3 Parametrization of 3D saccades . . . . . 71

viii Contents

2.5.4 3D Models: The saccade programmer. 2.5.5 3D Models: The saccade generator

2.6 Head-free Saccadic Gaze Shifts in 3D . 2.7 Conclusion 2.8 References ............... .

3 Integrating Anatomy and Physiology of the Primary Visual

74 77 81 84 88

Pathway: From LGN to Cortex 97 K. Funke, Z. F. Kisvarday, M. Volgushev, and F. Worgotter 3.1 Introduction...................... 97

3.1.1 The primary visual pathway: An overview. 98 3.1.2 Definitions: The receptive field . . . . 99

3.2 The LGN . . . . . . . . . . . . . . . . . . . . 100 3.2.1 General view and functional anatomy 100 3.2.2 Physiological properties of the LGN 104 3.2.3 Extra-retinal control of LGN function 116

3.3 Models of the LGN . . . . . . . . . . . . . . . 123 3.3.1 Basic membrane model of an LGN cell. 123 3.3.2 Models of the hyperpolarized thalamic state . 124 3.3.3 Models of the depolarized thalamic state. . . 128

3.4 The Visual Cortex . . . . . . . . . . . . . . . . . . . 131 3.4.1 Anatomical organization of the primary visual cortex 131 3.4.2 Basic response properties of visual cortical neurons. 142 3.4.3 Mechanisms of selectivity of cortical responses:

Orientation selectivity . . . . . . . . 149 3.4.4 Representations in the visual cortex . . . . . 155

3.5 Models of the Visual Cortex . . . . . . . . . . . . . . 160 3.5.1 Models of the temporal structure of cortical

responses ............... . 3.5.2 Models of cortical cell characteristics . 3.5.3 Models of cortical maps

3.6 References ................... .

4 Neural Principles of Preattentive Scene Segmentation: Hints from Cortical Recordings, Related Models, and

161 163 167 171

Perception 183 Reinhard Eckhorn 4.1 Introduction........................... 184

4.1.1 Preattentive scene segmentation is a prerequisite for object recognition ................ 184

4.1.2 Principles of neural coding beyond the classical receptive field . . . . . . . . . . . . . . . . . . . 185

4.1.3 Coupling beyond the classical receptive fields defines association fields . . . . . . . . . . . . . . . . . . .. 185

Contents ix

4.2 Properties of Synchronized Fast Cortical Oscillations (FCOs) ......................... 186 4.2.1 Sustained activation is required for the generation

of FCOs . . . . . . . . . . . . . . . . . . . . . . 186 4.2.2 Single neurons are differently involved in FCOs 186

4.3 Coding Contour Continuity . . . . . . . . . 187 4.4 Coding Region Continuity . . . . . . . . . . . . ..... 189 4.5 Coding the Separation of Adjacent Regions . . . . : 191 4.6 Spatially Restricted Synchronization Among FCOs . 192

4.6.1 Average zero-phase correlation within a cortical area 192 4.6.2 Average zero-phase correlation among two visual

cortex areas. . . . . . . . . . . . . . . . . . . . . . . 192 4.6.3 Why declines FCO coherence with cortical distance

and what are possible consequences for coding object continuity? ............. 195

4.6.4 Scene segmentation at consecutive levels of processing . . . . . . . . . . . . . . 197

4.7 Additional Properties of FCOs . . . . . . . . 199 4.7.1 Frequency and amplitude of FCOs are

highly variable . . . . . . . . . . . . . . . . . . . 199 4.7.2 Visual stimulation influences average oscillation

frequency of FCOs . . . . . . . . . 200 4.7.3 FCOs and temporal segmentation ...... 201

4.8 Stimulus-Locked Scene Segmentation . . . . . . . . . 202 4.8.1 Suppression of FCOs by fast stimulus-locked

activations ................... 203 4.8.2 Time courses of stimulus-locked and stimulus-

induced FCO-activity . . . . . . . . . 4.9 Early Labeling of Visual Objects by FCO- or

Rate-Coherence? 4.10 Appendix ................... . 4.11 References . . . . . . . . . . . . . . . . . . . .

5 Figure-Ground Segregation and Brightness Perception

204

207 208 210

at Illusory Contours: A Neuronal Model 217 E. Peterhans, R. van der Zwan, B. Heider, and F. Heitger 5.1 Introduction. 217 5.2 Methods......... 220 5.3 Results.......... 220

5.3.1 Neurophysiology 220 5.3.2 Computational model 224

5.4 Discussion........... 235 5.4.1 Occlusion cues . . . . 238 5.4.2 Occluding contours and surfaces 239

5.5 Acknowledgment 240 5.6 References................. 240

x Contents

6 Controlling the Focus of Visual Selective Attention 247 Ernst Niebur, Laurent ltti, and Christo! Koch 6.1 Introduction..................... 247 6.2 A Computational Model of The Dorsal Pathway 248

6.2.1 Model Assumptions 248 6.2.2 General architecture 249 6.2.3 Input features. . . 250 6.2.4 The saliency map . 252

6.3 Simulation Results . . . 257 6.3.1 Synthetic stimuli 257 6.3.2 Natural images . 258

6.4 Discussion........ 260 6.4.1 Psychophysical and physiological basis of the model 260 6.4.2 Limitations of the model. . . 265 6.4.3 Relationship to other models 266 6.4.4 Predictions 268

6.5 References............... 269

7 Activity-Gating Attentional Networks 277 J. Eggert and J. L. van Hemmen 7.1 Introduction................ 277

7.1.1 Different types of attention . . . 277 7.1.2 Why attentional processing at all? 278 7.1.3 Spotlight models . . . . . . . . . 280 7.1.4 The discussion forum metaphor. . 281

7.2 Activity-Gating Networks . . . . . . . . . 283 7.2.1 Working hypotheses about the coding of information 283 7.2.2 Implementation of neuronal ensembles . . . . . . 285 7.2.3 Computational units. . . . . . . . . . . . . . . . 288 7.2.4 Working hypotheses about the network function 290 7.2.5 Network architecture: Complementary

processing streams . . . . . . 294 7.2.6 Overall network organization 296

7.3 Results................. 298 7.3.1 Biased competition. . . . . . 298 7.3.2 Contour integration by laterally

transmitted expectation . . 301 7.3.3 Different types of attention . . . 301

7.4 Discussion................. 305 7.4.1 Microarchitecture and concurrent processing streams 305 7.4.2 Biased competition . . . . . . . . . 306 7.4.3 Origin of the attentional signal . . . . . . 306 7.4.4 Saliency and the focus of attention . . . . 307 7.4.5 Predictions of the model and conclusions 307

7.5 References...................... 308

Contents xi

8 Timing and Counting Precision in the Blowfly Visual System 313 Rob de Ruyter van Steveninck and William Bialek 8.1 Introduction........................... 313 8.2 Signal, Noise and Information Transmission in a Modulated

Poisson Process . . . . . . . . . . . . . . . 316 8.2.1 Description of the Poisson process 317 8.2.2 The modulated Poisson process . . 319 8.2.3 Correlation functions and spectra. 320 8.2.4 Shot noise . . . . . . . . . . . . . . 322 8.2.5 A Poisson process filtered by a random filter 324 8.2.6 Contrast transfer function and equivalent

contrast noise . . . . . . . . . . . . . . 326 8.3 The Early Stages of Fly Vision . . . . . . . . 327

8.3.1 Anatomy of the blowfly visual system 327 8.3.2 Optics............... 330 8.3.3 Reliability and adaptation . . . . 331 8.3.4 Efficiency and adaptation of the

photoreceptor-LMC synapse. . . . . . . . 339 8.4 Coding in a Blowfly Motion Sensitive Neuron . . 346

8.4.1 Retinal limitations to motion estimation. 348 8.4.2 Taking the fly outside: Counting and timing precision

in response to natural stimuli 355 8.5 Discussion and Conclusions 362 8.6 References............... 365

9 Paradigms for Computing with Spiking Neurons 373 Wolfgang Maass 9.1 Introduction........................... 373 9.2 A Formal Computational Model for a Network of

Spiking Neurons .................. 373 9.3 McCulloch-Pitts Neurons versus Spiking Neurons 375 9.4 Computing with Temporal Patterns .... 378

9.4.1 Conincidence detection .......... 378 9.4.2 RBF-Units in the temporal domain. . . . 380 9.4.3 Computing a weighted sum in temporal coding 381 9.4.4 Universal approximation of continuous functions

with spiking neurons in the temporal domain 383 9.4.5 Other computations with temporal patterns in

networks of spiking neurons . 385 9.5 Computing with a Space-Rate Code . . . . . . . . . . 386

9.5.1 Multilayer computations. . . . . . . . . . . . . 388 9.6 Analog Computation on Time Series in a Space-Rate Code 389 9.7 Computing with Firing Rates . . . . . . 390 9.8 Firing Rates and Temporal Correlations . . . . . . . . . . . 392

xii Contents

9.9 Networks of Spiking Neurons for Storing and Retrieving Information . . .

9.10 Computing on Spike Trains 9.11 Conclusions 9.12 References . . . . . . . . . .

Index

396 397 397 398

403

Contributors REINHARD ECKHORN, Biophysik Department, Philipps Universitat, Rent­hof 7, D-35032 Marburg, Germany

JULIAN EGGERT, Future Technology Research, HONDA R&D Europe, Carl Legien Strasse 30, D-63073 Offenbach/Main, Germany

KLAUS FUNKE, ZOLTAN F. KISVARDAY, MAXIM VOLGUSHEV, Abt. fUr Neurophysiologie, Medizinische Fakultat, Ruhr-Universitat Bochum, Uni­versitatsstrasse 150, D-44801 Bochum, Germany

J. LEO VAN HEMMEN, Physik Department, TU Munchen, D-85747 Gar­ching bei Munchen, Germany

CHRISTOF KOCH, Division of Biology, California Institute of Technology, Pasadena, CA 91125, USA

ERNST NIEBUR, Krieger Mind/Brain Institute, Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD 21218, USA

JOHN HERZ, NORDITA, Blegdamsvej 17, DK-2100 Copenhagen. Denmark

LAURENT ITTI, Computer Science Department, University of Southern Cal­ifornia, Hedco Neuroscience Building, Los Angeles, CA 90089-2520, USA

WOLFGANG MAASS, Institut fur Informationsverarbeitung, TU Graz, KIa­sterwiesgasse 32/II, A-80lO Graz, Austria

JOHN VAN OPSTAL, Medical Physics and Biophysics, University of Nij­megen, P.O. Box 9101, NL-6500 HB Nijmegen, Netherlands

ESTHER PETERHANS, RICK VAN DER ZWAN, BARBARA HEIDER, FRIED­RICH HEITGER, Neurology Department, Zurich University Hospital, CH-8091 Zurich, Switzerland

ROB DE RUYTER VAN STEVENINCK, WILLIAM BIALEK, NEC Research Institute, 4 Independence Way, Princeton, NJ 08540, USA

FLORENTIN WORGOTTER, Department of Psychology, University of Stir­ling, Stirling, FK9 4LA Scotland/UK