Fast Readout of Object Identity from Macaque Inferior Tempora Cortex

Chou P. Hung, Gabriel Kreiman,Tomaso Poggio, James J.DiCarlo

McGovern Institute for Brain Research,Brain and Cognitive Sciences, MIT

Object Recognition is difficult:trade-off between selectivity and invariance Selectivity

Many different images can correspond to the same type of object

Invariance Similar activation patterns can correspond

to different objects

The end station of the ventral stream in visual cortex is IT

Can we readout what the monkey is seeing?

Single electrode recordings Anterior inferior temporal cortex: highest visual area

in the ventral “what” pathway Spiking activity in AIT shows selectivity for complex

shapes

Can we “read-out” the subject’s object percept from IT? number of sites for reliable, real-time

performance temporal properties (onset + integration scale)

of object information neural code for different tasks invariance to object position, size, pose,

illumination, clutter recognition: ‘classification’ vs. ‘identification’? spatial scale of object information (single unit, multi-

unit, LFP) stability of these neuronal codes? improvement with experience? …

77 objects, 8 classes

Recording at each recording site during passive viewing

77 visual objects 10 presentation repetitions per object presentation order randomized and counter-

balanced

One-versus-all classification g classes (g=8): G1, …, Gg (toys, monkey

faces, vehicles, etc.) For each class i, build a binary classifier fi

(toys vs. rest, monkey faces vs. rest, etc.) sj labeled examples (j=1,…,n), For each example j, compute the output of

each classifier (e.g. pi=sj． fi ) Take prediction that maximizes pi

One-versus-all is not worse than other methods (Rifkin et al, 2003)

ij Gs

Comparison of different statistical classifiers

Decoding the population response

Categorization 8 groups

Pattern of mistakes made by the classifier

Very rapid read-out of object information

Categorization and Identification

IT representation is invariant to changes in position and size

IT representation is invariant to changes in position and size

IT representation is invariant to changes in position and size

Neural code in IT: time resolution

Neural code in IT: latency and integration time

Reading out another type of object info: scale and location

How are different kinds of information coded?

Reading out another type of object info: stimulus onset

Specific wiring significantly improves classifier performance

Extrapolation to novel pictures within the same categories

Strong overlap between the best neurons for categorization and identification

The SNR for categorization and identification are positively correlated

Invariance to scale and position