11/4/18
1
NeuralProcessingintheVentralVisualPathway
CS332VisualInformationProcessing
Earlyvisualprocessing
neuralprocessingenhancesintensitychangesintheimageprojectedontotheretina
V1
Hubel&Wieselfound3cellclassesinV1:
simplecomplexhypercomplex
retinalganglioncells
11/4/18
2
V1simple&complexcells
tuningcurve
simplecellsrespondbesttoedgesorbarsofaparticularposition,orientation,andsignofcontrast
complexcellshavelargerreceptivefieldsandare moretoleranttoposition
complexcellmay“pool”inputsfrommanysimplecellswithinreceptivefield
Kreiman,2013
SelectivityforstereoboundariesinV2
1-4
VonderHeydt&colleagues:
SomeV2cellsareselectivefortheorientation,contrast,andsideofborderownershipofanedge...foredgesdefinedbyluminanceorstereodisparity
Later,inareaV4,neuralresponsestostereodisparityappeartocorrespondmorecloselytoperceiveddepth
“anti-correlated”stereogram
11/4/18
3
V2andV4responsestocomplexshapes
Hegde &VanEssen,2007
Ventralvisualpathway
Progressingtohigherareasalongtheventralpathway:
� responselatencyincreases� receptivefieldsizeincreases� neuronsbecomeselectivetomorecomplexspatialpatterns� neuralresponsesbecomemoreinvarianttochangesinposition,scale,pose,etc.
11/4/18
4
FaceselectivecellsinITcortex
LocationsoffaceselectivecellsinIT,fromsinglecellrecordings
Desimoneetal.,1984 Tsao &Livingstone,2008
MRI vs. fMRI
manyimages(~every2secfor5mins)
lowspatialresolution(~1mm)
� increasedneuralactivityà increasedlocalbloodflow
à changeinoxygenationofhemoglobinà increaseinMRIsignal
� BloodOxygenationLevelDependent(BOLD)signalisanindirectmeasureofneuralactivity
� rawdata:~30,0003D“voxels”(eachvoxel:hundredsofthousandsofneurons)
functionalMagneticResonanceImaging(fMRI)
� bestspatialresolutionavailableformeasuringneuralactivitynoninvasivelyinthewholehumanbrain ...
11/4/18
5
9
~2s
functionalimages
time
~5min
fMRIexperiment
activationmap
FusiformFaceArea(FFA)inhumanbrain
11/4/18
6
FacepatchesinmacaqueITcortex
Tsao,Freiwald,Tootell,Livingstone,2006
Targetingneuronsinmiddlefacepatchusingsinglecellrecording
Tsao etal.2006
11/4/18
7
combinedmicrostimulation &fMRItomeasureconnectivityoffacepatches
ALAM
Thefacepatchnetwork
usedsinglecellrecordingtoprobeviewpointdependenceofneuralresponses
Otherobservations…• intactfacesyieldlargerneuralresponsesthanscrambledorinvertedfaces
• compositefaceeffect:greaterresponseforalignedvs.misalignedfaces
• ITneurons:responsetowholeface=sumofresponsestoparts• somefaceareasshowlargeincreaseinneuralresponseswhennaturalfacemovementsareadded,e.g.facialexpressions
humanfMRIstudies
dorsalpathwayventralpathway
Bernstein&Yovel,2015
11/4/18
8
Rapidobjectdetection/categorization
• 1,200images,halfcontainanimalsandhalfare“distractors”• respondasquicklyaspossible:doestheimagecontainananimalornot?• humansubjectswere~80%correct
Serre etal.(2007)
Ittakesabout100ms forvisualsignalsfromtheeyetoreachthefirstcorticalareasengagedinobject/facerecognition
Thorpe&Fabre-Thorpe(2001)
…
simplecells:detectedgesatdifferentpositions,orientations,scales
complexcells:moreinvarianttofeaturepositionandsize
HMAXmodelofrecognition
modelofprocessinginventralstreamforrapidobjectcategorization(100-150ms)
earlystagesare“hard-wired”
11/4/18
9
HMAXmodelofrecognition,cont’d
S2units:combinationsofC1unitsatdifferentorientationswithweightsthatarelearned
C2units:sameselectivityasS2unitswithmoretolerancetoposition&size(poolS2unitsofsameselectivitybutdifferentpositionsandsizes)
C2b,C3:poolinputswithMAX
S2b,S3,S4:combinemorecomplexfeatureswithweightsthatarelearned
HMAXmodelofrecognition,cont’d
• wiringandweightsbetweenCandSunitsatearlylevelsarealsolearned
e.g.C1à S2,S2bC2à S3
• unsupervisedlearningoffeaturecombinationsthatappearmostofteninnaturalimages
• learningofwiringandweightsfortop-levelobjectclassificationbysupervisedlearning
• goodmatchtoneuralresponses,V1à IT
• “neuraltuningsize”(numberofC1inputstoeachS2unit)accountsforholisticeffects(composite-face,face-inversion,whole-part)
• goodperformanceonnaturalimages
