fMRI Techniques to Investigate Neural Coding:

fMRA and MVPA

http://www.fmri4newbies.com/

Last Update: January 18, 2012Last Course: Psychology 9223, W2010, University of Western OntarioLast Update: January 18, 2012

Last Course: Psychology 9223, W2010, University of Western Ontario

Jody CulhamBrain and Mind Institute

Department of PsychologyUniversity of Western Ontario

Limitations of Subtraction Logic• Example: We know that neurons in the brain can be tuned

for individual faces

“Jennifer Aniston” neuron in human medial temporal lobeQuiroga et al., 2005, Nature

Limitations of Subtraction LogicF

irin

g R

ate

Fir

ing

Rat

e

Fir

ing

Rat

e

Act

iva

tion

Neuron 1“likes”

Jennifer Aniston

Neuron 2“likes”

Julia Roberts

Neuron 3“likes”

Brad Pitt Even though there are neurons tuned to each object, the population as a whole shows no preference

• fMRI resolution is typically around 3 x 3 x 6 mm so each sample comes from millions of neurons. Let’s consider just three neurons.

Two Techniques with “Subvoxel Resolution”

• “subvoxel resolution” = the ability to investigate coding in neuronal populations smaller than the voxel size being sampled

1. fMR Adaptation (or repetition suppression or priming)

2. Multivoxel Pattern Analysis (or decoding)

fMR Adaptation(or repetition suppression or priming…)

fMR Adaptation

• If you show a stimulus twice in a row, you get a reduced response the second time

Repeated

FaceTrial

Unrepeated

FaceTrial

Time

Hypothetical Activity inFace-Selective Area (e.g., FFA)

Act

ivat

ion

500-1000 msec

fMRI Adaptation

Slide modified from Russell Epstein

“different” trial:

“same” trial:

Block vs. Event-Related fMRA

Why is adaptation useful?

• Now we can ask what it takes for stimulus to be considered the “same” in an area

• For example, do face-selective areas care about viewpoint?

TimeA

ctiv

atio

n

Repeated Individual, Different Viewpoint

Viewpoint invariance:• area codes the face as the same despite the viewpoint change

Viewpoint selectivity:• area codes the face as different when viewpoint changes

LO pFs (~=FFA)

Grill-Spector et al., 1999, Neuron

Actual Results

And more…

• We could use this technique to determine the selectivity of face-selective areas to many other dimensions

Repeated Individual, Different

Expression

Repeated Expression,

Different Individual

Models of fMR Adaptation

Grill-Spector, Henson & Martin, 2006, TICS

Evidence for “Fatigue” Model

Data from: Li et al., 1993, J NeurophysiolFigure from: Grill-Spector, Henson & Martin, 2006, TICS

Evidence for Facilitation Model

James et al., 2000, Current Biology

Caveats in InterpretingfMR Adaptation Results

http://fmri4newbies.com/

fMRA Does Not Accurately Reflect Tuning

• MT+: most neurons are direction-selective (DS), high DS in fMRA

• V4: few (20%?) neurons are DS, very high DS in fMRA

• perhaps fMRA is more driven by inputs than outputs?

Tolias et al., 2001, J. Neurosci

http://fmri4newbies.com/

Basic Assumption/Hypothesis

• if a neuronal population responds equally to two stimuli, those stimuli should yield cross-adaptation

Ne

ura

l Re

spo

nse

Pre

dict

ed

fMR

I Re

spo

nse

A B C A-A B-B A-B C-A

Experimental Question

• the human lateral occipital complex (LOC) is arguably analogous/homologous to macaque inferotemporal (IT) cortex

• both human LOC and macaque IT show fMRI adaptation to repeated objects

• Does neurophysiology in macaque IT show object adaptation at the single neuron level?

Experiment 1Block Design Adaptation

Experiment 2Event-Related

Adaptation

Design

Sawamura et al., 2006, Neuron

Yes, neurons do adapt

Sawamura et al., 2006, Neuron

… but cross-adaptation is less clear

BLOCK

EVENT-RELATED

EXAMPLE A-A ADAPTA=B

B-A ADAPTA=B

WHOLEPOPULATION

A-AB-BC-AB-A

Sawamura et al., 2006, Neuron

Sawamura et al. Conclusions

• Evidence for adaptation at the single neuron level is clear

• Cross-adaptation is not as strong as expected, particularly for event-related designs

• They don’t think it’s just attention• Something special about repeated stimuli

Sept. 2008

Design

REP BLOCK (75% rep trials, 25% alt trials)AA BB CD EE FF GH II JJ…

ALT BLOCK (25% rep trials, 75% alt trials)AB CC DE FG HI JK LM NN…

Task: press

button for inverted

face

Summerfield et al., 2008, Nat Neurosci

Results

IndividualFFAROIs

SIG INTERACTION:stronger fMRA in blocks with freq.

reps

22%p<.001

9%p<.05

Summerfield et al., 2008, Nat Neurosci

Replication

• results were replicated with a different task

Task: press

button for small face

Summerfield et al., 2008, Nat Neurosci

New Explanation of fMRA

• “repetition suppression reflects a reduction in perceptual ‘prediction error’”

• mismatch between expectations and stimulus increases fMRI activation

• mismatch is higher on novel trials than repetition trials

Additional Caveats• Adaptation effects can be quite unreliable

– variability between labs and studies– even effects that are well-established in neurophysiology and

psychophysics don’t always replicate in fMRA• e.g., orientation selectivity in primary visual cortex

– David Heeger suggests that it may be critical to control attention

• The effect may also depend on other factors– e.g., time elapsed from first and second presentation

• days, hours, minutes, seconds, milliseconds?• number of intervening items

– attention (especially in block designs)– memory encoding

• Different areas may demonstrate fMRA for different reasons– reflected in variety of terms: repetition suppression, priming

So is fMRA dead? No.Criticism: fMRA may reflect inputs rather than outputs• Response: This is a general caveat of all fMRI studies.

Inputs are interesting too, just harder to interpret. Focus on outputs oversimplifies neural processing when presumably feedback loops are an essential component.

Criticism: fMRA may not reveal cross-adaptation even in populations that do show cross-coding

• Response: This suggests that caution is especially warranted when there is a failure to find adaptation (or a finding of “recovery from adaptation”). However, cross-adaptation can occur and is meaningful when it does. Many past fMRA studies have found it.

So is fMRA dead? No.Criticism: None of the basic models of fMRA seem to work.• Response: In some ways, it doesn’t matter. The essential

use of fMRA is to determine whether neural populations are sensitive to stimulus dimensions. The exact mechanism for such sensitivity may not be critical.

Criticism: fMRA, and maybe fMRI in general, is just responding to predictions.

• Response: Prediction is interesting too. Regarding fMRA, why do some brain areas make predictions about a stimulus while others don’t?

Multivoxel Pattern Analyses(or decoding or “mind reading”)

3 mm

3 mm

Voxels

• Modern scanner can collect ~150,000 voxels in 2 s

3 mm

lowactivity

highactivity

“Movement 1” or

“Movement 2” “Beep”

Next trialPreview Plan Execute ITI

Light

Difficulty with Standard fMRI analysisB

rain

Act

ivat

ion

(% B

SC

)

Time (seconds)

RLMovement 1Movement 2

RL

3 mm

3 mm

Voxel Pattern Information

Movement 1 Movement 23 mm

Standard Analysis

trial 1

trial 3

trial 2

trial 1

trial 2

trial 3

Movement 1 Movement 2

AverageSummedActivation

VoxelwiseActivityin ROI

Spatial Smoothing

• most conventional fMRI studies spatially smooth (blur) the data– increases signal-to-noise– facilitates intersubject averaging

• loses information about the patterns across voxels

No smoothing 4 mm FWHM 7 mm FWHM 10 mm FWHM

Effect of Spatial Smoothingand Intersubject Averaging

3 mm

3 mm

3 mm

Perhaps voxels contain useful information

• In traditional fMRI analyses, we average across the voxels within an area, but these voxels may contain valuable information

• In traditional fMRI analyses, we assume that an area encodes a stimulus if it responds more, but perhaps encoding depends on pattern of high and low activation instead

• But perhaps there is information in the pattern of activation across voxels

Multi-voxel pattern analysis (MVPA)

TrainingTrials

TestTrials

(not in training set)

trial 1

Can an algorithm correctly “guess” trial identity better than

chance (50%)?

trial 3

trial 2

trial 1

trial 2

trial 3

Movement 1 Movement 2

Coding in Voxel Patterns

• Simple experiment: Show subjects pictures of different objects (e.g., shoes vs. bottles) on different trials of different runs

Simple Correlation Analysis

• Measure within-category correlations– within bottles (B1:B2)– within shoes (S1:S2)

• Measure between-category correlations– between bottles: shoes (B1: S2; S1: B2)

• If within-category correlations > between-category correlations, conclude that area encodes different stimuli

First Demonstration

Haxby et al., 2001, Science

Haxby et al., 2001, Science

Haxby et al., 2001, Science

Decoding Algorithms

• Train algorithm to distinguish two object categories on a training set

• Test success of algorithm on distinguishing two object categories on a test set

• If algorithm succeeds better than chance, conclude that area encodes different stimuli

Norman et al., 2006, Trends Cogn. Sci.

MVPA Methods

• block or event-related data• resolution

– works even with moderate resolution (e.g., 3 mm isovoxel)

– tradeoff between resolution and coverage, SNR– 2 mm isovoxel recommended at 3 T– preprocessing

• usually steps apply (slice scan time correction, motion correction, low pass temporal filter)– EXCEPT: No spatial smoothing!

• Model single subjects, not combined group data (at least initially)

MVPA Methods

1. separate data into independent training and test sets– e.g., even and odd runs

e.g., iterate sequence of “leave one run out”

2. pick the area to analyze– ROI localizer– contrast in training set

3. train the classifier– input: beta weights from each voxel in area– variety of classifiers available– e.g., linear support vector machine

4. test the classifier– does classifier perform better than chance?

• e.g., chi-squared test

Summarized from Mur et al., 2009, Social Cognitive and Affective Neuroscience

simple 2D example

Classifier can act on single voxels. Conventional fMRI analysis would detect the difference.

Classifier would require curved decision boundary

Classifier can not act on single voxels because distributions overlapClassifier can act on combination of voxels using a linear decision boundary

Each dot is one measurement (trial) from one trial type (red circles) or the other (blue squares)

decision boundary

White and black circles show examples of correct and erroneous classification in the test set

9 voxels 9 dimensions Haynes & Rees, 2006, Nat Rev Neurosci

How can MVPA see patterns < 1 voxel?

Data from: Kamitami & Tong, 2005, Nat NeurosciFigure from: Norman et al., 2006, TICS

MVPA Searchlight

• define a spherical searchlight– optimal searchlight has radius = 4 mm

• contains 33 2-mm-isovoxel voxels

• compute multivariate effect within all possible locations within brain volume

• calculate voxelwise p values and threshold them at false discovery rate q values

Kriegeskorte, Goebel & Bandettini, 2006, PNAS

MVPA Searchlight

Kriegeskorte, Goebel & Bandettini, 2006, PNAS

MVPA Searchlight

Kriegeskorte, Goebel & Bandettini, 2006, PNAS

Does MVPA (decoding) make fMRA obsolete?

• MVPA allows us to address similar questions about what is coded in an area.

• MVPA may have some advantages (e.g., less susceptible to attentional confounds)

• MVPA utility depends on numerous factors (e.g., region size… are there enough voxels to get a meaningful pattern)

• MVPA requires clustering of neural populations and is sensitive to scanning parameters (voxel size); fMRA does not

• MVPA has the same problem as fMRA: it’s very hard to draw conclusions from a null result

Activation vs. Patterns

Mur et al., 2009, Social Cognitive and Affective Neuroscience