Representational Similarity Analyses:A Practical Guide for Functional MRI
ApplicationsHalle R. Dimsdale-Zucker*,x, Charan Ranganath*,x
*Center for Neuroscience, University of California, Davis, CA, United States; xDepartment of Psychology, University of
California, Davis, CA, United States
1. INTRODUCTION
1.1 Research Perspective
Over the past 50 years, we have seen massive progressin our understanding of the neurobiology of memory.Studies in animal models have revealed detailed relation-ships between neural coding by single cells in thehippocampus that are related to learning and behavior.Neuroimagingdparticularly functional magnetic reso-nance imaging (fMRI)dhas enabled researchers to relateglobal patterns of brain activity to cognitive processesthat support memory formation and retrieval. Untilrelatively recently, it was difficult to bridge the gapbetween these two approaches because of fundamentaldifferences in the kinds of questions asked in neuroimag-ing and single-unit neurophysiology.
In a canonical fMRI study of memory, researchersexamine whether the magnitude of brain activity duringmemory encoding or retrieval differs according towhether an item is subsequently remembered or missed(Diana et al., 2007). The relevant question posed by thesetypes of studies is whether “activation” [i.e., bloodoxygenation level dependent (BOLD) signal magnitude]is higher when a memory process is successful thanwhen it is unsuccessful. In single-unit recording,researchers also focus on the magnitude of neuralactivity (i.e., spike rates), but, unlike fMRI, the typicalexperimental question concerns the selectivity ofneurons, rather than the overall amount of activity.
Recent technical advances in neurophysiology haveimproved the ability of researchers to simultaneouslyrecord from large numbers of neurons. With thisapproach, neurophysiologists have been able to under-stand to examine what can be decoded from populationsof neurons, as opposed to single units (e.g., Leutgebet al., 2005). In parallel with the rise of population-based analysis approaches in neurophysiology, multi-variate analysis approaches have fundamentallychanged the kinds of questions posed in fMRI studiesof memory. In a typical fMRI data set, activity inany given brain region, such as the hippocampus,will be imaged across a reasonably large number ofvoxels. Rather than focusing on the mean level of activ-ity across a population of voxels as is done in traditionalapproaches, multivoxel pattern analysis (MVPA)approaches focus on an examination of patterns ofactivity across voxels within the population (Haxbyet al., 2001; Norman et al., 2006). MVPA has allowedneuroimaging researchers to focus on questions thatmore strongly parallel the kinds of questions thathave been addressed at the neuronal levels in rodents,thereby “connecting the branches of systems neurosci-ence” (Kriegeskorte et al., 2008a).
In this chapter, we will focus on a specific form ofMVPA known as representational similarity analysis(RSA) (Kriegeskorte et al., 2008a). RSA is a techniquethat is gaining ground as one of the primary dataanalysis approaches in cognitive neuroscience. Here,
509Handbook of In Vivo Neural Plasticity Techniques
we will consider how RSA can be used to revealfundamental insights into how memories are repre-sented in the human brain, go over experimental designfor RSA, and cover the pragmatic aspects of how toconduct RSA and how to avoid common analysis andinterpretational pitfalls.
1.2 What Is Representational SimilarityAnalysis, and How Does It Relate toOther Analysis Approaches?
In RSA, the dependent measure is the degree to whichvoxel patterns are similar across different experimentalconditions. Typically, researchers focus on pattern simi-larity (PS), although some researchers examine theinverse (i.e., dissimilarity) to parallel approaches usedin multidimensional scaling metrics (Kriegeskorteet al., 2008a,b). Thus, one would perform RSA andreport mean PS along the axis of the resultantsummary graph. Although the focus of this chapter ison the application of RSA to human fMRI data, one ofthe major strengths of RSA is that it is not limited totesting hypotheses from a single data type or species(Kriegeskorte et al., 2008b).
The basics of RSA are rooted in population vectoranalysis, which has long been a tradition in single-unitrecordings (Georgopoulos et al., 1986). The basic ideais that, rather than looking at the mean level of activityas measured with fMRI or with direct neural recordings,one can look at the distributed pattern of activity that isencoded across voxels or neurons. RSA is now begin-ning to also be applied to electrophysiological (EEG)recordings in humans, (e.g., Kaneshiro et al., 2015),but that is beyond the scope of the present chapter.Later, we will delve into the specifics of how thesecomputations are performed for fMRI data.
When evaluating whether or not to use RSA, itis important to understand what analytic power itprovides over analyses of overall signal magnitude orMVPA approaches that relate activity patterns to cate-gorical outcomes (e.g., prediction of whether one isrecalling an object or a face). Kriegeskorte et al. (2008a)operationalize these differences as addressing whetherthe neural response is directly related to a propertyof the stimulus (first-order isomorphism; more typicallythe goal of classifier-based multivariate analyses) orwhether it is related to understanding the link betweenthe stimulus and its representation (second-orderisomorphism; the goal of RSA).
Traditionally, fMRI data have been examined withunivariate analysesdso called because the analysesfocus on BOLD signal magnitude (i.e., “activation”) asthe single dependent measure. The relevant questionposed by univariate fMRI analysis is: “Does activationmagnitude in [Brain area X] differ between [Experimental
Condition A] and [Experimental Condition B].” In a massunivariate analysis, the researcher simultaneously runsunivariate analyses across every voxel in the brain. Forinstance, a researcher could test whether most of the vox-els in a given brain area show increased activation duringsuccessful recall of a studied word as compared with un-successful retrieval attempts. This is the expected patternif this region supports cognitive processes related toepisodic memory; in this example, a likely candidate toobserve this pattern would be the hippocampus (Dianaet al., 2007). A brain area might show changes in overallactivity due to processes that are correlated with success-ful retrieval (e.g., increased engagement of cognitive con-trol or increased efficiency in response selection). Thus,an increase in the amount of activity during recollectionof any event does not tell you anything about whethera brain area represents anything about specific eventsthat are recollected. The latter question is what memoryresearchers are typically most concernedwith, andwhereMVPA approaches such as RSA can be helpful.
Pattern classification approaches begin to solve thisproblem of utilizing voxel-by-voxel variability to under-stand cognition. Pattern classification relies on the ideathat there are systematic differences in the variabilityof voxels that differ between conditions of interest. Forexample, if individuals had studied either objects orfaces, the idea is that some proportion of voxels arepreferentially engaged when viewing a house, whereasothers are engaged when viewing a face. Intriguingly,these voxels may even be in the same region (Rhodeset al., 2004). This systematic difference in which voxelsare engaged, or the magnitude of their engagement,can be used by a classifier to predict whether on anygiven trial an individual was viewing a house or anobject (Haxby et al., 2001).
MVPA, which is a type of pattern classification, hasbeen extended beyond simple classification of objectcategories to answer questions about topics as diverseas binocular rivalry (Haynes and Rees, 2005) to lie detec-tion (Davatzikos et al., 2005). Different classificationschemes can be used, including linear classifiers, neuralnetworks, linear support vector machines, and GaussianNaıve Bayes classifiers (for a review and suggestions ofwhich method to use, see Norman et al., 2006). Regard-less of these implementation details, a huge advance-ment of MVPA over univariate analyses is the abilityto leverage the variability within a region to understandthe coding scheme of the brain. MVPA is most powerfulwhen looking for the presence or absence of a givencategory [e.g., object/face (Haxby et al., 2001)] or cogni-tive state [e.g., the category of information that is aboutto be remembered (Polyn et al., 2005)]. Where it fallsshort is in flexibly thinking about shared and dissimilarfeatures between items across a high-dimensional space(e.g., classification is not optimal for addressing whether
27. REPRESENTATIONAL SIMILARITY ANALYSES: A PRACTICAL GUIDE FOR FUNCTIONAL MRI APPLICATIONS510
a brain region maps a continuous dimension of experi-ence, such as relationships between events in spaceand time).
RSA is optimal for looking at a high-dimensionalrepresentational space where items can be related toone another in various ways, and it is particularly wellsuited for continuous, rather than discrete, relationshipsbetween items (Kriegeskorte et al., 2008a). For instance,in one of the first applications of RSA, Kriegeskorteshowed that it could be used to help understand andtest different models of visual representations acrossspecies (Kriegeskorte et al., 2008b). Because the goal ofRSA is to test whether or not the observed (neural)representation matches an idealized or theoreticalrepresentation, one can essentially test any reasonablehypothesis with the data and can even be used in anexploratory manner to help understand how the dataare structured. In the following sections, we will discusspractical issues for designing and analyzing with RSAand will talk about these different model comparisonapproaches in depth.
2. HOW TO DO REPRESENTATIONALSIMILARITY ANALYSIS
2.1 Overview
RSA is not a complicated method, but, as with anymethod, one can optimize experimental design anddata analysis to make best use of the method. In thissection, we will describe the issues to consider, startingfrom experimental design to interpretation of results.In brief, a task will need to be designed such thattrials can be isolated from one another, pairs oftrials from different conditions can be related to oneanother with a similarity metric (e.g., correlation), and,finally, comparisons between the summary values ofthese condition-wise similarity metrics are made andconclusions drawn. The details of these steps follow.
2.2 Moving From Univariate to Multivariate
One of the things that sets RSA apart frommore tradi-tional univariate analyses is that all trials are recom-bined based on their relationships to one another.Thus, one might compare all trials in one conditionagainst all trials in another condition in a univariateanalysis, whereas one might correlate trials within thesame experimental condition and compare correlationsbetween trials that share a particular feature and trialsthat do not share this feature in a multivariate analysis.To illustrate this further, let us continue with ourexample of a memory paradigm where participants arescanned while performing a recognition memory test
using a real example (Dimsdale-Zucker et al., 2018).A typical memory-related univariate contrast mightcompare activity between studied items that werecorrectly remembered during the scan from studieditems that were forgotten. This analysis can be of greatvalue in identifying regions [e.g., the hippocampus(Diana et al., 2007)] or processes [e.g., presence/absenceof recollection (Yonelinas, 2002)] that track memory forthe items. However, one could ask different questionsdfor instance, when recollection occurs, does a brainregion of interest (ROI) represent information from theoriginal episode?
To answer this question, we designed a study whereparticipants studied objects in two different virtual real-ity homes. Participants studied sets of 10 objects withinone of these two homes across a series of 20 videos.Thus, each object had a unique spatial (house) andepisodic (video) context (Fig. 27.1). At retrieval, partici-pants were simply asked to make a memory judgmentabout studied and unstudied objects while in the MRIscanner. To understand how spatial and episodic contex-tual information from encoding was represented atretrieval, we used RSA.
To address this question with RSA, we can testwhether items that were associated with the sameencoding context share neural similarity when they areretrieved. In theory, this question could be addressedby RSA if one is willing to assume that voxel patternscan reflect distributed neural representations in thehippocampus (or in any other brain region, for thatmatter). However, even if this assumption holds, RSAdoes not necessarily reflect representations that reflectthe goals of the experimenter (Stark et al., 2017). It ishere that sound experimental design is especiallyimportant.
Here, because, we were interested in understandingwhether having the same or different encoding contextinfluences the neural pattern observed at retrieval, wecompared the activity pattern elicited during retrievalof items that were each associated with the same studycontext (e.g., the umbrella and tent occurred in Brown-House/Video1 (Dimsdale-Zucker et al., 2018)). Likewise,we compared the patterns elicited during retrieval of theother “same context” trial pairs (e.g., umbrella/dumb-bells, umbrella/bean bag, umbrella/chandelier, tent/dumbbells, etc. in BrownHouse/Video1 and air plane/football helmet, airplane/telephone booth, etc. in Gray-House/Video1). As we will describe in the followingsection, these correlations, in isolation, are difficult tointerpret without a baseline for comparison. The mostobvious comparison would be to use the same trials butthis time examine correlations between all possible“different context,” trial pairsdthat is, trials were notassociated with the same encoding context (e.g.,umbrella/football helmet, umbrella/telephone booth,
2. HOW TO DO REPRESENTATIONAL SIMILARITY ANALYSIS 511
tent/football helmet, etc.) To reiterate, the same trials areincluded in each bin of the analysis, which is afundamental difference between RSA and univariateanalyses.
2.3 How Many Trials Do I Need?
As a general rule, an experiment should be designedsuch that one can get as many trial pairs as possible forthe relevant experimental question. What is the optimalnumber of trials? More is almost always the answer, andthe optimal number will depend of course on the under-lying effect size. As a general rule of thumb, one mightaim for at least 30 trial pairs in the smallest bin.
In the abovementioned example, it should be clearthat there will be more trial pairs in the different contextbins than in the same context bin. Why could thispresent a problem? We know that having more trialpairs should yield a better estimate of the true correla-tion value (see Section 2.8 for how these correlations
are computed), especially because having noisy trialscan have a more significant influence on the correlationswith low trial numbers. Thus, one might be concernedthat the observed patterns between the conditions maydiffer simply because of the differences in trial pairsbetween them if there is more instability in estimatingthe patterns from the bin with fewer trials. In practice,we have not seen this to be the case when one isaveraging across a reasonably large number of trialpairs.
If unequal bin sizes between conditions is a concern,one option is to randomly resample the condition withthe smaller number of trial pairs to match the numberof trial pairs between conditions. You can see anexample of this in Fig. 27.2 using data from a real studyconducted in our laboratory (Dimsdale-Zucker et al.,2018).
Another complication, in answering how many trialsare needed, is how one plans to analyze the data (i.e.,within a subject and within run, within a subject and
Encoding
Object recognition (fMRI)
Representational Similarity Analysis (RSA)
same videosame house
different videosame house
different videodifferent house
x10 objects
x20 videos
x10 objects
FIGURE 27.1 Experimental design from Dimsdale-Zucker et al. (2018). Participants encoded objects uniquely located within one of twovirtual reality homes (spatial contexts) across a series of 20 videos (episodic contexts). Next, they were scanned while performing an objectrecognition test, which required differentiating old and new objects presented without any contextual information. We used representationalsimilarity analyses to examine the similarity of voxel patterns elicited by each recollected object relative to other recollected objects that werestudied in the same (or different) spatial and episodic contexts. fMRI, functional magnetic resonance imaging. Figure adapted with permission of
author.
27. REPRESENTATIONAL SIMILARITY ANALYSES: A PRACTICAL GUIDE FOR FUNCTIONAL MRI APPLICATIONS512
between runs, across subjects). We will return to theseissues in Section 2.5.
2.4 How Should Trials Be Spaced?
We will cover issues related to modeling of both RSAand other fMRI designs in Section 2.5, but let us firstaddress the issue of trial spacing and randomizationbecause these both impact how well we are able touniquely account for the variance associated with eachtrial.
A large problem for RSA, and fMRI in general, is theinherently correlated nature of the data. This is because,even in the absence of a cognitive process, there areongoing biological and mechanical fluctuations in thesignal that can bemeasuredwithMRI. To reduce the over-lap in the signal that is measured at two time points, thetwo time points need to be reasonably far apart such thatthe variance accounted for by each time point can beuniquely estimated by the regression model. Classically,thismeant longspacingdon theorderof tensof secondsdbetween trials [slow event-related designs (Blamire et al.,1992)]. Later work revealed that by randomizing spacingbetween trials within a given range of both long and shortvalues (i.e., “jittering” the inter-trial interval (ITI); Burocket al., 1998; Clark et al., 1998; Dale and Buckner, 1997) onecan more efficiently estimate activation across trialswithin a particular experimental condition. Such fastevent-related designs are now standard in most modernfMRI studies. However, for RSA, because we aremodeling each trial in isolation, we want each trial tobe maximally isolated from all other trials.
In a jittered design, some trials would occur closer intime and others further apart, which is ideal when yourmodel can capitalize on the variance in time across trialswithin a condition, but it is not ideal for isolating trial-unique activity. Zeithamova et al. (2017) systematicallycompared the timing of stimuli on how well they wereable to use RSA to understand category-level differ-ences, item-level differences, and memory-relateddifferences across a set of canonical regions of interestknown to be sensitive to these features. Across the dura-tions they tested, both item-level and memory-relateddifferences were more robust with longer intervalsbetween stimuli (the longest stimulus onset asynchronythey used was 12 s) consistent with extant evidence thatlonger trial spacing may be better for estimation ofsingle-trial activity patterns (Visser et al., 2016); theability to detect shared category-level information didnot vary by timing condition. Jittering the onset timingof trials made a minimal difference in their ability todetect different representations. Thus, it seems that forRSA designs with many conditions, it is optimal tomaximally separate trials from one another rather thantrying to orthogonalize at the condition level as wouldbe carried out for univariate designs, although this issueis perhapsmore critical for within-run than between-runcomparisons. It is critical to strike a balance betweenwhat is best for the purposes of modeling and whatmakes the most sense psychologically to participants.AsMumford et al. (2014) has already written extensivelyabout, the spacing and order of trials should be run- andsubject unique to minimize the possibility of falsepositives.
CA1_body CA2_3_DG_body
Different VideoDifferent House
Different VideoSame House
Same VideoSame House
Different VideoDifferent House
Different VideoSame House
Same VideoSame House
0.000
0.002
0.004
0.006
Mea
n P
atte
rn S
imila
rity
(r)
Neural pattern similarity for spatial and event contexts
CA1_body CA2_3_DG_body
Different VideoDifferent House
Different VideoSame House
Same VideoSame House
Different VideoDifferent House
Different VideoSame House
Same VideoSame House
0.000
0.002
0.004
0.006
Mea
n P
atte
rn S
imila
rity
(r)
Neural pattern similarity for spatial and event contexts(A) (B)
FIGURE 27.2 (A) Mean pattern similarity (PS) including all possible trial pairs in each condition. Error bars represent standard error of themean. (B) Mean PS where each subject’s data have been resampled such that each condition has the same number of trial pairs. Although the datain panel B are noisier (e.g., larger error bars), the overall general pattern of interest (greater PS for items that occurred in both the same video andhouse in CA1 compared with reduced PS for these trials in CA2/3/DG) remains the same. Figure adapted from Dimsdale-Zucker et al. (2018) with
permission of author.
2. HOW TO DO REPRESENTATIONAL SIMILARITY ANALYSIS 513
2.5 How Should I Model my Data?
2.5.1 Within-Run Versus Between-Run Modeling
Once the task has been optimized and the datacollected, there are a series of choices that are made atanalysis that can impact the interpretation of the results.One that is often overlooked (seeMumford et al., 2014) iswhether you plan to compare voxel patterns across trialsthat occurred within the same run or trials that occurredin different runs. Either approach is reasonable, butcorrelating voxel patterns across trials within the samerun and across trials in different runs is not appropriate.
Although we would like to assume the BOLDresponse is only driven by neural influences, weknow that there are ongoing biological and mechanicalfluctuations that have a relatively slow drift over timewhen compared with the time course of cognitiveprocesses (Heeger and Ress, 2002). This necessarilymeans that neighboring trials share more variance,even if it is “uninteresting” variance that is unrelatedto the task. Not only do neighboring trials share morevariance, but trials within the same run are more similarto one another than trials in other runs again due to task-invariant reasons such as scanner drift. This means thatif both within- and between-run trials were includedwhen comparing PS, correlation values are beingartificially inflated. PS should be higher for conditionsthat include more correlations between trials in thesame run than for conditions that include morecorrelations between trials in different runs1.
The issue of whether to look at within-run PS orbetween-run PS is often determined by the task. In amemory design, or other paradigms where you mayhave little control over how many trials are in eachbin, it can often be preferable to do only between-runRSA because this should increase the total number oftrial pairs in each condition. However, for some designs,this does not make sense. For example, Jonker et al.(2018) had a design where they were interested incomparing the overlap in RSA between items atencoding and items during practice (either a restudyor a retrieval condition). In this design, to control forthe time between encoding and practice, all items wereencoded and practiced in the same run. Thus, RSAnecessarily had to be performed within the same run.
A related issue is whether it is better to have fewer,longer runs or more, shorter runs. For other classifica-tion techniques such as MVPA where one run is oftenleft out to verify the classification, it is advantageous
to have more runs (Coutanche and Thompson-Schill,2012). If you plan to do within-run RSA, longer runsare better because this means that you can have moreobservations in each run. For between-run RSA, shorterruns can also be preferable. Our standard practice isto do what makes the most sense for the cognitiveprocesses in the task and to achieve the cleanest datapossible (e.g., runs that are too long often have moremotion artifact).
2.5.2 Preprocessing
Whether the RSAwill be computed between trial pairsin the same run or across runs, the data cleaning and pre-processing procedures are similar. Indeed, the initialpreprocessing stages for RSA are nearly identical to thosefor standard univariate analyses (for a set of standardizedrecommendations for general preprocessing, see https://fmriprep.readthedocs.io/en/stable/). In our laboratory,we first start by running some type of data qualitycheck (e.g., http://cibsr.stanford.edu/tools/human-brain-project/artrepair-software.html) which generatesinformation about general noisy time points that can beincluded as spike regressors in our fMRI model andmotion parameters to again use in our fMRI model. Wethen typically coregister the data within each functionalrun, realign the functional data into structural space,and minimally smooth the data (e.g., with standard3 mm isotropic voxels, we might use a 2 mm full-widthat half maximum (FWHM) smoothing kernel). Althoughit may seem odd to smooth across voxels if we plan tolook at a voxel-wise pattern, a moderate degree of spatialsmoothing can enhance the signal-to-noise ratio andenhance classification accuracy (Op de Beeck, 2010) whilepreserving distributed pattern information (Kriegeskorteet al., 2010). If you are looking at voxel patterns in regionswhere anatomical precision is integral to the hypothesis(e.g., differences in voxel patterns between subfields ofthe hippocampus), no smoothing should be performed.
2.5.3 Single-Trial Modeling
One important consideration for RSA is how the dataare to be modeled. This is a potential concern because,due to the sluggishness of the BOLD signal, one mustdeconvolve overlapping responses to adjacent, closelyspaced trials. This is usually not a significant concernin standard univariate fMRI analyses, in which one usu-ally wants to estimate the average response across all of
1 It is conceivable that this confound could be avoided if you designed your task such that there are equal numbers of all conditions in all runs,
but, in practice, this is usually impossible with a memory experiment (and other designs that sort trials based on participants’ responses) in
which trials are sorted based on subject performance. Moreover, it is still not advisable to incorporate a systematic source of unmodeled
variance into any statistical analysis.
27. REPRESENTATIONAL SIMILARITY ANALYSES: A PRACTICAL GUIDE FOR FUNCTIONAL MRI APPLICATIONS514
the trials from a given condition in the run (Burock et al.,1998; Clark et al., 1998; Dale and Buckner, 1997).
In some experiments, one can model the data usingessentially the same approach that is used in a standardunivariate analysis. For instance, if there are multiplerepetitions of the condition of interest, and if you arecomputing pattern correlations across runs or acrosssubjects, it can be advantageous to model the datawith a regressor that models all of the trials within thecondition of interest. This approach is generally optimalbecause it is straightforward to obtain stable estimates ofactivity for each condition because the model is fit tomultiple observations. However, this approach mightnot be optimal for many memory paradigms. Forinstance, if you are presenting studied and novel imagesat retrieval, you do not want to repeatedly probeindividuals’ memory for each item. Even if there arerepetitions of items within a condition (e.g., you couldmodel all remembered trials together even if these areall different items), if you want to combine trial pairsacross different conditions (e.g., remembered trialsmay have been associated with different encoding con-texts), in which case, it may not be appropriate to modelthese trials together.
For cases in which it is not feasible to use a traditionaldesign and analysis approach (i.e., jittering ITI andrandomizing order of conditions), Mumford andcolleagues have written about several ways to estimateresponses to each trial in isolation (Abdulrahman andHenson, 2016; Mumford et al., 2012, 2014, 2015). Theseapproaches essentially involve running a differentgeneralized linear model (GLM) analysis for each trialthat is to be estimated. In eachmodel, one trial of interestis modeled separately from other trials. There aredifferent variants that differ in how the trials of nonin-terest are modeled to make them maximally orthogonalto the trial of current interest. In practice, there shouldnot be drastic differences between the differentmodeling approaches (Zeithamova et al., 2017), but itis worth choosing the specific modeling approach thatis most appropriate given your experimental question.
Even with the single-trial modeling approachesrecommended by Mumford, there are relatively strongcorrelations in estimated responses across trials thatare trials closely spaced in time (see Fig. 27.3 for anexample). Plotting a correlation matrix and looking tosee where the band of autocorrelation drops off can beinvaluable in determining howmany steps between trialpairs to remove.
Some have also suggested including the first-ordertemporal derivative as a way of soaking up theadditional hemodynamic differences with varying lagsbetween trials (Calhoun et al., 2004; Friston et al., 1998;Worsley and Taylor, 2006). However, when trials arerelatively close in time, these temporal derivatives can
actually contribute to colinearity between regressorsin the model. Therefore, for single-trial modelingapproaches, it is generally advised to not include thetemporal derivative (Abdulrahman and Henson, 2016;Mumford et al., 2012).
Regardless of how you choose to model your data, bythe end of this stage, the goal is to have a single betaimage (i.e., a matrix of parameter estimates for eachvoxel in the imaged region) for each trial, or trial type,of interest. These beta images will be used to computePS values (see trial-level cleaning for an argument forusing t-maps instead of betas for RSA).
2.6 How Do I Use Representational SimilarityAnalysis to Test a Hypothesis AboutSpecific Regions?
The goal of RSA and other multivariate techniques isto understand how information is represented. Often thegoal is to test a principled hypothesis about how a ROI
FIGURE 27.3 Graphical representation of all trial pairs for a singlesubject across four runs [for full data set, see Dimsdale-Zucker et al.(2018)]. Warmer colors (red) indicate higher pattern similarity (PS)(Pearson’s r); white lines indicate excluded trials (e.g., poor behavioralperformance, outlier beta value). Several patterns in the data are quitestriking: First, we observe the bright red diagonal; this is each trial’scorrelation with itself, which therefore has a value of 1. Moving just offof this diagonal, we can observe a band of approximately 4e5 trialsthat have lower correlations than all other trial pairs within each run.These trials should be removed from analysis if computing within-runPS because they are contaminated with autocorrelation. Finally, we canalso see an overall trend for higher PS within the same run than be-tween runs (notice that the rectangles for each of the four runs arewarmer in color than any of the between-run PS values).
2. HOW TO DO REPRESENTATIONAL SIMILARITY ANALYSIS 515
in particular represents information. To continue withthe example from our laboratory introduced earlier, weused an ROI-based RSA approach to ask how differentsubfields of the hippocampus represent contextualinformation when items are retrieved (Dimsdale-Zuckeret al., 2018). To do this, we measured activity patterns insubfields CA1 and CA2/3/dentate gyrus (DG) whenparticipants remembered items that had earlier beenstudied relative to different spatial (virtual homes) andepisodic (videos that occurred in these homes) contexts.We found that an item’s spatial context did not distin-guish the pattern of representation in the subfields.However, when we looked at items that either hadoccurred in the same or across different episodic con-texts, we found different patterns in CA1 and CA2/3/DG; in CA1, items that shared an episodic contextwere represented as more similar than those fromdifferent episodic contexts, whereas CA2/3/DGshowed the opposite pattern and differentiated betweenobjects that had occurred in the same episodic context.Taking an ROI-based RSA approach thus allowed us toanswer a question about how different subfields repre-sent contextual information and adjudicate betweencompeting hypotheses in the literature.
ROIs are typically delineated on the basis ofanatomical features (e.g., cortical delineations identifiedwith FreeSurfer), task-responsiveness (e.g., regionsaffiliated with a keyword on neurosynth.org), orconnectivity profiles (e.g., regions that are preferentiallyconnected to a node in a network). The only major issueto avoid is that an ROI should not be defined in amanner that is redundant with the PS analysis (Kriege-skorte et al., 2009). For instance, if you wanted to testwhether items that are remembered are more similarto one another than items that are forgotten, you shouldnot define ROIs on the basis of a univariate contrastbetween remembered and forgotten items.
Once the ROIs have been selected, it is necessary toextract activation estimates for each voxel in the ROIseparately for each trial or condition of interest. Forany given trial or condition, the corresponding voxelpattern vector is comprised of the collective set ofactivation estimates across the voxels. The voxel patternvectors across all the trials can be concatenated in avoxels-by-trials matrix. The dimensions of this matrixwill vary from ROI to ROI because ROIs are definedby anatomy and not absolute number of voxels. Wecan next compute a similarity metric (e.g., Pearson’s r)across each pair of voxel pattern vectors which will yielda square, trial-by-trial similarity matrix (see Section 2.8for other approaches to computing trial-wise similarity).If you are doing between-run correlations, this would bea trial-by-trial matrix for all trials across all runs.
On a procedural note, it is critically important tounderstand which trials fall into which conditions. In
our laboratory, we have found it handy to have aseparate matrix that shares the trial-wise dimension ofthe trial-by-trial correlation matrix across eitherrows or columns (typically, trials as rows has been ourlaboratory’s convention) and where all of the otherpotentially relevant trial information (e.g., the trial’sposition within encoding and within a run at retrieval,run membership, the subject’s response both atencoding and at retrieval, any encoding manipulationof interest, and any other task-relevant trial features) iscontained in the other dimension. This trial informationmatrix can then be used to index the trial pairs thatshould be extracted for various conditions of interest.
2.7 How Do I Use Representational SimilarityAnalysis to Test a Hypothesis That Is NotRegionally Specific?
If you wanted to call Batman for help, you couldilluminate the entire sky over the affected citydthis isakin to the ROI approach. Or, one could systematicallymove a spotlight across all points in that given areadthis is the general idea behind a searchlight. Eithershould result in Batman coming to the scene of thecrime, or, in our case, understanding the pattern withina ROI, but the size of the search space differs. SearchlightMVPA analyses (Kriegeskorte et al., 2006) are typicallyperformed in cases where one does not have a stronghypothesis about the region that will show the expectedeffect. With a searchlight, instead of looking across anentire anatomical region, you are instead creating asmall volume that is moved throughout either the wholebrain or throughout a particular ROI. For this reason, thesize and shape of the searchlight can have large conse-quences on the observed effects. Before we tackle theissue of selecting a shape and size, it is important tounderstand that the dimensions of the searchlightdetermine the smoothness of the pattern you are tryingto findda small searchlight is more likely to identifysmall, blocky patterns, whereas a larger searchlight issmoothing out the pattern that you are looking for (Etzelet al., 2013; Viswanathan et al., 2012).
In our laboratory, we are typically interested in look-ing at memory-related activity either in the hippocam-pus itself or in the surrounding medial temporalcortical regions. For this reason, our laboratory hasfavored relatively small searchlights with a diameter ofapproximately five voxels and that have a central nodewith arms of equal length (two voxels out from this cen-tral node) extending in three dimensions [see Fig. 27.4;for a published result with a searchlight like this, seeLibby et al. (2014)]. However, others have used spher-ical, diamond, or other shaped searchlights (for imple-mentations of some of these shapes, see http://cosmomvpa.org/). We then treat this searchlight volume
27. REPRESENTATIONAL SIMILARITY ANALYSES: A PRACTICAL GUIDE FOR FUNCTIONAL MRI APPLICATIONS516
as if it were a tiny ROI and extract the values from allvoxels within that searchlight volume, repeat this proce-dure across all trials for this searchlight volume, andeventually compute a pattern similarity value for thatsearchlight that is assigned to the central voxel node.Next, one can systematically move the searchlightmask across the brain, computing a voxel patternsimilarity analysis at each position. This procedure isrepeated until all locations within the search spacehave been sampled. We exclude any section of thesearchlight that extends beyond the volume of interest,and, if the total number of voxels in the searchlight spaceis ever less than nine voxels, we do not compute apattern correlation for this central node location. Thiscut-off ensures that estimated correlations will berelatively stable.
Unlike with ROI-based RSAwhere activity for pairs oftrials are extracted, with a searchlight it is more commonto subtract the searchlight maps for two conditions ofinterest. This yields maps for the entire searchlight spacewhere there is a difference in the voxel-wise pattern inthe searchlights. Although these maps can look likeunivariate contrast maps, their interpretation is different(Etzel et al., 2013). It is more appropriate to talk about asearchlight that is centered in a region rather than totalk about the region itself as being implicated in the rep-resentation (Etzel et al., 2013).
2.8 How Do I Quantify Similarity?
As suggested by the name, RSA is about analyzingsimilarity between patterns of activity evoked duringdifferent conditions. This can be a tricky issue, however,because similarity can be defined and measured indifferent ways. If we think about representing trialsimilarity values in the simplest way possible, we canthink about noting each trial’s value as a point in x/ycoordinate space and extending lines (pattern vectors)joining the origin to these points. We can then think
about summarizing the distance between these twopoints either relative to a line connecting them (thinkabout drawing the third arm of a triangle) or as the anglebetween these pattern vectors as measured at the origin(for an example, see Bobadilla Suarez and Love, 2017;Walther et al., 2015). Metrics that are similar to thinkingabout the line that would complete a triangle are dis-tance metrics, such as Euclidean distance andMahalanobis distance, and metrics that use the angleare Pearson’s correlation and cosine distances. Anythingthat shifts the distance to the origin (e.g., shifts in theMRI baseline) impact angle-related but not line-relatedmetrics, and changes in the length of the lines effectline-related but not angle-related metrics. We will startwith a description of Pearson’s correlation distancebecause it is probably the most commonly used distancemetric in RSA.
Pearson’s correlation was used in one of the firstapplications of RSA, and it is the most commonly usedsimilarity metric in current neuroimaging studies(Aguirre, 2007; Haxby et al., 2001; Kiani et al., 2007;Kriegeskorte et al., 2008b). Pearson’s correlation has theadvantage of being straightforward to compute and itis, mathematically speaking, invariant to differences inthe mean and variability of the data (Kriegeskorteet al., 2008a). Cosine distance is closely related to Pear-son’s correlation in that it is concerned with the angleof the pattern vectors. In fact, it is identical to Pearson’scorrelation in the case where the mean pattern hasbeen removed from each voxel (also called cocktail-blank removal; Misaki et al., 2010; Walther et al., 2015).In practice, many readers are unfamiliar with cosinedistance, whereas Pearson’s correlation is well known.However, Pearson’s correlation is more often used as ameasure of association between two variables, ratherthan a similarity metric, which can lead to some intuitivemisinterpretations. For instance, PS between two trialscould, in theory, be associated with an r value of �0.8.Intuitively, this would imply a strong association
FIGURE 27.4 Example searchlight with a diameter of five voxels. Figure courtesy of Alex Clarke. For data published with this searchlight, see Clarkeet al. (2016).
2. HOW TO DO REPRESENTATIONAL SIMILARITY ANALYSIS 517
between two variables, but, in the context of RSA, thenegative correlation means that there is very lowsimilarity between the voxel patterns. This interpretivetrap can be avoided by plotting the correlation distancemetric, 1 � r, which rescales the values from between0 and þ2.
An alternative to angle-related measures areEuclidian distance metrics (Edelman et al., 1998).Euclidian distance is a “true” distance metricdthat is,it is based on the Euclidian geometric system and isinterpreted as the distance between two points in amultidimensional space (Walther et al., 2015). Mahala-nobis distance is closely related but quantifies thedistance between points as a function of standarddeviations from the mean (Kriegeskorte et al., 2006). Inessence, Mahalanobis distance takes the covariancestructure of the entire data set into account, whereasEuclidean distance can be computed without estimatingcovariance.
Tests of the reliability of these various distance met-rics have not revealed systematic differences (althoughfor a more thorough discussion of when there aredifferences, see Walther et al., 2015) and RSA toolboxes(e.g., CoSMoMVPA, http://cosmomvpa.org/) allowthe use of different distance metrics usually by changing
one line of code. We therefore encourage researchers topick the metric that is most appropriate for thequestion(s) of interest and to consider what has beenused in similar studies.
2.9 How Can I Test Hypotheses Using VoxelPattern Information?
Having computed PS values for each condition ortrial bin of interest, these data can be used to testhypotheses. There are at least two major ways to usePS values to summarize the trial-wise patterns forRSA. One approach is to compose a theoretical correla-tion matrix, or expectancy matrix, that one thinkssummarizes the relationships between conditions (seeFig. 27.5CeE for an example Clarke et al., 2016; Kriege-skorte et al., 2008a). This approach is often useful whenthe relationships between conditions are complex or tocompare the observed data against the theoreticalpredictions of different computational models. Theobserved trial-wise RSA matrix is then correlated withthe expectancy matrix to obtain what is called asecond-order similarity matrix. Currently, best practiceis to use Kendall’s Tau to evaluate second-ordersimilarity (Nili et al., 2014).
“apple”
“pencil”
“chair”
same-room similarity
- diff-room similarity
context similarity
(bedroom)
(bedroom)
(kitchen) 0.07
0.08
0.09
Sim
ilari
ty
L Ant. HF L PRC
same-room diff-room
immediate delayed immediate delayed
* † †
(A)
(C)
(D)
(E)
(B)
FIGURE 27.5 Different ways to test hypotheses with representational similarity analysis. (A) Extraction of patterns from a region of interest ofinterest for words studied in different room contexts. (B) Summarized pattern similarity (PS) values in left anterior hippocampus (L. Ant. HF) andleft perirhinal cortex (L PRC) subdivided based on when an item was tested (immediate/delayed) and whether or not it occurred in the same ordifferent room context. (C) Extraction of patterns using a searchlight for studied objects. (D) Resultant PS matrix for all searchlights. (E) Predictedmatrix [or representational dissimilarity matrix (RDM)] that would be correlated with the observed patterns. Smaller matrices on the right showdifferent potential models that could fit the data. (A-B) Figure adapted from Ritchey et al. (2015) with permission of author. (C-E) Figure adapted from
Clarke et al. (2016) with permission of author.
27. REPRESENTATIONAL SIMILARITY ANALYSES: A PRACTICAL GUIDE FOR FUNCTIONAL MRI APPLICATIONS518
For simpler experimental questions, you can directlyextract the similarity values for pairs of trials withindifferent conditions, average across them, and comparethe mean level of PS between conditions [see Fig. 27.5Aand B for an example (Ritchey et al., 2015)]. Note thatif Pearson’s r is the chosen similarity metric, thecorrelations must be z-transformed (Fisher transform)before they can be submitted to a parametric statisticaltest. This is because Pearson’s coefficients are boundedbetween �1 and þ1 and therefore are not normallydistributed.
In general, you can conduct group-level analyses ofPS values in the same manner as traditional univariatefMRI analyses. Typically, group analyses focus on asummary statistic approach in which a single meancontrast value is computed for each participant andentered into a group t-test. A preferable alternative isto take a mixed model analysis approach. In a mixedmodel, each trial pair within the condition is enteredinto the model as a single PS value, rather than averagedacross a condition. The model then separately estimatesvariance attributable to the variable of interest, varianceattributable to intersubject variability, and intertrialvariance within each subject’s data set. In general,these models are better suited to differentiate betweeninteresting and uninteresting sources of variance in thedata. A detailed treatment of mixed effects models isbeyond the scope of this chapter, but interested readerscan choose from several excellent papers on this topic(Barr et al., 2013; Bates, 2010; Bates et al., 2014; Singmannet al., 2015).
When using a searchlight analysis or examining dataacross several ROIs, it is important to correct formultiple comparisons. The most straightforwardapproach is to use nonparametric approaches suchas permutation tests. These approaches often involvesome sort of shuffling procedure to create a nulldistribution that is specific to the range of variance inthe data and then the observed statistic being evaluatedfor significance can be compared to this generateddistribution. Others have written more extensivelyabout the nuances of computing and reporting permuta-tions and other nonparametric statistics [(Etzel andBraver, 2013; Etzel et al., 2009) for an exampleapplication of this, see (Huffman and Stark, 2017)].
3. PITFALLS, PROBLEMS, AND PROPOSEDSOLUTIONS
3.1 Noise Reduction
Although it is not a widely recognized issue in RSA,we feel it is important to consider the fact that fMRIdata are inherently noisy. Even a large perceptual or
cognitive signal will, at best, elicit an observed BOLDsignal change around 2%e4% (Pillai and Zaca, 2011).In typical memory studies, signal changes are typicallyaround 0.25%e0.5%. Therefore, it can be important totake some additional steps to remove sources of noiseand nuisance variance from the data. Although ourlaboratory routinely employs many of these techniques,it is also our experience that these steps may have smalleffects. We have never seen noise reduction methodsintroduce an effect where one did not exist before,and we would be concerned if they did! We have seen,however, that careful data cleaning can enhancesignal-to-noise ratios and sensitivity to relatively smallsignals in task- and pattern-based fMRI.
3.2 Voxel-Level Cleaning
One cleaning step is to identify and remove noisyvoxels that consistently have fluctuations that exceedsome range of the data (e.g., greater than 3 standarddeviation mean signal change across all runs). Suchlarge fluctuations are usually caused by motion artifacts.For example, a voxel near the edge of the brain or neara zone of signal dropout might alternate betweensampling the brain and sampling nonbrain regionsover the course of scanning. One way to identify thesevoxels is to look at each voxel’s time course. If thereare truly voxels that are contributing noise, their meansignal (or mean standard deviation in signal) shouldlie outside of the distribution of all other voxels. Howev-er, it is important to be sure that this variability is notcorrelated with your task (e.g., motion that co-occurswith a particular trial type). Typically, our laboratoryhas used cut-offs of 3e5 standard deviations outside ofthe range of the rest of the data. In our experience, onerarely needs to exclude voxels, and large fluctuationsin many voxels are often indicative of a deeper issue(e.g., excessive movement artifact, misalignment ofROIs to functional data).
Instead of excluding voxels that are most likelycontributing noise, an alternative approach is to only usethe voxels that are most likely to be contributing signal.This approach, called feature selection, is relativelycommon in classification-based multivariate approaches(Norman et al., 2006), especially when the entire brainis being used for classification. Feature selection is notcommonly used in RSA, and, in our limited tests, wehave not found it to make a significant difference.
If you use feature selection, it is important to selectvoxels in an unbiased manner (Kriegeskorte et al.,2009). For example, if you scanned a separate run ofanother memory task where you could identifymemory-sensitive voxels unrelated to memory for itemsin the task at hand, or if you had a separate localizer runto identify voxels that were selective to another
3. PITFALLS, PROBLEMS, AND PROPOSED SOLUTIONS 519
manipulation in your task, using these separate data toidentify voxels would not be problematic. However, itis rare that we have the luxury of extra time in the scan-ner to collect these additional data.
3.3 Trial-Level Cleaning
Scaling up from the level of voxels, we can also thinkabout noise at the trial level. When you separately modelsingle-trial activation values, there will always be sometrials that are poorly fit by the model, or trials that, forwhatever reason, have extreme activation estimates.Our approach in removing outlier trials is identical, inprinciple, to removing outlier voxels. That is, to identifyan outlier trial, we look at the distribution of beta values(e.g., by computing the mean beta value for each trialwithin a ROI) both for each subject, and, across the entiresample of subjects, to determine a threshold cut-off. Wehave also tried using a more standardized approach todetermine a cut-off based on a z-score approach orpercentile rank. What is important is that you pick ametric that is sensible for your data and be consistentin its application. If you are removing more than about
10%e15% of the betas for each participant, this typicallyindicates that there are other issues with the data. It isalso a good idea to visually inspect the betas that areidentified for removal. For example, our laboratorydetected a scanner artifact (see Fig. 27.6) that resultedin “stripey” patterns in the modeled betas. These betaswere clearly nonneural sources of noise and should beeliminated from the rest of the analysis pipeline.
Another approach to reduce the influence of noisyvoxels is to use t-maps rather than beta maps forextracting patterns of interest (Walther et al., 2015).Whereas feature selection removes entire voxels fromthe analysis, one can instead include all the voxels anduse t-values to weight the analysis toward voxels ortrials that are generally more informative. A t-statisticis simply a beta that is scaled by the variance. Thismeans that the most extreme voxels are penalizedmore than stable voxels because they vary more fromtime point to time point. Again, this approach shouldnot produce a pattern where one did not exist previouslybut should enhance sensitivity to detect task-dependentPS effects (see Fig. 27.7 for an example in ventral visualcortex).
FIGURE 27.6 Example of a “stripey” beta that was identified as an outlier and removed from analysis pipeline. Visualized in FSL (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki). Unpublished data, courtesy of Marika C. Inhoff.
27. REPRESENTATIONAL SIMILARITY ANALYSES: A PRACTICAL GUIDE FOR FUNCTIONAL MRI APPLICATIONS520
In memory paradigms, it is often necessary tocompare pattern correlations to conditions that haveunequal numbers of trial pairs. For instance, one couldtest whether hippocampal PS is higher for items thatwere associated with the same context than for trialsthat were associated with different contexts. Unfortu-nately, in this design, it would likely be the case thatthere are more trial pairs associated with different thanthe same context. A frequent concern in reviews, andin the general community, is whether an analysis mightbe biased to show higher correlations for the conditionwith fewer trial pairs than for the condition with moretrial pairs. One way to address this concern is to rerunanalyses in which you equate the number of trial pairsby subsampling conditions from the conditions withmore trials (see Fig. 27.2 for an example). Even if theconditions vary in the number of trial pairs, resultsshould remain stable if there are a reasonably largenumber of trial pairs in each condition.
3.5 Influence of Univariate Activity onMultivariate Patterns
In general, researchers expect univariate andmultivar-iate fMRI analyses to reveal qualitatively different kindsof information (Arbuckle et al., 2018). A concern that isfrequently expressed is whether a PS difference isconfounded with a difference in the overall magnitude
of activation between the two conditions. One mightnot necessarily want to separate the two influences,however. From a computational perspective, there is noreason to think that different representations in a neuralnetwork should be associated with equal overall firingrates. Nonetheless, many researchers believe that RSAcan only be interpreted if pattern differences are notdriven by overall activation magnitude.
As noted earlier, Pearson’s r, Spearman’s r, and cosinedistance are magnitude insensitive. For example, ifcondition B shows the same pattern but more activationthan condition A (i.e., the angle is preserved but thevector length differs), then the r value will be þ1, justas if the two conditions were associated with the sameactivation values. That said, even when using metricsthat are mathematically insensitive to activation magni-tude, activation differences could nonetheless confoundvoxel PS estimates.
This is particularly the case when one is using asearchlight or an inappropriate ROI. For example,imagine a searchlight analysis in which the searchlightcenters on a point between two anatomically distinctregions, A and B. Region A shows uniformly highactivity in the memory condition but not in the controlcondition and Region B shows low activity in bothconditions. RSA on the searchlight region, even usingPearson’s r, would reveal a pattern difference acrossthe memory and control conditions that are entirelydriven by differences between overall (univariate)activation magnitude across the two regions spannedby the searchlight. Even in this case, the interpretationis not straightforward. One might conclude that a uni-variate effect in Region A is driving a spurious patterndifference between the memory and control conditions.Alternatively, one might conclude that there is an under-lying representation in the memory condition that isdistributed across a single brain region or network thatis comprised of both Region A and Region B. Bothconclusions are equally valid, depending on one’sperspective.
From a practical standpoint, the issue of univariateactivation confounds is often irrelevant. In some cases,the trial pairs are simply the recombination of allpossible trials from one side of a univariate contrast(e.g., rearranging all remembered items based on theirencoding condition, e.g., Dimsdale-Zucker et al., 2018).Of course, if there was a univariate difference betweentwo different encoding conditions (e.g., had we foundlarge activation differences between videos 1 and 3and all trial pairs of interest came from these twovideos), then there could be a univariate confound tobe concerned about; however, with many conditions itis unlikely that this confound would arise by chance.In other cases, researchers have attempted to pull apartthe univariate and multivariate contributions either by
FIGURE 27.7 Informal comparison between representationalsimilarity analysis (RSA) using betas (warm colors) versus t-maps(cool colors) comparing repeated presentations of the same item ascompared with presentations of a different item. Notice that the RSAusing t-maps does a better job of identifying PS effects in ventral visualcortex, as would be expected in this comparison. Unpublished data,
courtesy of Marika C. Inhoff.
3. PITFALLS, PROBLEMS, AND PROPOSED SOLUTIONS 521
FIGURE 27.8 Correlations between differences in reaction time between pattern similarity (PS) trial pairs and PS values with linear fit lines (blue) in CA1(A-C) and CA23DG (D-F). RT,reaction time. Figure adapted from Dimsdale-Zucker et al. (2018) with permission of author.
27.REPRESENTATIO
NALSIM
ILARITYANALYSES:A
PRACTIC
ALGUID
EFORFUNCTIO
NALMRIAPPLIC
ATIO
NS
522
removing the mean condition-wise effect (Macevoy andEpstein, 2009; Misaki et al., 2010) or showing that the in-formation contained in the univariate and multivariateeffects differ (Aly and Turk-Browne, 2016).
3.6 Influence of Subject-Specific Confounds
In 2013, Todd et al. (2013) showed that trial-wisereaction time differences (RT) could account for patterndifferences between conditions, even if, on average, thetwo conditions do not show an RT difference. This seem-ingly counterintuitive finding reflects the fact thatpattern classifiers can be driven by both positive andnegative activation differences. Imagine a case whereone subject is 100 ms faster in Condition A than B,whereas another subject is 100 ms faster on ConditionB than A. A brain area where activation is sensitive toRTcould be used to classify A and B states for both sub-jects, despite the fact that, across subjects, the averageRTs are equal across conditions. In this case, we wouldnot know whether accurate classification betweenConditions A and B reflects interesting differences inunderlying representations or whether it is a result ofa relatively uninteresting effect that is related to individ-ual variations in response latency or task difficulty.
Todd et al. considered whether this is an issue fornonclassification multivariate techniques such asRSA. Their conclusion was that RSA should be lesslikely to suffer from these types of confounds thanclassification-based methods. In the paper, they simplysuggest including reaction time in a linear regressionmodel to account for between-condition reaction timedifferences. Fig. 27.8 shows an example data set (Dims-dale-Zucker et al., 2018) where we included a randomeffect of reaction time in our mixed models (which isan essentially equivalent approach to linear regression).As you can see, in this data set it is clear that whether ornot we account for reaction time does not change theobserved pattern of effects. However, this a reasonableand straightforward control analysis to routinelyperform if there are reaction time differences betweenconditions of interest.
4. REPRESENTATIONAL SIMILARITYANALYSIS: WHERE DO WE STAND?
Since its introduction to the human imaging world(Kriegeskorte et al., 2006, 2008a), the number of publica-tions using RSA or other multivariate techniques hassteadily increased. It has been particularly useful inmoving the field forward from localization of processeswithin regions to understanding how these regionsrepresent information. Although we still know little
about how neural activity drives large-scale differencesin multivoxel activity patterns, RSA is a valuable toolfor bridging the gap between human fMRI and invasiverecording studies in animal models (Kriegeskorte et al.,2008b). RSA is now also beginning to be applied tosingle-unit recording studies of memory (McKenzieet al., 2016) and scalp EEG (Kaneshiro et al., 2015)showing that is a highly flexible analysis technique. Inthe future, we hope to see a continued use of RSAand other multivariate techniques to enhance ourunderstanding of the mind and brain.
References
Abdulrahman, H., Henson, R.N., 2016. Effect of trial-to-trial variabilityon optimal event-related fMRI design: implications for Beta-seriescorrelation and multi-voxel pattern analysis. NeuroImage 125,756e766. https://doi.org/10.1016/j.neuroimage.2015.11.009.
Aly, M., Turk-Browne, N.B., 2016. Attention stabilizes representationsin the human hippocampus. Cerebr. Cortex 26 (2), 783e796.https://doi.org/10.1093/cercor/bhv041.
Arbuckle, S.A., Yokoi, A., Pruszynski, J.A., Diedrichsen, J., 2018.Stability of representational geometry across a wide range of fMRIactivity levels. BioRxiv 266585. https://doi.org/10.1101/266585.
Barr, D.J., Levy, R., Scheepers, C., Tily, H.J., 2013. Random effects struc-ture for confirmatory hypothesis testing: keep it maximal. J. Mem.Lang. 68 (3), 255e278. https://doi.org/10.1016/j.jml.2012.11.001.
Bates, D., 2010. lme4: Mixed-Effects Modeling With R. Springer,Madison, WI.
Bates, D., Machler, M., Bolker, B., Walker, S., 2014. Fitting linear mixed-effects models using lme4. J. Stat. Software 1e48. https://doi.org/10.18637/Jss.V067.I01.
Blamire, A.M., Ogawa, S., Ugurbil, K., Rothman, D., McCarthy, G.,Ellermann, J.M., et al., 1992. Dynamic mapping of the human visualcortex by high-speed magnetic resonance imaging. Proc. Natl.Acad. Sci. U. S. A. 89 (22), 11069e11073.
Bobadilla Suarez, S., Love, B.C., November 2017. Measures of neuralsimilarity. In: Presented at the Society for Neuroscience, Washing-ton, D.C. Retrieved from: http://www.abstractsonline.com/pp8/#!/4376/presentation/18122.
Calhoun, V.D., Stevens, M.C., Pearlson, G.D., Kiehl, K.A., 2004. fMRIanalysis with the general linear model: removal of latency-induced amplitude bias by incorporation of hemodynamic deriva-tive terms. NeuroImage 22 (1), 252e257. https://doi.org/10.1016/j.neuroimage.2003.12.029.
Clark, V.P., Maisog, J.M., Haxby, J.V., 1998. fMRI study of face percep-tion and memory using random stimulus sequences. J. Neurophy-siol. 79 (6), 3257e3265. https://doi.org/10.1152/jn.1998.79.6.3257.
Clarke, A., Pell, P.J., Ranganath, C., Tyler, L.K., 2016. Learning warpsobject representations in the ventral temporal cortex. J. Cognit.Neurosci. 28 (7), 1010e1023. https://doi.org/10.1162/jocn_a_00951.
Coutanche, M.N., Thompson-Schill, S.L., 2012. The advantage of brieffMRI acquisition runs for multi-voxel pattern detection across runs.NeuroImage 61 (4), 1113e1119. https://doi.org/10.1016/j.neuroimage.2012.03.076.
Dale, A.M., Buckner, R.L., 1997. Selective averaging of rapidly pre-sented individual trials using fMRI. Hum. Brain Mapp. 5 (5),329e340. https://doi.org/10.1002/(SICI)1097-0193(1997)5:5<329::AID-HBM1>3.0.CO;2-5.
Davatzikos, C., Ruparel, K., Fan, Y., Shen, D.G., Acharyya, M.,Loughead, J.W., et al., 2005. Classifying spatial patterns of brain activ-ity with machine learningmethods: application to lie detection. Neu-roImage 28 (3), 663e668. https://doi.org/10.1016/j.neuroimage.2005.08.009.
Diana, R.A., Yonelinas, A.P., Ranganath, C., 2007. Imaging recollectionand familiarity in the medial temporal lobe: a three-componentmodel. Trends Cognit. Sci. 11 (9), 379e386. https://doi.org/10.1016/j.tics.2007.08.001.
Dimsdale-Zucker, H.R., Ritchey, M., Ekstrom, A.D., Yonelinas, A.P.,Ranganath, C., 2018. CA1 and CA3 differentially support sponta-neous retrieval of episodic contexts within human hippocampalsubfields. Nat. Commun. 9 (1), 294. https://doi.org/10.1038/s41467-017-02752-1.
Edelman, S., Grill-Spector, K., Kushnir, T., Malach, R., 1998. Towarddirect visualization of the internal shape representation space byfMRI. Psychobiology 26 (4), 309e321. https://doi.org/10.3758/BF03330618.
Etzel, J.A., Braver, T.S., 2013. MVPA permutation schemes: permuta-tion testing in the land of cross-validation. In: Presented at the3rd International Workshop on Pattern Recognition in Neuroimag-ing, Philadelphia, PA. https://doi.org/10.1109/PRNI.2013.44.
Etzel, J.A., Gazzola, V., Keysers, C., 2009. An introduction to anatomicalROI-based fMRI classification analysis. Brain Res. 1282 (Suppl. C),114e125. https://doi.org/10.1016/j.brainres.2009.05.090.
Georgopoulos, A.P., Schwartz, A.B., Kettner, R.E., 1986. Neuronal pop-ulation coding of movement direction. Science 233 (4771),1416e1419.
Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L.,Pietrini, P., 2001. Distributed and overlapping representations offaces and objects in ventral temporal cortex. Science 293 (5539),2425e2430. https://doi.org/10.1126/science.1063736.
Haynes, J.-D., Rees, G., 2005. Predicting the stream of consciousnessfrom activity in human visual cortex. Curr. Biol. 15 (14),1301e1307. https://doi.org/10.1016/j.cub.2005.06.026.
Heeger, D.J., Ress, D., 2002. What does fMRI tell us about neuronalactivity? Nat. Rev. Neurosci. 3 (2), 142e151. https://doi.org/10.1038/nrn730.
Huffman, D.J., Stark, C.E.L., 2017. The influence of low-level stimulusfeatures on the representation of contexts, items, and their mne-monic associations. NeuroImage 155 (Suppl. C), 513e529.https://doi.org/10.1016/j.neuroimage.2017.04.019.
Jonker, T.R., Dimsdale-Zucker, H.R., Ritchey, M., Clarke, A., Ranga-nath, C., 2018. Neural reactivation in parietal cortex enhances mem-ory for episodically-linked information (under review).
Kaneshiro, B., Perreau Guimaraes, M., Kim, H.-S., Norcia, A.M.,Suppes, P., 2015. A representational similarity analysis of the dy-namics of object processing using single-trial EEG classification.PLoS One 10 (8), e0135697. https://doi.org/10.1371/journal.pone.0135697.
Kiani, R., Esteky, H., Mirpour, K., Tanaka, K., 2007. Object categorystructure in response patterns of neuronal population in monkeyinferior temporal cortex. J. Neurophysiol. 97 (6), 4296e4309.https://doi.org/10.1152/jn.00024.2007.
Kriegeskorte, N., Cusack, R., Bandettini, P., 2010. How does an fMRIvoxel sample the neuronal activity pattern: compact-kernel or com-plex spatiotemporal filter? NeuroImage 49 (3), 1965e1976. https://doi.org/10.1016/j.neuroimage.2009.09.059.
Kriegeskorte, N., Goebel, R., Bandettini, P., 2006. Information-basedfunctional brain mapping. Proc. Natl. Acad. Sci. U. S. A. 103 (10),3863e3868. https://doi.org/10.1073/pnas.0600244103.
Kriegeskorte, N., Mur, M., Bandettini, P., 2008a. Representational sim-ilarity analysis - connecting the branches of systems neuroscience.Front. Syst. Neurosci. 2 https://doi.org/10.3389/neuro.06.004.2008.
Kriegeskorte, N., Mur, M., Ruff, D.A., Kiani, R., Bodurka, J., Esteky, H.,et al., 2008b. Matching categorical object representations in inferiortemporal cortex of man and monkey. Neuron 60 (6), 1126e1141.https://doi.org/10.1016/j.neuron.2008.10.043.
Kriegeskorte, N., Simmons, W.K., Bellgowan, P.S.F., Baker, C.I., 2009.Circular analysis in systems neuroscience: the dangers of doubledipping. Nat. Neurosci. 12 (5), 535e540. https://doi.org/10.1038/nn.2303.
Leutgeb, S., Leutgeb, J.K., Moser, M.-B., Moser, E.I., 2005. Place cells,spatial maps and the population code for memory. Curr. Opin.Neurobiol. 15 (6), 738e746. https://doi.org/10.1016/j.conb.2005.10.002.
Libby, L.A., Hannula, D.E., Ranganath, C., 2014. Medial temporal lobecoding of item and spatial information during relational binding inworking memory. J. Neurosci. 34 (43), 14233e14242. https://doi.org/10.1523/JNEUROSCI.0655-14.2014.
Macevoy, S.P., Epstein, R.A., 2009. Decoding the representation of mul-tiple simultaneous objects in human occipitotemporal cortex. Curr.Biol. 19 (11), 943e947. https://doi.org/10.1016/j.cub.2009.04.020.
McKenzie, S., Keene, C., Farovik, A., Blandon, J., Place, R.,Komorowski, R., Eichenbaum, H., 2016. Representation of mem-ories in the corticalehippocampal system: results from the applica-tion of population similarity analyses. Neurobiol. Learn. Mem.https://doi.org/10.1016/j.nlm.2015.12.008.
Misaki, M., Kim, Y., Bandettini, P.A., Kriegeskorte, N., 2010. Compari-son of multivariate classifiers and response normalizations forpattern-information fMRI. NeuroImage 53 (1), 103e118. https://doi.org/10.1016/j.neuroimage.2010.05.051.
Mumford, J.A., Davis, T., Poldrack, R.A., 2014. The impact of studydesign on pattern estimation for single-trial multivariate patternanalysis. NeuroImage. https://doi.org/10.1016/j.neuroimage.2014.09.026.
Mumford, J.A., Poline, J.-B., Poldrack, R.A., 2015. Orthogonalization ofregressors in fMRI models. PLoS One 10 (4), e0126255. https://doi.org/10.1371/journal.pone.0126255.
Op de Beeck, H.P., 2010. Against hyperacuity in brain reading: spatialsmoothing does not hurt multivariate fMRI analyses? NeuroImage49 (3), 1943e1948. https://doi.org/10.1016/j.neuroimage.2009.02.047.
Pillai, J.J., Zaca, D., 2011. DTI and fMRI: review of complementarytechniques. In: Faro, S.H., Mohamed, F.B., Law, M., Ulmer, J.L.(Eds.), Functional Neuroradiology: Principles and ClinicalApplications. Springer, New York, pp. 803e830.
27. REPRESENTATIONAL SIMILARITY ANALYSES: A PRACTICAL GUIDE FOR FUNCTIONAL MRI APPLICATIONS524
Viswanathan, S., Cieslak, M., Grafton, S.T., 2012. On the geometricstructure of fMRI searchlight-based information maps. Quantita-tive Biology. Retrieved from: http://arxiv.org/abs/1210.6317.
Walther, A., Nili, H., Ejaz, N., Alink, A., Kriegeskorte, N.,Diedrichsen, J., 2015. Reliability of dissimilarity measures formulti-voxel pattern analysis. NeuroImage. https://doi.org/10.1016/j.neuroimage.2015.12.012.
Worsley, K.J., Taylor, J.E., 2006. Detecting fMRI activation allowingfor unknown latency of the hemodynamic response. NeuroImage29 (2), 649e654. https://doi.org/10.1016/j.neuroimage.2005.07.032.
Yonelinas, A.P., 2002. The nature of recollection and familiarity: a re-view of 30 years of research. J. Mem. Lang. 46 (3), 441e517.https://doi.org/10.1006/jmla.2002.2864.
Zeithamova, D., de Araujo Sanchez, M.-A., Adke, A., 2017. Trial timingand pattern-information analyses of fMRI data. NeuroImage.https://doi.org/10.1016/j.neuroimage.2017.04.025.
