The effect of encoding strategy on the neural correlates of memoryfor faces
Lori J. Bernstein, Sania Beig, Amy L. Siegenthaler, Cheryl L. Grady *Rotman Research Institute, Baycrest Centre for Geriatric Care, Uni�ersity of Toronto, 3560 Bathurst Street, Toronto, Ont., Canada M6A 2EI
Received 6 March 2000; received in revised form 4 December 2000; accepted 23 March 2001
One aspect contributing to episodic memory perfor-mance (i.e. conscious recollection of particular episodesor events that occurred in a person’s experience (seeRef. [60]) is the degree to which the new material hasbeen incorporated into existing knowledge. Craik andcolleagues [12,14] have shown that individuals are bet-ter at remembering an item when the item has beenprocessed in a way that promotes abstract, semantic, orassociative organization (deep encoding) than when ithas been processed superficially (shallow encoding). Itis believed that memory retention depends on the extent
to which the learner has developed a system to analyzeand enrich new information. Deeper levels of process-ing are concerned with pattern recognition and extrac-tion of meaning, and the deeper the level, the greaterdegree of semantic or abstract analysis, which yieldsmore robust memory traces. The levels of processingeffect is robust, being observed for words (e.g. Ref.[12]) and pictures of objects (e.g. Refs. [13,24], but seeRef. [30]) as well as for faces (e.g. Ref. [7]). Althoughthe levels of processing effect is much stronger andmore consistent for words than for less verbal material(pictures and faces), it has been found for all thesetypes of stimuli.
Cognitive psychological theories, lesion studies, andimaging experiments support the notion that encodingand retrieval are dissociable forms of memory, each ofwhich can be differentially affected by a variety of
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–98 87
factors, such as brain damage or disease (e.g. Refs.[4,17,53], for review). In addition, imaging studies haveshown that the neural correlates of encoding and recog-nition can be differentially affected by encoding instruc-tions such as those promoting different levels ofprocessing. For instance, evidence from positron emis-sion tomography (PET) reveals different patterns ofregional cerebral blood flow (rCBF) during incidentalshallow and deep encoding of words and pictures incontrast with intentional learning (i.e. when partici-pants are explicitly told to try to learn the items)[24,33,52]. In one of these studies [24], deep processingengaged left anterior medial prefrontal cortical areas,left medial temporal regions, and bilateral posteriorextrastriate cortex. Intentional learning was associatedwith left ventrolateral prefrontal cortex, left premotorcortex, and bilateral ventral extrastriate cortex. Nybergand colleagues [46] found more left medial temporalactivity during retrieval when the items had been stud-ied under incidental deep task instructions. Althoughnone of these studies compared brain activity duringboth encoding and retrieval as a function of encodingmanipulation, the evidence suggests that encoding andretrieval processes are subserved by different systemswhich can both be modulated by encoding strategy.
The present study was designed to examine whetherthe manner in which unfamiliar faces are studied influ-ences the neural networks involved in encoding andrecognition. Although the neural correlates of face per-ception and memory have been investigated quite ex-tensively (e.g. Refs. [10,11,21–23,26,27,29,31,32,35,39,56]), no study has yet investigated the neural correlatesof different types of encoding strategies using faces asstimuli. We contrasted several strategies which areknown to influence memory — two incidental ones(deep and shallow) and an intentional one. It is possiblethat different encoding strategies may engage the samememory system, with participating regions being moreactive during one condition in comparison with an-other. Alternatively, different strategies of encodinginformation may engage different memory systems thatreflect individual strategies in encoding the item, andthese systems could be anatomically dissociable. Al-though the reviewed findings are consistent with thehypothesis that different encoding strategies may en-gage different neural circuits when readily nameableobjects or words are used as stimuli, it is not knownwhether such effects on neural activity can be general-ized to other types of stimuli, such as faces.
In examining the effects of encoding strategy onbrain activity we have focused on the roles of twospecific regions of cortex. One region is left prefrontalcortex, which is reported to be active during encodingof both objects and words [24,35] and sometimes duringface encoding (e.g. Ref. [27]), but not always [35,41]. Itwas hoped that the present study would confirm a role
for left prefrontal gyrus in encoding of unfamiliar faces.If the left prefrontal cortext is indeed a general encod-ing area, then it should be active during all encodingconditions, regardless of strategy. The second area ofinterest is the fusiform gyrus. Evidence from neuropsy-chological (for review see Ref. [40]) and neuroimaging(e.g. Refs. [25,26,31,49,56]) studies suggest that thefusiform is specialized for face and/or object percep-tion. However, it remains unclear if activation of thisarea reflects an automatic processing of faces, beingactive regardless of what type of face task is required ofthe participant, or whether it can be affected by thetype or extent of processing in which the person isengaged (e.g. Refs. [36,65]). If the fusiform is sensitiveto controlled processes then we should observe differ-ences in rCBF during different levels of encoding orrecognition, or perhaps in the relation of activation inthis area to performance.
2. Materials and methods
2.1. Participants
Twelve right-handed individuals (mean age 23.4years; range 21–28; six male, six female) participatedfor pay. They provided informed consent, and theexperiment was conducted according to University ofToronto and Baycrest Centre for Geriatric Care guide-lines. None of these participants had any sign of brainabnormality based on structural MRIs. In addition tothese 12 individuals, ten age and educated-matchedparticipants were given the face encoding and recogni-tion tasks (described below), but were not scannedduring any of the conditions. Through a screeningquestionnaire, it was determined that all twenty-twoparticipants had normal or corrected-to-normal visionand none had a history of neurological or psychiatricdisease or brain injury. The average education of par-ticipants was 17 years. Every participant performedwithin normal limits on several neuropsychologicaltests: the Mill–Hill Vocabulary Test [50], Beck Depres-sion Inventory [5], Mini Mental Status Examination[19], and F-A-S verbal fluency [6].
2.2. Procedure
Eighty-eight faces were obtained from a high schoolyearbook (also used in Ref. [26]). All were black-and-white photograph quality (Grayscale), were cropped toremove hair and clothing, and portrayed mostly Cau-casians. The faces were presented against a black back-ground. Each face subtended approximately 4° of visualangle with participants approximately 60 cm from thecomputer monitor.
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–9888
All stimuli were presented on a monitor attached toa PC running Superlab software (Cedrus, Phoenix,AZ). During the encoding conditions, 24 faces (halfwomen, half men) were presented one at a time for twoseconds with an inter-stimulus interval (ISI) of 2 s.Between trials, a white fixation cross against a blackbackground was presented and participants were in-structed to look at it at all times. Responses for alltasks were button presses made with the right indexfinger for the left mouse button or right middle fingerfor the right mouse button.
There were three encoding conditions. The faces weredivided into three lists that were assigned to each of thethree encoding conditions in a counterbalanced way. Inone condition, participants were asked to judge whethereach face was pleasant or unpleasant (‘deep’ level ofprocessing), pressing one of two mouse buttons toindicate their decision. This strategy was chosen be-cause it has been shown to be effective in improvingrecognition performance (e.g. Refs. [7,58]) even thoughit may also be considered a decision which requires anemotional judgment. Although there was no right orwrong answer for the pleasantness of each face, a pilotexperiment showed that approximately half of the facesin each condition were judged as pleasant and half asunpleasant. In a second condition, participants wereasked to report whether the face was oriented primarilyto the right or to the left (‘shallow’ level of processing),indicating their decision by pressing one of two mousebuttons. Half of the faces were oriented to the right andthe other half to the left. This strategy was intended torequire a judgment utilizing perceptual processing,analogous to the usual verbal shallow tasks such asdeciding whether there is a letter ‘a’ in a word. In thedeep and shallow conditions, participants were not toldthey would be tested later for their memory of thesefaces; thus, these two conditions are considered inciden-tal encoding tasks. During the third encoding condi-tion, participants were told to study the faces becausememory for them would be tested later but no specificstrategy was provided (‘intentional learning’). Partici-pants were instructed to press the left button for eachface during the encoding condition. Questioning afterthe experiment revealed that most participants couldnot generally articulate the strategies they used duringmemorization, or they claimed not to use any onestrategy in particular. When a strategy was specified, itwas usually looking for distinctive features of the faceor thinking of a person of whom the unfamiliar facereminded them. Each participant performed all threeencoding conditions; the order of these conditions wascounterbalanced across participants. There were at leasttwo practice trials given for each type of encodingcondition to allow participants to become accustomedto the timing and to make sure they understood theinstructions. If a participant needed additional practice
items, the same two-item practice trials were shownagain. The faces shown during the practice trials werethe same in all three encoding conditions and were nota part of the experimental set of faces.
After all three encoding conditions were completed, arecognition test was given, divided into three separatelists. In each recognition list, participants saw 32 facesone at a time, half of which were presented at encoding(‘old’ faces), and half of which had not been seenpreviously (‘new’ faces). Participants judged whetherthe face was one they had seen before in any of theconditions. The recognition test was divided into threeseparate scans so that brain activity during recognitionof faces encoded under different instructions could becompared. The recognition lists were presented in thesame order in which encoding tasks had been per-formed. That is, if a participant performed the shallowencoding task first, then the first recognition list wouldcontain faces from the shallow encoding task. As in theencoding conditions, the stimuli were presented for twoseconds with an ISI of 2 s. Participants had as long asfour seconds to make a response, and they were told tomake their best guess if they were not sure. No empha-sis was made on speed of response.
Recognition accuracy was calculated by computingthe proportion of hits (number of ‘old’ responses to oldfaces divided by the total number of old faces, in thiscase 16) minus the proportion of false alarms (numberof ‘old’ responses to new faces divided by the totalnumber of new faces, or 16). For mean reaction times(RT), all correct trials (responding ‘old’ to old facesand ‘new’ to new faces) were included. The ten age-matched participants, who were not scanned, and thetwelve participants who were scanned, were not signifi-cantly different on any of the behavioral measures (nogroup by condition interaction, F�1 in all cases). Toincrease the power, therefore, the data from all 22participants were combined and were analyzed by usinga repeated measures ANOVA with encoding strategy asthe repeated measure. Both sets of data (PET partici-pants or all participants together) are reported in theresults section.
2.3. PET
Changes in rCBF were measured using PET duringeach of the six conditions (three encoding, three recog-nition). Before the first encoding and after the lastrecognition scan, baseline conditions were administeredin which participants were instructed to press the leftmouse button to 24 distorted and skewed faces pre-sented with the same timing as the experimental tasks.Although these distorted faces were not recognized asfaces by a separate group of pilot observers, some ofthe PET participants reported in post-scan questioningthat they realized during the second baseline scan (i.e.
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–98 89
the second time they saw them after the encoding andrecognition scans), that a few of the control stimuliwere distorted faces.
PET scans, with injections of 40 mCi [15O]water eachand separated by 11 min, were performed on a GEMSPC2048-15B tomograph, which has a reconstructed res-olution of 6.5 mm in both transverse and axial planes.This tomograph allows 15 planes, separated by 6.5 mm(center to center), to be acquired simultaneously. Emis-sion data were corrected for attenuation by means of atransmission scan obtained at the same levels as theemission scans. A thermoplastic mask that was moldedto each participant’s head and attached to the scannerbed helped to minimize head movement during thescans. Each task started 15 s before isotope injectioninto a forearm vein through an indwelling catheter andcontinued throughout the 1-min scanning period. Eightscans were obtained in each participant during theexperiment, one for each encoding and recognitioncondition, and the two baseline scans.
Integrated regional counts were used as an index ofrCBF. For each participant, all images were first regis-tered to the initial scan to correct for head motionduring the experiment using AIR [66]. Then each imagewas spatially transformed to an rCBF template con-forming to Talairach and Tourneaux [59] stereotaxicspace using SPM95 (SPM95, Wellcome Department ofCognitive Neurology, London) implemented in Matlab(Mathwords, Sherborn, MA). After transformation,each image was smoothed with a 10-mm isotropicGaussian filter.
Ratios of rCBF to global cerebral blood flow (CBF)within each scan for each participant were computedand analyzed using a multivariate analysis, partial leastsquares or PLS (for a more complete description seeRef. [43]), to identify spatially distributed patterns ofbrain activity related to the different task conditions.PLS operates on the covariance between brain voxelsand the experimental design to identify a new set ofvariables (so-called latent variables or LVs) that opti-mally relate the two sets of measurements. Brain activ-ity patterns are displayed (e.g. Fig. 1) as singularimages which show the brain areas that covary with thecontrast or contrasts that contribute to each LV. Eachbrain voxel has a weight, known as a salience, that isproportional to these covariances, and multiplying therCBF value in each brain voxel for each participant bythe salience for that voxel, and summing across allvoxels gives a score for each participant for each condi-tion on a given LV. Comparison of these scores acrossconditions indicates the effect of task on brain activity.The significance for each LV as a whole was assignedby using a permutation test [15,43]. Because saliencesare derived in a single analytic step, no correction formultiple comparisons of the sort done for univariateimage analyses is required. In addition to the permuta-
tion test, a second and independent step in PLS analysisis to determine the reliability of the saliences for thebrain voxels characterizing each pattern identified bythe LVs. To do this, all saliences were submitted to abootstrap estimation of the standard errors [16,54].Peak voxels with a salience to standard error ratiogreater than or equal to 2.0 (corresponding to a 95%confidence interval [54]) were considered reliable. Localmaxima for the brain areas with reliable saliences oneach LV were defined as the voxel with a salience/SEratio higher than any other voxel in a 2-cm cubecentered on that voxel. These maxima are reported interms of brain region, or gyrus, and Broadman area(BA) as defined in the Talairach and Tournoux atlas[59].
Fig. 1. The brain areas that show differential activity related to faceencoding and recognition are shown. The image at the top of thefigure presents activations imposed on standard MRI slices rangingfrom z= −28 (top left) to z=40 (bottom right) [59], separated by4-mm intervals. Positive saliences are indicated in white and representareas where there was greater activity during encoding than duringrecognition. Black areas represent areas where there was greateractivity during recognition than during encoding. Right is right onthe image. See Table 2 for local maxima of these regions. The graphat the bottom of the figure shows the mean brain scores on this LV.Positive mean brain scores were found in the encoding conditions,where activity was increased in the brain regions shown in white (i.e.those with positive salience on the LV). Negative mean brain scoreswere found in the recognition conditions, where activity was in-creased in the brain areas shown in black (i.e. those with negativesalience on the LV).
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–9890
The PET data were submitted to two types of PLSanalysis. The first, Task PLS, reveals the patterns ofcovariation that occur across the task conditions asdescribed above. Three task analyses were performed:one compared the three encoding and the three recogni-tion conditions, a second contrasted the encoding con-ditions and the average of the control conditions, and athird compared the three recognition tasks and theaverage of the control conditions. The second type ofPLS analysis, Behavioral PLS, reveals patterns of co-variation between brain activity and the behavioraldata (i.e. accuracy) (e.g. Refs. [42,55]). Task PLS re-veals which areas are active while performing a particu-lar task, and can tell us that conditions 1 and 2 haveareas in common which are in contrast to conditions 3and 4. However, this analysis reveals nothing abouthow those areas are related to performance, only thatthose areas are active in those conditions. BehavioralPLS supplements the Task PLS because it can tell usnot only that an area is used in a particular conditionbut that the area is correlated with good (or poor)recognition performance. It is quite possible for there tobe some areas that are active during a particular taskbut which are not related to individual differences inperformance on that task. We carried out this behav-ioral analysis using the accuracy measures (hits-falsealarms) of the 12 participants and all six of the memorytasks (encoding and recognition) to examine similaritiesand/or differences in brain/behavior correlations acrossconditions.
3. Results
3.1. Beha�ioral data
During the encoding tasks, participants were signifi-cantly faster in making judgments about the orientationof the face than the pleasantness of the face (831 vs.1025 ms, F(1,11)=7.36; P�0.02). Although a speededresponse was not emphasized, the RT difference sug-gests that participants were not engaged to the sameextent in the two conditions. This indicates that thepleasantness decision required more effort or otheradditional processing than judging face orientation.
The recognition task was relatively difficult (seeTable 1). There was a significant effect of encodingstrategy (F(2,40)=4.65, P�0.05) on recognition accu-racy. A Helmert contrast revealed a significant differ-ence between recognition of deeply encoded facesversus those from the shallow encoding and intentionallearning conditions (F(1,21)=8.95, P�0.01) and didnot show a significant difference between the learn andshallow encoding conditions (F�1). Reaction times inthe recognition conditions did not differ significantlyfrom each other.
Numbers in ‘()’ are means for PET alone.a The accuracy measures are shown as proportion hits minus
proportion false alarms such that 0 is chance performance and 1 isperfect performance.
3.2. PET data
3.2.1. Task effectsThe task analysis contrasting encoding and recogni-
tion revealed significant patterns of activity (Fig. 1),which showed a main effect of encoding versus recogni-tion (P�0.001), identifying brain regions that distin-guished the three encoding conditions from the threerecognition conditions regardless of the type of encod-ing instructions. Areas of increased rCBF during en-coding included left ventral and medial frontal areas,bilateral inferior and middle temporal areas, left amyg-dala, bilateral fusiform, and right inferior parietal andangular gyri. Areas of increased activity during recogni-tion included right middle and superior frontal areas, aleft inferior frontal area (superior to the one activeduring encoding), left posterior cingulate gyrus, rightsuperior and medial parietal (precuneus) cortex, andbilateral lingual gyri. Maximal coordinates of theseregions are shown in Table 2. Because these areas aremore (or less) active during encoding or recognition,regardless of encoding strategy and resulting overalllevel of memory performance, they represent areas thatappear necessary for face memory in general.
Because there was a main effect of encoding versusrecognition, it seemed sensible to perform two addi-tional task PLS analyses which compared encoding andrecognition separately with the average of the twobaseline scans (see Fig. 2A and B). When includingencoding and the average baseline in the analysis, onesignificant pattern of activation was identified that dif-ferentiated deep encoding from baseline (P�0.01).Deep encoding was associated with increased activity inthe left amygdala, anterior cingulate gyrus, and bilat-eral inferior frontal cortex, whereas the baseline condi-tions had relatively more activity in bilateral inferiortemporal and lateral parietal areas (Fig. 2A). Coordi-nates are listed in Table 3. The shallow and learnconditions contributed little to this pattern of activity,as indicated by the mean brain scores for these condi-tions (which were near zero, Fig. 2A). Thus, the areasidentified by this LV that showed increased activityduring the pleasantness judgement (e.g. left amygdala,anterior cingulate gyrus and bilateral inferior frontal
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–98 91
cortices) and those with increased activity during thecontrol task (e.g. temporal and occipital regions)showed essentially equivalent activity during the shallowencoding and intentional learning conditions. In addi-tion, three of the regions with greater activity duringdeep encoding were similar in location to areas identifiedin the previous analysis as having greater activity duringencoding in general. These three regions were left infe-rior prefrontal cortex, left amygdala, and right thalamus(compare Tables 2 and 3).
In contrast, when the analysis included all recognitionconditions and the average baseline, all recognitionconditions showed the same pattern compared to base-line (P�0.01). This pattern was characterized by bilat-eral frontal activation during recognition and greateroccipital and occipito-temporal area activation duringthe baseline tasks (Fig. 2B). Coordinates are listed in
Table 4. In summary, the particular strategy of faceencoding had an effect on brain activity during thosetasks, particularly during deep encoding. For recogni-tion, on the other hand, the manner in which faces wereencoded did not appear to differentially distinguishpatterns of brain activity; instead, a general recognitionpattern was identified in comparison to viewing non-facestimuli.
3.2.2. Brain–beha�ior correlationsWhile the task PLS analyses reveal different brain
patterns associated with some conditions compared withothers, they cannot reveal how the brain activation isrelated to behavioral performance. The behavioral PLS,however, is able to examine such correlations. In doingso, one significant pattern was found, shown in Fig. 3,with maximum coordinates listed in Table 5. This pat-tern distinguished the deep memory conditions fromboth the shallow and learn conditions (P�0.001). Bet-ter recognition performance for deeply-encoded faceswas associated with increased activity in the left inferiortemporal, left inferior and medial frontal, and bilateralparietal regions during both encoding and recognition(Table 5). Alternatively, if these same areas were activeduring the other two encoding and recognition condi-tions, then recognition performance was poorer. Incontrast, good recognition performance for shallowly-encoded and intentionally-learned faces depended uponactivation in the left cerebellum, bilateral posteriorfusiform, bilateral hippocampi and amydalae, and rightinferior temporal areas during both encoding and recog-nition (Table 5). If these latter areas were active duringdeep encoding and deep recognition, recognition for thedeeply-encoded faces was poorer.
Finally, we were curious whether there were any areasthat showed differential activity during encoding v.recognition and that were also correlated with behavior.To do this, the overlap between the reliable regions fromthe task analysis and the behavioral analysis was deter-mined by multiplying the saliences from the two LVs(Fig. 4) [47]. Several areas with increased activity duringencoding were found to be related to performance butduring different conditions. The left amygdala and bilat-eral fusiform gyri were active across encoding tasks butwere correlated with better recognition performanceafter shallow encoding and learning. Several regions inthe temporal lobe and a region in medial dorsal frontalcortex also were active in all encoding tasks but werecorrelated with better recognition for deeply encodedfaces. There were also some areas with increased activityduring all the recognition tasks which were found to berelated differentially to performance. Activity in areas inmiddle temporal regions, right inferior frontal and leftmiddle frontal areas was correlated with better perfor-mance in the deep recognition conditions (compare Fig.1, Fig. 3A and Fig. 4).
Table 2Local maxima of areas with activity differentiating encoding andrecognition of faces
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–9892
Fig. 2. (A) The brain areas where deep encoding was distinguished from the average baseline conditions are shown. White areas are those wherethere was greater activity during the deep encoding condition and black regions are those with more activity during the baseline compared to deepencoding. See Table 3 for local maxima of these regions. The graph to the right of the figure shows the mean brain scores on this LV. The meanscores for the shallow encoding and intentional learning conditions are near zero, indicating that these conditions contribute little to this patternof activity. (B) The brain areas where recognition was distinguished from the average baseline conditions are shown. White areas are those wherethere was greater activity during all three recognition conditions, and black regions are those with more activity during the baseline compared torecognition. See Table 4 for local maxima of these regions. The graph to the right of the figure shows the mean brain scores on this LV.
4. Discussion
The primary pattern in the task PLS analysis distin-guished encoding and recognition regardless of theencoding strategy. During all three encoding strategiesparticipants employed a mainly ventral network ofregions in frontal and in temporal cortex, including leftinferior frontal and bilateral medial temporal areas.This is consistent with previous studies which haveshown increased activity in ventral occipital, temporal,and left prefrontal regions during face encoding [2,28].Although there was not a main effect of strategy acrossthe memory conditions when encoding was comparedto recognition, when the three encoding conditions werecompared to the average of the baseline conditions,there was a significant difference between the deepencoding condition and baseline. This pattern showedthat the left amygdala, anterior cinguate gyrus, andbilateral inferior frontal areas were more active duringdeep encoding. It is not surprising that the left amyg-
dala was more active during the deep encoding condi-tion, as assessment of pleasantness is a type ofemotional judgment, and the amygdala is thought to beimportant in processing of emotional stimuli and facialexpressions (e.g. Refs. [1,8,37,44,64]). As to whetherthese areas are more associated with deep processing asopposed to emotional processing per se cannot bedetermined with the current study. An experimentwhich includes contrasting types of ‘deep’ encodingstrategies – some involving emotional judgments, somenot-would be one possible way of examining this. Re-gardless of the specific mechanism involved, the relativelack of differential brain activity related to the threeencoding strategies contrasts with the previous resultsof Grady et al. [23] who found distinct changes in rCBFthat characterized deep and shallow encoding as well asintentional learning of words and pictures (see also Ref.[34]). The discrepancy between our finding and that ofGrady et al. [23] may have arisen because our manipu-lation produced only a modest levels-of-processing ef-
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–98 93
fect on behavioral performance, although the magni-tude of this improvement was consistent with previousstudies (e.g. Refs. [7,58]). However, the brain activityresults were consistent with the behavioral results inthat the deep encoding condition produced both adistinct brain activity pattern and better recognitionperformance. It is also possible that a given strategy forencoding faces does not trigger the same neural systemsas those that are engaged when using a similar strategyfor encoding words and nameable pictures. In compar-ing previous studies, results indicate that memory forunfamiliar faces is not as affected by study instructionsas memory for words and easy-to-name pictures (e.g.Refs. [7,12,24]). This may relate to the fact that somestimuli, in this case unfamiliar faces, cannot be encodedsemantically as easily as others (see Ref. [3]). It shouldbe noted that although levels of processing effects on
Table 4Local maxima of areas with activity differentiating recognition andbaselines
BAHem ZRegion, gyrus YX
Recognition�a�erage of baselinesFrontal
Inferior 040−4446/10L18−44 289LMiddle/Inferior
R 8/9 42 2812Middle/Inferior04228Middle 10R
20R 9 20 54Superior32L 4 −36 −6Precentral
4RPrecentral 28−1648Temporal
Middle R 39 32 −64 24Occipital
R 19Fusiform 36 −68 −16Hippocampal R 19 26 −50 4Cingulate L 24 −4 −6 32
L 18Middle −42 −86 16Middle R 18 24 −90 16Fusiform R 19 40 −64 −4
L 18Cuneus −2 −92 12Insula R 34 6 8
brain activity have been found at both encoding (e.g.Refs. [24,35]) and during retrieval (e.g. Refs. [46,52]),overall it appears that the effects of encoding strategygenerally have stronger brain effects during encodingthan during recognition.
The task PLS analysis also revealed that in contrastwith encoding, recognition of faces appears to involvemore dorsal regions, including bilateral prefrontal, rightparietal and premotor areas. A number of studies havereported enhanced rCBF in right frontal cortex duringrecognition (e.g. Refs. [51,57,62]). In contrast, encodingoften has been associated with increased rCBF in theleft frontal region (hemispheric encoding/retrievalasymmetry, or HERA) (e.g. Ref. [61]). In the currentstudy, encoding was associated with enhanced rCBFpredominantly in the left inferior frontal areas andrecognition was associated with increases in rCBF in
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–9894
bilateral frontal regions, more extensively on the rightside, particularly in anterior frontal cortex. This patternof activity, although not conforming to a stringentHERA prediction, is consistent with the idea of somelateralization of different memory operations. Further-
more, this result is supported by evidence that encodingof faces and other stimuli such as pictures can engage amore symmetrical set of regions (e.g. Refs. [24,35]). Infact, our results shed additional light on prefrontalactivity during encoding and suggest that there are two
Fig. 3. (A) Brains areas where activity was correlated with performance levels as a function of particular conditions are shown. White areas arethose which are associated with good performance if active during encoding and recognition of faces studied under shallow encoding or intentionlearning instructions, and poor performance if active during deep encoding and recognition. Black areas are the opposite; namely, increasedactivity in these areas is associated with good performance during deep encoding and recognition, and with poor performance during shallow andintentional learning conditions. (B) Plots of brain scores from the PLS analysis and accuracy for each of the encoding (top three panels) andrecognition conditions (bottom three panels). During the shallow and learning conditions (two left and two center panels) there are positivecorrelations between scores and accuracy indicating that increased activity in brain regions shown in white (with more positive salience) arecorrelated with better recognition performance. Conversely, during the deep conditions (two right panels), the correlations are negative indicatingthat increased activity in the regions with negative salience, shown in black, is associated with better performance.
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–98 95
Table 5Local maxima of areas where activity is differentially correlated withperformance in memory tasks
BA XRegion, gyrus YHem Z
Positi�e correlation with performance in shallow and intentionallearn conditions, negati�e correlation with performance in deepconditionsOccipital
LFusiform 19 −20 −72 −20Fusiform 18L −38 −84 −12
18 24R −82Fusiform −16LCuneus 18/19 −4 −82 28
Lingual 18R 6 −70 0Temporal
21Middle 50R −40 −438 −34Superior 8L −20
−20L −32Hippocampus −16RHippocampus 22 −14 −20
−22Amygdala −4L −1628R −6Amygdala −16
LCerebellum −8 −48 −16Midbrain L −12 −14 −16
Positi�e correlation with performance in deep conditions, negati�ecorrelation with performance in shallow and intentional learnconditionsOccipital
In the current study, the task PLS analysis revealedenhanced rCBF in the fusiform gyrus, mainly in middleregions, during encoding compared to the recognitionconditions. This is consistent with many studies whichhave shown increased rCBF in this region for tasksinvolving the processing of faces (e.g. Refs. [25,28,31]).The fact that the fusiform gyrus was differentiallyactive during encoding and recognition suggests thatthis area can be modulated by controlled processes.That is, although the fusiform may indeed be special-ized for face processing, it is not necessarily involved inthe same manner for every kind of task. This indicatesthat it plays a role in perceiving the face (processing thefeatures, for instance), and further that this processingcapacity is engaged to a greater degree during initialpresentation of the face than during repeatedpresentations.
One possible explanation for why there is more mid-dle fusiform activation during encoding than recogni-tion is that the difference reflects an effect of attention.That is, it may be that the encoding condition is moreperceptually demanding than recognition, and as such,more activation in the fusiform is observed (e.g. Ref.[65]). Another possibility is that more activity duringencoding could actually indicate a reduction in activityduring recognition, consistent with monkey and humanwork showing that some brain areas are less active thesecond time a stimulus is encountered as compared withthe first time, perhaps reflecting object memory (e.g.Refs. [18,38,63]). Consistent with this interpretationwas our finding of an additional fusiform area that hadgreater activity during the baseline conditions com-pared to recognition (coordinates: 30, −48, −12), acontrast that also compares novel to more familiaritems. This fusiform region with reduced activity duringrecognition compared to baseline was similar in loca-tion to the midfusiform region that was active duringface encoding (coordinates: 14, −42, −12). However,the fusiform results cannot be explained solely in termsof such a general novelty effect because there also wasa posterior fusiform area that was more active duringrecognition than during the baseline (coordinates: 36,−68, −16), where the non-face stimulus patterns arepresumably most novel. Thus, it may be that differentareas of the fusiform gyrus perform different functions.For instance, it may be that the middle fusiform ismore related to processing novel stimuli, whether thosebe newly seen faces or non-face stimuli, whereas poste-rior fusiform is more related to the process of recogniz-ing previously seen faces.
The observed effects of deep versus shallow encodingon recognition performance for faces are consistentwith previous behavioral studies (e.g. Ref. [7]) showingimproved performance when material has been pro-cessed at a deeper, semantic level. At the neural level,according to our behavioral PLS analysis, participants
prefrontal regions that are more active during faceencoding than during recognition but that have differ-ent laterality. One is a ventral prefrontal area in the lefthemisphere (coordinates: −48, 42, 12, see Table 2 andFig. 1) that appears to be important during encodingregardless of the type of stimulus, whether verbal orvisual (e.g. Refs. [28,48,57]). However, there is anotherprefrontal region that is more dorsal and posterior tothe first region whose laterality appears to be a functionof the type of stimulus that is encoded. This area showsmore right hemisphere activation for visual materialsuch as faces, and more left hemisphere activation forverbal material (e.g. Refs. [35,41]). We also found thisprefrontal region to be bilaterally active, although moreon the right, during encoding, particularly during thedeep encoding condition (coordinates: 44, 16, 24, seeTable 3 and Fig. 2A).
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–9896
were best at recognizing faces studied under deep en-coding instructions when rCBF was increased in leftinferior temporal and frontal areas and in bilateralmiddle temporal and parietal cortices. Some of theseregions are involved in semantic processing of wordsand pictures, for example, left inferior temporal and leftfrontal areas (e.g. Refs. [9,20,24]). This suggests thatsome ‘semantic’ areas are also important for deep en-coding of faces even if the actual deep processing taskis different. On the other hand, increased activity in theleft amygdala was related to better performance in theshallow and learn conditions, consistent with otherreports of a role for this region in implicit processing ofemotion in faces (e.g. Refs. [45,64]. In addition, goodrecognition performance for shallowly-encoded faceswas observed when there was increased blood flow inbilateral fusiform, medial temporal, and right inferiortemporal areas. The same pattern was true for theintentionally-encoded faces. Thus, not only is theredifferential fusiform activation during encoding andretrieval, but we also found that posterior fusiformactivation is differentially correlated with performancein different memory strategy conditions. Interestingly,these correlations between brain activity and recogni-tion accuracy were found in posterior regions of thefusiform gyrus, and not in the midfusiform gyrus, ashas been reported by Kuskowski and Pardo [36]. Dif-ferences between our study and theirs that could ac-count for the discrepant results include differentanalyses (multivariate PLS versus univariate correla-
tions), and the fact that they scanned participants onlyduring encoding and not during recognition. Our datasuggest that activation of the fusiform gyrus during aface-processing task does not necessarily ensure goodtask performance. Posterior fusiform activity appearsto be more related to perceptual conditions than deep(emotive or semantically encoded) conditions whereasmidfusiform activity may be unrelated to performancelevel. This result underscores the importance of examin-ing directly the relation between brain activity andbehavior. An analysis of task effects shows what areasare active during the condition regardless of how theseregions are related to performance, whereas a behav-ioral analysis reveals which areas are specifically corre-lated with individual differences in performance. Ourresults clearly show that a region may be active duringa number of task conditions (as the fusiform wasduring encoding in general), but nevertheless may con-tribute differently to task performance in these condi-tions (compare Fig. 1 and Fig. 4).
In conclusion, the results have highlighted severalimportant aspects of face memory. First, encoding andrecognition of unfamiliar faces involve different net-works of brain activity. Second, there is an effect ofencoding strategy on blood flow during encoding offaces, mainly during deep encoding. There is no effectof encoding strategy on blood flow during recognition.Third, the effect of different encoding strategies onbrain activity is consistent with the behavioral effects,namely, that recognition of deeply encoded faces is
Fig. 4. Areas of overlap between those differentiating encoding and recognition (Fig. 1) and those correlated with behavioral performance (Fig.3A). Regions with a threshold �4 are shown. In this overlap image areas that are the same color in Figs. 1 and 3 (i.e. both white or both black)are shown in white, and those areas with different colors in Figs. 1 and 3 are shown in black. White arrows point to areas with greater activationduring encoding than during recognition and where increased activity was correlated with better performance in shallow and intentional learningconditions. Black arrows point to areas with greater activation during encoding than during recognition and where increased activity wascorrelated with better performance in the deep conditions. Black circles indicate areas with greater activation during recognition than duringencoding and where activity was correlated with better performance in the deep conditions.
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–98 97
greater than those encoded in the shallow and learnconditions and brain activity during deep encoding isdistinct from that in the other two encoding conditions.Finally, brain areas that are active across all encodingtasks may nevertheless be related differently to individ-ual differences in memory performance due to encodingstrategy.
Acknowledgements
The authors would like to thank the staff of the PETCentre at the Centre for Mental Health and Addiction,University of Toronto, for their technical assistance incarrying out this experiment, and Dr Claude Alain forcomments on an earlier version of the manuscript. Thiswork was supported by the Ontario Mental HealthFoundation, the Canadian Institutes of Health Re-search, and the Alzheimer Society of Canada.
References
[1] Aggleton JP, editor. The Amygdala: Neurobiological Aspects ofEmotion, Memory, and Mental Dysfunction. New York: Wiley,1992.
[2] Andreasen NC, O’Leary DS, Arndt S, et al. Neural substrates offacial recognition. Journal of Neuropsychiatry and Clinical Neu-roscience 1996;8:139–46.
[3] Baddeley AD. The trouble with levels: a reexamination of Craikand Lockhart’s framework for memory research. PsychologicalReview 1978;85:139–52.
[4] Bauer RM, Tobias B, Valenstein E. Amnesic disorders. In:Heilman KM, Valenstein E, editors. Clinical Neuropsychology.New York: Oxford University Press, 1993:523–602.
[6] Benton AL, Hamsher KD. Multilingual Examination. IowaCity, IA: University of Iowa, 1976.
[7] Bower GH, Karlin MB. Depth of processing pictures of facesand recognition memory. Journal of Experimental Psychology1974;103:751–7.
[8] Breiter HC, Etcoff NL, Whalen PJ, et al. Response and habitua-tion of the human amygdala during visual processing of facialexpression. Neuron 1996;17:875–87.
[9] Cabeza R, Nyberg L. Imaging cognition: an empirical review ofPET studies with normal subjects. Journal of Cognitive Neuro-science 1997;9:1–26.
[10] Courtney SM, Ungerleider LG, Keil K, Haxby JV. Object andspatial visual working memory activate separate neural systemsin human cortex. Cerebral Cortex 1996;6:39–49.
[11] Courtney SM, Ungerleider LG, Keil K, Haxby JV. Transientand sustained activity in a distributed neural system for humanworking memory. Nature 1997;386:608–11.
[12] Craik FIM, Lockhart RS. Levels of processing: a framework formemory research. Journal of Verbal Learning and Verbal Behav-ior 1972;11:671–84.
[13] Craik FIM, Simon E. Age differences in memory: the roles ofattention and depth of processing. In: Poon LW, Fozard JL,Cermak L, Arenberg D, Thompson LW, editors. New Directionsin Memory and Aging: Proceedings of the George TallandMemorial Conference. Hillsdale, NJ: Lawrence Erlbaum,1980:95–112.
[14] Craik FIM, Tulving E. Depth of processing and the retention ofwords in episodic memory. Journal of Experimental PsychologyGeneral 1975;104:268–94.
[15] Edgington ES. Randomization Tests. New York: MarcelDekker, 1980.
[16] Efron B, Tibshirani R. Bootstrap methods for standard errors,confidence intervals and other measures of statistical accuracy.Statistical Science 1986;1:54–77.
[17] Eustache F, Desgranges B, Laville P, et al. Episodic memory intransient global amnesia: encoding, storage, or retrieval deficit?Journal of Neurology, Neurosurgery and Psychiatry1999;66:148–54.
[18] Fahy FL, Riches IP, Brown MW. Neuronal activity related tovisual recognition memory: long-term memory and the encodingof recency and familiarity information in the primate anteriorand medial inferior temporal and rhinal cortex. ExperimentalBrain Research 1993;96:457–72.
[19] Folstein MF, Folstein SE, McHugh PR. ’Mini-mental state’. Apractical method for grading the cognitive state of patients forthe clinician. Journal of Psychiatric Research 1975;12:189–98.
[20] Grady CL. Neuroimaging and activation of the frontal lobes. In:Miller BL, Cummings JL, editors. The Human Frontal Lobes:Function and Disorders. New York: Guilford Press, 1999:196–230.
[21] Grady CL, Horwitz B, Pietrini P, et al. The effect of taskdifficulty on cerebral blood flow during perceptual matching offaces. Human Brain Mapping 1996;4:227–39.
[22] Grady CL, Maisog JM, Horwitz B, et al. Age-related changes incortical blood flow activation during visual processing of facesand location. Journal of Neuroscience 1994;14:1450–62.
[23] Grady CL, McIntosh AR, Bookstein F, Horwitz B, RapoportSI, Haxby JV. Age-related changes in regional cerebral bloodflow during working memory for faces. NeuroImage 1998;8:409–25.
[24] Grady CL, McIntosh AR, Rajah MN, Craik FIM. Neuralcorrelates of the episodic encoding of pictures and words. Pro-ceedings of the National Academy of Science USA1998;95:2703–8.
[25] Halgren E, Dale AM, Sereno MI, Tootell RB, Marinkovic K,Rosen BR. Location of human face-selective cortex with respectto retinotopic areas. Human Brain Mapping 1999;7:29–37.
[26] Haxby JV, Horwitz B, Ungerleider LG, Maisog JM, Pietrini P,Grady CL. The functional organization of human extrastriatecortex: a PET-rCBF study of selective attention to faces andlocations. Journal of Neuroscience 1994;14:6336–53.
[27] Haxby JV, Ungerleider LG, Horwitz B, Maisog JM, RapoportSI, Grady CL. Face encoding and recognition in the humanbrain. Proceedings of the National Academy of Science USA1996;93:922–7.
[28] Haxby JV, Ungerleider LG, Horwitz B, Maisog JM, RapoportSI, Grady CL. Storage and retrieval of new memories for facesin the intact human brain. Proceedings of the National Academyof Science USA 1996;93:922–7.
[29] Haxby JV, Ungerleider LG, Horwitz B, Rapoport SI, Grady CL.Hemispheric differences in neural systems for face working mem-ory: a PET-rCBF study. Human Brain Mapping 1995;3:68–82.
[30] Intraub H, Nicklos S. Levels of processing and picture memory:the physical superiority effect. Journal of Experimental Psychol-ogy: Learning, Memory and Cognition 1985;11:284–298.
[31] Kanwisher N, McDermott J, Chun MM. The fusiform face area:a module in human extrastriate cortex specialized for face per-ception. Journal of Neuroscience 1997;17:4302–11.
[32] Kapur N, Friston KJ, Young A, Frith CD, Frackowiak RSJ.Activation of human hippocampal formation during memory forfaces: a PET study. Cortex 1995;31:99–108.
[33] Kapur S, Craik FIM, Tulving E, Wilson AA, Houle S, BrownGM. Neuroanatomical correlates of encoding in episodic mem-
L.J. Bernstein et al. / Neuropsychologia 40 (2002) 86–9898
ory: levels of processing effect. Proceedings of the NationalAcademy of Science USA 1994;91:2008–11.
[34] Kapur S, Rose R, Liddle PF, et al. The role of the left prefrontalcortex in verbal processing: semantic processing or willed action?Neuroreport 1994;5:2193–6.
[35] Kelley WM, Miezin FM, McDermott KB, et al. Hemisphericspecialization in human dorsal frontal cortex and medial tempo-ral lobe for verbal and nonverbal memory encoding. Neuron1998;20:927–36.
[36] Kuskowski MA, Pardo JV. The role of the fusiform gyrus insuccessful encoding of face stimuli. Neuroimage 1999;9:599–610.
[37] LeDoux JE. Emotional memory systems in the brain. Be-havioural Brain Research 1993;58:69–79.
[38] Li L, Miller EK, Desimone R. The representation of stimulusfamiliarity in anterior inferior temporal cortex. Journal of Neu-rophysiology 1993;69:1918–29.
[39] McCarthy G, Puce A, Gore JC, Allison T. Face-specific process-ing in the fusiform gyrus. Journal of Cognitive Neuroscience1997;9:605–10.
[41] McDermott KB, Buckner RL, Petersen SE, Kelley WM, SandersAL. Set- and code-specific activation in frontal cortex: an fMRIstudy of encoding and retrieval of faces and words. Journal ofCognitive Neuroscience 1999;11:631–40.
[42] McIntosh AR., Mapping cognition to the brain through neuralinteractions. Memory 1999;7:523–548.
[43] McIntosh AR, Bookstein FL, Haxby JV, Grady CL. Spatialpattern analysis of functional brain images using partial leastsquares. NeuroImage 1996;3:143–57.
[44] Morris JS, Frith CD, Perrett DI, et al. A differential neuralresponse in the human amygdala to fearful and happy facialexpressions. Nature 1996;383:812–5.
[45] Morris JS, Ohman A, Dolan RJ. A subcortical pathway to theright amygdala mediating ‘unseen’ fear. Proceedings of the Na-tional Academy of Sciences USA 1999;96:1680–5.
[47] Nyberg L, Persson J, Habib R et al. Large scale neurocognitivenetworks underlying episodic memory. Journal of CognitiveNeuroscience 2000;12:163–173.
[48] Owen AM, Milner B, Petrides M, Evans AC. Memory for objectfeatures versus memory for object location: a positron-emissiontomography study of encoding and retrieval processes. Proceed-ings of the National Academy of Science USA 1996;93:9212–7.
[49] Puce A, Allison T, McCarthy G. Electrophysiological studies ofhuman face perception. III: Effects of top-down processing onface-specific potentials. Cerebral Cortex 1999;9:445–58.
[51] Rugg MD, Fletcher PC, Frith CD, Frackowiak RSJ, Dolan RJ.Differential activation of the prefrontal cortex in successful andunsuccessful memory retrieval. Brain 1996;119:2073–84.
[52] Rugg MD, Fletcher PC, Frith CD, Frackowiak RSJ, Dolan RJ.Brain regions supporting intentional and incidental memory: aPET study. Neuroreport 1997;8:1283–7.
[53] Sagar HJ, Cohen NJ, Corkin S, Growdon JH. Dissociationsamong processes in remote memory. Annals of the New YorkAcademy of Sciences 1985;444:533–5.
[54] Sampson PD, Streissguth AP, Barr HM, Bookstein FL. Neu-robehavioral effects of prenatal alchohol: Part II. Partial leastsquares analysis. Neurotoxicology and Teratology 1989;11:477–91.
[55] Schreurs BG, McIntosh AR, Bahro M, Herscovitch P, Sunder-land T, Molchan SE. Lateralization and behavioral correlationof changes in regional cerebral blood flow with classical condi-tioning of the human eyeblink response. Journal of Neurophysi-ology 1997;77:2153–63.
[56] Sergent J, Ohta S, MacDonald B. Functional neuroanatomy offace and object processing. A positron emission tomographystudy. Brain 1992;115:15–36.
[57] Shallice T, Fletcher P, Frith CD, Grasby P, Frackowiak RS,Dolan RJ. Brain regions associated with acquisition and retrievalof verbal episodic memory. Nature 1994;368:633–5.
[58] Smith AD, Winograd E. Adult age differences in rememberingfaces. Developmental Psychology 1978;14:443–4.
[59] Talairach J, Tournoux P. Co-Planar Stereotaxic Atlas of theHuman Brain. New York: Thieme, 1988.
[60] Tulving E. Elements of Episodic Memory. New York: OxfordUniversity Press, 1983.
[61] Tulving E, Kapur S, Craik FIM, Moscovitch M, Houle S.Hemspheric encoding/retrieval asymmetry in episodic memory:positron emission tomography findings. Proceedings of the Na-tional Academy of Science USA 1994;91:2016–20.
[62] Tulving E, Kapur S, Markowitsch HJ, Craik FIM, Habib R,Houle S. Neuroanatomical correlates of retrieval in episodicmemory: auditory sentence recognition. Proceedings of the Na-tional Academy of Science USA 1994;91:2012–5.
[64] Whalen PJ, Rauch SL, Etcoff NL, McInerney SC, Lee MB,Jenike MA. Masked presentations of emotional facial expres-sions modulate amygdala activity without explicit knowledge.Journal of Neuroscience 1998;18:411–8.
[65] Wojciulik E, Kanwisher N, Driver J. Covert visual attentionmodulates face-specific activity in the human fusiform gyrus:fMRI study. Journal of Neurophysiology 1998;79:1574–8.
[66] Woods RP, Cherry SR, Mazziotta JC. Rapid automated al-gorithm for aligning and reslicing PET images. Journal of Com-puter Assisted Tomography 1992;16:620–33.
