Recollection- and Familiarity-Based Decisions Reflect Memory Strength

We used event-related fMRI to investigate whether recollection- and familiarity-based memory judgments are modulated by the degree of visual similarity between old and new art paintings. Subjects performed a flower detection task, followed by a Remember/Know/New surprise memory test. The old paintings were randomly presented with new paintings, which were either visually similar or visually different. Consistent with our prediction, subjects were significantly faster and more accurate to reject new, visually different paintings than new, visually similar ones. The proportion of false alarms, namely remember and know responses to new paintings, was significantly reduced with decreased visual similarity. The retrieval task evoked activation in multiple visual, parietal and prefrontal regions, within which remember judgments elicited stronger activation than know judgments. New, visually different paintings evoked weaker activation than new, visually similar items in the intraparietal sulcus. Contrasting recollection with familiarity revealed activation predominantly within the precuneus, where the BOLD response elicited by recollection peaked significantly earlier than the BOLD response evoked by familiarity judgments. These findings suggest that successful memory retrieval of pictures is mediated by activation in a distributed cortical network, where memory strength is manifested by differential hemodynamic profiles. Recollection- and familiarity-based memory decisions may therefore reflect strong memories and weak memories, respectively.


INTRODUCTION
Functional brain imaging studies have shown that medial temporal, parietal and prefrontal cortices are involved in recognition memory of prior episodes (Rugg and Wilding, 2000;Rugg and Yonelinas, 2003). The functional role that these regions play in memory retrieval, however, is still debated. Specifi cally, it is unclear whether recollection, the retrieval of specifi c information about a past experience, and familiarity, a sense that an event has been previously experienced (Tulving, 1985), are mediated by dissociated neural systems or separate strong memories from weak memories. Some studies suggest that separate cortical networks (Yonelinas et al., 2005) and differential activation in parietal cortex (Vilberg and Rugg, 2007) mediate these two distinct memory processes, whereas other studies suggest that recollection and familiarity refl ect differences in the strength of a common memory trace (Donaldson, 1996;Dunn, 2004;Gonsalves et al., 2005;Wixted, 2007).
In most episodic retrieval studies written words have been used as stimuli (e.g., Cabeza et al., 2001;Henson et al., 1999) and the neural correlates of retrieving pictures from memory are not fully understood. We have recently conducted a series of event-related fMRI studies to investigate the neural mechanisms of retrieving pictorial information from long-term memory and the effects of visual similarity between old and new pictures. Subjects memorized paintings  or unfamiliar Asian faces  and 4 days later performed an old-new recognition memory task in the MR scanner. The old pictures were presented with new ones that were visually similar, somewhat similar or visually different from the old paintings. Consistent with our hypothesis, subjects were slower and less accurate to reject new, visually similar paintings. We found activation in a distributed cortical network that included face-and object-selective regions in the visual cortex, as well as prefrontal areas where the old paintings evoked stronger activation than the new ones, regardless of their visual similarity. Moreover, activation elicited by new paintings in parietal cortex was reduced with decreased similarity to the old items, whereas in the hippocampus and precuneus, stronger responses were evoked by the new, visually different paintings. We concluded that recognition memory is mediated by classifi cation of new items as a match or a mismatch, based on their degree of visual similarity to old items Yago and Ishai, 2006). Our old-new task, however, did not address the issue of memory processes and the extent to which the observed behavioral and neural responses were due to recollection-or familiarity-based memory decisions.
To test whether recollection and familiarity judgments were infl uenced by the degree of visual similarity between old and new pictures, we used event-related fMRI with portraits, landscapes and abstract compositions by painters with a unique style (Figure 1). Based on our previous study, we predicted that subjects would correctly reject more new paintings that were visually different from the old ones, and would make more false alarms to new, visually similar paintings. Furthermore, we hypothesized that if recollection and familiarity judgments about complex pictures refl ect strong memories and weak memories, respectively, correctly remembered items would be associated with stronger neural activation than known items.

Subjects
Twenty-one normal, right-handed subjects (11 females, mean age 25 years) with normal or corrected to normal vision participated in the study. All subjects gave informed written consent for the procedure in accordance with protocols approved by the University Hospital of Zurich. The subjects, students from the University of Zurich, had no formal art education and reported visiting art museums once a year or less. Post-scan questionnaires revealed that all subjects were unfamiliar with the paintings and had not seen them prior to the experiment.

Stimuli and tasks
Stimuli were displayed using Presentation (www.neurobs.com, version 9.13) and were projected with a magnetically shielded LCD video projector onto a translucent screen placed at the feet of the subject. During the study phase, subjects performed a fl ower detection task on 20 portraits by Modigliani and Renoir, 20 landscapes by Pissarro and Van Gogh and 20 abstract compositions by Kandinsky and Miro (Figure 1). Each painting was presented for 3 s and subjects had to indicate whether it includes any fl owers by pressing one of two buttons. In each of the three time series collected during the study phase, paintings (4 epochs of 15 s each) alternated with 15-s fi xation epochs.
We then collected the anatomical images and after this 10-min scan, subjects performed a surprise memory test, in which the 60 old paintings were randomly presented with 60 new paintings. The new paintings were either visually similar or visually different from the old paintings. The degree of visual similarity between the new paintings and the old ones was assessed in a separate behavioral pilot (for details see Yago and Ishai, 2006). Each painting was presented for 3 s and subjects had to make a Remember/Know/New decision by pressing one of three buttons. Subjects were instructed to respond "Remember" if they were absolutely convinced that the picture was presented during the fl ower detection task; "Know" if the picture seemed familiar; and "New" if the picture was not presented in the fl ower detection task. In each of the three time series collected in the test phase, paintings (8 epochs of 15 s each) alternated with 15-s fi xation epochs. In both study and test, each run included one category of paintings (i.e., portraits, landscapes or abstract compositions).
High-resolution spoiled gradient recalled echo structural images were collected in the same session for all the subjects (180 axial slices, TR = 20 ms, TE = 2.3 ms, fi eld of view = 220 mm, acquisition matrix = 224 × 224, reconstructed voxel size = 0.9 mm × 0.9 mm × 0.75 mm). These high-resolution anatomical images were collected after the study phase and provided detailed anatomical information for the region-of-interest (ROI) analysis and for 3D normalization to the Talairach and Tournoux (1998) atlas.

Data analysis
Accuracies and reaction times were computed for each subject, category of paintings (portraits, landscapes, abstract paintings) and response type (Yes/No during the fl ower detection task; Remember/Know/New during the memory retrieval test). ANOVA was used to compare the various conditions.
Functional MRI data were analyzed in BrainVoyager QX Version 1.8 (Brain Innovation, Maastricht, The Netherlands). All volumes were realigned to the fi rst volume, corrected for motion artefacts and spatially smoothed using a 5-mm FWHM Gaussian fi lter. The main effects during the study and test were analyzed using multiple regression (Friston et al., 1995). Based on the contrast of paintings vs. fi xation, a set of ROIs was anatomically defi ned for each subject with clusters that showed a signifi cant effect (p < 0.0001, uncorrected). These regions included the inferior occipital gyrus (IOG), fusiform gyrus (FG), dorsal occipital cortex (DOC), intraparietal sulcus (IPS), inferior frontal gyrus (IFG), insula and the anterior cingulate cortex (ACC). The contrasts of Remember vs. Know and Remember vs. New further revealed signifi cant activation in the precuneus and in two medial temporal lobe structures, the parahippocampal cortex (PHC) and the hippocampus. In each subject and each ROI, the mean parameter estimates were calculated separately for each response type (Yes/No during fl ower detection task; Remember/Know/New during memory test) and were used for between-subjects random-effects analyses.
Finally, we tested whether reaction times and fMRI activation during the study phase could predict subsequent behavioral and neural responses to the old paintings during the test phase. Thus, responses during the fl ower detection task were sorted based on subsequent Remember and Know judgments subjects made during the retrieval test.

Study phase
Behavioral data. In this phase subjects were presented with portraits, landscapes and abstract paintings and indicated whether each painting contained any fl owers. The behavioral data collected while subjects performed the task in the scanner are shown in Figure S1. As most paintings did not include fl owers, the proportion of No responses was higher than the proportion of Yes responses [t(124) = 10.9, p < 0.00001 for portraits, landscapes and abstract paintings]. Furthermore, subjects made signifi cantly faster Yes responses than No responses [t(120) = 3.3, p < 0.001 for portraits, landscapes and abstract paintings]. Interestingly, subjects reported seeing fl owers in more than 20% of the abstract paintings, which do not depict natural objects, but rather use purely visual forms of line, color and shape. Moreover, it took subjects signifi cantly longer to decide whether these abstract compositions contained fl owers. Thus, the differences in response latencies between Yes abstract responses and Yes responses to portraits and landscapes were signifi cant [t(36) = 4.2, p < 0.001 and t(36) = 2.9, p < 0.01, respectively], as well as the difference in response latencies between No responses to abstract paintings and No responses to portraits [t(40) = 4.4, p < 0.001].
Imaging data. The main effect, namely responses evoked by all paintings as compared with the fi xation baseline, revealed activation within a distributed cortical network that included multiple visual, parietal and prefrontal regions ( Figure S2). Signifi cant activation was found in the DOC (mean Talairach (8,7,53). Within all regions, activation evoked by Yes and No responses during the fl ower detection task was virtually identical, ruling out differential effects of attention during task performance. We then tested whether behavioral and neural responses during study could predict subsequent memory performance during test. We found that shorter response latencies for paintings with fl owers predicted subsequent Remember judgments during the memory test. In terms of the neural response, we found that Yes responses during the fl ower detection task resulted in similar activation in the FG during subsequent Remember and Know judgments ( Figure S3).

Test phase
Behavioral data. Ten minutes after performing the fl ower detection task, subjects performed a surprise memory task. In this test phase the paintings from the study phase were randomly presented with new paintings that were either visually similar or visually different from the old ones (see Figure 1) and subjects made Remember/Know/New decisions. The behavioral data collected while subjects performed the task in the scanner are shown in Figure 2. The proportion of remember responses to the old items was signifi cantly higher than both know and new responses [t(61) = 5.8, p < 0.000001 in both comparisons]. Consistent with our prediction, the proportion of false alarms, namely remember and know responses to new items, signifi cantly decreased with decreased visual similarity between the old and the new items [remember new similar vs. remember new different, t(61) = 5.7, p < 0.000001; know new similar vs. know new different, t(61) = 10.5, p < 0.000001]. In terms of response latencies, know decisions took signifi cantly longer than both remember and new responses [know vs. remember, t(187) = 9.0, p < 0.000001; know vs new, t(187) = 8.2, p < 0.000001 for all paintings]. Finally, consistent with our prediction, subjects responded signifi cantly faster to the new, visually different than to the new, visually similar paintings [t(61) = 4.9, p < 0.00001].
We then analyzed the activation evoked by correct and incorrect responses in all ROIs (Figure 4). Within the IPS, hits, namely correct responses to old items, evoked stronger activation than misses (p < 0.05), correct rejection of new paintings (p < 0.01); and false alarms (p < 0.001).
To test our hypothesis about visual similarity, we compared correct responses to old paintings with correct responses to new, visually similar and new, visually different items (Figure 6). Consistent with our previous study , we found that within the IPS, new, visually different paintings evoked less activation than new, visually similar ones (p < 0.01). To further understand the effect of visual similarity on memory decisions, we compared remember responses to old items (correct responses) with remember responses to new, visually similar items (false alarms). Within the FG, the difference between these responses was not signifi cant, however in the IPS, remember responses to old items evoked signifi cantly stronger activation than remember responses to new, visually We then contrasted the memory responses. Interestingly, comparing correct remember judgments with correct know judgments revealed activation predominantly in the precuneus (Figure 5). Analysis of the BOLD response within this region indicated not only stronger responses to remembered items, but also a BOLD latency shift, with a signifi cantly earlier peak for recollection (5.6 s) than familiarity [10.3 s, t(40) = 3.1, p < 0.004]. Contrasting correctly remembered pictures with correct rejections of new pictures further revealed a similar pattern of activation, albeit with a lower threshold. Activation in the precuneus was stronger and peaked significantly earlier during remember (5.7 s) than new [10.2 s, t(40) = 3.37, p < 0.002] judgments. Finally, contrasting correct know responses with correct rejection of new items revealed similar hemodynamic response profi les and the peak of the BOLD response (11.1 and 10.2 s, respectively) was not statistically signifi cant [t(40) = 0.54, p = 0.59].
Comparing correct remember judgments with correct know judgments also revealed activation in the medial temporal lobe, namely in the PHC and the hippocampus (Figure 6). In both regions, correctly remembered paintings evoked signifi cantly stronger activation than correctly known items [t(40) = 2.75, p < 0.009 in the PHC and t(40) = 2.47, p < 0.018 in the hippocampus]. The difference between remembered and new paintings, however, was not statistically signifi cant [t(40) = 1.55, p = 0.13 in the PHC and t(40) = 1.16, p = 0.25 in the hippocampus].

DISCUSSION
We investigated the neural correlates that mediate recognition memory of portraits, landscapes and abstract paintings. During the study phase, subjects performed an attention demanding fl ower detection task. The behavioral data showed that response latencies were signifi cantly shorter for the representational paintings, i.e., portraits and landscapes, in which familiar objects were clearly depicted. In contrast, the visual  (Ishai et al., 2007). The fl ower detection task evoked activation in a distributed cortical network, in which paintings with and without fl owers evoked virtually identical responses, ruling out differential effects of attention during encoding. Interestingly, shorter response latencies for paintings with fl owers predicted subsequent Remember judgments during the memory test, suggesting that fast identifi cation and detection of the fl ower targets facilitated later recollection of information about these paintings. Stronger activation in the FG for paintings with fl owers predicted subsequent Remember and Know judgments, whereas reduced activation for paintings without fl owers predicted subsequent Know judgments.
The surprise Remember/Know/New memory retrieval test revealed that most of the old paintings were correctly recognized and that responses to the new items depended on their visual similarity to the old ones. Consistent with our hypothesis, the proportion of false alarms, namely remember and know responses to new paintings, was signifi cantly reduced with decreased visual similarity. Consistent with previous fi ndings (e.g., Dewhurst and Conway, 1994), correct and incorrect know responses were associated with signifi cantly longer latencies, suggesting that subjects hesitated before deciding that a picture looked familiar but they could not recollect additional information about its prior experience.
The recognition memory task evoked activation within a distributed cortical network that included similar visual, parietal and prefrontal regions to those activated during the study phase. Within DOC, IOG, FG, IPS and IFG, correctly remembered old items elicited stronger activation than both correctly known and new items. In some of these regions, greater activation for remembered than known words has been previously found (Henson et al., 1999), suggesting that recollection elicits enhanced activation than familiarity regardless of stimulus format. When remember judgments were directly contrasted with know judgments, a signifi cant cluster of activation was found in the precuneus, a region implicated in many episodic memory retrieval (e.g., Fletcher et al., 1995;Shannon and Buckner, 2004;Yago and Ishai, 2006;Yonelinas et al., 2005) and visual imagery (Ishai et al., 2000a(Ishai et al., , 2002Mechelli et al., 2004) studies. Within the precuneus, correctly remembered paintings not only elicited stronger activation than known items, but a latency shift of the BOLD response was observed, with a signifi cantly earlier peak during recollection than familiarity-based judgments. Within the temporal lobe, activation in the hippocampus and PHC was stronger during recollection than during familiarity-based memory decisions. Our fi ndings provide empirical evidence in support of a recent perspective according to which the Remember/Know procedure separates strong memories from weak memories Wixted, 2007). It is of interest that within the PHC and the hippocampus, activation evoked by remembered paintings was not statistically signifi cant from activation evoked by new items. Our fMRI fi ndings are consistent with a previous report, in which activation in the hippocampus was recorded in epileptic patients using depth electrodes. Some hippocampal neurons increased their fi ring rate in response to old pictures, whereas other neurons signaled novelty by increased fi ring rate in response to new pictures (Rutishauser et al., 2006). Taken collectively, our fi ndings that recollection evoked stronger activation than familiarity within multiple regions suggest that the memory decisions refl ect memory strength and not independent memory processes.
Activation in parietal cortex during the memory test revealed stronger responses to the old, correctly remembered paintings than to the new items, consistent with previous ERP and fMRI studies (e.g., Curran and Cleary, 2003;Kahn et al., 2004;Wilding, 2000). Furthermore, activation within the IPS was reduced with decreased similarity between the new paintings and the old ones. The IPS, traditionally considered a region of the dorsal frontoparietal attention network, was implicated in many cognitive studies of attention, particularly in target detection tasks (Corbetta et al., 2000;Kincade et al., 2005;Shulman et al., 2001) and the segmentation of old from new items (Pollmann et al., 2003). Numerous recognition memory studies have further shown that posterior parietal cortex does not merely detect old items but, rather, mediates higher order cognitive processes associated with memory retrieval (Konishi et al., 2000;Shannon and Buckner, 2004;Wheeler and Buckner, 2003). Taken collectively, our previous Yago and Ishai, 2006) and current fi ndings suggest not only that the parietal cortex mediates the old/new effect, but also processes the degree of visual similarity between old and new items. These fi ndings support the "mnemonic accumulator" hypothesis, according to which recognition memory decisions are based on the integration of sensory signals .
Models of recognition memory assume that recollection and familiarity are independent processes during retrieval (Yonelinas, 2002). Evidence for such neuroanatomical dissociation came from studies in which subjects were instructed to rate their memory confi dence (Yonelinas et al., 2005) or to indicate the amount of recollected information (Vilberg and Rugg, 2007). Future studies will determine the extent to which the dissociation of the neural correlates of recollection and familiarity could be generalized across various experimental paradigms. Although remember and know responses are exclusive, recollected items are also familiar ones. It is therefore highly likely that retrieval of mnemonic information about complex pictures is modulated by activation within a distributed neural system, where memory strength modulates the neural response. The redundant relationship between recollection and familiarity has been corroborated by neuropsychological (Knowlton, 1998) and electrophysiological (Yovel and Paller, 2004) studies. Our current fi ndings suggest that the same cortical structures are activated during recollection-and familiarity-based judgments and are consistent with models of a continuum of mnemonic information on which the subject establishes a criterion (Donaldson, 1996;Dunn, 2004;Gonsalves et al., 2005). This criterion, as shown in our current and previous studies Yago and Ishai, 2006), depends on the degree of visual similarity between old and new items. A distributed neural system for recognition memory is consistent with recent neuroanatomical fi ndings in amnesic patients, indicating that the ability to recollect remote memories depends not only on the medial temporal lobe but on widely distributed neocortical areas in the occipital, parietal and prefrontal lobes (Bayley et al., 2005;Squire and Bayley, 2007). Furthermore, a distributed memory network is not only physiologically and ecologically plausible, but also confi rms with previous fMRI studies, showing that the representation of objects and faces in the human brain is not modular, but rather distributed across a wide expanse of cortex (Haxby et al., 2001;Ishai et al., 1999Ishai et al., , 2000b. Within this distributed neural system, the memory strength is manifested by differential BOLD responses during recollection and familiarity judgments. Future studies will determine the extent to which these two memory processes are manifested by differential patterns of effective connectivity among regions.
In summary, our results show that recollection-and familiarity-based memory decisions depend on the degree of visual similarity between old and new items. Furthermore, recognition memory of complex pictures is mediated by activation within a distributed cortical network, where remembered and known items evoke differential BOLD responses that refl ect their memory strength.