Who Can You Trust? Behavioral and Neural Differences Between Perceptual and Memory-Based Influences

Decisions about whether to trust someone can be influenced by competing sources of information, such as analysis of facial features versus remembering specific information about the person. We hypothesized that such sources can differentially influence trustworthiness judgments depending on the circumstances in which judgments are made. In our experiments, subjects first learned face-word associations. Stimuli were trustworthy and untrustworthy faces, selected on the basis of consensus judgments, and personality attributes that carried either the same valence (consistent with face) or the opposite valence (inconsistent with face). Subsequently, subjects rated the trustworthiness of each face. Both learned and perceptual information influenced ratings, but learned information was less influential under speeded than under non-speeded conditions. EEG data further revealed neural evidence of the processing of these two competing sources. Perceptual influences were apparent earlier than memory influences, substantiating the conclusion that time pressure can selectively disrupt memory retrieval relevant to trustworthiness attributions.

these fi ndings do not pinpoint the timing of trustworthiness judgments, given that judgments about 100-ms faces may be completed after faces disappear.
Obtaining neural measures of trustworthiness-relevant perceptual processing can more precisely defi ne the mechanisms by which judgments are made. In particular, neuropsychological and functional magnetic resonance imaging (fMRI) data have implicated the amygdala, which has been linked with threat detection (Phelps and LeDoux, 2005). Trustworthiness judgments were less systematic in patients with amygdala damage than in controls (Adolphs et al., 1998). Moreover, lower trustworthiness ratings correlated with higher amygdala activation in healthy individuals, even when they did not make explicit judgments during scanning, suggesting that amygdala activation to untrustworthy faces is automatic (Winston et al., 2002). The idea that amygdala processing can subserve threat assessment based on facial features, independently of cognitive analysis based on other information sources, also fi ts with evidence that amygdala activation correlated more highly with averaged consensus trustworthiness ratings than with an individual's own idiosyncratic judgments (Engell et al., 2007).
In contrast to these fMRI studies, other studies have enlisted the higher temporal resolution of electroencephalography for assaying the precise timing of relevant processes. For example, differences in event-related potentials (ERPs) have been reported in comparisons between emotional faces and neutral faces (Eimer and Holmes, 2007). These differences appeared in frontal scalp recordings beginning approximately 200 ms after face onset. We hypothesized that parallel effects might be found if consensus trustworthy (CT) faces were compared to consensus untrustworthy (CU) faces.
Who can you trust? Behavioral and neural differences between perceptual and memory-based infl uences INTRODUCTION Upon making new acquaintances, individuals must decide how much trust to place in them. Given that harm can come from unwarranted trust in some social situations, an ability to make accurate trustworthiness judgments is benefi cial. There have been many investigations of the neurocognitive systems that process trustworthiness-relevant information from a single source, such as perceptual features or previous experience. In real-world situations, however, many sources of information are available simultaneously, and they are not necessarily congruent. For example, information from stereotypes based on physical features may confl ict with learned information about an individual. How are trustworthiness judgments made when multiple relevant sources of information tug in different directions? Moreover, do certain circumstances bias which sources can assert control over a judgment? We hypothesized that time pressure may be one critical factor that infl uences what can be brought to bear on trustworthiness and related judgments. For example, perceptual information may rapidly become available to guide judgments, whereas additional time may be required to retrieve relevant information.
Like judging a book by its cover, a person's trustworthiness can be judged using facial information alone. Ratings of novel faces tend to be remarkably consistent across individuals (Engell et al., 2007), although the exact nature of the relevant feature dimensions is unclear. Some evidence suggests that facial expression may be particularly infl uential in trustworthiness judgments (Winston et al., 2002;Todorov, 2008;Oosterhof and Todorov, 2009). This perceptual information processing appears to be quite rapid, because systematic trustworthiness judgments can be made after a fl eeting 100-ms face presentation (Willis and Todorov, 2006). However, Previously learned information can also infl uence trustworthiness judgments, probably through a combination of conscious and nonconscious memory retrieval. In one study, positive or negative descriptions associated with faces impeded the later learning of new associations with the same faces when the new information was inconsistent with the original information (Carlston and Skowronski, 1994). This effect occurred even when subjects could not explicitly recall the initial descriptions. Similarly, trustworthiness judgments were affected by the valence of descriptions associated with faces, even in amnesic patients who could not recall the descriptions (Johnson et al., 1985).
Whereas both learned and perceptual information can infl uence trustworthiness judgments, it is unclear how these sources of information interact to yield the ultimate behavioral decision. The current study provides new insight into the nature of these perceptual and memory-based processes. In Experiment 1, behavioral evidence showed that the two sources were differentially sensitive to a response deadline, and ERPs in Experiment 2 provided further information about the timing of the two sources.

Subjects
Twenty-four students participated for course credit (10 men and 14 women; mean age = 19 years).

Materials
A total of 128 faces were classifi ed as either CT or CU based on ratings from a separate group of 35 subjects. Faces were paired with words selected from a list of eight positive and eight negative personality traits (Anderson, 1968), with specifi c face-word pairings randomized across subjects.

Experimental procedures
Subjects completed eight blocks, each consisting of a training phase, a rating phase, and a recall phase. In the training phase, subjects saw 16 faces, each paired with a different word (four CT faces with positive traits, four CT faces with negative traits, four CU faces with positive traits, and four CU faces with negative traits). Subjects were instructed to remember each word-face pairing. At the beginning of the rating phase, subjects were advised whether ratings would be speeded or deliberate (four blocks each, pseudorandom order). For speeded ratings, subjects were required to produce a trustworthiness rating within 1500 ms of face onset using a 5-point scale. For deliberative ratings, subjects were required to wait at least 3000 ms before responding (during which time a red X was visible above the face). In the recall phase, subjects were shown each face and given unlimited time to produce the associated word. Each block included a unique set of 16 faces and the same 16 words. Procedures were approved by the Northwestern University Institutional Review Board and all subjects gave informed consent.

RESULTS
Ratings were infl uenced by consensus face trustworthiness and by word valence as predicted (Figure 1). CT faces were rated as more trustworthy than CU faces by 0.65 points on the 5-point scale [main effect of consensus trustworthiness: F(1,23) = 20.5, p < 0.001].
Faces previously paired with positive words were rated 1.57 points more trustworthy than those paired with negative words [main effect of word valence: F(1,23) = 96.4, p < 0.001] There was no interaction between consensus trustworthiness and word valence [F(1,23) < 1].
Subjects tended to be infl uenced more either by facial trustworthiness or by word valence. That is, subjects who rated faces highly differently as a function of previously paired word valence tended to differentiate less between CT and CU faces (r = −0.59, p < 0.01). This negative correlation was more pronounced in the deliberative condition (r = −0.68, p < 0.001) than in the speeded condition (r = −0.32, p > 0.1).
In the speeded condition, reaction times were faster when consensus trustworthiness and word valence were consistent (both positive or both negative, 1291 ms), than when the information confl icted across sources [1382 ms, t(23) = 2.9, p < 0.01]. Reaction times did not differ between consistent versus confl icting information in the The infl uence of word valence, as measured by the average difference in ratings between faces paired with positive versus negative words, was attenuated in the speed rating condition compared to the deliberative rating condition. In contrast, the infl uence of consensus face rating, as measured by the average difference in ratings between consensus trustworthy (CT) and consensus untrustworthy (CU) faces, was similar between conditions. deliberative condition [3829 ms vs. 3871 ms, respectively, t(23) < 1, reaction times calculated from face onset].

DISCUSSION
Perceptual information relevant to trustworthiness judgments was apparently accessible sooner than information retrieved from memory. Memory for personality attributes that had earlier been associated with faces infl uenced trustworthiness ratings in the expected manner, but when subjects were forced to decide quickly, this infl uence was reduced compared to when there was ample time to decide.
An alternative explanation for these results is that speeded ratings were more random than deliberative ratings. However, if the speeded condition simply encouraged random responding, then perceptual infl uences would also decline in this condition compared to the deliberative condition. Yet, there was no evidence that perceptual infl uences on trustworthiness were disrupted in the speeded condition compared to the deliberative condition.
The slowing of responses when information from the two sources confl icted suggests that perceptual and memory-based processes compete in their infl uence on trustworthiness decisions. The negative correlation between the infl uence of previously learned information and the infl uence of consensus facial information suggests further that competition exists between these two sources, and that individuals differ in the weights given to them in making trustworthiness decisions.
Experiment 2 was designed to provide further information about the processing relevant for perceptual and memory-based infl uences. In particular, we sought converging evidence for the hypothesis that trustworthiness-relevant perceptual information is available sooner than information retrieved from memory.

Subjects
Twenty-two individuals participated for monetary compensation (3 men and 19 women, mean age = 22 years). Data from fi ve of these subjects were not used because of equipment malfunction (one subject) or frequent blinking artifacts (four subjects). The pattern of behavioral results was similar regardless of whether these subjects were included.

Materials
Stimuli were the same as those used in Experiment 1, except that there were 144 faces and they were classifi ed somewhat differently. Faces in the top 33% of consensus trustworthiness were classifi ed as CT, those in the middle 33% as consensus neutral (CN), and those in the bottom 33% as CU. On average, 49% of the CU faces, 51% of the CN faces and 50% of the CT faces were paired with positive words.

Experimental procedures
The procedure was similar to that of Experiment 1. There were nine blocks, each with a training phase, a rating phase, and a recall phase, and EEG was recorded during the rating phase. Timing parameters in the rating phase were the same in each block; each face was displayed for 500 ms, and subjects could enter responses within 4000 ms from face onset. There was a fi xation period of 1500 ms between the end of the rating period and the onset of the next face.
EEG was recorded continuously from tin electrodes at 59 locations covering the scalp. Four other channels monitored eye movements and blinks, and trials with such artifacts were excluded in ERP analyses (average of 15% rejected per subject, SE = 0.02%). Reference electrodes were placed over the mastoid bones behind each ear. Recordings were referenced to right mastoid online and were re-referenced offl ine to averaged mastoids. Impedance was brought below 5 kΩ and signals were amplifi ed with a band pass of 0.05-200 Hz and a 1000-Hz sampling rate. ERP epochs were from 100 ms before to 1000 ms after stimulus onset, and epochs were baseline-corrected by subtracting the average voltage of the 100-ms prestimulus interval from all data points in the epoch.
Analyses focused on EEG data from a cluster of frontal electrodes and a cluster of central/parietal electrodes, based on previous research indicating that emotional expression infl uences frontal ERPs and that memory retrieval typically infl uences central and parietal ERPs. Data were analyzed by calculating mean amplitudes over 200-ms intervals beginning 200 ms post-stimulus, when the earliest effects were expected. Procedures were approved by the Northwestern University Institutional Review Board and all subjects gave informed consent.

Behavioral results
As in Experiment 1, trustworthiness judgments were infl uenced by both word valence and consensus trustworthiness. Faces paired with positive words were rated 0.99 points higher than faces paired with negative words [F(1,16) = 18.8, p < 0.001]. Face type also infl uenced ratings, with CT faces given ratings 0.48 points higher than CN faces, which were in turn given ratings 0.29 points higher than CU faces [F(2,32) = 26.5, p < 0.001]. In addition, there was a signifi cant interaction of face trustworthiness and word valence, whereby word valence was increasingly infl uential as face trustworthiness increased [F(2,32) = 11.1, p < 0.01].
Again, subjects who made highly different ratings between CT and CU faces were less infl uenced by word valence (r = −0.48, p = 0.05). Subjects who were better able to recall at least the correct valence of word associates during the test phase also tended to be more infl uenced by word valence during the rating phase (r = 0.73, p < 0.001). There was also a correlation between the percentage of faces associated with the specifi c correct word during the test phase and the infl uence of word valence (r = 0.61, p < 0.05). In addition, negative words were more likely to be correctly remembered in the test phase than positive words [t(16) = 3.8, p < 0.01], but there was no effect of consensus face trustworthiness on memory for words [F(1,16) < 1].

EEG results
Faces of different consensus trustworthiness levels elicited ERPs that clearly diverged in the time period between 200 and 400 ms after stimulus onset, as shown in Figure 2A. To assay correlates of memory-based processes, rating-phase trials were sorted as a function of memory performance in the test phase ( Figure 2B). This memory performance was likely driven both by encoding in the initial test phase, and rehearsal during the rating phase. Regardless, because the words were not repeated during the rating phase, ERP correlates of test-phase memory performance must refl ect processing of previously learned information. In line with previous studies of ERP correlates of successful memory retrieval (Voss and Paller, 2008), ERPs from the parietal cluster were more positive at 400-600 ms when corresponding words were remembered compared to when they were forgotten [t(16) = 2.4, p < 0.05]. This difference continued in the interval from 600-800 ms [t(16) = 2.1, p = 0.05].
It is possible that the relatively early frontal ERP divergence between CT and CU faces was driven by subjects' ultimate behavioral ratings, rather than by perceptual features per se. In this case, one would expect to fi nd the same difference between ERPs to faces previously paired with positive versus negative words, as this manipulation affected behavioral judgments even more strongly than perceptual features. However, this comparison revealed no differences prior to 600 ms [t(16) < 1]. Instead, a later effect at 800-1000 ms was found (Figure 2C), in which parietal ERPs to faces shown previously with negative words were more positive than those to faces shown previously with positive words [t(16) = 2.1, p = 0.05]. Electrode clusters used for statistical analyses are displayed in Figure 2D.

DISCUSSION
Separate ERP correlates of trustworthiness-relevant perceptual and memory processes were identifi ed. As expected, signals of perceptual processing appeared sooner than signals of memory processing. In addition, ERPs associated with consensus ratings were not simply an indirect refl ection of behavioral judgments; the same ERPs were not found when comparing responses to faces previously paired with positive words to those paired with negative words even though paired-word valence had a strong infl uence on behavioral judgments.

GENERAL DISCUSSION
Experiment 1 demonstrated that trustworthiness ratings were infl uenced by perceptual features and by memory retrieval, and that the infl uence of memory declined when people were forced to make ratings quickly. Converging results from Experiment 2 showed that neural correlates of perceptual information processing and memory retrieval relevant to trustworthiness can be measured separately, and that signals of perceptual processing appeared sooner than those of memory retrieval. Together, these results suggest that perceptual information relevant to face trustworthiness is processed more quickly and can infl uence behavior sooner than memorybased information. Three relevant electrical signals were identifi ed: an early frontal correlate of consensus trustworthiness, a later correlate of memory retrieval with a parietal topography, and an even later correlate of word valence that also exhibited a parietal topography. The fi rst signal is likely to refl ect the analysis of facial expressions, as observed previously (e.g., Eimer and Holmes, 2007). Other results have shown that subtle facial expressions can drive trustworthiness judgments (Winston et al., 2002;Todorov, 2008;Oosterhof and Todorov, 2009). ERPs may have been most positive for CT faces because of unambiguously happy expressions for these faces compared to neutral and untrustworthy faces. It is presently unclear whether consistent trustworthiness ratings generally arise from the gross categorization of facial expression or whether more subtle cues are operative. The ERP correlate of consensus trustworthiness found in the current study does not refl ect subjects' behavioral ratings alone, and as such it displays a pattern similar to that of amygdala activation in fMRI studies (Engell et al., 2007).
The second ERP signal, a correlate of later memory retrieval, occurred with a latency and distribution paralleling effects found in other memory paradigms (Voss and Paller, 2008). This resemblance adds weight to the hypothesis that infl uences on trustworthiness judgments from conscious memory retrieval occur later than relevant perceptual processes.
The late difference found in the comparison of ERPs to faces previously paired with positive versus negative words may refl ect better memory for negative words. Though it appears later than typical ERP correlates of conscious memory retrieval (Voss and Paller, 2008), the pattern of increased parietal positivity for faces previously paired with (better remembered) negative words fi ts with such an account.
The longer response times in the speeded condition when there was confl ict between perceptual features and learned information suggests that there is some integration of information from these two sources and that this integration may be more effi cient or straightforward when the information is consistent. The negative correlations between perceptual and memory-based infl uences suggest that individuals weight certain information sources more heavily at the expense of others, and that the pattern of source weights varies across people. Previous studies of the neural inte-gration of confl icting information, with Stroop procedures (West and Alain, 1999;Liotti et al., 2000;Fruhholz et al., 2009) and face evaluation (Fruhholz et al., 2009;Schiller et al., 2009), relied only on perceptual information. The current study goes beyond these prior studies in showing that neural measures can be used to monitor the separate processing of information from perceptual and memory-based sources.
The interplay of the neurocognitive processes investigated here is likely quite common in everyday life; when individuals make rapid judgments about others, even in the absence of a formal rating, they may be highly infl uenced by perceptual features and discount memory-based sources of information. Stereotyping based on physical features may be a particularly salient example of this; individuals may have learned information contrary to stereotypes, but because of the time required to access this information, time pressure or cognitive load may often produce stereotype-based errors (Sherman et al., 1998;Correll et al., 2002). Initial personal judgments are quite persistent over time (Ambady and Rosenthal, 1993). Such judgments can infl uence behavior not only in the context of artifi cial social interactions but also in real-world situations such as when one makes fi nancial decisions with input from professional consultants, chooses among multiple job applicants, or selects which political candidates to support (Langlois et al., 2000;Todorov et al., 2005;van 't Wout and Sanfey, 2008). Thus, understanding the nature of the perceptual and memory-based neurocognitive processes pertaining to trustworthiness has wideranging implications.
The current results are consistent with evidence that perceived and remembered information infl uences trustworthiness judgments through separate neural processes (Engell et al., 2007;Todorov et al., 2007;Todorov and Olson, 2008). Our fi ndings further provide behavioral and neural evidence that the perceptual processing precedes memory retrieval. Accordingly, information from perceptual analysis can infl uence trustworthiness assessments well before relevant information has been retrieved from memory. More generally, this conclusion may apply in the same way when information from perception and memory sources can be brought to bear in a variety of decision-making circumstances.