Who can you trust? Behavioral and neural differences between perceptual and memory-based influences

Upon making new acquaintances, individuals must decide how much trust to place in them. Given that harm can come from unwarranted trust in some social situations, an ability to make accurate trustworthiness judgments is beneficial. There have been many investigations of the neurocognitive systems that process trustworthiness-relevant information from a single source, such as perceptual features or previous experience. In real-world situations, however, many sources of information are available simultaneously, and they are not necessarily congruent. For example, information from stereotypes based on physical features may conflict with learned information about an individual. How are trustworthiness judgments made when multiple relevant sources of information tug in different directions? Moreover, do certain circumstances bias which sources can assert control over a judgment? We hypothesized that time pressure may be one critical factor that influences what can be brought to bear on trustworthiness and related judgments. For example, perceptual information may rapidly become available to guide judgments, whereas additional time may be required to retrieve relevant information.

Like judging a book by its cover, a person’s trustworthiness can be judged using facial information alone. Ratings of novel faces tend to be remarkably consistent across individuals (Engell et al., 2007 ), although the exact nature of the relevant feature dimensions is unclear. Some evidence suggests that facial expression may be particularly influential in trustworthiness judgments (Winston et al., 2002 ; Todorov, 2008 ; Oosterhof and Todorov, 2009 ). This perceptual information processing appears to be quite rapid, because systematic trustworthiness judgments can be made after a fleeting 100-ms face presentation (Willis and Todorov, 2006 ). However, these findings do not pinpoint the timing of trustworthiness judgments, given that judgments about 100-ms faces may be completed after faces disappear.

Obtaining neural measures of trustworthiness-relevant perceptual processing can more precisely define the mechanisms by which judgments are made. In particular, neuropsychological and functional magnetic resonance imaging (fMRI) data have implicated the amygdala, which has been linked with threat detection (Phelps and LeDoux, 2005 ). Trustworthiness judgments were less systematic in patients with amygdala damage than in controls (Adolphs et al., 1998 ). Moreover, lower trustworthiness ratings correlated with higher amygdala activation in healthy individuals, even when they did not make explicit judgments during scanning, suggesting that amygdala activation to untrustworthy faces is automatic (Winston et al., 2002 ). The idea that amygdala processing can subserve threat assessment based on facial features, independently of cognitive analysis based on other information sources, also fits with evidence that amygdala activation correlated more highly with averaged consensus trustworthiness ratings than with an individual’s own idiosyncratic judgments (Engell et al., 2007 ).

In contrast to these fMRI studies, other studies have enlisted the higher temporal resolution of electroencephalography for assaying the precise timing of relevant processes. For example, differences in event-related potentials (ERPs) have been reported in comparisons between emotional faces and neutral faces (Eimer and Holmes, 2007 ). These differences appeared in frontal scalp recordings beginning approximately 200 ms after face onset. We hypothesized that parallel effects might be found if consensus trustworthy (CT) faces were compared to consensus untrustworthy (CU) faces.

Previously learned information can also influence trustworthiness judgments, probably through a combination of conscious and nonconscious memory retrieval. In one study, positive or negative descriptions associated with faces impeded the later learning of new associations with the same faces when the new information was inconsistent with the original information (Carlston and Skowronski, 1994 ). This effect occurred even when subjects could not explicitly recall the initial descriptions. Similarly, trustworthiness judgments were affected by the valence of descriptions associated with faces, even in amnesic patients who could not recall the descriptions (Johnson et al., 1985 ).

Whereas both learned and perceptual information can influence trustworthiness judgments, it is unclear how these sources of information interact to yield the ultimate behavioral decision. The current study provides new insight into the nature of these perceptual and memory-based processes. In Experiment 1, behavioral evidence showed that the two sources were differentially sensitive to a response deadline, and ERPs in Experiment 2 provided further information about the timing of the two sources.

Twenty-four students participated for course credit (10 men and 14 women; mean age = 19 years).

A total of 128 faces were classified as either CT or CU based on ratings from a separate group of 35 subjects. Faces were paired with words selected from a list of eight positive and eight negative personality traits (Anderson, 1968 ), with specific face-word pairings randomized across subjects.

Subjects completed eight blocks, each consisting of a training phase, a rating phase, and a recall phase. In the training phase, subjects saw 16 faces, each paired with a different word (four CT faces with positive traits, four CT faces with negative traits, four CU faces with positive traits, and four CU faces with negative traits). Subjects were instructed to remember each word-face pairing. At the beginning of the rating phase, subjects were advised whether ratings would be speeded or deliberate (four blocks each, pseudorandom order). For speeded ratings, subjects were required to produce a trustworthiness rating within 1500 ms of face onset using a 5-point scale. For deliberative ratings, subjects were required to wait at least 3000 ms before responding (during which time a red X was visible above the face). In the recall phase, subjects were shown each face and given unlimited time to produce the associated word. Each block included a unique set of 16 faces and the same 16 words. Procedures were approved by the Northwestern University Institutional Review Board and all subjects gave informed consent.

Ratings were influenced by consensus face trustworthiness and by word valence as predicted (Figure 1 ). CT faces were rated as more trustworthy than CU faces by 0.65 points on the 5-point scale [main effect of consensus trustworthiness: F(1,23) = 20.5, p < 0.001]. Faces previously paired with positive words were rated 1.57 points more trustworthy than those paired with negative words [main effect of word valence: F(1,23) = 96.4, p < 0.001] There was no interaction between consensus trustworthiness and word valence [F(1,23) < 1].

The type of rating phase, speeded or deliberative, had no overall influence on ratings [F(1,23) < 1]. However, the effect of word valence was greater in the deliberative condition than the speeded condition [1.80 points vs. 1.34 points, respectively, interaction of rating phase type and word valence: F(1,23) = 14.6, p = 0.001]. In contrast, the CT/CU effect was very similar across rating phase type [0.63 vs. 0.68 points, F(1,23) < 1]. Type of rating phase altered the word-valence influence reliably more than the consensus-face-trustworthiness influence [t(23) = 2.9, p < 0.01].

Subjects tended to be influenced more either by facial trustworthiness or by word valence. That is, subjects who rated faces highly differently as a function of previously paired word valence tended to differentiate less between CT and CU faces (r = −0.59, p < 0.01). This negative correlation was more pronounced in the deliberative condition (r = −0.68, p < 0.001) than in the speeded condition (r = −0.32, p > 0.1).

In the speeded condition, reaction times were faster when consensus trustworthiness and word valence were consistent (both positive or both negative, 1291 ms), than when the information conflicted across sources [1382 ms, t(23) = 2.9, p < 0.01]. Reaction times did not differ between consistent versus conflicting information in the deliberative condition [3829 ms vs. 3871 ms, respectively, t(23) < 1, reaction times calculated from face onset].

Perceptual information relevant to trustworthiness judgments was apparently accessible sooner than information retrieved from memory. Memory for personality attributes that had earlier been associated with faces influenced trustworthiness ratings in the expected manner, but when subjects were forced to decide quickly, this influence was reduced compared to when there was ample time to decide.

An alternative explanation for these results is that speeded ratings were more random than deliberative ratings. However, if the speeded condition simply encouraged random responding, then perceptual influences would also decline in this condition compared to the deliberative condition. Yet, there was no evidence that perceptual influences on trustworthiness were disrupted in the speeded condition compared to the deliberative condition.

The slowing of responses when information from the two sources conflicted suggests that perceptual and memory-based processes compete in their influence on trustworthiness decisions. The negative correlation between the influence of previously learned information and the influence of consensus facial information suggests further that competition exists between these two sources, and that individuals differ in the weights given to them in making trustworthiness decisions.

Experiment 2 was designed to provide further information about the processing relevant for perceptual and memory-based influences. In particular, we sought converging evidence for the hypothesis that trustworthiness-relevant perceptual information is available sooner than information retrieved from memory.

Twenty-two individuals participated for monetary compensation (3 men and 19 women, mean age = 22 years). Data from five of these subjects were not used because of equipment malfunction (one subject) or frequent blinking artifacts (four subjects). The pattern of behavioral results was similar regardless of whether these subjects were included.

Stimuli were the same as those used in Experiment 1, except that there were 144 faces and they were classified somewhat differently. Faces in the top 33% of consensus trustworthiness were classified as CT, those in the middle 33% as consensus neutral (CN), and those in the bottom 33% as CU. On average, 49% of the CU faces, 51% of the CN faces and 50% of the CT faces were paired with positive words.

The procedure was similar to that of Experiment 1. There were nine blocks, each with a training phase, a rating phase, and a recall phase, and EEG was recorded during the rating phase. Timing parameters in the rating phase were the same in each block; each face was displayed for 500 ms, and subjects could enter responses within 4000 ms from face onset. There was a fixation period of 1500 ms between the end of the rating period and the onset of the next face.

EEG was recorded continuously from tin electrodes at 59 locations covering the scalp. Four other channels monitored eye movements and blinks, and trials with such artifacts were excluded in ERP analyses (average of 15% rejected per subject, SE = 0.02%). Reference electrodes were placed over the mastoid bones behind each ear. Recordings were referenced to right mastoid online and were re-referenced offline to averaged mastoids. Impedance was brought below 5 kΩ and signals were amplified with a band pass of 0.05–200 Hz and a 1000-Hz sampling rate. ERP epochs were from 100 ms before to 1000 ms after stimulus onset, and epochs were baseline-corrected by subtracting the average voltage of the 100-ms prestimulus interval from all data points in the epoch.

Analyses focused on EEG data from a cluster of frontal electrodes and a cluster of central/parietal electrodes, based on previous research indicating that emotional expression influences frontal ERPs and that memory retrieval typically influences central and parietal ERPs. Data were analyzed by calculating mean amplitudes over 200-ms intervals beginning 200 ms post-stimulus, when the earliest effects were expected. Procedures were approved by the Northwestern University Institutional Review Board and all subjects gave informed consent.

As in Experiment 1, trustworthiness judgments were influenced by both word valence and consensus trustworthiness. Faces paired with positive words were rated 0.99 points higher than faces paired with negative words [F(1,16) = 18.8, p < 0.001]. Face type also influenced ratings, with CT faces given ratings 0.48 points higher than CN faces, which were in turn given ratings 0.29 points higher than CU faces [F(2,32) = 26.5, p < 0.001]. In addition, there was a significant interaction of face trustworthiness and word valence, whereby word valence was increasingly influential as face trustworthiness increased [F(2,32) = 11.1, p < 0.01].

Again, subjects who made highly different ratings between CT and CU faces were less influenced by word valence (r = −0.48, p = 0.05). Subjects who were better able to recall at least the correct valence of word associates during the test phase also tended to be more influenced by word valence during the rating phase (r = 0.73, p < 0.001). There was also a correlation between the percentage of faces associated with the specific correct word during the test phase and the influence of word valence (r = 0.61, p < 0.05). In addition, negative words were more likely to be correctly remembered in the test phase than positive words [t(16) = 3.8, p < 0.01], but there was no effect of consensus face trustworthiness on memory for words [F(1,16) < 1].

Faces of different consensus trustworthiness levels elicited ERPs that clearly diverged in the time period between 200 and 400 ms after stimulus onset, as shown in Figure 2 A. These differences gave rise to a significant main effect in the frontal cluster [F(2,32) = 4.2, p < 0.05], driven by a greater positivity for CT faces compared to CN faces [t(16) = 2.4, p < 0.05] and CU faces [t(16) = 2.6, p < 0.05]. This divergence remained significant at 400–600 ms [F(2,32) = 4.4, p < 0.05], again driven by a greater positivity for CT faces compared to CN faces [t(16) = 2.6, p < 0.05] and CU faces [t(16) = 2.5, p < 0.05].

To assay correlates of memory-based processes, rating-phase trials were sorted as a function of memory performance in the test phase (Figure 2 B). This memory performance was likely driven both by encoding in the initial test phase, and rehearsal during the rating phase. Regardless, because the words were not repeated during the rating phase, ERP correlates of test-phase memory performance must reflect processing of previously learned information. In line with previous studies of ERP correlates of successful memory retrieval (Voss and Paller, 2008 ), ERPs from the parietal cluster were more positive at 400–600 ms when corresponding words were remembered compared to when they were forgotten [t(16) = 2.4, p < 0.05]. This difference continued in the interval from 600–800 ms [t(16) = 2.1, p = 0.05].

It is possible that the relatively early frontal ERP divergence between CT and CU faces was driven by subjects’ ultimate behavioral ratings, rather than by perceptual features per se. In this case, one would expect to find the same difference between ERPs to faces previously paired with positive versus negative words, as this manipulation affected behavioral judgments even more strongly than perceptual features. However, this comparison revealed no differences prior to 600 ms [t(16) < 1]. Instead, a later effect at 800–1000 ms was found (Figure 2 C), in which parietal ERPs to faces shown previously with negative words were more positive than those to faces shown previously with positive words [t(16) = 2.1, p = 0.05]. Electrode clusters used for statistical analyses are displayed in Figure 2 D.

Separate ERP correlates of trustworthiness-relevant perceptual and memory processes were identified. As expected, signals of perceptual processing appeared sooner than signals of memory processing. In addition, ERPs associated with consensus ratings were not simply an indirect reflection of behavioral judgments; the same ERPs were not found when comparing responses to faces previously paired with positive words to those paired with negative words even though paired-word valence had a strong influence on behavioral judgments.

Experiment 1 demonstrated that trustworthiness ratings were influenced by perceptual features and by memory retrieval, and that the influence of memory declined when people were forced to make ratings quickly. Converging results from Experiment 2 showed that neural correlates of perceptual information processing and memory retrieval relevant to trustworthiness can be measured separately, and that signals of perceptual processing appeared sooner than those of memory retrieval. Together, these results suggest that perceptual information relevant to face trustworthiness is processed more quickly and can influence behavior sooner than memory-based information.

Three relevant electrical signals were identified: an early frontal correlate of consensus trustworthiness, a later correlate of memory retrieval with a parietal topography, and an even later correlate of word valence that also exhibited a parietal topography. The first signal is likely to reflect the analysis of facial expressions, as observed previously (e.g., Eimer and Holmes, 2007 ). Other results have shown that subtle facial expressions can drive trustworthiness judgments (Winston et al., 2002 ; Todorov, 2008 ; Oosterhof and Todorov, 2009 ). ERPs may have been most positive for CT faces because of unambiguously happy expressions for these faces compared to neutral and untrustworthy faces. It is presently unclear whether consistent trustworthiness ratings generally arise from the gross categorization of facial expression or whether more subtle cues are operative. The ERP correlate of consensus trustworthiness found in the current study does not reflect subjects’ behavioral ratings alone, and as such it displays a pattern similar to that of amygdala activation in fMRI studies (Engell et al., 2007 ).

The second ERP signal, a correlate of later memory retrieval, occurred with a latency and distribution paralleling effects found in other memory paradigms (Voss and Paller, 2008 ). This resemblance adds weight to the hypothesis that influences on trustworthiness judgments from conscious memory retrieval occur later than relevant perceptual processes.

The late difference found in the comparison of ERPs to faces previously paired with positive versus negative words may reflect better memory for negative words. Though it appears later than typical ERP correlates of conscious memory retrieval (Voss and Paller, 2008 ), the pattern of increased parietal positivity for faces previously paired with (better remembered) negative words fits with such an account.

The longer response times in the speeded condition when there was conflict between perceptual features and learned information suggests that there is some integration of information from these two sources and that this integration may be more efficient or straightforward when the information is consistent. The negative correlations between perceptual and memory-based influences suggest that individuals weight certain information sources more heavily at the expense of others, and that the pattern of source weights varies across people. Previous studies of the neural integration of conflicting information, with Stroop procedures (West and Alain, 1999 ; Liotti et al., 2000 ; Fruhholz et al., 2009 ) and face evaluation (Fruhholz et al., 2009 ; Schiller et al., 2009 ), relied only on perceptual information. The current study goes beyond these prior studies in showing that neural measures can be used to monitor the separate processing of information from perceptual and memory-based sources.

The interplay of the neurocognitive processes investigated here is likely quite common in everyday life; when individuals make rapid judgments about others, even in the absence of a formal rating, they may be highly influenced by perceptual features and discount memory-based sources of information. Stereotyping based on physical features may be a particularly salient example of this; individuals may have learned information contrary to stereotypes, but because of the time required to access this information, time pressure or cognitive load may often produce stereotype-based errors (Sherman et al., 1998 ; Correll et al., 2002 ). Initial personal judgments are quite persistent over time (Ambady and Rosenthal, 1993 ). Such judgments can influence behavior not only in the context of artificial social interactions but also in real-world situations such as when one makes financial decisions with input from professional consultants, chooses among multiple job applicants, or selects which political candidates to support (Langlois et al., 2000 ; Todorov et al., 2005 ; van ’t Wout and Sanfey, 2008 ). Thus, understanding the nature of the perceptual and memory-based neurocognitive processes pertaining to trustworthiness has wide-ranging implications.

The current results are consistent with evidence that perceived and remembered information influences trustworthiness judgments through separate neural processes (Engell et al., 2007 ; Todorov et al., 2007 ; Todorov and Olson, 2008 ). Our findings further provide behavioral and neural evidence that the perceptual processing precedes memory retrieval. Accordingly, information from perceptual analysis can influence trustworthiness assessments well before relevant information has been retrieved from memory. More generally, this conclusion may apply in the same way when information from perception and memory sources can be brought to bear in a variety of decision-making circumstances.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

This research was supported by a National Science Foundation Graduate Research Fellowship to John D. Rudoy, and by research grants from the National Science Foundation (BCS 0518800 and BCS 0818912).