One key issue in face study is to understand how emotional expression is represented in the human visual system. According to the classical cognitive model (Bruce and Young, 1986) and neural model (Haxby et al., 2000), emotional expression is consider to be represented and processed independent of facial identity. This view is supported by several lines of evidence. First, neuropathological studies have reported that some brain injury patients exhibit selective impairments of ability to recognize expression but not face, or vice versa (e.g., Bruyer et al., 1983; Young et al., 1993; Palermo et al., 2011), indicating a double dissociation between the representations of identity and expression. Second, early behavior studies found no difference in reaction times when making expression matching decisions to familiar and unfamiliar faces (e.g., Bruce and Young, 1986; Ellis et al., 1990), suggesting that expression processing does not depend on the familiarity of facial identity. Finally, single-unit recording and fMRI studies have found that expression may be processed in the superior temporal sulcus, located in lateral occipitotemporal cortex, whereas the process of facial identity preferentially involves the face fusiform area (FFA), located in inferior occipitotemporal cortex (e.g., Hasselmo et al., 1989; Haxby et al., 2000; Winston et al., 2004).

Despite substantial evidence supporting independent representation of expression, there are a growing number of studies suggesting that expression representation is related to facial identity. Using a Ganer's speeded-classification task, variation in identity was found to interfere with the performance of expression classification (Schweinberger and Soukup, 1998; Schweinberger et al., 1999) or vice versa (Ganel and Goshen-Gottstein, 2004, for a recent further discussion, see Wang et al., 2013). On the other hand, happy expression was reported to improve the familiarity ratings of face (Baudouin et al., 2000) and the identity recognition speed (e.g., Kaufmann and Schweinber, 2004). Furthermore, Chen et al. (2011) compared effects of all basic facial expressions on identity recognition, and observed the different reaction times when faces are matched in different expression conditions, indicating the modulation of expression on the processing of face identity. In fact, besides emotional expression in face, body expressions and emotional cues in the background scene could also affect the processing of facial identity (Van den Stock and de Gelder, 2012, 2014), suggesting an general interaction mechanism underlying the processing of emotion and facial identity. In support of these behavior studies, single-cell studies have identified certain cells in monkey STS area (e.g., Sugase et al., 1999) and amygdale area (e.g., Gohard et al., 2007) which are responsive to both identity and expression stimuli, indicating that representations of emotional expression and facial identity may be overlapped.

In fact, concerning the relationship between expression and identity representations, besides the radical view of complete independence or the opposing view of highly dependence, there is also a third view to describe representation of expression, i.e., there are two types of expression representations, one depends on facial identity while the other is independent of facial identity. In a pioneer study, Fox and Barton (2007) have provided the evidence for this view of partial dependence, using the cross-identity expression adaptation. Adaptation refers to the phenomenon that prolonged exposure to a given stimulus would result in a subsequent perceptual bias (i.e., aftereffect). For instance, after viewing a unidirectional moving stimulus, an observer will perceive a subsequent stationary stimulus as moving in the opposite direction. As visual adaptation can reflect short-term responsiveness changes in the human neural system, it has been well developed to study the processes of low-level visual properties (for review see Durgin and Proffitt, 1996; Anstis et al., 1998; Clifford, 2002), and complex visual stimuli, such as various facial dimensions (e.g., Leopold et al., 2001; Zhao and Chubb, 2001; Rhodes et al., 2003; Hsu and Young, 2004; Yamashita et al., 2005, for review see Webster et al., 2004; Webster and MacLeod, 2011). As far as expression adaptation are concerned, within a series of ambiguous expression images morphing between two expressions (e.g., happy and angry faces), adaptation to one expression (e.g., happy face) will distort the perception of observers to facilitate perceiving the ambiguous images as the other expression (i.e., angry expression). The magnitude of expression aftereffect can be indexed by the differences in response proportions of subjects before and after adaptation.

In Fox and Barton's adaptation experiments, expression aftereffect is measured using adaptors and tests that were both the same and different identities. The expression aftereffect is strongest when the identities of the adaptor and test faces were the same, however, it is still present, but much reduced when the adaptor and test faces differ in facial identity. These results were interpreted as evidence that there are two different neural representations in expression system, i.e., identity-dependent and identity-independent expression representations. If the adaptor and test faces are congruent, both identity-dependent and identity-independent expression representations are adapted and contribute to the expression aftereffect; if the adaptor and test faces are incongruent, only identity-independent expression representation contributes to the expression aftereffect. This explains why the expression aftereffect in incongruent condition is much weaker than that in congruent condition.

In support of Fox and Barton, several subsequent works replicated this cross-identity expression adaptation with a variety of experimental paradigms and stimuli (Ellamil et al., 2008; Campbell and Burke, 2009; Vida and Mondloch, 2009; Pell and Richards, 2013). The cross-identity expression aftereffect was found to occur generally in five basic expressions with approximately the same extent of transfer, suggesting different expression representations depend on identity in a similar way (Campbell and Burke, 2009). A recent study further reported that the overlapping expression representations (e.g., the representation for similar emotional features of disgusted and angry faces) are also identity-dependent, it was found that the adaptation to a disgusted face would bias perception away from angry face and this cross-emotion expression adaptation generalizes across identity (i.e., Pell and Richards, 2013). Additionally, Vida and Mondloch (2009) shows that children (5–9-year-olds) display an adult-like transfer of expression aftereffect across identities.

The above studies provided strong evidence for the existence of identity-dependent and identity-independent expression representations, little is, however, known about the nature of these two different types of expression representations, especially about their sensitivity to expression configuration. Expression can be viewed as stereotyped geometrical variations in facial configurations that correspond to well-defined action patterns (Webster and MacLeod, 2011). Although there is only a small set of six basic expressions, one expression can be conveyed with different varieties of expression configurations within an emotion category to present subtle emotional information (Rozin et al., 1994). For instance, humans can open their lips and show their teeth to express a high-intensity smile, or they can simply crinkle the corner of their eye to express a weak-intensity smile. So far, there is relatively little knowledge regarding the sensitivity to expression configuration of the identity-dependent and identity-independent expression representations, this issue is, however, helps to obtain a better understanding for the processing mechanism of the human expression system, as well as for an understanding for the functional difference between the identity-dependent and identity-independent expression systems.

The expression aftereffect is an effective measure to assess the sensitivity of expression system to expression configuration. Fox and Barton (2007) have showed that the magnitude of expression aftereffect was almost not affected when the adaptor was changed to a picture of the same expression in the same individual but that differs from the image used to create the morphed test stimuli, indicating that the expression aftereffect is not sensitive to the subtle change in the expression configuration to some extent. However, the different expression images of the same expression by the same person are still highly similar; this finding cannot well exclude the influence of expression configuration on expression aftereffects. In contrast, several other studies suggested that the expression aftereffect depends on the expression configuration. Butler et al. (2008) showed that the expression aftereffect can be produced by emotional facial features, but not by the same emotional features in a hybrid face with inconsistent expression configurations. On the other hand, Skinner and Benton (2010) reported that adaptation to the anti-expression, which is defined by the physically complementary expression configuration, would induce a significant expression aftereffect on the original emotional expression matching to the anti-expression (see also Cook et al., 2011; Juricevic and Webster, 2012). As the anti-expression does not convey the emotional information, this observation also reflects that the expression aftereffect is closely related to the spatial configuration of expression.

Although the previous studies have provided the cue about how expression configuration influences expression aftereffect, there is no studies directly examining this issue in a framework comprising identity-dependent and identity-independent elements. This is important because the identity-dependent and identity-independent expression representations may reside in different brain areas (Fox and Barton, 2007), and have different sensitivities to expression configuration (Harris et al., 2012, 2014). Morris et al. (1996) and Thielscher and Pessoa (2007) have previously reported that responses in the amygdala can be modulated by changes in the emotion's intensity of expression images, suggesting that the amygdala contain neuron population that tuned to the expression configuration. In contrast, in the recent studies, Harris found that amygdala area is relatively robust to expression change within the same emotion category in comparison to posterior superior temporal sulcus (pSTS), using static expression images (Harris et al., 2012) or dynamic movie as stimuli (Harris et al., 2014). Although the results from above studies are not completely consistent, the current evidence indicates that the brain areas may have the different neural sensitivities to the expression configuration.

In current study, we used the cross-identity expression adaptation (Fox and Barton, 2007) to investigate the sensitivities of identity-dependent and identity-independent expression representations, respectively. The experimental logic for adaptation is as follows: Given that the adaptation with the repedition of the completely same stimulus would reduce the neural response and induce the aftereffect to the maximum extent, the reduction of aftereffect should be observed when a particular dimension of stimulus is changed, provided that the underlying neural system codes that dimension. The aftereffect would become weak because the altered stimulus activates a new, non-adapted neural representation. In contrast, the aftereffect would remain the same if the underlying neural system is insensitive to differences along the altered stimulus dimension (Grill-Spector and Malach, 2001). As far as our research question is concerned, if the identity-dependent neural representation is sensitive to the expression configuration, the identity-dependent expression aftereffect should be reduced when the adaptor is changed to a expression configuration different from the tests; if identity-dependent neural representation is regardless of the processing of expression configuration, the expression aftereffect should be robust to the change in the expression configuration. The logic for identity-independent expression representation is similar.

Two experiments were performed in this study. In Experiment 1, we measured the expression aftereffect in an orthogonal 2 × 2 design, manipulating whether the adaptor and test faces exhibited the same or different facial identities, and independently, whether the expression configurations on the identities were the same or different. Experiment 1 thus includes the following 2 × 2 combinations (same or different: facial identity and expression configuration between adaptor and test faces): same identity/same configuration, same identity/different configuration, different identity/same configuration, different identity/different configuration. Given this factorial design, if identity-dependent and identity-independent expression representations have similar neural sensitivities to expression configuration, it is expected to observe that expression configuration change has similar effect on expression aftereffect regardless of the identities of adaptors and tests. In other word, there should be no the significant interaction between facial identity and expression configuration. On the contrary, if the sensitivities of these two expression representations are different, the interaction is expected to be observed. More specifically, if identity-dependent expression representation is sensitive to expression configuration, the expression aftereffect in same identity/different configuration condition should be weaker than that in the same identity/same configuration condition, otherwise, the aftereffect size in the two conditions should be approximately the same. Similarly, if identity-independent expression aftereffect is sensitive to expression configuration, the expression aftereffects in different identity/different configuration is expected to be weaker than that in different identity/same configuration condition, otherwise, we expect to observe the approximately same aftereffect in these two conditions.

In Experiment 2, we repeated the different identity/different configuration experimental condition using schematic line face as the adaptor and real faces as tests, which minimized the similarity in face features between adaptor and tests. Using these dissimilar stimuli, interference is expected to be excluded from the identity-dependent expression representation, and the effect of expression configuration on identity-independent expression representation to the expression aftereffect can be accurately evaluated.

The expression aftereffect is measured in the following way. We created morph series using two expression images of the same person. The images in the middle range of the morph series thus showed a recognizable identity but an ambiguous expression cue. After adapted to one expression, the subject was instructed to judge which expression the ambiguous morphing expression resembled. Generally, adaptation to one expression increased the possibility the ambiguous expression was judged as the other expression.

Experiment 1 included 4 adapting conditions as described in the introduction part, in which test faces were always the same among conditions, but the identity and/or the expression configuration of the adaptor were manipulated to create the different adapting conditions. The four conditions are respectively termed as SI/SC, SI/DC, DI/SC, and DI/DC, where the first two letters indicates whether adaptor and test face are of the same identity (SI: Same Identity) or not (DI: Different Identity), and last two letters indicates whether the expression configuration of adaptor are the same with that of test face (SC: Same Configuration) or not (DC: Different Configuration). For instance, SI/SC refers to the condition in which both the identity and the expression configuration of adaptor are identical to those of test face, whereas DI/DC means that adaptor differentiates with the test face in both identity and expression configuration.

Although Experiment 1 shows no significant difference in aftereffect size between different identity/different configuration and different identity/same configuration conditions, indicating identity-dependent expression aftereffect is robust to expression configuration change. The statistical power for this observation is, however, not high, which is due to the interference of identity-dependent expression representation. Here, we further confirmed this observation using schematic line face as the adaptor and real faces as tests. It is well established that the adaptation effect is modulated by the perceptual similarity between adaptor and test. As the line face is totally dissimilar with real face and thus only conveys the emotional information of an expression, using the line face as adaptor could exclude the possible contribution from the identity-dependent neural representation to expression aftereffect. Besides, the line face enables us to manipulate more precisely the expression configuration and/or expression intensity. If the identity-independent expression aftereffect is robust to the variance in expression configuration, it can be predicted that the magnitude of the expression aftereffect should be approximately equal, even with different expression configurations. Otherwise, the aftereffect magnitude should differ significantly from different expression configurations.

Our results first confirmed that the expression aftereffect is reduced when the facial identity is manipulated, a finding that several studies have observed before. This observation does not relate to the specific expression pairs, as the similar expression aftereffect pattern was observed in both the happy-angry and in surprised-disgusted expression pairs. Campbell and Burke (2009) also found that the extent of the aftereffect reduction across identities was almost the same for five basic emotional expressions, although the processing of these different expressions appears to involve different neural mechanisms and visual areas (e.g., Posamentier and Abdi, 2003). This suggests the processing of individual facial expressions depend on facial identity similarly.

The reduction in the cross-identity expression aftereffect is not simply due to change in expression configuration between adaptor and test faces, although a change in expression configuration does influence expression aftereffect. The result in Experiment 1 shows that the expression aftereffect is still reduced even when the adaptor and test faces have identical expression configurations, suggesting the functional influence from facial identity on expression aftereffect. Such observation is consistent with Ellamil et al.'s (2008) work, which used artificial faces of angry and surprised expressions as adaptors and found similar reduction in the adaptation effect when the adaptor and test faces had the same expression morphing prototype but different facial textures and contours. Together, our and Ellamil et al.'s (2008) observation consolidates Fox's proposal of the identity dependent neural representation.

We further determined how a change in the expression configuration influences the identity- dependent and identity-independent expression aftereffects, respectively. This was achieved using real faces as adaptors in Experiment 1 and using line faces as adaptors in Experiment 2. The results were consistent in two experiments. Experiment 1 found that a change in expression configuration impaired the identity-dependent expression aftereffect, but not the identity-independent expression aftereffect. The Experiment 2 further confirmed the observation of Experiment 1 using the line faces as adaptor. We found that the identity-independent expression aftereffect is consistently robust to the variance in the expression configuration relative to the identity-independent expression aftereffect.

Why do identity-dependent and identity-independent expression systems show different sensitivities to expression configuration? The possible explanation is that these two systems depend on the different facial components to process emotional expressions. As the structural reference hypothesis stated (Ganel and Goshen-Gottstein, 2004), the face structure information is not only important to identity discrimination, but it is also used by observers as a reference to compute and recognize expressions. The identity-dependent expression system may rely more on the facial shape and/or structure information (e.g., local edge, facial contour) to perform an expression configuration analysis, thus it is sensitive to change in spatial configurations of expressions. In support of this notion, Neth and Martinez (2009) have demonstrated that simple structural changes in emotionless faces could induce the virtual perception of facial expression, with shorter vertical distance between eyes and nose resulting in the perception of anger, and the larger distance leading to the perception of sadness. On the other hand, the identity-independent expression system seems to depend on the information of emotion category; thus, it is more robust to the detailed variance in the configuration properties of the face. Just as in daily life, although the happy expression can be displayed with different configurations or different intensities in the face, all these expressions would be perceived as a happy emotion within a certain range of expression intensities and configurations. It is possible for the identity-dependent expression system to interpret the spatial configuration of an expression, as well as for identity-independent neural system to receive the input of identity-dependent expression system and extract the emotional information from the facial expression.

There is a long-term debate whether facial expressions are represented as belonging to discrete categories of emotion or as continuous dimensions (see Bruce and Young, 2012; Harris et al., 2012). The category model is supported by the evidence that faces within a category were discriminated more poorly than faces in different categories that differed by an equal physical amount (Etcoff and Magee, 1992; Calder et al., 1996). In contrast, the observation that human can accurately perceive differences in intensity (Calder et al., 1996) and variation (Rozin et al., 1994) of a given emotional expression is consistent with the continuous model. Our result supports a synthesis of the above two models by showing that expression is represented in both a categorical and a continuous ways. Specifically, the identity-independent expression system is robust to variance in expression configurations within same emotion category, suggesting it may perceive emotional expressions in a category manner; In contrast, the high sensitivity to expression configuration supports a continuous manner in the identity-dependent expression system.

We suggest two possible candidates of brain areas responsible for identity dependent expression representation. One possible candidate of brain area responsible for the identity-dependent expression representation is the posterior superior temporal sulcus (STS). A recent fMRI adaptation study investigated adaptation to both facial identity and expression (Winston et al., 2004). Adaptation to identity but not expression was found in the fusiform face area, and adaptation to expression but not identity was found in the middle STS. In the posterior STS, there were large adaptation effects to identity and a smaller adaptation effect to expression. These observations suggest that the posterior STS area may encode both identity and expression. Future researches are desirable to examine whether the adaptation effect is sensitive to the change in expression configuration in the same person in the posterior STS. The other candidate for the identity-dependent expression representation is the FFA. Although this area is generally believed to be responsible for the processing of static information of the face in identity discrimination (Haxby et al., 2000; Winston et al., 2004), the FFA is also found to be sensitive to variations in expression even when attention was directed to identity (Ganel et al., 2005). The findings fit with our behavior data and suggest an interactive system for the processing of expression and identity in the FFA area.

It is relatively difficult to infer the possible brain area related to identity-independent expression representation. Although Winston et al. (2004) found that repeating emotional expressions across different face pairs led to reduced signal in the middle STS, showing the identity-independent processing of expressions in the middle STS. However, the middle STS is sensitive to the dynamic and transient changes in facial features, which contrasts with the robustness of the identity-independent expression aftereffect to expression configuration observed in current study. We suggest that the amygdala is a possible candidate. This area is highly related to the processing of emotional expressions, and has previously found to be insensitive to expression change within the same emotion category (Harris et al., 2012), which well fits with our result observed in identity-independent expression aftereffect. On the other hand, it is worth pointing out that the amygdala may be not the only area involved in identity-independent processing of expressions. As amygdala is generally related to processing of fear and happy expressions (Morris et al., 1996; Blair et al., 1999), and the other expressions, such as disgusted expression, could not induce the activation in the amygdala (Phillips et al., 1997). The disgusted expression, however, shows the similar result as the fear expression in first experiment. This appears to suggest that other brain areas besides amygdala were also involved in identity-independent expression processing. It would be interesting for future studies to investigate whether the activation of the other expression-related brain areas are robust to the change in expression configuration using an fMRI adaptation paradigm.