# TYPICAL AND ATYPICAL PROCESSING OF GAZE

EDITED BY : Chris Ashwin and Paola Ricciardelli PUBLISHED IN : Frontiers in Psychology and Frontiers in Neuroscience

#### Frontiers eBook Copyright Statement

The copyright in the text of individual articles in this eBook is the property of their respective authors or their respective institutions or funders. The copyright in graphics and images within each article may be subject to copyright of other parties. In both cases this is subject to a license granted to Frontiers. The compilation of articles constituting this eBook is the property of Frontiers.

Each article within this eBook, and the eBook itself, are published under the most recent version of the Creative Commons CC-BY licence. The version current at the date of publication of this eBook is CC-BY 4.0. If the CC-BY licence is updated, the licence granted by Frontiers is automatically updated to the new version.

When exercising any right under the CC-BY licence, Frontiers must be attributed as the original publisher of the article or eBook, as applicable.

Authors have the responsibility of ensuring that any graphics or other materials which are the property of others may be included in the CC-BY licence, but this should be checked before relying on the CC-BY licence to reproduce those materials. Any copyright notices relating to those materials must be complied with.

Copyright and source acknowledgement notices may not be removed and must be displayed in any copy, derivative work or partial copy which includes the elements in question.

All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For further information please read Frontiers' Conditions for Website Use and Copyright Statement, and the applicable CC-BY licence.

ISSN 1664-8714 ISBN 978-2-88963-605-1 DOI 10.3389/978-2-88963-605-1

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# TYPICAL AND ATYPICAL PROCESSING OF GAZE

Topic Editors: Chris Ashwin, University of Bath, United Kingdom Paola Ricciardelli, University of Milano Bicocca, Italy

Citation: Ashwin, C., Ricciardelli, P., eds. (2020). Typical and Atypical Processing of Gaze. Lausanne: Frontiers Media SA. doi: 10.3389/978-2-88963-605-1

# Table of Contents


Michelle Jarick and Renee Bencic

# Editorial: Typical and Atypical Processing of Gaze

#### Chris Ashwin<sup>1</sup> \* and Paola Ricciardelli <sup>2</sup> \*

*<sup>1</sup> Centre for Applied Autism Research, Department of Psychology, University of Bath, Bath, United Kingdom, <sup>2</sup> Department of Psychology, University of Milano-Bicocca, Milan, Italy*

Keywords: eyes, gaze processing, gaze direction, mental states, attention, autism spectrum disorder

**Editorial on the Research Topic**

#### **Typical and Atypical Processing of Gaze**

Eye gaze represents a central source of socially meaningful cues which helps convey key information about the focus of others attention. Gaze direction is used for non-verbal communication, which includes making inferences about the mental states, intentions, beliefs, and desires of other people. The direction of gaze is such an automatic and powerful cue that it triggers the observers' own attention in the same direction, even when that information is uninformative, and allows individuals to build up shared representation of the inner and outside world.

Despite the importance of gaze processing in the social world we still know surprisingly little about how we process this information, including the cognitive and neural mechanisms and the factors affecting gaze processing. This includes about how gaze processing is different in various disorders with difficulties in social cognition, such as autism, which is reported to involve reduced expertise for perceiving gaze direction (Ashwin et al., 2009). The contributions of this Research Topic have utilized various research methodologies and techniques testing the effects of different factors on gaze processing toward a better understanding about the underlying mechanisms across different types of samples e.g., typical controls, brain-damaged patients or those with various disorders. The studies in this eBook also test how we integrate the different types of information conveyed by gaze with the context and our environment. The studies and reviews are briefly summarized along with some remarks about their contributions.

# FACTORS AFFECTING GAZE PROCESSING IN TYPICAL SAMPLES

A number of factors related to gaze processing were investigated by the studies reported, with the first section including studies involving typical control participants. The role of situational factors was examined using an innovative experimental design in the study by Balsdon et al., where participants were asked either to make a left/right gaze judgement (single task—directional) or direct/indirect gaze judgement (two-interval task—non-directional). Results showed that observers integrated the same sensory information from head orientation and gaze direction in a flexible manner, giving them a different weight depending on the kind of judgment made in-line with previous studies (e.g., Ricciardelli and Driver, 2008). The effects of eccentricity of gaze processing was tested in a study by Yokoyama and Takeda, where they investigated whether the gaze cuing effect requires foveal vision. Results showed that this effect occurs even when the gaze stimuli appear in peripheral vision up to 5◦ of eccentricity from the attentional focus. The study by Awad et al. tested the importance of the context and the emotional expressions of the face being judged on gaze processing, reporting that the perception of gaze direction depended on both the face

#### Edited and reviewed by:

*Rufin VanRullen, Centre National de la Recherche Scientifique (CNRS), France*

#### \*Correspondence:

*Chris Ashwin c.ashwin@bath.ac.uk Paola Ricciardelli paola.ricciardelli@unimib.it*

#### Specialty section:

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology*

Received: *25 September 2019* Accepted: *31 October 2019* Published: *19 November 2019*

#### Citation:

*Ashwin C and Ricciardelli P (2019) Editorial: Typical and Atypical Processing of Gaze. Front. Psychol. 10:2576. doi: 10.3389/fpsyg.2019.02576*

**4**

eccentricity and its emotion. Together, these studies demonstrate that you do not need to be looking directly at someone for their gaze to affect your attention, which may be how gaze perception is often done in real-life social situations.

Tipples et al. investigated whether a gaze-liking effect, which involves an increase in the likeability of objects which have been repeatedly gazed at (e.g., Bayliss et al., 2006), can also occur for verbal descriptions of looking behavior without the need of attentional shifts. The authors compared the effect of pointing gestures and arrow cues and the results provide partial support for the gaze specificity hypothesis. However, the liking effect was enhanced for gaze cues compared to arrows indicating that eye gaze is a highly effective cue. The effect of eye contact in memory recognition has been shown in the study by Lanthier et al., who manipulated several contextual factors (i.e., live eye contact, the temporal dynamics and the communicative meaning of the gaze, and gender) during a word memory recognition task and reported that eye contact improves recognition, but only in females. A study by Capellini et al. showed that experiencing social exclusion alters joint attention behavior. They used the Cyberball manipulation to induce feelings of ostracism. The socially excluded participants had reduced attentional shift toward averted gaze than those who were included, although pointing arrows elicited an attentional orienting response in both groups. This may be because the ostracized participants perceive averted gaze as a further sign of exclusion. These last three studies showed that gaze processing can modify other cognitive processes such as word processing and memory and even judgments about the self.

The effects of task demands were investigated by McCrackin and Itier who measured EEG during tasks of emotion recognition, direction of attention, and gender discrimination involving images of individuals with direct or averted gaze. They report differences in accuracy and also neural processing as measured by ERP's, based on whether the gaze in the individuals being perceived was direct or averted across the tasks. The article by Jarick and Bencic measured physiological arousal during a live task with dyads of strangers and found greater arousal when both participants were making eye contact with each other (i.e., both sending and receiving gaze information), compared to other conditions where only one person was sending/receiving gaze information or no gaze information was communicated between them. Both the studies described in this paragraph helped reveal about the underlying neural and physiological effects of gaze direction and eye contact.

# TYPICAL AND ATYPICAL GAZE IN DISORDERS AND PATIENTS

Lopis and Conty investigated whether there might be enhanced memory for face-name associations in people with Alzheimer's disease when images of other individuals contain direct gaze than averted gaze, based on previous reports of direct gaze improving memory for subsequent faces and verbal information. They did not find enhanced memory for face-name associations in Alzheimer patients, but across all participants, they found that direct gaze context produced greater memory for faces and for names presented independently. Therefore, some gaze effects are preserved in both normal aging and Alzheimer's disease. A study by Tsuji and Shimada used psychophysics measures to investigate the impact of socially anxious tendencies on gaze perception. They tested the presence of a negative bias when judging the valence (positive or negative) of gaze in emotionally ambiguous faces using only the eye region of either disgusted, happy or neutral faces. They found that lower negative intensity ratings of gaze were associated with higher social anxiety scores, suggesting that those with higher phobia tendencies experience more anxiety toward the gaze of others, likely because they perceive it as threatening.

The importance of eye gaze in driving attention was examined in a sample of patients with unilateral neglect in the study by Rato et al. The authors used a modified version of a cancellation test, where the targets to be canceled were direct gaze or averted gaze stimuli appearing among the distracters with closed eyes. Their findings speak in favor of the effectiveness of direct gaze to capture attention and improve neglect, which is characterized by an attentional orienting deficit. The article by Del Bianco et al. included an eyetracking paradigm used to investigate attention to faces in adults with and without autism spectrum disorder (ASD), and reports that people with ASD showed initial attention to faces comparable to controls, and they had even greater fixation times on bodies and faces than controls, which is contrary to what is typically reported and theorized in ASD but may reflect greater processing demands of social information which require greater attention. Interestingly, those with ASD were more influenced than controls by the different task instructions by showing variable responses across conditions than controls. These studies on pathological and atypical individuals showed the importance of gaze perception in humans because many of the samples showed intact processing of the gaze of others and typical effects of gaze on other cognitive processes.

# REVIEWS ABOUT GAZE PROCESSING

Three of the contributions to the Research Topic were literature reviews about gaze processing. Clifford and Palmer provide a review about the phenomenon of adaptation to the gaze direction of others, which is a perceptual aftereffect that occurs after viewing someone gazing in a specific direction for a long period of time (i.e., after viewing a specific gaze direction subsequent gaze directions are perceived differently). They report about studies investigating whether the effect occurs at higher or lower levels of processing and proposed potential mechanisms of adaptation and gaze coding, and how gaze adaptation is reported to be different in autism. Hietanen reviews the evidence from studies reporting about the effects of eye contact on affective reactions, and proposes that the physiological and cognitive reactions of emotional arousal are clearer for studies involving the implicit

processing of eye contact compared to those involving explicit processing and offers some interesting ideas about the potential underlying mechanisms. A review by Cañigueral and Hamilton highlights the dual functioning of eye gaze for having roles in both perceiving social information and signaling social information to others, and with the roles being more evident during face-to-face interactions over typical lab experiments using only one mode of communication (e.g., a participant viewing a computer screen with image of someone gazing). They propose the Interpersonal Gaze Processing model and discuss differences in interactive gaze functioning in autism. Together, these reviews give a new and updated view of the literature about the neural, physiological and cognitive

#### REFERENCES


mechanisms of gaze processing and provide some directions for future research.

# AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

# FUNDING

CA was supported by The Leverhulme Trust, grant no. RFG-2015-400. PR was supported by Fondazione Cariplo and Regione Lombardia, grant no. 2016-0925.

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Ashwin and Ricciardelli. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Affective Eye Contact: An Integrative Review

Jari K. Hietanen\*

Human Information Processing Laboratory, Faculty of Social Sciences, University of Tampere, Tampere, Finland

In recent years, many studies have shown that perceiving other individuals' direct gaze has robust effects on various attentional and cognitive processes. However, considerably less attention has been devoted to investigating the affective effects triggered by eye contact. This article reviews research concerning the effects of others' gaze direction on observers' affective responses. The review focuses on studies in which affective reactions have been investigated in well-controlled laboratory experiments, and in which contextual factors possibly influencing perceivers' affects have been controlled. Two important themes emerged from this review. First, explicit affective evaluations of seeing another's direct versus averted gaze have resulted in rather inconsistent findings; some studies report more positive subjective feelings to direct compared to averted gaze, whereas others report the opposite pattern. These contradictory findings may be related, for example, to differences between studies in terms of the capability of directgaze stimuli to elicit feelings of self-involvement. Second, studies relying on various implicit measures have reported more consistent results; they indicate that direct gaze increases affective arousal, and more importantly, that eye contact automatically evokes a positively valenced affective reaction. Based on the review, possible psychological mechanisms for the positive affective reactions elicited by eye contact are described.

#### Edited by:

Paola Ricciardelli, Università degli Studi di Milano Bicocca, Italy

#### Reviewed by:

Isabelle Mareschal, City, University of London, United Kingdom Yumiko Otsuka, University of New South Wales, Australia

#### \*Correspondence:

Jari K. Hietanen jari.hietanen@uta.fi

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 10 March 2018 Accepted: 08 August 2018 Published: 28 August 2018

#### Citation:

Hietanen JK (2018) Affective Eye Contact: An Integrative Review. Front. Psychol. 9:1587. doi: 10.3389/fpsyg.2018.01587 Keywords: affect, arousal, brain, emotion, eye contact, gaze, face, psychophysiology

# INTRODUCTION

Other individuals' gaze is a powerful social stimulus. Gaze direction is used to regulate interaction, to facilitate communicational goals, and to express intimacy and social control, to name some of its important functions in the modulation of social interaction processes (Kleinke, 1986). Most importantly, perhaps, other individuals' directed gaze signals their direction of attention. We use others' gaze to discriminate and infer where they have directed their attention—what or who are they looking at. Arguably, the most important discrimination is whether other individuals have directed their eyes toward me or away from me. Seeing other individuals' eyes directed at me indicates, with a high probability, that they are attending to me, whereas seeing other individuals' averted gaze signals their attention to be directed away from me.

Extensive lines of research have shown that others' gaze direction has effects on an observer's own attention. Direct gaze has been shown to induce attention orienting toward faces (von Grünau and Anston, 1995; Senju et al., 2005; Conty et al., 2006; Doi et al., 2009; Shirama, 2012; Böckler et al., 2014; Lyyra et al., 2017; for a critical view regarding the results from visual search studies, see Cooper et al., 2013), whereas seeing another individual with a gaze directed away from oneself triggers the re-orienting of one's visuospatial attention in the gazed-at direction (e.g., Friesen and Kingstone, 1998; Driver et al., 1999; Hietanen, 1999; Langton and Bruce, 1999; for a review, see Frischen et al., 2007).

**7**

However, the present article aims to review research concerning the effects of others' gaze direction on observers' affective responses. This research has received considerably less attention compared to research on the effects of perceived gaze direction on attention. Yet, in our every-day life, we often associate these two. When speaking in front of an audience, a direct gaze cast even by just one of the listeners may feel pleasant and comforting, whereas a gaze aversion by an individual you are approaching in a party makes you feel insecure and uncomfortable. On the other hand, waiting for the last bus at the bus stop at midnight and seeing somebody looking at you may evoke very negative feelings, whereas you feel relaxed when this individual averts gaze away from you. Thus, one's interpretation of the meaning of another's gaze is, of course, contingent upon a number of antecedent, concurrent, and anticipated contextual factors. Moreover, the gazer's verbal and non-verbal behavior, most importantly the verbal content and facial expressions, can have a great influence on the meaning attributed to his or her gaze. In a classic study by Ellsworth and Carlsmith (1968), participants were interviewed by an experimenter, who looked either directly at the participant's eyes or to her left or right ear a fixed number of times. In addition, the verbal content of the interview was manipulated to be either positive or negative. The results showed that, in the positive context, the participants in the direct gaze group evaluated both the interview and the interviewer more positively as compared to those in the averted gaze group. The result was exactly the opposite in the negative context; the evaluation was more positive in the averted gaze than in the direct gaze group. However, there are many instances and situations that could be regarded as relatively socially neutral. Is there evidence that others' gaze direction would elicit affective reactions in the observers in these kinds of situations, and if so, is direct gaze (eye contact) perceived as affectively more positive or more negative as compared to averted gaze?

In many animal species, perception of direct gaze triggers protective behavior and elicits threat or fighting responses (Emery, 2000; Skuse, 2003). In humans, direct gaze is used for control, and accordingly, it has been linked to potency, dominance, and power (Argyle et al., 1974; Hall et al., 2005), characteristics that sometimes elicit negative feelings in others. However, humans have a fundamental need for belongingness and for forming and maintaining social relationships, and the fulfillment of these needs is intrinsically positive (Maslow, 1943; Baumeister and Leary, 1995; Eisenberger et al., 2003). Because another's direct gaze signals attention and social inclusion (Wirth et al., 2010), one could presume that direct gaze would evoke positive affective reactions.

In the present article, I review research in which the effect of another individual's gaze direction on a perceiver's affective reactions has been investigated. I will focus on well-controlled laboratory experiments in which other contextual factors possibly influencing perceivers' affect have been eliminated or minimized. This approach is similar to that employed by numerous studies from the past few decades, in which the recognition of and affective reactions in response to human expressions of emotions—whether expressed in facial expressions, body movements, and posture, or in vocal prosody—were investigated by presenting carefully prepared stimuli to participants. Although the number of studies investigating affective responses to gaze stimuli is still relatively modest, two important themes seem to emerge from this review. The first one is that affective reactions elicited by another individual's direct versus averted gaze appear to result in different, and often, opposite findings when investigated by using explicit and implicit measures. Second, while explicit evaluations seem to result in heterogeneous findings, studies relying on different types of implicit measures seem to provide a more consistent pattern, indicating that direct gaze evokes a more positive affective reaction as compared to averted gaze.

In the following sections, I will first review the existing research by classifying studies according to the methodology used in measuring participants' affective reactions. Subsequently, I will deal with different possible explanations for the affective eye contact effect, and finally, I will discuss the possibility that the affective and attentional effects of direct gaze may be intertwined and that the affective reactions elicited by direct gaze should be incorporated into recent models that describe and explain different types of "eye contact" or "watching eyes" effects (cf. Senju and Johnson, 2009; Conty et al., 2016).

# EXPLICIT AFFECTIVE FEELINGS ELICITED BY GAZE STIMULI

Since long, social psychological research has investigated how an individual's gaze behavior influences other people's evaluations concerning his or her characteristics. Although these studies did not focus on observers' affective reactions in response to others' gaze direction, and although the stimuli often contained many other uncontrolled variables that possibly influenced participants' evaluations, these studies deserve to be mentioned briefly before focusing on the target studies of the present review. In such studies, participants typically watched films depicting either one individual or two interacting individuals, while the filmed individual's extent of eye contact with the camera or the other individual appearing on the film was manipulated. In some studies, participants had real, live encounters with collaborators. After watching the films or after the encounters, they were asked to evaluate the target individual on various characteristics. In general, the results showed that an individual making eye contact was evaluated more favorably as compared to an individual not making eye contact. Additionally, within limits, the degree of positive evaluations correlated positively with the extent of eye contact. These evaluations included characteristics such as likability, competence, attractiveness, intelligence, credibility, and potency (e.g., Argyle et al., 1974; Abele, 1981; Shrout and Fiske, 1981; for reviews, see Kleinke, 1986; Hall et al., 2005). Within this research tradition, other studies presented well-controlled facial stimuli with neutral expressions to participants and the researchers manipulated only the gaze direction. The results from these types of studies revealed an association between direct gaze and more positive evaluations. For example, higher liking

ratings were observed for faces with direct versus averted gaze, both when using photographs of real people (Mason et al., 2005) and virtual avatars (Kuzmanovic et al., 2009) with dynamic gaze shifts (e.g., gaze shifting from averted to direct gaze or vice versa).

Importantly for the present review, there are also few studies in which participants were asked to directly evaluate their own affective feelings in response to neutral faces with direct and averted gaze. However, unlike the studies described above, the results from these studies seem to reveal a much more inconsistent pattern of the effects of gaze direction on affective responses. Wirth et al. (2010) showed participants 2.5-min "movies" with direct and averted gaze faces. In the direct-gaze stimuli, photographs of real faces with direct gaze were interspersed with occasional closed eyes pictures (i.e., creating an illusion of blinking eyes), whereas, in the averted-gaze stimuli, the gaze was alternating between the stimulus face looking to the left and right. After watching the movies, participants were asked to rate, among other things, their positive mood (friendly, happy, and good) and negative mood (unpleasant, sad, bad, and unfriendly). The results showed that the participants who watched the averted gaze film (a between-subject design) reported significantly more negative feelings than did those who watched the direct gaze film. Further, the gaze direction did not influence participants' ratings of their positive feelings. More recently, feelings of distress and of being excluded were measured in a study in which participants were presented photographs of faces with direct or averted gaze (Leng et al., 2018). A single trial consisted of a stimulus sequence of direct gaze (1000 ms) and closed-eyes (800–1000 ms) followed by either direct gaze or averted gaze (left or right, all presented for 1500 ms). All participants were presented with both types of sequences (a within-subject design). The results showed that participants felt more distressed and more excluded when looking at the sequences ending with averted gaze as compared to sequences ending with direct gaze. However, it should be noted that, in these studies, the affective feelings were rated after participants had been asked to evaluate the extent to which they were looked at by the stimulus face (Wirth et al., 2010) and the extent to which they felt ignored and excluded while watching the stimuli (feelings of ostracism). It is possible that these rating tasks influenced participants' responses regarding their affective responses.

Despite these possible confounding factors, however, compatible results have been observed in studies without preceding tasks that may have led to biased affective ratings. Faces with static direct gaze have been shown to elicit more pleasant subjective feeling states (i.e., higher ratings of subjective affective valence) as compared to faces with averted gaze, both when pictures of real human faces (Uono and Hietanen, 2015) or pictures showing the eye-region of animated realistic looking faces (Experiment 2, Chen et al., 2017a) were used as stimuli. However, it should be noted that the study by Chen and colleagues also included eye-region stimuli with closed eyes. In fact, these closed-eyes stimuli elicited even higher valence ratings than direct-gaze stimuli did. In a study by Marschner et al. (2015), virtual characters shifted their gaze (dynamic gaze shifts) ending up with either a direct or an averted gaze. In this study, the facial expression was also manipulated. The results showed that, when embedded in a neutral facial expression, gaze direction did not have an effect on participant's ratings of their subjective affective valence. However, in the context of happy faces, direct gaze increased valence ratings; whereas, in the context of angry faces, direct gaze decreased valence ratings in comparison to averted gaze (Marschner et al., 2015).

In two studies, Hietanen and colleagues presented neutral faces with direct and averted (static) gaze in two different presentation modes; either live (presented through a liquid crystal window) or as images on a computer monitor, and they compared the effects of gaze direction on selfratings of affective valence between these modes of stimulus presentation. In both studies, the gaze direction in the images did not have any effect on subjective valence ratings; whereas, for live faces, direct gaze elicited lower affective valence (albeit still positive) as compared to averted gaze or closed eyes (Hietanen et al., 2008; Pönkänen et al., 2011a).

To sum up, at this stage of research, it is difficult to analyze the reasons for the diverse effects observed in different studies. So far, only a few studies have been conducted on this topic, with several differences in the stimuli used. Interestingly, however, most of these studies relied on the same method to measure subjective affective feelings; the Self-Assessment Manikin (SAM) scales for affective valence (unpleasant–pleasant) and affective arousal (calm–aroused) (Bradley and Lang, 1994). On attempting to identify a pattern behind the studies and their results described above, one would be tempted to argue that the more the stimuli resembled the gaze during natural interaction, the less likely direct gaze was to evoke relatively more positive affective feelings as compared to averted gaze. Studies in which participants were presented still images of faces or eye-regions (Uono and Hietanen, 2015; Chen et al., 2017a) reported higher valence ratings to direct versus averted gaze. A study that presented stimuli with dynamic gaze aversions resulted in no effect of gaze direction (Marschner et al., 2015). Studies using real, live faces as stimuli resulted in lower valence ratings to direct than averted gaze did (Hietanen et al., 2008; Pönkänen et al., 2011a). In one of these previous studies, it was suggested that lower valence ratings to direct than averted gaze when facing another, live individual could be due to the enhanced feelings of self-involvement and of the uneasiness caused by being watched by another individual (Pönkänen et al., 2011a). Thus, one could speculate that the more capable the stimuli are of evoking the feeling of being looked at by another, the less positive are the conscious self-evaluations regarding the valence of one's affective feeling state. However, of course, this is a very speculative suggestion considering the present stage of research.

In the following sections, I review research that has employed different types of implicit measures to investigate affect-related responses to another individual's direct and averted gaze. These measures include both behavioral and physiological

measurements. As we will see, these studies seem to provide a more consistent pattern of findings regarding the effects of gaze direction on affective reactions.

# BEHAVIORAL PARADIGMS WITH IMPLICIT MEASURES

Lawson (2015) conducted seven experiments by using the Implicit Association Test (IAT) to measure implicit affective evaluations of direct and averted gaze. In the IAT, participants' task is to categorize stimuli belonging to two pairs of categories. Lawson asked participants to categorize pictures of faces in which the individual was looking either toward or away from them (static gaze), and to classify affectively positive and affectively negative words. Two response keys were used. In one condition ("congruent" sorting condition), the task instructions required participants to press one key if the stimulus was a face looking toward or if the stimulus was a positive word, and to press the other key if the stimulus was a face looking away or if the stimulus was a negative word. In the other condition ("incongruent" sorting condition), the associations between the response key nominations and the categories were changed, i.e., looking toward/negative word vs. looking away/positive word. By comparing the speed of categorization in the two sorting conditions, it is possible to investigate the strength of implicit associations between the target categories (i.e., direct and averted gaze) and the positive and negative valence of the words (Greenwald et al., 2003; Nosek et al., 2005). The results showed that the categorization times were shorter in the congruent than in the incongruent conditions. In other words, participants more easily implicitly associated faces looking toward (direct gaze) and faces looking away (averted gaze) with positivity and negativity, respectively, than the other way around. Importantly, this pattern of results was observed when the faces were shown in full frontal view, when the head was rotated to the left or right side, and even when the frontal-view faces were presented upside-down. Strikingly, even angry faces with direct and averted gaze were more readily associated with positivity and negativity, respectively, than the other way around. These results are in striking contrast with those of studies relying on self-reports. As cited above, a study by Marschner et al. (2015) showed that, while direct gaze increased valence ratings for happy faces in comparison to averted gaze (but no effect of gaze direction on neutral faces), direct gaze decreased valence ratings for angry faces. Lawson (2015) also tested whether similar results would be obtained if the face stimuli were replaced by arbitrary geometrical shapes of two different colors (associated with "looking at you" and "looking to the side" labels) or by arrows pointing toward or away. Even in these conditions, participants implicitly evaluated "looking at you" stimuli more positively than "looking to the side" stimuli, albeit the magnitude of this effect was significantly smaller than that observed in the experiments that used facial stimuli.

In the IAT study by Lawson (2015), the task instructions directed participants' attention to the gaze direction, and the task instructions activated the concepts of "looking at you" and "looking to the side" because the face stimuli were presented with these labels and the response selection was based on discriminating between these categories. Thus, an important question is whether faces with direct gaze, compared to faces with other gaze directions, would also be more positively associated in conditions in which the task instructions do not require participants to attend to the gaze direction and do not imply that the stimuli are related to "looking at you" and "looking to the side."

Dubey et al. (2015) used a novel choose-a-movie (CAM) paradigm in which participants saw two colored boxes on the screen in each trial. They could open one of the boxes (with a varying number of locks) and then watch a movie clip associated with that box. During preceding familiarization trials, participants had learnt the mapping between the color of the box and the category of the movie that was shown when the box was opened. Three categories of movies were prepared. In direct gaze movies, an individual looked up (toward the camera) and smiled. Averted gaze movies showed exactly the same stimulus, but they were filmed with a camera positioned such that the individual appeared to be looking away from the camera. In object movies, household objects were slowly rotating on a turntable. The results showed that participants were prepared to put in more effort to watch direct gaze movies as compared to averted gaze (and object) movies. The authors interpreted their results in the context of the social motivation and reward gained from seeing social stimuli, but the results can also be interpreted to reflect more positive affective reactions to direct versus averted gaze. Interestingly, Dubey et al. (2015) also examined the same in individuals with autism spectrum disorder (ASD), and these adults showed a significant reduction in their preference for direct gaze.

Chen et al. (2017a) employed the affective priming paradigm to investigate the automatic affective evaluations elicited by gaze stimuli. In this study, direct-gaze, averted-gaze, and closed-eyes stimuli (eye-region "letterbox" stimuli) were briefly presented (masked 13-ms and unmasked 100-ms presentation times) as primes, followed by positive and negative words as targets (with prime-target onset delays of 150 ms and 300 ms). Participants' task was to ignore the primes and to evaluate the words as affectively positive or negative, as quickly as possible. Thus, unlike in the study by Lawson (2015), in this study, participants' attention was not directed to the gaze direction, and concepts related to the stimulus face's focus of attention were not activated. The results by Chen et al. (2017a) showed that the response (categorization) times for positive words were significantly shorter when they were presented after direct-gaze rather than closed-eyes primes, whereas the response times to negative words were significantly shorter when they were presented after closed-eyes rather than direct-gaze primes. For both targets, the response times after averted-gaze primes were numerically between those observed after direct-gaze and closed-eyes primes, but these response times did not differ statistically significantly from those.

In the affective priming literature, the effect of prime category on the affective categorization of targets is typically interpreted to show that the prime automatically activates affective evaluation

and facilitates the processing of affectively congruent targets (Fazio, 2001; Klauer and Musch, 2003). Thus, Chen et al. (2017a) interpreted their results to indicate that seeing a direct gaze automatically activated more positive evaluations than did seeing closed eyes. Interestingly, Chen and colleagues also measured explicit affective feelings evoked by the stimuli. In Experiment 1, after the affective priming experiment, participants rated the affective valence of their subjective feelings in response to the prime stimuli. Intriguingly, these results showed exactly the opposite pattern of results; more positive evaluations to closed-eyes than to direct-gaze stimuli. Experiment 2 confirmed this pattern of results for the explicit ratings independent of whether the stimuli were shown briefly (100 ms, similar to when shown as primes in the affective priming paradigm) or for a longer time (until an explicit rating response was made). The authors concluded that their results indicated that the perception of mere eye gaze automatically activates observers' emotions and that the instinctual "gut feeling" to eye contact is positive.

The three studies described in this section provide consistent support for a view that direct gaze is implicitly associated with a more positive affect and that it automatically activates a more positive affective reaction as compared to averted gaze. The study by Lawson (2015) showed this even when the direct and averted gazes were embedded in an angry face. This finding would suggest that another individual's self-directed attention, even if associated with a hostile intention, would be more positively evaluated as compared to not receiving his or her attention. However, it is possible that due to the behavioral task, the participants' attention was focused on gaze direction to such an extent that the effect of the facial expression was minimized. Moreover, the effect of facial expression might have been minimized because the participants were presented faces with only one type of expression during the experiment, either angry faces or happy faces (between-subject design). In future, it would be advisable to investigate if facial expressions modulate the effect of gaze direction on affective reactions when participants are presented expressions from more than just one category of emotions (in a within-subject design), and when the task instructions do not draw participants' attention to the gaze direction. This could be achieved by employing the affective priming paradigm, for example.

# AUTONOMIC AROUSAL AND AMYGDALA ACTIVATION

Physiological arousal is a fundamental component of affective responses (Plutchik, 1980). Several studies have reported that sympathetic skin conductance responses (SCRs)—a robust indicator of affective arousal (Critchley, 2002)—are greater in conditions with another individual's direct gaze rather than averted gaze or closed eyes (e.g., Nichols and Champness, 1971; Hietanen et al., 2008; Helminen et al., 2011; Pönkänen et al., 2011b; Myllyneva and Hietanen, 2015). Pupil dilation, another index of physiological arousal, has also been shown to be larger in response to direct- versus averted-gaze stimuli (Porter et al., 2006). Further, in another study, the dynamics of the pupil dilation response correlated with the length of time participants felt comfortable to look at faces with direct gaze; the longer periods of direct gaze participants preferred, the faster was the increase in pupil dilation (Binetti et al., 2016).

Affective arousal is controlled by the amygdala (Mangina and Beuzeron-Mangina, 1996; LeDoux, 2000; Williams et al., 2005; Laine et al., 2009). Thus, in line with the psychophysiological findings mentioned above, amygdala activation has been linked with the processing of gaze direction as well as emotion. Imaging studies in humans have shown that, not only are amygdala responses to facial emotional expressions modulated by gaze direction (Adams et al., 2003; Sato et al., 2004; Hadjikhani et al., 2008; Ewbank et al., 2010; Adams et al., 2012), but amygdala activation is also responsive to gaze direction in emotionally neutral faces. Studies have reported greater right amygdala activation in response to direct rather than averted gaze (Kawashima et al., 1999; Wicker et al., 2003). Recently, a study examined amygdala activation in healthy participants and in a cortically blind patient, and the results showed greater activation in the right amygdala in response to images of (neutral) faces with direct gaze as compared to faces with averted gaze, both in healthy participants and in the cortically blind patient (Burra et al., 2013). These findings suggest that amygdala responsivity does not even require an intact primary visual cortex. Other studies have also shown that functional coupling between activations in the right fusiform gyrus, an area specialized in face processing, and the right amygdala is greater for direct than for averted gaze (George et al., 2001).

However, even if increased physiological arousal and amygdala activation to direct gaze is interpreted to reflect an affective response, it is more difficult to say anything about whether this response is related to a positive or negative affective response. Unlike earlier views that associate the amygdala with the processing of negative (threatening) information, more recent views have emphasized its role in processes related to affective arousal and affective attention, both positive and negative (Hoffman et al., 2007; Pessoa, 2010). For example, intra-cerebral event-related potentials recorded from the human amygdala have shown enhanced responses to eye-region stimuli expressing both fear and happiness (Meletti et al., 2012). Moreover, other studies have shown the opposite pattern of results; greater amygdalar activation in response to averted than to direct gaze (e.g., Straube et al., 2010; Sauer et al., 2014). Thus, greater amygdala activation in response to direct versus averted gaze is difficult to interpret in terms of valenced affective reactions.

Nevertheless, it is evident that the amygdala plays a central role in mediating the affective arousal response and attentional allocation to direct gaze. A subcortical processing tract from the superior colliculus to the amygdala, through the pulvinar, is likely to be involved in detecting eyes and processing information about gaze direction (Senju and Johnson, 2009; Tamietto et al., 2012; Nguyen et al., 2014, 2017; Soares et al., 2017). In a functional magnetic resonance imaging (fMRI) study conducted on rhesus monkeys, Hoffman and colleagues showed that a part of the amygdala, called the lateral extended amygdala (LEA, comprising the central nucleus and the bed nucleus of the stria

terminalis) was specifically sensitive to gaze direction (Hoffman et al., 2007). The central nucleus of the amygdala sends fibers to centers controlling autonomic arousal (LeDoux, 2000; Laine et al., 2009), and therefore, it is thought to play a central role in heightening arousal and orienting attention (Davis and Whalen, 2001). Interestingly, in Hoffman et al.'s (2007) study on monkeys, LEA activation was stronger in response to averted rather than direct gaze, and recordings of SCRs also showed greater responses to averted rather than direct gaze. However, as cited above, several studies in humans have reported greater SCRs in response to seeing another individual's direct gaze rather than averted gaze. Based on these results, it could be presumed that, in humans, the LEA plays a central role in increased autonomic arousal responses and attention orienting to direct gaze. Dysfunction in these nuclei could result in direct gaze not being affectively arousing and not grabbing visual attention. Studies with patients suffering from amygdala lesions have shown that they do not look at the eye region the same way as controls do (Spezio et al., 2007), and that they do not show gaze-cued attention orienting (Akiyama et al., 2007).

# THE BRAIN REWARD NETWORK

Some neuroimaging studies have reported the effects of gaze direction on the activation of the brain systems implicated in the processing of reward. The "classic" reward network includes the ventral striatum (nucleus accumbens and ventral pallidum) and the orbitofrontal cortex, but areas such as the insula and anterior cingulate cortices have also been suggested to be parts of this network (Rolls, 2000; Schultz, 2006; Berridge and Kringelbach, 2015). Now, as another individual's direct gaze signals his or her communicative intent and social inclusion, and as imaging studies have shown that social interaction with others activates the striatum (Báez-Mendoza and Schultz, 2013; Pfeiffer et al., 2014), one would expect that seeing another individual's direct gaze would activate the reward system.

In an event-related fMRI study by Kampe et al. (2001), participants were shown images of faces with their eyes directed either at or away from them. After the imaging session, participants were instructed to rate the attractiveness of the stimulus faces. The results showed that, indeed, the gaze direction had an effect on the activation of the ventral striatum. However, interestingly, this activation was also dependent on facial attractiveness. For stimuli with direct gaze, ventral striatum activation increased as a function of facial attractiveness, whereas, for averted-gaze stimuli, activation decreased with increasing attractiveness (Kampe et al., 2001). Therefore, the authors suggested that facial attractiveness acted as a social reward. A direct gaze from an attractive face signals a possibility for an upcoming social interaction with an attractive individual, and thus, it anticipates a social reward. Instead, a direct gaze from an unattractive face may lead to the anticipation of an unwanted social interaction. Thus, so far, direct gaze has not been shown to activate the ventral striatum. Therefore, future studies need to examine if the activation of the ventral striatum in response to the direct gaze of faces occurs irrespective of their attractiveness. Further, in Kampe et al.'s (2001) study, the face stimuli were images of static faces. Therefore, it is possible that, for example, dynamic shifts of gaze toward the viewer could elicit enhanced ventral striatum activation independent of facial attractiveness.

In fact, in one study, anterior insula activation was observed only in response to dynamic shifts of gaze, but not in response to static images (Ethofer et al., 2011). Ethofer et al. (2011) measured participants' brain activation in response to dynamic gaze when they were performing a gender categorization task. Findings revealed that gaze shifts toward the viewer resulted in greater activation within the right anterior insula as compared to gaze shifts away from the viewer. Interestingly, a connectivity analysis revealed an increase in the functional coupling of the right posterior superior temporal sulcus (pSTS)—a central region in gaze processing—with the anterior insula when the gaze shifted toward rather than away from the viewer. Notably, there was also a highly significant difference between the hemispheres in terms of the structural connectivity between the pSTS and the anterior insula. Specifically, in the left hemisphere, only infrequent connections were found between the pSTS and anterior insula. Further, in a study in which the participants were looking, via a large mirror, at a live individual sitting in the scanning room, greater activation in the anterior insula, anterior cingulate, and globus pallidus was reported in response to direct rather than averted gaze (Cavallo et al., 2015). Finally, in a study measuring electroencephalographic activity in response to dynamic gaze shifts, source localizing analyses showed a cluster of sources in the orbitofrontal cortex, in which the activity was greater in response to the dynamic gaze that shifted from averted to direct gaze than from direct to averted gaze, specifically between 190 and 220 ms after stimulus onset (Conty et al., 2007).

In sum, neuroimaging studies have shown that seeing a direct gaze results in greater activation of the various components of the reward system as compared to seeing an averted gaze. These results could be considered as evidence supporting the view that gaze direction can trigger affective processing and that direct gaze elicits more positive affective reactions compared to those elicited by averted gaze. However, great cautiousness is warranted in interpreting these results. Apart from the ventral striatum, the association between reward and the functioning of the other brain areas mentioned above is complicated by the fact that these areas are also involved in many other cognitive, affective, and interoceptive functions, and, at the present stage of research, it is difficult to know whether the findings described above are related to affective reactions elicited by gaze or to some other processes like selfreferential processing (see, e.g., Northoff et al., 2006; Herbert et al., 2011).

# ASYMMETRIC FRONTAL CORTICAL ACTIVITY

More direct brain research evidence associating gaze direction with affective valence comes from studies reporting the effects

of gaze direction on the activation of the brain systems implicated in the processing of affect and motivational states. There is a considerable line of research associating asymmetric frontal alpha-band electroencephalographic (EEG) activity to emotional and motivational processes. The relatively greater activation of the left versus the right frontal cortex has been linked to positively valenced affect and activation of the approach-related motivational system, whereas the opposite pattern of frontal asymmetric activation has been linked to negative affect and activation of the avoidance system (Davidson, 1984, 2004; Harmon-Jones, 2003, 2004; Harmon-Jones et al., 2006; Van Honk and Schutter, 2006). Most of this research has investigated the association between resting state frontal EEG activity, and trait affect and trait motivation, but other studies have examined asymmetric frontal EEG activity in response to affective and motivationally significant stimuli (for a review, see Harmon-Jones and Gable, 2018).

Few studies have shown that seeing another individual's gaze direction has an effect on observers' frontal EEG asymmetry. For instance, Hietanen and colleagues measured the hemispheric asymmetry in the frontal EEG activity in response to seeing another, live individual with direct and averted gaze (Hietanen et al., 2008; Pönkänen et al., 2011b). Findings revealed that seeing another individual's direct gaze elicited greater relative left-sided frontal EEG activity as compared to seeing averted gaze. These results provide evidence in favor of the fact that direct gaze elicits greater activation in brain mechanisms associated with approach motivation and positive affect as compared to averted gaze. In fact, in Hietanen et al.'s (2008) study, another individual's averted gaze elicited right-sided, avoidance-related frontal EEG asymmetry. Interestingly, in their study, subjective ratings of affective valence were also measured, and they indicated that averted gaze was rated as slightly more pleasant as compared to direct gaze. Thus, in this study too, implicit (physiology) and explicit (self-rating) measures resulted in incongruent patterns of results.

The frontal EEG asymmetry response to gaze direction has been shown to be modulated by personality and neuro-psychiatric disorders. Uusberg et al. (2015) measured EEG asymmetry in response to a live individual's gaze in participants with varying degrees of neuroticism according to the Five Factor Model. The results showed that, in participants scoring low on neuroticism, direct gaze elicited greater left-sided frontal EEG asymmetry as compared to averted gaze, as observed in the two studies mentioned above. However, in participants scoring high on neuroticism, direct gaze elicited greater right-sided frontal EEG asymmetry as compared to averted gaze. In another study, the frontal EEG asymmetry response to gaze direction was investigated in adolescents with clinically diagnosed social anxiety disorder (Myllyneva et al., 2015). The results showed marginally greater left-sided frontal EEG response to direct gaze in control participants as compared to the clinical group. ASD have also been shown to influence the frontal EEG asymmetry response to gaze. Kylliäinen et al. (2012) investigated children with ASD and control children, and showed that, in the control children, direct gaze elicited greater left-sided frontal asymmetry than closed eyes did; whereas, in ASD children, the gaze direction did not have an effect on frontal EEG asymmetry responses. These three studies indicate that the increased negativity to direct gaze associated with neuroticism, social anxiety, and autism (Campbell and Rushton, 1978; Senju and Johnson, 2009; Moukheiber et al., 2010) is reflected in the frontal EEG asymmetry responses.

# STARTLE REFLEX MODULATION

The startle reflex is an automatic defensive reaction to abrupt and strong stimuli. A convenient way to investigate the startle reflex is to measure electromyographic (EMG) eyeblink responses (Lang et al., 1990; Bradley et al., 1999; Grillon and Baas, 2003) or heart rate (HR) acceleration responses (Graham and Clifton, 1966; Graham, 1992; Holand et al., 1999; Richter et al., 2011) triggered by an acoustic startle probe. Interestingly, simultaneously presented affective foreground stimuli can modulate the magnitude of the reflex. The eyeblink and the cardiac acceleration responses are increased in an unpleasant context and decreased in a pleasant context (e.g., Vrana et al., 1988; Bradley et al., 1993; Bradley and Lang, 2000; Ruiz-Padial et al., 2005; Roy et al., 2009; Sánchez et al., 2009; Ramírez et al., 2010; Richter et al., 2011).

Two studies have investigated the modulatory effect of perceived gaze direction on the magnitude of the startle reflex. In one study, acoustic startle probes were presented to male participants while pictures of nude females with direct and averted gaze were shown as foreground stimuli (Lass-Hennemann et al., 2009). Affectively positive nude bodies decreased participants' eyeblink response. The gaze direction also had an effect, in that the attenuation was smaller for pictures with direct rather than averted gaze. This would suggest that averted gaze was perceived as more positive than direct gaze was. However, the authors suggested that the effect of gaze direction was due to its effect on attention. Direct gaze grabbed attention to the faces and therefore, the effect of the nude bodies was reduced in the context of direct gaze.

More recently, the effect of gaze direction on startle reflex modulation was investigated by presenting loud auditory stimuli while a live model's direct- and downward-gaze stimuli were presented through a liquid crystal window (Chen et al., 2017b). In this study, both eyeblink startle and cardiac reflexes were measured. The results showed that the magnitude of the eyeblink startle and cardiac reflexes decreased when measured in the context of a direct versus downward gaze. Interestingly, in this study, the participants also self-evaluated the valence of their subjective feelings while looking at the stimulus faces. Similar to other previous studies measuring both explicit and implicit affective reactions (e.g., Hietanen et al., 2008; Chen et al., 2017a), this study found that direct gaze was rated as slightly less positive as compared to downward gaze, although the difference was not statistically significant. In sum, the results of this study provide further evidence in support of

the view that another individual's gaze direction elicits affective reactions, and that, compared to averted gaze, direct gaze automatically elicits more positive affective responses in the viewer.

# FACIAL ELECTROMYOGRAPHY

Measurements of EMG responses from the facial muscles involved in producing facial emotional expressions have been widely used as a method to investigate the valence of automatic affective reactions (Cacioppo et al., 1986; Tassinary and Cacioppo, 1992; Dimberg and Thunberg, 1998; Dimberg et al., 2000). Affectively positive stimuli increase the activity of the Zygomaticus major (smile) and decrease activity of the Corrugator supercilii muscle (furrows between the eyebrows), whereas negative stimuli increase the activity of the Corrugator supercilii muscle (Cacioppo et al., 1986; Larsen et al., 2003).

Previous studies have reported that an expressor's gaze direction can modulate the facial EMG responses elicited by the emotional facial expression (Schrammel et al., 2009; Rychlowska et al., 2012; Soussignan et al., 2013), but, in these studies, no effect of gaze direction was observed in response to neutral faces. Hietanen et al. (2018) argued that the reason for the lack of a mere gaze direction effect could be that, in these previous studies, the stimuli were images of human faces or animated virtual characters. A viewer knows that an image of a face presented on a computer monitor does not look back. This argument was supported by their previous experiments that showed that, while psychophysiological responses (electroencephalographic and autonomic responses) to direct versus averted gaze had been observed to differ when a live individual was presented as a stimulus, there was no effect of gaze direction on responses to pictures of the same individual (Hietanen et al., 2008; Pönkänen et al., 2011a,b). Therefore, Hietanen et al. (2018) investigated the effect of another individual's gaze direction on participants' facial responses by showing a live individual with a neutral expression as a stimulus. In their study, not only did the model individuals vary their gaze direction, but the participants were also allowed to look either directly at the model individual or slightly away from him or her, at a pre-determined fixation spot. This lateral fixation spot was placed such that the participants were able to see, from the corner of their eye, whether the model individual had a direct gaze or not. The results showed that the zygomatic responses were greater in response to another individual's direct versus averted gaze when the participant was looking toward the other as well as when the participant was looking slightly away. However, the participant's own gaze direction also had an effect; the zygomatic response to the model's direct gaze was greater during the former (i.e., a genuine eye contact) as compared to the latter condition.

Thus, measurements of facial EMG responses have also provided evidence that, in a neutral context, another individual's gaze direction elicits affective reactions, and that, compatible with the other findings reviewed above, direct gaze seems to elicit a positive affective reaction. However, as Hietanen et al. (2018) stressed in their discussion, we cannot know for sure about the extent to which the observed facial reactions reflect automatic affective reactions or highly automatized affiliative facial responses triggered by communicative motivations during social interaction.

# THE AFFECTIVE EYE CONTACT EFFECT: POSSIBLE MECHANISMS

The present review has shown that studies using explicit and implicit measures have provided somewhat contradicting findings regarding whether direct gaze elicits more positive or less positive affective reactions as compared to control-gaze stimuli. Many of the studies relying on explicit self-evaluations reported higher valence ratings to averted-gaze or closed-eyes stimuli as compared to ratings in response to direct-gaze stimuli, whereas studies using different kinds of implicit measures consistently showed more positive affective reactions to direct gaze than to averted gaze. How can we explain these discrepancies in the results of explicit and implicit measurements?

People's explicit responses are known to be susceptible to motivational biases and individuals may lack introspective access to their implicit affective reactions. In fact, correlations between explicit and implicit measures increase as a function of increasing spontaneity of self-reports (Hofmann et al., 2005). There is plenty of evidence in the area of social cognition research on how explicit and implicit processes can be not only complementary but also oppositional (Frith and Frith, 2008). Introspection of one's own feelings to direct-gaze stimuli may, for example, evoke uncertainness because one cannot be sure about the gazer's intentions and the reasons for being the target of his or her attention. Another individual's direct gaze may also increase self-directed attention and self-awareness (Hietanen and Hietanen, 2017). This, in turn, may lead to critical evaluation of the self and to a negative affective state (Duval and Wicklund, 1972). Thus, even if one's initial and automatic response to direct gaze was affectively positive, it could be suppressed by more controlled evaluations, and it may even be biased in a negative direction. This idea is compatible with the views proposing that socio-cognitive functions depend on the workings of two systems; one responsible for the detection of socially relevant actions, which relies on automatic processing; and another responsible for social evaluation, which relies on more controlled processing (Spunt and Lieberman, 2013; Vogeley, 2017).

Thus, the reviewed research provides considerably strong evidence that eye contact automatically elicits positive affective reactions. However, an essential question that emerges is why eye contact triggers positive affective reactions. In the following paragraphs, four different possible factors behind the affective eye contact effect are characterized. For an illustration, see **Figure 1**.

In principle, affective reactions to gaze could be triggered by the perception of eyes—by low-level visual cues related, e.g., to luminance distribution (Kobayashi and Kohshima, 1997; Langton et al., 2000; Ando, 2002) analyzed from the eyes—processed by subcortical mechanisms described in the section dealing with amygdala activation in response to gaze. According to the fast-track modulator model of eye contact presented by Senju and Johnson (2009), direct gaze is detected

and processed by a subcortical pathway, involving the superior colliculus, pulvinar, and amygdala. Senju and Johnson (2009) suggested that this pathway then modulates the functioning of the social brain network and the network involved in mentalizing. In the context of the present review, one could argue that the subcortical pathway also modulates the networks involved in regulating affective responses. Thus, according to this view, the affective reactions in response to gaze would be triggered by the visual information analyzed from the gazers' eyes.

In the introduction of this article, when discussing about why gaze direction should be expected to have any affective effects, a higher-level explanation was suggested based on human beings' fundamental need for belongingness and for forming and maintaining social relationships (Maslow, 1943; Baumeister and Leary, 1995; Eisenberger et al., 2003). It was suggested that, as direct gaze indicates attention and social inclusion (Wirth et al., 2010), it is likely to be perceived as a positive social signal. Thus, another possibility is that, rather than based on visual information from the eyes, the affective effects reflect the "perception" of other individuals' attention directed to the self. In fact, there is evidence supporting this latter view. First, previous studies have shown greater autonomic arousal responses and greater relative left-sided frontal EEG activity in response to direct versus averted gaze when participants saw a live individual, but not when they saw an image of a face on a computer monitor (Hietanen et al., 2008; Pönkänen et al., 2011b). As noted above, images do not look back. More direct evidence in support of this view was provided by Myllyneva and Hietanen (2015, Experiment 1), who measured SCRs in response to a live individual's gaze direction in two conditions. In one condition, the participant and the model individual were able to see each other normally; whereas, in the other condition, the participant was led to believe that a half-silvered mirror was placed between the participant and the model in such a way that the model could not see the participant. The results showed greater SCRs to direct than to averted gaze when the participants believed that the model was able to see them, but not when the participants believed that the model could not see them. Moreover, in Experiment 2, the authors manipulated the visibility of the model's eyes. In three different experimental blocks, the model wore a different pair of sunglasses: a pair without lenses (eyes visible), a pair of normal sunglasses with dark lenses (eyes not visible, but the participants knew that the model was able to see them), and an identical pair of sunglasses with dark lenses, but with lenses covered from inside (eyes not visible and the participants knew that the model was not able to see them). The results showed greater SCRs to direct gaze/head orientation as compared to averted gaze/head orientation, both when the eyes were visible and when the participants were wearing normal sunglasses. However, when the model was wearing opaque sunglasses, there was no effect of gaze direction/head orientation on SCRs. These results strongly indicate that the enhanced physiological responses to another individual's direct gaze reflect the awareness of being attended

to by another individual, rather than as responses to the visual appearance of directly looking eyes.

A third possible mechanism relates to the enhanced self-directed attention triggered by eye contact. Recently, Conty et al. (2016) postulated that eye contact initiates, via self-directed attention, a self-referential mode of information processing, i.e., a heightened processing of stimuli in relation with the self. This postulation was confirmed later, based on the findings of a study that showed that, in a task requiring participants to complete sentences by choosing a pronoun (first singular, first plural, third singular, or third plural), a gaze stimulus presented before each trial influenced the selection of the pronouns; specifically, direct gaze (eye contact) increased the use of first-person pronouns (Hietanen and Hietanen, 2017). Now, as self-referential processing is associated with positive affect (self-positivity) (Baumeister, 1998; Heine et al., 1999), it could be postulated that the positive affective reactions elicited by eye contact could be mediated by the effects of eye contact on self-reference.

A fourth possibility is that the affective responses to gaze reflect responses to interaction. Recently, in the field of social attention, particularly in studies investigating eye movements and fixations when looking at real people versus images, it has been reported that the gazing patterns can be very different between these conditions. It has been suggested that the key difference between watching pictorial and live stimuli is in the possibility for bidirectional sending and receiving information, i.e., possibility for interaction (Laidlaw et al., 2011; Risko et al., 2012; Wu et al., 2013). Possibility for interaction has also been suggested to play a role in triggering the autonomic affective responses to eye contact (Myllyneva and Hietanen, 2016). In a recent experiment, as described in the previous section, Hietanen et al. (2018) measured participants' facial EMG responses when they were allowed to look either directly at the model individual or slightly away from him or her. As described above, the zygomatic response to the model individual's direct gaze was greater when participants were looking at the model as compared to when their gaze was slightly averted. Thus, despite the fact that, in both conditions, participants were able to see that the model's attention was on them (belief of being seen), the reciprocated direct gaze (eye contact) resulted in the strongest zygomatic response. Moreover, in that study, Hietanen and colleagues also measured SCRs, which revealed that the autonomic arousal response was greater to the model's direct versus averted gaze only when participants were looking toward the model individual, but not when they were not reciprocating the direct gaze. The authors interpreted this finding to suggest that enhanced affective arousal to another individual's direct gaze is conditional to (a) an observer's perception and understanding that another individual's attention is directed to him or her and (b) an observer simultaneously directing his or her own gaze toward the other individual and understanding that the latter perceives that he or she is being seen by the observer.

Thus, it is possible that, for example, the previous results by Myllyneva and Hietanen (2015) showing no effect of another individual's gaze direction (direct versus averted gaze) on affective arousal responses when the participant believed that the model could not see him or her, was not only due to the self's understanding of not being seen by the other, but also due to self's understanding of not being able to communicate to the other that "I am looking at you." Even though looking at each other's eyes between two motionless individuals does not involve "behavior" as such, there is nevertheless an interaction—coordinated, reciprocal, and joint activity (for theoretical definition of interaction, see De Jaegher et al., 2010). In eye contact, the parties have chosen to simultaneously direct their attention toward the other and they both know about it. De Jaegher et al. (2010) emphasized that an essential characteristic of interaction is the engagement between the agents. Genuine eye contact definitely fulfills this criterion. Schilbach and colleagues have also emphasized the importance of interaction and emotional engagement as a fundamental factor differentiating between natural encounters with another individual and situations where an individual is merely observing another without a possibility to interact (Gangopadhyay and Schilbach, 2012; Schilbach et al., 2013).

At the present stage of research, it is difficult to evaluate the relative importance of the possible mechanisms listed above regarding the affective eye contact effect. It is possible that all these factors—low-level visual cues of direct gaze, receiving others' attention, self-referential processing, and interaction contribute to the positive affective reactions elicited by eye contact. It is also possible that the contribution of these factors vary depending on the way affective reactions are probed in experiments. For future studies, an important aim would be to investigate the specific contribution of these factors on different measures indexing affective reactions.

# AFFECTIVE EYE CONTACT INVESTIGATED WITH IMAGES VERSUS LIVE FACES

One more important issue related to the suggested mechanisms behind the affective eye contact effects deserves attention. If these effects reflect the influence of receiving others' attention, interaction, self-referential processing, or some combination of these, how have affective effects also been observed in studies where participants have been shown images of faces, i.e., faces which do not attend to or interact with the observer? As suggested previously, one possibility is that "the belief of being watched" may be an intrinsic property of direct gaze, possibly based on both human evolution and overlearning during early life, and that it is embedded in the perception of direct gaze (Conty et al., 2016). This would explain why direct gaze in pictorial stimuli is also capable of eliciting automatic positive affective responses. However, if so, the next problem is how to explain the findings of some studies that revealed that, while a live individual's gaze direction influenced these reactions, this was not observed when the same participants were shown the same facial stimuli as images (e.g., Hietanen et al., 2008; Pönkänen et al., 2011b). Notably, this difference is unlikely to relate to some low-level differences in the stimuli used in these studies. For example, both in studies reporting (Lawson, 2015; Chen et al., 2017a) and not

reporting (Hietanen et al., 2008; Pönkänen et al., 2011b) affective effects by gaze direction, the stimuli were static images of faces without any dynamic gaze shifts. Additionally, even when the stimuli were video clips in which the model appears very similar (e.g., with occasional blinks) as compared to when shown live through a liquid crystal window, direct gaze does not result in enhanced autonomic responses (Lyyra et al., 2018), unlike that observed in studies that used a live model individual (Hietanen et al., 2008; Helminen et al., 2011; Pönkänen et al., 2011b; Myllyneva and Hietanen, 2015). Moreover, as cited above, the results of Myllyneva and Hietanen (2015, Experiment 1) showing the effect of a live individual's gaze direction on affective arousal responses when participants believed that the model was able to see them, but not when they believed that the model could not see them, speak against the possibility that low-level visual differences in live versus pictorial stimuli could explain these differences in the results.

One possibility may be related to the nature of responses measured in different studies. The studies reporting no effects of gaze direction with pictures (while observing the effects with live faces; Hietanen et al., 2008; Pönkänen et al., 2011b) measured physiological responses indexing autonomic arousal and the activation of the affective-motivational brain systems, while the studies showing an effect with face pictures relied on behavioral measures sensitive to the cognitive-affective associations between gaze direction and affective information (i.e., Lawson, 2015; Chen et al., 2017a). Perhaps, contextual information about perceiving (just) a picture, regulates the physiological response systems through a top–down process, which inhibits these responses when there is no actual possibility or need to prepare the system for interaction (cf., Myllyneva and Hietanen, 2016).

Another possibility is that these discordant findings are related to attention allocation toward stimuli and cognitive load during stimulus presentation. In studies where pictorial-gaze stimuli have not evoked affective (physiological) responses, participants have been passive observers without a cognitive task (Hietanen et al., 2008; Pönkänen et al., 2011b). Thus, attentional resources could have been directed, not only to the gaze stimuli, but also to the contextual situation where one is facing a computer monitor and is being presented with pictures. Instead, in studies where pictorial-gaze stimuli were observed to have affective effects (i.e., using the affective priming paradigm and implicit association test; Lawson, 2015; Chen et al., 2017a), participants' attention was directed to a primary cognitive task—affective categorization of the stimuli. Moreover, in the affective priming paradigm (Chen et al., 2017a), participants were even instructed to ignore the gaze stimuli, that is, the primes. In fact, the possibility that these discrepant findings are related to attention and cognitive load during stimulus presentation was directly tested in a study by Conty et al. (2010). They reasoned that direct gaze might evoke amygdala-mediated autonomic arousal response when face stimuli are presented secondary to a main task. To this end, they presented pictorial direct-gaze, averted-gaze, and closedeyes stimuli concomitantly with a demanding word-spelling task or a simple letter decision task. The results showed greater SCRs to direct gaze compared to averted gaze and closed eyes in the context of the demanding task, but no effect of gaze direction was observed in the context of the simple task. Conty and colleagues interpreted their results referring to the fast-track modulator model proposed by Senju and Johnson (2009) and suggested that, without cognitive load, the arousal response mediated by the subcortical route is inhibited by cortical top–down control.

# CONCLUSION AND FUTURE DIRECTIONS

I started this review by emphasizing the role of gaze perception in allowing a perceiver to infer the direction of another individual's attention. Indeed, substantial literature has shown the effects of another individual's gaze direction on the perceiver's own attention; another individual's direct gaze attracts a perceiver's visual attention and gaze toward the other's eyes. The attentional effects and prioritized processing of direct gaze have also been central in recent models attempting to describe the various effects of direct gaze and eye contact on cognitive processing (Senju and Johnson, 2009; Conty et al., 2016). The present review shows that there is accumulating evidence that eye contact automatically activates affective systems. Thus, it is likely that the affective processes and reactions also play a role in the various "eye contact effects" (cf., Senju and Johnson, 2009) and "watching eyes effects" (cf., Conty et al., 2016) described previously. However, the initiation of affective processes elicited by eye contact were not explicitly described in these models. For example, Conty et al. (2016) postulated that eye contact initiates, via self-directed attention, a self-referential mode of information processing, i.e., a heightened processing of stimuli in relation with the self, and that this leads to the enhancement of self-awareness, memory effects, activation of pro-social behavior, and positive appraisals of others. Now, as eye contact seems to trigger positively valenced affective processing and bodily responses, it is possible that these reactions contribute to the advantageous effect of direct gaze on memory, pro-social behavior, and evaluation of others (for reviews of these effects, see Senju and Johnson, 2009; Conty et al., 2016). The advantageous effects of positive affect, in general, on memory, pro-social behavior, and individual perception are welldocumented in the literature (Forgas and Bower, 1987; George, 1991; Ashby et al., 2002).

This research field is abundant with interesting questions waiting to be investigated. Given that research has started to reveal automatic positive affective reactions to eye contact, these findings can pave the way for a broader investigation of these effects in various types of social encounters. Positive affect is known to positively influence performance on a variety of cognitive tasks (Isen, 1999), possibly via increased brain dopamine levels (Ashby et al., 1999). Although longer periods of eye contact may be disruptive for cognitive performance and may lead to gaze aversion (presumably to decrease cognitive load, e.g., Doherty-Sneddon and Phelps, 2005), shorter periods of eye contact could, indeed, trigger positive affective reactions, thus leading to improved cognitive performance and facilitation of social interaction. A particularly interesting issue relates to the possible effects of eye contact on therapeutic change, via positive affective reactions. In the field of psychotherapy,

positive affect has been suggested to play a role as a generator of therapeutic change by facilitating cognitive flexibility (Fitzpatrick and Stalikas, 2008).

More specific issues deserving future research relate, for example, to the role of physical presence in the affective eye contact effect. If positive affective influences reflect the understanding of being attended by others or the possibility for reciprocal interaction, what role does physical presence play in eliciting these affective reactions? Will eye contact elicit positive affective reactions when seeing another via a telecommunication application as well? So far, everyday experiences suggest that this might not be the case, but this may be due to the typical technical limitations (e.g., location of the camera in relation to the screen and time-delay in transmitting the video signal). Another, highly interesting field relates to human interaction with robots. Presently, technology in the field of robotics is developing fast and social robots are starting to appear. Just within a few years, we may be interacting not only with fellow humans but also with robots in our homes, workplaces, and in places offering various services. How do we react affectively to robots, and is eye contact with a robot capable

### REFERENCES


of eliciting similar kinds of positive affective reactions as eye contact with another human does? If it turns out that eye contact with robots also generates positive affective reactions in humans, this could have huge potential in terms of using robots to increase people's well-being and to alleviate negative states of feelings, for example, in people having difficulties in forming and maintaining social relations with others (e.g., due to psychological problems or physical handicaps) or in people suffering from loneliness.

John Heron once wrote: "The most fundamental primary mode of interpersonal encounter is the interaction between two pairs of eyes and what is mediated by this interaction" (Heron, 1970, p. 244). The present review suggests that this encounter not only opens a door for the meeting of minds, but it does it in an inherently positive way.

### AUTHOR CONTRIBUTIONS

The author confirms being the sole contributor of this work and approved it for publication.


gaze motion processing. Neuropsychologia 45, 3024–3037. doi: 10.1016/j. neuropsychologia.2007.05.017




**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Hietanen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

fpsyg-09-02111 October 30, 2018 Time: 15:19 # 1

# Socially Anxious Tendencies Affect Impressions of Others' Positive and Negative Emotional Gazes

Yuki Tsuji1,2 and Sotaro Shimada<sup>1</sup> \*

<sup>1</sup> Department of Electronics and Bioinformatics, School of Science and Technology, Meiji University, Kawasaki, Japan, <sup>2</sup> Japan Society for the Promotion of Science, Tokyo, Japan

Socially anxious tendencies have potential to become social anxiety disorder (SAD), which is characterized by fear of social situations associated with being evaluated or embarrassed by others. In particular, others' gazes induce social anxiety. People with SAD have a negative interpretation bias toward ambiguous emotions in others' faces; however, negative interpretation bias toward ambiguous emotions in others' gazes has not been fully investigated. We used an impression judgment task to examine negative interpretation bias toward others' gazes among people with socially anxious tendencies. We generated emotionally ambiguous gazes (positive, negative, and neutral) using a morphing technique with 10% steps (neutral, 10–100% negative, and 10–100% positive). Participants (all male) were asked to judge whether the stimulus was positive or negative. Each participant's level of social anxiety was examined using the Japanese version of the Social Phobia Inventory (SPIN-J), which measures three symptom dimensions: fear, avoidance, and physiological arousal. To examine the influence of socially anxious tendencies in the impression judgment task, we calculated the point of subjective equality (PSE) using a two-step logistic curve fitted to individual participant's responses. The negative emotional intensity of the PSE became lower as the fear score became higher (p < 0.05). This result suggests individuals with a high tendency toward social anxiety tend to interpret subtle negative emotional gazes as a negative emotion and regard these gazes as a threat.

#### Edited by:

Paola Ricciardelli, Università degli Studi di Milano Bicocca, Italy

#### Reviewed by:

Colin W. G. Clifford, University of New South Wales, Australia Marco Tullio Liuzza, Università degli Studi Magna Graecia di Catanzaro, Italy

> \*Correspondence: Sotaro Shimada sshimada@meiji.ac.jp

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 04 June 2018 Accepted: 12 October 2018 Published: 01 November 2018

#### Citation:

Tsuji Y and Shimada S (2018) Socially Anxious Tendencies Affect Impressions of Others' Positive and Negative Emotional Gazes. Front. Psychol. 9:2111. doi: 10.3389/fpsyg.2018.02111 Keywords: social anxiety disorder, gaze perception, emotional gazes, impression, morphing

# INTRODUCTION

To guess emotion is one of important abilities in social interaction. Humans are able to appropriately judge emotions from facial expressions, with this ability extending beyond cultural boundaries (Darwin, 1872; Ekman et al., 1969). However, impressions received from facial expressions vary by individual characteristics. The constructed emotion theory suggests that categorizing one's own or others' emotions depends on integrating information from the inside world (e.g., interoception, memorized or imagined representations, and concepts) and the outside world (e.g., five senses) to obtain meaning (Russell, 2003; Barrett et al., 2007; Barrett, 2012). According to this theory, suitably guessing others' emotion from others' facial expressions requires the categorized emotion and the other person's emotion to correspond. Cognitive models of social anxiety suggest that socially anxious individuals have an attentional bias for negative social cues fpsyg-09-02111 October 30, 2018 Time: 15:19 # 2

that could indicate social rejection or threat (Rapee and Heimberg, 1997). In social situations, individuals with high socially anxious (HSA) tendencies tend to pay attention to themselves as social objects rather than focusing on people around them (Clark and Wells, 1995). This bias is thought to activate negative self-beliefs (e.g., "others dislike me"). These theories and cognitive models suggest individuals with social anxiety tend to use their inside world information as clues for judging emotions, thereby making it difficult to suitably guess others' emotions. A defining feature of social anxiety disorder (SAD) or social phobia is avoidance or excessive fear of situations associated with evaluation or embarrassment by others (American Psychiatric Association [APA], 2000). Some studies reported that individuals with HSA tendencies or SAD were more likely to evaluate or misinterpret facial expressions as threatening (Dimberg et al., 1986; Pozo et al., 1991; Winton et al., 1995; Dimberg, 1997; Heuer et al., 2010; Vassilopoulos, 2011). In particular, clinical studies indicate that patients with SAD recognized others' facial expressions as a threat (Heinrichs and Hofmann, 2001; Hirsch and Clark, 2004).

Several studies have used Likert-type self-report scales to rate the perceived intensity of facial emotional expressions during neuroimaging or following experimental procedures as a control condition (for a review see Staugaard, 2010). However, studies using morphing techniques to generate faces expressing varying intensities of emotion provide inconsistent behavioral evidence for the effect of social anxiety on facial emotion identification. For example, Joormann and Gotlib (2006) reported that individuals with HSA tendencies had a lower threshold for identifying angry faces relative to healthy participants. In contrast, Montagne et al. (2006) reported that healthy participants had a lower threshold for identifying angry faces relative to those with HSA tendencies. However, individuals with HSA rate negative emotional facial expressions as more negative than individuals with low socially anxious (LSA) tendencies or healthy participants (Dimberg and Christmanson, 1991; Dimberg and Thunberg, 2007; Schofield et al., 2007; Goldin et al., 2009). Furmark et al. (2009) reported that participants with social phobia showed more anxiety responses to angry or neutral faces than healthy controls. Other studies reported no association between social anxiety and identifying facial expressions (Philippot and Douilliez, 2005; Schofield et al., 2007).

Individuals with high trait anxiety tend to classify blended angry and disgusted expressions as disgusted expressions (Richards et al., 2002). Socially anxious individuals fear being evaluated as incompetent or disgusting more than they fear provoking others' anger (American Psychiatric Association [APA], 2000). Cognitive models of social anxiety suggest that socially anxious individuals activate negative self-beliefs (e.g., incompetent or disgust) in response to perceived social threats (Clark and Wells, 1995; Rapee and Heimberg, 1997). Individuals with HSA also tend to rate disgusted faces as more negative than angry faces (Amir et al., 2010). Therefore, others' disgusted expressions may induce activity of negative self-beliefs in individuals with socially anxious tendencies. Patients with SAD also confused ambiguous or neutral emotional expressions with negative emotions (e.g., anger, threat, disgust) relative to healthy people (Bell et al., 2011). These findings suggest that individuals with HSA have a negative interpretation bias toward disgusted, ambiguous, and happy facial expressions relative to those with LSA.

Mathews et al. (2003) reported that participants with high trait anxiety showed enhanced orienting to the gaze cued location of faces with fearful expressions, relative to other expressions. Similar effects have been found when comparing high and low state anxiety (Holmes et al., 2006). Anxiety is associated with enhanced attentional cuing by fearful eye gazes (Fox et al., 2007), and social anxiety is associated with increased orientation to facial threats (Mogg and Bradley, 2002; Mogg et al., 2007) and aversion to direct eye gaze (Schulze et al., 2013a). The perception of direct gaze also varies as a function of trait anxiety, with a bias toward perceiving slightly averted gazes as direct gazes in clinically and non-clinically anxious individuals (Schulze et al., 2013a,b). Despite the fact that the gazes of other people commonly induce social anxiety (Den Boer, 2000), the way in which people with SAD interpret others' emotional gazes has not been fully investigated.

The present study examined the influence of socially anxious tendencies on subjective impressions of emotional gazes. We investigated responses or impressions to positive or negative emotional gazes of varying emotional intensities using morphing techniques. Previous reports indicated that individuals with clinical or subclinical social anxiety rated negative emotional facial expressions, especially disgusted faces (Amir et al., 2010), as more negative (Dimberg and Christmanson, 1991; Dimberg and Thunberg, 2007; Schofield et al., 2007; Goldin et al., 2009). These individuals also classified ambiguous emotional facial expressions as negative (Melfsen and Florin, 2002; Bell et al., 2011), and rated positive emotional facial expressions as less pleasant (Straube et al., 2004). Therefore, we hypothesized that socially anxious tendencies would modulate subjective impressions of ambiguity of others' emotional gazes (disgusted or happy). We assumed that using a Likert scale to respond to the impression of the stimulus might allow ambiguous answers, and would not clarify the influence of socially anxious tendencies on subjective impressions of emotional gazes. Therefore, we used a forced two-choice task (impression judgment task) to clarify this issue.

# MATERIALS AND METHODS

# Participants

McLean et al. (2011) reported that there were no significantly difference of the lifetime and 12-month prevalence rates of SADs across gender. We assumed that there was no difference of impressions response to emotional gazes across gender. We collected data from male participants to match the gender of participants and that of stimuli. Participants were 32 healthy male volunteers (mean age 21.4 ± 1.21 years). All participants had normal or corrected-to-normal vision, and provided written informed consent to participate in this study. The study protocol was approved by the Ethics Committee of the School of Science and Technology, Meiji University. This study was conducted according to the principles and guidelines of the Declaration of Helsinki.

# Social Anxiety Rating

fpsyg-09-02111 October 30, 2018 Time: 15:19 # 3

Each participant's level of social anxiety was measured with the 17-item Japanese version of the Social Phobia Inventory (SPIN-J). Each item is rated on a 5-point Likert-type scale, giving a total score of 0–68. The SPIN-J measures three symptom dimensions: fear, avoidance, and physiological arousal (Connor et al., 2000). The SPIN-J is unique in that it contains a physiological subscale. This subscale may be particularly important in Japan because East Asian patients with anxiety disorders tend to somaticize their symptoms (Kirmayer, 2001). The fear and avoidance scale displayed a good internal consistency (Cronbach's alpha = 0.83,0.81, respectively), but the arousal scale displayed a poor internal consistency (Cronbach's alpha = 0.27) in the present sample.

# Gaze Stimuli

The experimental stimuli were grayscale images of the human eye region. We generated prototypical emotional (happy and disgusted) and neutral gazes. These prototypes were produced from pictures the eyes of four Japanese male volunteers (mean age 23.0 ± 1.15 years) using Adobe Photoshop CS6.0 software. We used eye region of natural smile as positive emotional gazes, that of disgusted face as negative emotional gazes and that of neutral face as neutral gazes, respectively. When we took a picture of each actor's neutral face, we instructed them to keep as expressionless as possible. We took a picture of the actors' natural smiles, when they began to spontaneous laughter. To take a picture of each actor's disgusted face, we asked them to think of an aversive episode. Independent raters, who were 15 male volunteers (mean age 22.0 ± 2.42 years), chose among six emotions (happiness, sadness, anger, disgust, surprise, fear) to describe happy gazes and disgusted gazes and they classified these gazes into positive or negative emotion. The happy gazes were the most frequently judged as happiness (57.0 ± 6.20%) and also chosen as a positive emotion (60.0 ± 6.36%). The disgusted gazes were the most frequently judged as anger (56.7 ± 6.67%), the second frequently judged as disgust (30.0 ± 6.55%), and also chosen as a negative emotion (100 ± 0.00%). Thus, we were confident that experimental stimuli were interpreted as a negative or positive stimulus.

The images occupied 3.4◦ × 13.4◦ of the visual field (4.7 × 19 cm). We morphed each emotion prototype with a neutral image using 16 reference points (three points at equal intervals on the upper and lower eyelids, one point at the inner corners of the eyes and one point at the outer canthus) to generate continua with 100 emotion intensities. We selected 10 intensities for each emotion (10%–100% in 10% steps), giving a total of 84 images. The stimuli were presented at the center of a 27-inch LCD monitor using E-Prime software (Psychology Software Tools, PA, United States), and viewed at a distance of approximately 80 cm. We displayed part of the experimental stimuli (each volunteer's 100% positive, neutral, and 100% negative gazes) before beginning the experiment. Participants were instructed that these stimuli showed a full smile, expressionlessness, and visible distaste, respectively. The 84 stimuli are provided as **Supplementary Material**.

#### Procedure

In each trial, the experimental stimulus was displayed for 1.0 s. Participants were instructed to fixate on the eye region of the stimulus during stimulus presentation, and indicate their impression of the stimulus as negative or positive, corresponding to emotional intensities for positive or negative (impression judgment task). After participants provided their answers, a fixation cross was displayed for 1.5 s and then the next trial was initiated (**Figure 1**). An experimental session consisted of 84 trials. Each session for each participant lasted approximately 5 min, and each participant underwent four experimental sessions. All 84 gaze stimuli were presented in each session, giving a total of 336 trials over the four sessions. The order of stimulus presentation was random in each session.

# Data Analysis

Participants were asked to judge whether their impression of each experimental stimulus was positive or negative. We plotted participants' mean responses using graphs, with the negative response rate as the vertical axis and the emotional intensity of the experimental stimulus as the horizontal axis. We observed a plateau around neutral.

To examine the influence of socially anxious tendencies in judgment, we fitted non-linear regression curves to individual participant responses. We used two models of curve fit as a preliminary analysis: a logistic curve and a two-step logistic curve. The logistic curve represented the entire shape of participants' responses, whereas the two-step logistic curve represented the entire shape of participants' responses and a plateau (**Figure 2**). We estimated goodness of fit for each model by calculating the Akaike Information Criterion (AIC) (Akaike, 1998) and Bayesian Information Criterion (BIC) (Schwarz, 1978). The AIC and BIC for the two-step logistic curve fitted to individual participant's responses were smaller than when the logistic curve was fitted. The results of curve fit using the twostep logistic curve model showed that goodness of fit was better relative to the other model. Therefore, we selected the two-step logistic curve model using the following formula

$$P(\mathbf{x}) = \frac{a}{1 + \exp\{-b\left(\mathbf{x} - \mathbf{c}\right)\}} + \frac{1 - a}{1 + \exp\{-d\left(\mathbf{x} - \mathbf{c}\right)\}}.$$

The first term represents the sigmoid shape on the negative emotional side. The second term represents the sigmoid shape on the positive emotional side. In the first term, x is the emotional intensity, P (x) the probability of negative judgment, a the rate of negative judgment of negative emotional gazes, b the steepness of the fitted curve, and c the inflection point. In the second term, 1 − a indicates the rate of negative judgment of ambiguous emotional gazes, d the steepness of the fitted curve, and e the inflection point. Curve fit was performed using a non-linear least squares method (a trust-region algorithm), provided by the Curve Fitting Toolbox in MATLAB R2015b (The MathWorks Inc., Natick, MA, United States). We used Spearman's rank fpsyg-09-02111 October 30, 2018 Time: 15:19 # 4

FIGURE 1 | Timeline of the trial. The experimental stimulus was displayed for 1.0 s, after which a fixation cross appeared for 1.5 s. The next trial began after participants provided their answers.

correlation coefficients to examine correlations between SPIN-J scores and the point of subjective equality (PSE), and between SPIN-J scores and the parameter of the curve fit model. To control for type 1 error, Bonferroni correction was applied for correlation analyses.

# RESULTS

**Figure 2** shows the means of participants' responses and the fitted curve.

We found a significant negative correlation between the fear score (SPIN-J subscale) and PSE (ρ = −0.53, p < 0.05; **Figure 3A**). This indicated that the intensity of negative emotion giving a negative impression became lower as the fear score became higher. We found significant negative correlations between b (which indicates the sigmoid curve's slope at the negative emotional side) and the SPIN-J score (ρ = -0.57, p < 0.05; **Figure 3B**) and fear score (ρ = -0.53, p < 0.05). This indicated that the sigmoid curve's slope at the negative emotional side became gentler as the SPIN-J score (especially fear score) became higher. There were no significant correlations between other SPIN-J subscales and PSE or other parameters of the fitted curve.

# DISCUSSION

The present study examined the influence of socially anxious tendencies on impressions of others' positive and negative emotional gazes. The present results showed that the negative emotional intensity that was regarded as a negative impression became lower as the SPIN-J fear subscale score became higher. Button et al. (2013) performed an expression classification task in which the experimental stimuli were facial expressions that were changed using morphing technology; participants with HSA tendencies more often misclassified emotional expressions than those with LSA tendencies. A previous study reported that children with socially anxious tendencies classified neutral faces as emotional faces (Melfsen and Florin, 2002). Another study reported that patients with SAD more often classified neutral faces as angry faces than healthy people (Bell et al., 2011). Consistent with previous studies, our results suggest that individuals with a high tendency toward social anxiety interpreted subtle intensities of negative gazes as negative expressions.

The amygdala activation related with social threat perception in individuals with HSA tendencies. A positron emission tomography study reported that the amygdala response to public speaking decreased following cognitive behavioral therapy or administration of selective serotonin reuptake inhibitors (Furmark et al., 2002). Selective serotonin reuptake inhibitors are the most commonly prescribed class of drugs for depression, anxiety, and obsessive-compulsive disorder (Lockhart and Guthrie, 2011). Some functional magnetic resonance imaging (fMRI) studies have suggested that increased activity of the amygdala is related to the degree of social anxiety (Straube et al., 2004; Cooney et al., 2006; Stein and Stein, 2008). People with SAD showed increased activation of the amygdala more often in response to angry expressions than to neutral expressions. In those with SAD, activation of the amygdala response to angry faces also increased compared with the response to neutral faces (Straube et al., 2004). Some studies have reported that the amygdala has a role in increasing responses to fearful faces (Whalen et al., 1998; Liddell et al., 2005), and amygdala activation is assumed to be related to social threats, such as fearful gazes (Kanat et al., 2015). Individuals with SAD showed increased amygdala activity in response to neutral or ambiguous emotional faces compared with healthy people (Cooney et al., 2006). The activity of the amygdala in response to emotional faces suggested that individuals with SAD regarded emotional faces as socially threatening. Our fpsyg-09-02111 October 30, 2018 Time: 15:19 # 5

results, which showed that the negative emotional intensity (which was regarded as the negative impression) became lower as the SPIN-J fear subscale score became higher, suggested that individuals with a high tendency toward social anxiety recognized negative emotional gazes of subtle intensity as social threats. In addition, our results showed that the sigmoid curve's slope at negative emotional side gentled as the SPIN-J score (especially fear score) became higher. Some studies suggested that individuals with HSA tendencies attribute excessive psychological cost to disgusted, angry, sad, or neutral faces (Schofield et al., 2007; Douilliez et al., 2012; Button et al., 2013). Our results also indicated that socially anxious tendencies increased negative impressions of negative emotional gazes.

There was the bump on the positive emotional side (left side in **Figure 2**) in the mean negative response rates. The independent raters classified happy gazes into various emotions. Several studies have reported associations between a smile and various emotions other than positive emotions (Niedenthal et al., 2010; Ambadar et al., 2009; Calvo et al., 2013a,b). For example, a smile may be associated with negative emotions such as social dominance, irony, ridicule, or embarrassment. The bump on the positive emotional side suggested that 100% positive emotional gazes were associated with other negative emotions. Although some studies reported associations between positive emotions and negative impressions, our results showed no significant correlation between the SPIN-J scores and the parameters of sigmoid shape on the positive emotional side. Other studies suggested that although individuals with HSA tendencies attributed excessive psychological cost to negative or neutral faces, such excessive psychological cost was not attributed to happy faces to the same extent (Schofield et al., 2007; Button et al., 2013). Consistent with these findings, our results suggested that individuals with a high tendency toward social anxiety had no excessive psychological cost attributed to positive emotional gazes compared with negative emotional gazes.

The present results suggest that others' ambiguous emotional gazes induced more anxiety in individuals with a high tendency toward social anxiety; consequently, individuals with a high tendency toward social anxiety perceived these emotional gazes as threatening. Some fMRI studies have shown that neutral, ambiguous, or negative facial expressions can induce hyperactivation of the amygdala in individuals with SAD (Cooney et al., 2006; Blair et al., 2008; Labuschagne et al., 2010). Further studies of amygdala function would complement the present findings, and increase our understanding of negative interpretation bias in individuals with socially anxious tendencies toward others' ambiguous emotional gazes.

# CONCLUSION

This study aimed to reveal the effects of socially anxious tendencies on impressions of others' positive and negative emotional gazes. We found that the negative emotional intensity that was regarded as a negative impression became lower as the socially anxious tendencies became higher. Therefore, we suggest that negative emotional gazes of subtle intensity may induce social anxiety in individuals with a high tendency toward social anxiety. Individuals with a high tendency toward social anxiety are more likely to have a negative interpretative bias toward negative emotional gazes of even subtle intensity, and interpret these subtle intensity negative emotional gazes as threats.

# AUTHOR CONTRIBUTIONS

fpsyg-09-02111 October 30, 2018 Time: 15:19 # 6

YT and SS designed the experiments and wrote the manuscript. YT performed the experiments, collected the data, and analyzed the data.

# FUNDING

This work was supported by JSPS KAKENHI Grant Nos. 17J08259 (awarded to YT) and 16H02839 (awarded to SS).

#### REFERENCES


#### ACKNOWLEDGMENTS

We thank Audrey Holmes, MA, from Edanz Group (www. edanzediting.com/ac) for editing a draft of this manuscript.

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2018.02111/full#supplementary-material


fpsyg-09-02111 October 30, 2018 Time: 15:19 # 7


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Tsuji and Shimada. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Adaptation to the Direction of Others' Gaze: A Review

Colin W. G. Clifford\* and Colin J. Palmer

School of Psychology, University of New South Wales Sydney, Sydney, NSW, Australia

The direction of another person's gaze provides us with a strong cue to their intentions and future actions, and, correspondingly, the human visual system has evolved to extract information about others' gaze from the sensory stream. The perception of gaze is a remarkably plastic process: adaptation to a particular direction of gaze over a matter of seconds or minutes can cause marked aftereffects in our sense of where other people are looking. In this review, we first discuss the measurement, specificity, and neural correlates of gaze aftereffects. We then examine how studies that have explored the perceptual and neural determinants of gaze aftereffects have provided key insights into the nature of how other people's gaze direction is represented within the visual hierarchy. This includes the level of perceptual representation of gaze direction (e.g., relating to integrated vs. local facial features) and the interaction of this system with higher-level social-cognitive functions, such as theory of mind. Moreover, computational modeling of data from behavioral studies of gaze adaptation allows us to make inferences about the functional principles that govern the neural encoding of gaze direction. This in turn provides a foundation for testing computational theories of neuropsychiatric conditions in which gaze processing is compromised, such as autism.

#### Edited by:

Paola Ricciardelli, Università degli Studi di Milano Bicocca, Italy

#### Reviewed by:

Sebastian Pannasch, Technische Universität Dresden, Germany Alice O'Toole, The University of Texas at Dallas, United States

#### \*Correspondence:

Colin W. G. Clifford colin.clifford@unsw.edu.au

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 10 June 2018 Accepted: 22 October 2018 Published: 09 November 2018

#### Citation:

Clifford CWG and Palmer CJ (2018) Adaptation to the Direction of Others' Gaze: A Review. Front. Psychol. 9:2165. doi: 10.3389/fpsyg.2018.02165 Keywords: gaze direction, visual adaptation, social attention, face perception, sensory coding

# INTRODUCTION

Eye gaze signals play a critical role in human communication and interaction (Argyle and Cook, 1976). To an observer, the direction of your gaze reveals where you are looking and hence what you are looking at. This might be an object of shared attention or it might be the observer him or herself. The direction of your gaze is thus a strong social signal to your intentions and future actions (Baron-Cohen, 1995), and gaze plays a role in many social behaviors that rely on interpersonal coordination of attention and behavior, such as learning, and joint action (Frith and Frith, 2008). Understanding the mechanisms by which another's gaze is perceived and interpreted has consequently become an active area of interest in the burgeoning field of social neuroscience (Nummenmaa and Calder, 2009).

The perception of gaze direction is an interesting phenomenon to study in part because it sits at the interface between visual perception and social cognition. Psychophysics and neuroimaging research has begun to reveal how the human visual system extracts information about another person's focus of attention from the stream of sensory signals that are relayed from the retina to the cortex (Langton, 2010), the role played by sub-cortical structures such as the superior colliculus, amygdala and pulvinar (e.g., in signaling eye contact; Senju and Johnson, 2009), and the interaction of these systems with higher-level attentional and cognitive processes (Carlin and Calder, 2013).

**29**

Gaze direction is a component of our social experience that is relatively tractable for experimental research, as it can be defined along a continuous dimension (e.g., with horizontal deviations of the eyes ranging from approximately 40◦ leftward to 40◦ rightward) and has an identified cortical basis in the superior temporal sulcus (STS, Carlin and Calder, 2013). In this way, perception of gaze direction may serve as an important model system for social neuroscience.

The phenomenon of visual adaptation to gaze direction demonstrates that our perception of other people's gaze is a remarkably plastic process that can be affected by the recent history of stimulation (Jenkins et al., 2006; Seyama and Nagayama, 2006). Adaptation is an overloaded term that refers to three inter-related elements: procedure, process and percept (Wade and Verstraten, 2005). The procedure of adaptation is exposure to a particular diet of sensory stimulation. In response to changes in stimulation, our sensory systems change the way that they process incoming information. These changes in sensory processing give rise to measurable aftereffects in our perception. Adaptation is well-established as a fundamental characteristic of low-level sensory processing, readily apparent for sensory properties like luminance, orientation, and color (Webster, 2015). It is only relatively recently, however, that adaptation to higher-level visual qualities associated with faces and objects has been explored (Clifford and Rhodes, 2005). In the context of eye gaze, an observer who has been adapted to a series of faces displaying averted gaze will tend to display marked changes in their perception of others' gaze direction, such as whether they judge a given face stimulus as looking at them or not (Jenkins et al., 2006; Seyama and Nagayama, 2006). Investigation of the details of this phenomenon has provided important insights into how the direction of another person's gaze is coded in the visual system.

Here, we review the literature on gaze adaptation and discuss its implications for our understanding of gaze processing in the human brain. The following section focuses on establishing the nature of gaze aftereffects. We begin with methods to measure the effects of adaptation on gaze perception, both in terms of perceptual effects and their neural correlates. These measures allow us to ask to what extent the effects of adaptation are specific to eye gaze, and at what level(s) of the visual processing hierarchy they are mediated. Importantly, the determinants and phenomenology of adaptation are also diagnostic as to the processes by which our brains represent the direction of the gaze of others. Correspondingly, Section "What Gaze Aftereffects Reveal About the Sensory Coding of Gaze Direction" focuses on how gaze aftereffects can be used experimentally to probe the sensory coding of gaze direction in the brain. The properties of gaze adaptation allow us to characterize the representation of gaze direction in the neurotypical human visual system in terms of a simple channel structure, and identify functional mechanisms that the sensory coding of gaze direction may rest upon. This, in turn, allows us to test theoretically motivated hypotheses about gaze processing in clinical populations such as people with autism spectrum disorder (ASD).

# THE NATURE OF GAZE AFTEREFFECTS

# Measurement of Gaze Aftereffects

Gaze aftereffects are generally 'repulsive' or 'negative.' That is to say, following adaptation to a series of faces with gaze averted in a particular direction, the perceived gaze direction of a subsequently presented face is repelled away from this adapting direction when compared to how that same face was perceived in an unadapted baseline condition. Robust gaze aftereffects are evident with various techniques of measurement. For example, using a forced-choice judgment of gaze as leftward or rightward, Seyama and Nagayama (2006) found that adaptation to gaze averted horizontally by 35◦ biased the perception of subsequently presented test faces throughout the range ± 4 ◦ such that they were more likely to be reported as gazing in the opposite direction to the adaptor. Similarly, using a forced-choice categorization of gaze as leftward/direct/rightward, Jenkins et al. (2006) found that adaptation to 25◦ averted gaze tended to cause test stimuli averted 5–10◦ to the same side as the adaptor to be reported as gazing directly at the observer.

More recent studies have used a continuous rather than categorical measure of perceived gaze direction, requiring participants to adjust an on-screen pointer to indicate the direction that a face appears to be looking (Palmer and Clifford, 2017a,b; Palmer et al., 2018). Using a pointer has the advantage of allowing the effects of adaptation to be measured metrically (e.g., how much the perceived gaze direction of a given face shifts in degrees as a consequence of adaptation), and across the whole gamut of physically realizable gaze directions. These studies have consistently found that, for adaptors averted by 25◦ , the strongest aftereffects are observed for test stimuli averted by around 10◦ to the same side as the adaptor. The peak magnitude of these aftereffects is approximately 8◦ , which corresponds to roughly half the width of the participant's head at the viewing distance of 50 cm used in these studies. Thus, adaptation caused gaze directed at the participant's ear to appear to be directed straight at them! This is illustrated in **Figure 1**.

Gaze aftereffects can be surprisingly long lasting, surviving up to 24 h when there is no testing immediately after adaptation (Kloth and Rhodes, 2016; Kloth et al., 2017) but decaying with repeated testing (Kloth and Schweinberger, 2008; Kloth et al., 2017). Most studies of gaze adaptation have investigated gaze averted horizontally. However, Cheleski et al. (2013) demonstrated comparable degrees of adaptation to gaze averted vertically or obliquely. Adaptation has also been demonstrated to gaze vergence, the relative deviation of the eyes that indicates the depth at which someone is fixating (Stiel et al., 2014).

# Neural Correlates of Gaze Adaptation

The neural correlates of gaze adaptation have been investigated using both functional magnetic resonance imaging (fMRI: Calder et al., 2007) and electroencephalography (EEG: Schweinberger et al., 2007; Kloth and Schweinberger, 2010). Using fMRI, Calder et al. (2007) assessed how adaptation to gaze averted left or right by 25◦ affected the blood oxygenation level dependent (BOLD) response to test faces with gaze averted 10◦ left, 10◦ right or direct.

They observed adaptation of the BOLD response in the anterior STS and inferior parietal lobule (IPL) specific to faces with gaze averted to the same side as the adaptor. These effects showed significant lateralization to the right hemisphere.

Schweinberger et al. (2007) used EEG to measure eventrelated potentials in response to gaze averted 5◦ left, 5◦ right or direct following adaptation to gaze averted left or right by 25◦ . They observed effects specific to gaze direction not in the N170 but in later (∼250–350 ms) occipitotemporal components. These findings were confirmed and extended by Kloth and Schweinberger (2010), who also observed direction-specific adaptation in a late (∼400–600 ms) positive centroparietal component.

Together, these studies demonstrate the sensitivity of adaptation as a technique to dissociate the neural systems coding different directions of gaze. Future studies combining the relatively high spatial resolution of fMRI with the temporal precision of EEG or magnetoencephalography (MEG) could further our understanding of the mechanisms involved. Consistent with the effect of adaptation to gaze direction on the BOLD response in humans, single-unit recording studies in macaques have identified cells in the anterior STS that respond selectively to the gaze direction of observed faces (Perrett et al., 1985; De Souza et al., 2005). The neural correlates of gaze adaptation in humans thus fit with a picture of gaze-selective processing of faces in higher-order visual pathways in temporal cortex (Carlin and Calder, 2013). However, the effects of prolonged or repeated presentation of faces on the responses of gaze-sensitive cells is yet to be examined with extracellular recording techniques. In addition, sub-cortical structures have been implicated in gaze processing, such as the superior colliculus, amygdala, and pulvinar in signaling eye contact (Senju and Johnson, 2009), but whether gaze aftereffects reflect changes in processing within gaze-specific sub-cortical systems is yet to be determined.

# Specificity of Gaze Adaptation

A fundamental question regarding gaze adaptation is whether the effects are specific to the visual processing of eye gaze. It is helpful to decompose this question into two parts. Firstly, is adaptation occurring at a level at which the direction of gaze is represented, rather than being generated earlier in the visual processing hierarchy? If so, are the effects specific to eye gaze or do they generalize to other directional cues to social attention (e.g., pointing gestures)?

#### Is Gaze Adaptation 'High-Level'?

It has been suggested that perceptual face aftereffects may in large part be generated by the adaptation of early visual mechanisms, such as those that represent local orientation (e.g., Dickinson and Badcock, 2013). While such early mechanisms are insensitive to faces per se, they provide the input to higher, face-selective levels of visual processing. Thus, adaptation of lowlevel mechanisms could change the input to those representations coding facial attributes explicitly. If the effects of adaptation on the representation of gaze direction were simply inherited from earlier levels of processing then this would undoubtedly make them rather less interesting theoretically, although of course no less compelling perceptually.

To reduce the effects of adaptation at low-level, retinotopic stages of processing on face aftereffects, studies typically introduce a size change between adapting and test stimuli such that corresponding regions no longer overlap spatially (e.g., Zhao and Chubb, 2001). When Jenkins et al. (2006) doubled the size of the adapting stimuli while leaving test size the same, this manipulation had very little effect on the pattern of gaze aftereffects they observed. Gaze adaptation has also been shown to survive changes in viewpoint (i.e., yaw rotation of the head) between adaptor and test, which provides an alternative means of disrupting low-level correspondences (Jenkins et al., 2006; Palmer and Clifford, 2017b).

However, the importance of controlling for the inheritance of low-level adaptation effects is illustrated by a study of gaze adaptation under inter-ocular suppression (Stein et al., 2012). Stein et al. (2012) used the technique of continuous flash suppression (Tsuchiya and Koch, 2005) to render their adapting gaze stimuli perceptually invisible. To achieve this, they presented the adaptors to only one eye while the corresponding region of retina in the other eye was stimulated with a stream of continuously changing, colorful patterns. They then presented

fully visible test stimuli. When there was no size change between adaptor and test, Stein et al. (2012) found significant aftereffects of adaptation to the perceptually invisible stimuli, even when the test was presented only to the eye opposite to the adaptor. However, when a size difference of 25% was introduced between adaptor and test, gaze aftereffects to invisible stimuli were abolished. Thus, the study of Stein et al. (2012) not only demonstrates that awareness of the adapting stimulus is required for gaze-specific aftereffects to be generated, it highlights the importance of introducing a manipulation such as a size change between adaptor and test in order to avoid the effects of adaptation at lower levels of processing propagating up through the visual hierarchy.

One can also consider different levels of visual processing at which adaptation might occur within the domain of face perception. Our sense of where other people are looking depends on the integration of different facial features, namely integration of information about the two eyes (Nguyen et al., 2018) and integration of information about eye direction and head orientation (Wollaston, 1824; Otsuka et al., 2016). Thus, key to defining the nature of gaze aftereffects is whether adaptation acts on representations of where other people are looking that are derived from the integration of different facial cues.

First, Stiel et al. (2014) investigated whether gaze adaptation occurs at a level of representation within the visual hierarchy at which information about the deviations of the stimulus's two eyes is integrated. They used two different adapting conditions in which each eye alternated in deviation between 20◦ left or right. In the 'averted' condition, each eye was always deviated in the same direction as the other such that gaze alternated between adapting faces in a series from left to right with eyes parallel. In the 'vergent' condition, the deviation of the two eyes was always opposite such that gaze alternated between adapting faces from converged (i.e., 'cross-eyed') to diverged. Crucially, the behavior of each eye, when considered independently, was the same in the two adapting conditions – alternating between leftward and rightward deviated with a period of 3 s. However, in the 'averted' condition the two eyes alternated in phase with one another (left, right, left, right . . .) whereas in the 'vergent' condition they alternated in anti-phase (converged, diverged, converged, diverged . . .). Because the deviation of each individual eye in each adaptation condition followed the same duty cycle, a difference between these two conditions in the resulting aftereffects could be attributed to adaptation of an integrated representation of the two eyes' gaze, rather than adaptation to the features of either eye alone.

Stiel et al. (2014) compared the effects of these two adaptation conditions on both the perception of gaze direction and the perception of gaze vergence. They observed a greater increase in the range of test gaze directions categorized by observers as directed at them following 'averted' compared to 'vergent' adaptation. Conversely, the range of test gaze vergences categorized as parallel (as opposed to convergent or divergent) was greater for the 'vergent' than the 'averted' adapted condition. This specificity indicates that both adaptation to gaze direction and gaze vergence occur at a level of processing at which information from the two eyes is integrated.

Furthermore, the results of Stiel et al. (2014) demonstrated that adaptation occurs not only to the two eyes as a unitary stimulus but also to the individual eyes independently. In particular, comparison of the cross-adaptation conditions to an unadapted baseline revealed significant effects both of 'vergent' adaptation on perception of gaze direction and of 'averted' adaptation on vergence perception, even though there was a size difference between adaptor and test. Such two-way crossadaptation is indicative of adaptation at a size-tolerant level of representation of the individual eyes that occurs prior to the integration of information from the two eyes to extract a unique direction and depth of fixation.

More recently, Palmer and Clifford (2018) investigated whether adaptation occurs at a level of visual processing that follows the integration of eye and head cues to gaze direction. Their participants were adapted on faces that evoke the Wollaston illusion, in which the direction that the face appears to look differs from its actual eye deviation due to the influence of head rotation on perceived gaze direction (Wollaston, 1824). They compared across sets of faces that were exactly matched in the lower-level features of the image, but appeared to be looking in different directions due to differences in the conjunction of head rotation and eye deviation. The changes in participants' perception of gaze direction following adaptation were consistent with habituation having occurred to the perceived gaze direction of the Wollaston faces, where this is dependent on integration of eye deviation and head rotation, rather than to the actual deviation of the eyes. This indicates that adaptation operates within a higher-level, integrated representation of gaze direction, which relies on holistic processing of the face, rather than to specific features of the eye-region alone.

Similarly, adaptation to gaze direction can occur across a set of face images that differ substantially in their head rotation and eye deviation but maintain a constant direction of gaze relative to the viewer (Palmer and Clifford, 2017b). This suggests that the visual system codes the direction that other people are looking relative to ourselves as a higher-level or 'abstract' perceptual property, independent of the particular combination of head orientation and eye deviation that combine to signal a given direction of gaze in the current moment.

#### Is Gaze Adaptation Distinct From Other Directional Effects in High-Level Vision?

The direction of another person's attention can be signaled to us by visual cues other than their eye deviation, such as their body rotation and pointing gestures. In addition, gaze direction can play a similar role to certain non-social cues, such as arrows, in directing the spatial focus of our own visual attention. Thus, there is a question of whether aftereffects following adaptation to gaze direction reflect changes in processing specific to gaze, or whether they are indicative of changes in processing in more general spatial or directional representations. The latter might include 'social attention' mechanisms that are agnostic to the particular cues that signal the direction of others' attention (Cooney S. et al., 2015; Lawson and Calder, 2016), or even more generic (i.e., non-social) directional mechanisms. This

question was partly addressed in the two original studies of gaze aftereffects. Jenkins et al. (2006) showed that gaze adaptation did not affect performance on a subsequent line bisection task, while Seyama and Nagayama (2006) found that adaptation to arrows did not induce gaze aftereffects. Subsequently, Bayliss et al. (2011) found that gaze adaptation had a direction-specific effect on how subsequently presented faces cued shifts in the subjects' spatial attention, whereas adapting to a pointing stimulus did not. More recently, Palmer and Clifford (2017b) measured perceived gaze direction after adaptation to heads all turned to the same side over the range 0–50◦ but wearing dark glasses so that their eyes were not visible. They found that adaptation to turned heads did not induce significant gaze aftereffects, even though adaptation to turned heads does induce marked aftereffects in perceived head direction (Fang and He, 2005). Together these results indicate a degree of independence between representations of gaze direction and other directional or spatial cues.

While adaptation to gaze is specific from other types of directional stimuli, it generalizes from one face to another. For example, gaze adaptation does not show specificity for the sex of the face stimuli, such that there is no aftereffect contingent on the sex of the test following adaptation to a stimulus ensemble consisting of males with gaze averted 25◦ to one side and females 25◦ to the other (Kloth et al., 2015). Similarly, it is also common for studies of gaze adaptation to test the effects of adaptation on the perception of different identities to those adapted on (e.g., Palmer and Clifford, 2017a). Furthermore, gaze adaptation appears to be independent from identity processing in that it has been found to be unimpaired in patients with prosopagnosia (Duchaine et al., 2009).

# High-Level Influences on Gaze Adaptation

A small number of studies have examined how gaze aftereffects are influenced by higher-level aspects of the social context.

Interestingly, the strength of gaze adaptation appears to depend on the observer's belief that the person used as the adapting stimulus can actually see (Teufel et al., 2009, 2013). The original study to report this effect (Teufel et al., 2009) involved an ingenious deception such that participants were led to believe that a pre-recorded video they were watching was actually a live camera feed from an adjacent room. In this video, participants saw a person gazing to the side while wearing a pair of goggles. In one condition, participants were led to believe that the goggles were transparent, and in another condition, that the goggles were opaque. Thus, participants believed that the adaptor could see in one condition but not the other, despite the visual features in the images being identical. The results revealed stronger gaze aftereffects when participants believed that the adaptor could see, suggesting that participants' explicit beliefs about the person used as the adapting stimulus modulated the degree of adaptation (**Figure 2**). This result suggests that higherlevel cognitive factors can determine whether the mechanisms that represent gaze direction are recruited to process a given visual stimulus or not. The interaction between theory of mind

and social-perceptual mechanisms is discussed further in Teufel et al. (2010).

In their more recent study, Teufel et al. (2013) employed a similar set-up, except that this time participants were told that they were watching a pre-recorded video. In this experiment there was no significant difference in the magnitude of aftereffects between the two different types of adaptor, which the authors interpreted as indicating that mechanisms mediating theory of mind might be more strongly engaged when a stimulus video is actually believed to be a live link than when it is known to be a pre-recorded video.

The magnitude of gaze aftereffects has also been shown to be susceptible to modulation by the emotion of the adapting face, such that bigger aftereffects are evident following adaptation to happy compared to surprised faces (Seyama and Nagayama, 2006). This effect of emotion was evident despite the fact that Seyama and Nagayama were careful to use only information carried by the eyebrows and mouth to convey emotion while the eyes themselves were identical between the happy and surprised adapting stimuli. It is unclear by what mechanism emotional facial expression modulates the size of gaze aftereffects, but these results suggest, like those of Teufel and colleagues, that the engagement of adaptable representations of other people's gaze direction (e.g., perhaps relating to the salience of the eye region) depends on the broader social context.

### Function of Gaze Aftereffects

fpsyg-09-02165 November 7, 2018 Time: 16:56 # 6

Adaptation is generally held to offer functional benefits to the processing of sensory information. What benefits might adaptation confer to the processing of gaze direction?

In general, adaptation allows sensory systems to be selfcalibrating in their mapping of aspects of the environment onto patterns of neural response and offers them the potential to optimize their coding properties to match the prevailing diet of stimulation (Clifford, 2005). Self-calibration encompasses recalibration and error-correction. Recalibration refers to changes in coding in response to changes in the environment. Error-correction is the process of adapting to changes in the system itself in an unchanging environment. Sensory systems, of course, have direct access neither to the state of the environment nor to their own internal state. However, they can remain self-calibrating by adapting their coding properties to keep the distribution of sensory response patterns constant (Benucci et al., 2013). In some aspects of visual processing, such as color vision, self-calibration allows the visual system to recalibrate to routine changes in the environmental illumination and thus achieve a high degree of constancy in the perception of the color of objects despite large changes in the wavelength distribution of light incident on the retina (Webster, 2015). In other visual modalities where strong adaptation is also observed, such as the processing of spatial orientation, the statistical distribution of environmental stimuli is typically less volatile and so the principal functional benefits of adaptation are in error-correction and optimization of coding efficiency. Given that the distribution of others' gaze directions to which one is exposed is presumably fairly stable over time, it seems reasonable to assume that functional benefits of adaptation to gaze direction are also likely to be best understood in terms of error-correction and coding efficiency rather than recalibration to environmental changes. However, in the laboratory, artificially biasing the distribution of gaze directions to which an individual is exposed provides an excellent opportunity to measure the perceptual effects of gaze adaptation and, in turn, to make inferences about the underlying processes.

# WHAT GAZE AFTEREFFECTS REVEAL ABOUT THE SENSORY CODING OF GAZE DIRECTION

In the previous section, we saw that gaze aftereffects are indicative of a neural system that represents gaze direction as a property of the world abstracted from specific face features (e.g., combinations of head and eye direction), generalize across facial identities but are distinct from other high-level directional representations, and are recruited flexibly depending on the social context. In the current section, we examine how gaze aftereffects have substantiated a framework for understanding the sensory coding of gaze direction in the visual system, including the computational mechanisms that underlie these effects.

# Frame of Reference of the Adapted Representations

One characteristic of sensory coding that the adaptation paradigm can be used to probe is the frame of reference in which the nervous system represents information about other people's direction of gaze. An important distinction can be drawn between a first-person reference frame, in which gaze direction is coded relative to the observer, and a second-person reference frame, in which gaze direction is coded relative to an axis of the stimulus (e.g., relative to the orientation of the head or body of the individual being observed). See **Figure 3** for an illustration. Palmer and Clifford (2017b) adapted participants to a set of face images that maintained a particular direction of gaze in one reference frame, while varying the direction of gaze in the other reference frame. To test whether gaze adaptation involved representations in a first-person reference frame, they showed participants a series of face images that shared a particular direction of gaze relative to the observer (25◦ averted), but varied in their direction of gaze relative to the head of the stimulus. To test for gaze adaptation in a secondperson reference frame, participants were shown stimuli that had eyes deviated 25◦ relative to the stimulus head, but where the

reference frame). In contrast, their eye direction relative to their own head is 45◦ (the 'second person' reference frame). (B) Across these face images, the direction of gaze is in the same rightwards direction relative to the viewer, but signaled by different combinations of head and eye angle. There is evidence that the brain contains representations of where other people are looking relative to oneself that are engaged regardless of the particular head and eye direction that combine to signal this direction of gaze in a given image (Palmer and Clifford, 2017b; Clifford, 2018; discussed in Section "Specificity of Gaze Adaptation").

direction of gaze relative to the viewer varied. The pattern of aftereffects they observed revealed adaptation specifically of firstperson representations of gaze direction, with no evidence for adaptation in a second-person reference frame.

Whereas Palmer and Clifford (2017b) manipulated the yaw rotation of the head (i.e., around its vertical axis), Seyama (2006) investigated the effect of rotation in the image plane. Seyama found that adaptation to 35◦ horizontally averted gaze of a set of upside-down faces generated an aftereffect with upright test faces in a first-person reference frame. In other words, adaptation to upside-down faces gazing to their left (and therefore the observer's left) produced a similar pattern of aftereffects as adaptation to upright faces gazing to their right (the observer's left).

However, Seyama (2006) found that adaptation to faces rotated by 90◦ in the image plane also generated an aftereffect with upright test faces. Adaptation in a first-person reference frame should produce no aftereffect under these conditions, as the adapting stimuli were gazing upward or downward in the observer's frame of reference whereas the test stimuli were gazing left or right. Instead, the results are consistent with adaptation in a second-person reference frame.

Seyama (2006) interpreted his results as evidence for adaptation occurring at both first- and second-person levels of representation. However, it is notable that in their study Palmer and Clifford (2017b) found no evidence for adaptation in a second-person reference frame and that the second-person aftereffects reported by Seyama (2006) were on average only around 40% of the magnitude of perceptual aftereffects reported in the original study of Seyama and Nagayama (2006). Thus, it appears that the adaptive coding of gaze direction in the human visual system probably occurs primarily, although not exclusively, in a first-person frame of reference.

# Using Adaptation to Reveal the Structure of the Neural Channels Coding Gaze Direction

Our perceptual experience contains a variety of information about the external world, and a fundamental question in neuroscience is how this information is represented or encoded in neural activity. This question can be considered both for lowerlevel perceptual properties, like the orientation of contours in our field of view, and more complex perceptual properties, like the gaze direction of a face. A strategy that appears to be employed across different levels of the visual hierarchy is population coding, whereby a perceptual property is represented in terms of the relative activity of a set of sensory neurons that are each tuned to different locations along the relevant stimulus dimension (Suzuki, 2005). For instance, area V1 contains neurons that are selective in their responding to edge-orientations, including neurons that respond most strongly to vertical edges, and neurons that respond most strongly to particular off-vertical edges (Hubel and Wiesel, 1968). In this way, the particular orientation of a presented stimulus can be encoded in terms of the relative firing rates that it elicits across this set of neurons.

The perceptual effects of adaptation can be understood in terms of how the pattern of activation elicited by a stimulus across a sensory population is modified by selective changes in the responsiveness of sensory neurons. In psychophysics, the concept of sensory channels refers to cell populations in the nervous system that display different tuning along a given stimulus dimension. Adaptation can be modeled as a reduction in the responsiveness of a set of sensory channels proportional to how strongly each channel is engaged by the adapting stimulus (Graham, 1989). In this way, a given test stimulus will produce a different pattern of activation across sensory channels before adaptation compared to after adaptation, due to uneven changes in the sensitivity of the channels. The perceptual effects of adaptation reflect how altered channel sensitivities affect the 'end result' of population coding, so can be used to probe how a stimulus property is represented across a system of channels. In the context of gaze perception, single-cell recording studies in macaque monkeys have identified cells in the temporal cortex (specifically, the anterior STS) that not only respond selectively to faces but also respond differentially to the gaze direction of the face (Perrett et al., 1985; De Souza et al., 2005). In humans, haemodynamic responses in anterior STS similarly indicate the existence of distinct cell populations tuned to different gaze directions (Calder et al., 2007; Carlin et al., 2011), and various sub-cortical areas are also implicated in the rapid detection of direct eye contact (Senju and Johnson, 2009). Studies that investigate the perceptual effects of adaptation to gaze direction have built upon these findings by providing important new insights into how gaze direction may be encoded across a set of gaze-selective sensory channels (Clifford, 2018).

One functional characteristic of gaze processing of which aftereffects can be diagnostic is the number of sensory channels involved in representing gaze direction. The earliest studies showed dissociable perceptual aftereffects of adaptation to leftward and rightward averted gaze (Jenkins et al., 2006; Seyama and Nagayama, 2006) indicating that there are dissociable neural mechanisms coding these directions, likely located in the right anterior STS of the human brain (Calder et al., 2007). The relative activation of two opponent channels tuned to leftward and rightward gaze could in principle code the full range of horizontal gaze directions that we encounter, with direct gaze represented by equal activation of these channels and increasingly averted gaze represented by increased activation in one channel over the other. However, psychophysical adaptation studies further suggest the existence of a channel coding explicitly for gaze directed at the observer, pointing to the existence of at least three sensory channels coding for horizontal gaze direction (Calder et al., 2008; Palmer and Clifford, 2017a).

Calder et al. (2008) investigated the range of test gaze directions that observers categorized as being directed at them, and observed opposite effects of adaptation on perceived gaze direction depending on whether adaptation was to a series of faces (i) all gazing directly at the observer or (ii) alternating between leftward and rightward gaze averted by 25◦ . Calder et al. (2008) reasoned that, within each of these two adaptation conditions, channels tuned to leftward and rightward gaze should be engaged to the same extent. Consequently, in a two-channel

opponent system, the two adaptation conditions should have qualitatively the same effect on the range of test gaze directions categorized as direct. However, adaptation to direct gaze was found to narrow the range of test gaze directions categorized as direct whereas adaptation to alternating leftward and rightward gaze broadened the range (see Lawson et al., 2009, 2011 for analogous findings regarding adaptation to body and head direction, respectively). Calder et al. reasoned that the most parsimonious account of this pattern of data was a system of three broadly tuned channels tuned to leftward, rightward and direct gaze, respectively. Under such a system, adaptation to direct gaze would engage primarily the direct channel, causing the range of subsequent test directions perceived as direct to narrow. Conversely, adaptation to alternating leftward and rightwards gaze would preferentially engage the leftward and rightward channels, causing the range of gaze directions perceived as direct to broaden.

The use of a categorical measure of perceived gaze direction (e.g., left/direct/right) requires observers to adopt a decision criterion as to where precisely the boundaries between categories lie. It is conceivable that adaptation might affect such subjective category boundaries in a systematic way, rather than the perceptual experience of gaze direction per se (Storrs, 2015). For example, if exposure to a series of directly gazing faces served as a repeated reference as to what constitutes direct gaze then it might both narrow the range of test faces categorized as direct and make the location of the subjective category boundaries less variable on a trial-by-trial basis. If this were the case, then one might in principle be able to account for the pattern of data reported by Calder et al. (2008) within the framework of a two-channel opponent system. In other words, adaptation to direct gaze might not affect perception of gaze direction, but rather how a given perceived direction of gaze is categorized.

Palmer and Clifford (2017a) revisited the question of what channel structure underlies the coding of horizontal gaze direction using a different response method. Participants were required to use a pointer to indicate perceived direction, avoiding the need for them to adopt subjective category boundaries in their responding. Although not finding clear evidence of an aftereffect of adaptation to direct gaze (see also Kloth and Schweinberger, 2010), Palmer and Clifford observed in their data a novel characteristic diagnostic of the existence of a direct channel. Specifically, they found that the magnitude of aftereffects to 25◦ averted gaze was tuned for test direction, with the maximum aftereffects evident for test stimuli averted 10–15◦ to the same side as the adaptor. This finding was replicated in Palmer and Clifford (2017b) and Palmer et al. (2018).

Using computational modeling to simulate hypothetical channel structures, Palmer and Clifford (2017a) demonstrated that the tuning of aftereffect magnitude for test direction is characteristic of a system comprising a small number of broadly tuned mechanisms whose activity is subject to divisive normalization (**Figure 4**). Specifically, leftwards and rightwards channels are combined in opponent fashion, as in a simple opponent model, but this opponent signal is divided by the sum of the signals across all (leftward, rightward, and direct) channels. Their simulations supported the intuitive notion that the effects of adaptation on perception should be most evident when the test stimulus engages channels differentially affected by the adapting stimulus. This leads to distinct predictions for the tuning of aftereffects in two-channel opponent and threechannel systems. For example, following adaptation to leftward averted gaze, a system of only two opponent channels would produce the strongest aftereffects for direct test stimuli, as this is the direction for which the strongly adapted (leftwards) and relatively unadapted (rightward) channels are equally engaged. In a three-channel system, however, the strongest aftereffects following adaptation to leftwards averted gaze would be evident for a test direction where leftwards and direct channels are equally engaged, i.e., moderately averted gaze to the same side

as the adaptor, as observed empirically. The findings of Palmer and Clifford (2017a) thus support the conclusions of Calder et al. (2008) that the coding of horizontal gaze direction is inconsistent with the operation of a two-channel opponent system but can be parsimoniously accounted for within a three-channel framework.

It is interesting to note the architectural similarity between the model of gaze processing proposed by Palmer and Clifford (2017a) and the channel structure of the early stages of the color vision pathway (Gegenfurtner and Sharpe, 1999). Normal human color vision is subserved by three classes of retinal cone photoreceptor with overlapping bands of wavelength selectivity. Signals from these three chromatic channels are combined in opponent fashion in the sub-cortical visual pathway and subject to normalization (Solomon and Lennie, 2005). Palmer and Clifford's computational modeling similarly highlights the roles of channel opponency and divisive normalization in the coding of gaze direction. Divisive normalization is a form of gain control, ensuring that the relative activation across channels provides a code that is robust to variation in the absolute level of stimulation. In the present context, the encoded gaze direction is normalized to the pooled activity across gaze-selective sensory channels. This ensures that the encoded gaze direction is not affected by extraneous variables that might influence activity across these channels (e.g., stimulus contrast), but rather relates only to the proportional difference in activity between gazeselective channels. Divisive normalization has been argued to be a canonical feature of nervous system function (Carandini and Heeger, 2012), though it is relatively unexplored in the context of higher-level, social vision. In general, the effect of adaptation on sensory coding is complicated by its potential to act on both driving and suppressive mechanisms (Solomon and Kohn, 2014). Here, the precise tuning profile of gaze aftereffect magnitude generated by the model to fit the empirical data arises because signals from the adapted channel(s) feed into not only the driving mechanism ('N' in **Figure 4**) but also into the normalization signal ('D' in **Figure 4**).

In summary, attempts to account for the specific characteristics of gaze aftereffects (e.g., their tuning across test directions) has substantiated a computational framework for understanding the sensory coding of gaze direction in the visual system.

# Gaze Adaptation in People With Autism Spectrum Disorder

Autism spectrum disorder is a heterogeneous developmental condition whose characteristics include atypicalities in social communication and interaction (American Psychiatric Association, 2013), including gaze-based behaviors (Baron-Cohen, 1995). To understand the social mind in ASD, it is important to study both the perceptual mechanisms that furnish the individual with information about other people (e.g., the eye direction detector; **Figure 2**) as well as more cognitive processes that employ or interpret this information within the broader social context (e.g., theory of mind). There is now a significant body of research investigating perceptual function in ASD (e.g., Simmons et al., 2009; Marco et al., 2011), and many theories emphasize how systematic differences in the processing of sensory information may contribute to diverse features of this condition (e.g., Happé and Frith, 2006; Mottron et al., 2006; Pellicano and Burr, 2012; Rosenberg et al., 2015; Palmer et al., 2017). However, the sensory mechanisms that underlie perceived gaze direction are yet to be comprehensively examined in ASD, despite the relevance of gaze perception to many social-cognitive functions. As we have seen, perceptual adaptation is a useful tool for probing how information about other people's direction of gaze is flexibly encoded in the visual system.

Reduced effects of adaptation to gaze direction have been observed in ASD both in children (Pellicano et al., 2013) and adults (Lawson et al., 2017). Palmer et al. (2018) also recorded perceptual aftereffects in a sample of adults with ASD, and fitted a computational model of the sensory coding of gaze direction to this data to characterize the function of specific mechanisms involved in gaze processing. This included simulating the effect of varying either the degree of divisive normalization ('1-w' in **Figure 4**) or the degree of channel adaptability in the model of horizontal gaze coding proposed by Palmer and Clifford (2017a). A recent hypothesis is that altered divisive normalization processes may contribute to a wide array of the behavioral consequences in ASD (Rosenberg et al., 2015), though it had not been explored whether any differences in divisive normalization computations were apparent in the context of perceived gaze direction. On the basis of their computational modeling, Palmer et al. (2018) predicted that a reduction in the degree of divisive normalization in the gaze system should lead to a broader tuning profile of gaze aftereffect magnitude as a function of test direction (**Figure 5**). In contrast, reduced sensitivity to the recent history of sensory stimulation more generally (i.e., reduced adaptability of sensory channels) would lead to a reduction in the overall magnitude of perceptual aftereffects, rather than a change in the tuning profile as a function of test direction.

However, 27 adults with a diagnosis of ASD showed no difference from matched neurotypical controls in either the overall magnitude of their gaze aftereffects or the degree of divisive normalization inferred from fitting the model to their data. On the basis of a Bayesian statistical analysis, Palmer et al. (2018) concluded that their results provide strong support for there being no difference between ASD and control groups in how the effects of adaptation differ across test directions. Nor was there a significant correlation between the strength of adaptation or normalization at an individual level and autistic features (ADOS and AQ scores). As described in the previous section, the perceptual effects of adaptation to averted gaze can be indicative of several functional mechanisms, including (i) the flexible adjustment of channel gain in response to the recent history of sensory stimulation, (ii) the divisive normalization of sensory responses, and (iii) the channel structure coding for horizontal gaze direction. Thus, the robust magnitude and profile of perceptual aftereffects observed in this study is a testament to the typical coding of other people's gaze direction in the visual system in adults with ASD, despite the social-cognitive differences that are characteristic of this condition.

direction. (B) The simulated effects of channel adaptability on perceptual aftereffects. The plotted lines show the simulated aftereffects for a series of models with the same degree of normalization, but where exposure to the adapting stimulus results in either a stronger or weaker change in subsequent channel sensitivities. The degree of channel adaptability scales the magnitude of perceptual aftereffects, but has a less distinct effect on the tuning of aftereffects across stimulus gaze

directions compared to the effect of varying normalization shown in (A). The observation by Palmer et al. (2018) of strong gaze aftereffects in adults with ASD appears at odds with previous findings of reduced effects of adaptation to gaze direction in per se. DIRECTIONS

ASD both in children (Pellicano et al., 2013) and adults (Lawson et al., 2017). A methodological difference between these studies is the method used to quantify shifts in perceived gaze direction associated with adaptation. In Palmer et al. (2018) participants indicated the perceived direction of gaze metrically, by setting the rotation of a pointer, while in the previous two studies participants made a categorical judgment as to whether the face was looking directly toward them or not. The difference between studies may therefore reflect a difference between groups in how gaze directions are categorized. For instance, when using categorical measures of gaze perception, a given shift in perceived gaze direction following adaptation will be most apparent when it occurs across category boundaries (e.g., perceived gaze direction shifting from an angle of gaze consistently categorized as 'averted' to one consistently categorized as 'direct'). The effects of adaptation may thus be less distinct in individuals who classify a wider range of gaze deviations as direct, or who have a less sharp transition between what they judge as being 'direct' and 'averted' gaze. In the two earlier studies of gaze adaptation in ASD (Pellicano et al., 2013; Lawson et al., 2017), participants with ASD more commonly categorized gaze as direct at baseline compared to neurotypical controls. Thus it might be that the reduced effects of adaptation reported in these studies reflect a greater difficulty in detecting adaptation effects in the ASD group when using categorical measures, rather than a difference in the effects of adaptation on the coding of gaze direction

# CONCLUSION AND FUTURE

In this review, we have discussed the marked changes in perception of gaze direction that occur following adaptation to faces with a particular direction of gaze (Jenkins et al., 2006; Seyama and Nagayama, 2006). These aftereffects can be measured robustly and are evident across faces with a range of different test gaze directions (Palmer and Clifford, 2017a,b; Palmer et al., 2018). There is psychophysical evidence that gaze aftereffects reflect habituation of neurons that are specifically involved in the representation of other people's gaze direction, rather than merely being inherited from changes in processing at lower levels of the cortical visual hierarchy or reflecting habituation at the level of more generic directional representations (Jenkins et al., 2006; Stein et al., 2012). Moreover, gaze aftereffects are indicative of a level of visual processing in which different facial features are integrated to produce our sense of where others look (e.g., head and eye direction; Stiel et al., 2014; Palmer and Clifford, 2017b; Clifford, 2018). Adaptation to gaze direction is associated with changes in the neural processing of faces that are detectable in both haemodynamic responses (Calder et al., 2007) and scalp potentials (Schweinberger et al., 2007; Kloth and Schweinberger, 2010). The effects of adaptation on cortical processing have been

observed in the right anterior STS and right IPL (Calder et al., 2007), consistent with a view of gaze-specific processing emerging in higher-level visual pathways in temporal cortex (Carlin and Calder, 2013). However, the contribution of sub-cortical areas of the 'social brain' to gaze aftereffects, such as the superior colliculus, amygdala, and pulvinar (Senju and Johnson, 2009), is currently unknown.

Exploring the determinants of gaze aftereffects has provided insights into how other people's direction of gaze is encoded in the visual system, and the functional and computational mechanisms upon which this process depends (Calder et al., 2008; Palmer and Clifford, 2017a). This work has focused so far on the representation of horizontal gaze direction across gaze-selective sensory channels. Gaze is a multi-dimensional phenomenon, however, such that the focus of another person's gaze can be described in spherical coordinates relative to their face, with both polar and azimuthal angles as well as a particular depth of fixation. Further work thus remains to investigate the effects of adaptation on the perception of gaze deviations along the vertical and oblique axes (Cheleski et al., 2013). This is an important extension, not only because the geometry of the eyes and head is very different along the horizontal and vertical directions, but also because of the different social signals conveyed by horizontally and vertically averted gaze. For example, downwards gaze can signal shame or embarrassment (Darwin, 1965), and gaze can be averted downwards while still being directed at the viewer (Lawson et al., 2011). Similarly, there is a question of how adaptable mechanisms that carry information about the depth of fixation (i.e., gaze convergence: Stiel et al., 2014) combine with mechanisms that represent gaze direction, to jointly specify the focus of other people's gaze in three-dimensional space (Nguyen et al., 2018). In addition, the positional specificity of gaze aftereffects (e.g., retinotopic or spatiotopic), and by implication the spatial receptive field properties of the mechanisms that represent gaze direction, are currently unknown.

Perceptual aftereffects are also apparent for cues to the direction of other people's social attention other than the eyes, including heads (Fang and He, 2005), static and walking bodies (Lawson et al., 2009; Benton et al., 2016), and pointing hands (Cooney S.M. et al., 2015). There is some evidence for a degree of cross-adaptation between different directional cues, namely from head to body direction but not vice versa (Cooney S. et al., 2015), which may indicate overlap in the mechanisms representing

#### REFERENCES


other people's direction of social attention derived from different bodily features. However, Lawson and Calder (2016) found no significant cross-adaptation either way between head and body direction. Even if the mechanisms prove to be distinct, further research should determine whether or not the channel structures and functional processes underlying the coding of gaze direction and other indicators of social attention share similar principles of organization. For example, while the narrowband model of head viewpoint coding proposed by Chen et al. (2010) contrasts with the broadband model of perceived gaze direction implemented in Palmer and Clifford (2017a), a broadband model might also be able to account for the effects of adaptation on the perception of head and body direction (Lawson et al., 2009, 2011).

Finally, the perceptual adaptation paradigm is useful as a method for assessing plasticity and the sensory coding of gaze direction in conditions with apparent differences in the response to other people's eye gaze, such as autism (Pellicano et al., 2013; Lawson et al., 2017; Palmer et al., 2018), schizophrenia (Tso et al., 2012), and social anxiety disorder (Jun et al., 2013). Understanding the function of basic sensory mechanisms in gaze perception may form an important complement to research on higher-level social cognition in these conditions (Baron-Cohen, 1995; Green et al., 2015).

# AUTHOR CONTRIBUTIONS

CC wrote the initial draft of the manuscript. CP made extensive revisions to the original draft and prepared all of the figures. CC and CP approved the final version of the manuscript.

# FUNDING

This work was supported by Australian Research Council Discovery Project DP160102239.

# ACKNOWLEDGMENTS

This manuscript grew out of an invited lecture on gaze adaptation given by CC to the Biannual Forum of the Japanese Psychonomic Society, whose sponsorship is gratefully acknowledged. We are grateful to two reviewers for their constructive comments.


system. J. Exp. Psychol. Gen. 137, 244–261. doi: 10.1037/0096-3445.137. 2.244


direction. Proc. R. Soc. Lond. B Biol. Sci. 223, 293–317. doi: 10.1098/rspb.1985. 0003


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Clifford and Palmer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Task Dependent Effects of Head Orientation on Perceived Gaze Direction

#### Tarryn Balsdon1,2 \* and Colin W. G. Clifford<sup>1</sup>

<sup>1</sup> School of Psychology, University of New South Wales, Sydney, NSW, Australia, <sup>2</sup> Laboratory of Perceptual Systems and Laboratory of Cognitive Neuroscience, Department of Cognitive Studies, École Normale Supérieure, PSL University, CNRS, Paris, France

The perception of gaze direction involves the integration of a number of sensory cues exterior to the eye-region. The orientation of the head is one such cue, which has an overall repulsive effect on the perceived direction of gaze. However, in a recent experiment, we found the measured effect of head orientation on perceived gaze direction differed within subjects, depending on whether a single- or two-interval task design was employed. This suggests a potential difference in the way the orientation of the head is integrated into the perception of gaze direction across tasks. Four experiments were conducted to investigate this difference. The first two experiments showed that the difference was not the result of some interaction between stimuli in the two-interval task, but rather, a difference between the types of judgment being made across tasks, where observers were making a directional (left/right) judgment in the single-interval task, and a non-directional (direct/indirect gaze) judgment in the two-interval task. A third experiment showed that this difference does not arise from observers utilizing a non-directional cue to direct gaze (the circularity of the pupil/iris) in making their non-directional judgments. The fourth experiment showed no substantial differences in the duration of evidence accumulation and processing between judgments, suggesting that observers are not integrating different sensory information across tasks. Together these experiments show that the sensory information from head orientation is flexibly weighted in the perception of gaze direction, and that the purpose of the observer, in sampling gaze information, can influence the consequent perception of gaze direction.

Keywords: gaze perception, head orientation, dual-route model, social vision, response bias

# INTRODUCTION

Human observers are especially good at judging the direction of another's gaze, with empirical measures suggesting judgments of gaze direction can be about as accurate as human visual acuity would permit (Cline, 1967; Jenkins and Langdon, 2003). This ability is important for social interactions, where information about where someone else is looking gives observers insight into the contents of other's thoughts. It has been suggested that the human eye evolved to facilitate accurate gaze perception, with increased contrast between the white sclera and the dark pupil,

#### Edited by:

Paola Ricciardelli, Università degli Studi di Milano-Bicocca, Italy

#### Reviewed by:

Andrea Marotta, La Sapienza University of Rome, Italy Nuala Brady, University College Dublin, Ireland Sarah Cooney, Trinity College Dublin, Ireland, in collaboration with reviewer NB

#### \*Correspondence:

Tarryn Balsdon tarryn.balsdon@ens.fr; tarryn.balsdon@gmail.com

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 16 July 2018 Accepted: 22 November 2018 Published: 06 December 2018

#### Citation:

Balsdon T and Clifford CWG (2018) Task Dependent Effects of Head Orientation on Perceived Gaze Direction. Front. Psychol. 9:2491. doi: 10.3389/fpsyg.2018.02491

**42**

in comparison to other primates (Kobayashi and Kohshima, 1997, 2001). This allows for increased accuracy in judging gaze direction based on the iris eccentricity (Anstis et al., 1969; Emery, 2000) and the contrast polarity of the dark pupil against the white sclera (Ricciardelli et al., 2000; Sinha, 2000). In addition to cues from the eye region, observers integrate a number of other cues exterior to the eye region, such as the orientation of the head, and the emotional expression of the looker. Some authors have therefore suggested that perceived gaze direction may be computed in a more holistic manner (Tanaka and Farah, 1993; Tanaka and Simonyi, 2016) in order to estimate the overall attentional direction of the looker, or by integrating information in a hierarchical manner, whereby more global information (such as head orientation) is utilized when local information (such as the relative position of the pupil) is unavailable (Perrett et al., 1985, 1992).

When the head is oriented directly toward the observer, the observer can judge the direction of gaze based on the relative position of the pupil within the eye opening, for example, a leftward pupil indicates leftward gaze. However, the task of judging gaze direction becomes more complicated if the head is not oriented directly toward the observer. When viewing real human faces, Gibson and Pick (1963) found observers' perception of gaze was biased in the opposite direction to head orientation, and similar results have been found with tightly controlled realistic face stimuli (Anstis et al., 1969; Otsuka et al., 2014, 2015). In contrast, the Wollaston effect (Wollaston, 1824) shows gaze direction tends to be perceived in the same direction as head orientation in artificial stimuli where the same eye-region is placed in the context of differently oriented heads. Similar results have been found when cartoon faces are moved laterally within a cartoon head, simulating the face eccentricity changes that coincide with head turn (Todorovic, 2009). Thus, the orientation of the head can have both a repulsive and an attractive effect on perceived gaze direction.

The dual route model, illustrated in **Figure 1**, offers a functional account of these two seemingly opposite effects of head orientation on perceived gaze direction (Otsuka et al., 2014, 2015, 2016). The indirect repulsive effect results from the effect of head turn on the eye region information: As the head rotates, the information from the eye region projected to the observer changes in a number of ways, for example, the projected shape of the eyes change, some of the eye may be obscured by the bridge of the nose, and importantly, the amount of visible sclera changes such that, as the head rotates to the right there is increased sclera visible to the right of the pupil, in the same manner as if gaze had shifted to the left in a direct facing head. In this way, the information within the eye region changes with head rotation in a similar manner as if gaze had shifted in the opposite direction to the head, such that the observer may perceive gaze to be directed more in the opposite direction of head rotation. This indirect repulsive effect is mitigated somewhat by a direct attractive effect of head orientation on perceived gaze direction. The orientation of the head acts as a coarse scale spatial cue to gaze direction, causing observers to perceive gaze more in the same direction as head orientation. These cues to gaze direction are weighted differently, resulting overall in a stronger repulsive effect of

a coarse scale spatial cue to gaze direction, attracting perceived gaze in the same direction as head orientation.

head orientation on perceived gaze direction in naturalistic faces, which becomes stronger if the observer is shown only the eye region (thereby weakening the information contributing to the direct attractive effect; Otsuka et al., 2014, 2015).

changes in a similar manner as when gaze is averted in the opposite direction. Head orientation also has a direct effect on perceived gaze direction, acting as

The overall repulsive effect of head orientation on perceived gaze direction (in naturalistic faces) has been measured in a number of ways. Most previous experiments have used a singleinterval design, where the observer is shown a single stimulus and asked to make some judgment about the direction of gaze. Gibson and Pick (1963) asked observers to decide if gaze was directed at them or not, Otsuka et al. (2015) asked observers to judge whether gaze was directed left, right, or direct, with respect to themselves, and Anstis et al. (1969) and Otsuka et al. (2016) asked observers to report the exact direction of gaze, for example, by orienting a pointer in the same direction as the gaze of the stimulus. These experiments produce similar results, using different stimuli and different response types, all measuring an overall repulsive effect of head orientation on perceived gaze direction. This repulsive effect can also be demonstrated simply by examining the example stimuli in **Figure 2**, where the same gaze deviations in differently oriented heads are not perceived as gazing in the same direction. Rather, the typical viewer will perceive gaze in the leftward oriented head as more rightward than the same degree of gaze offset in the rightward oriented head.

In a recent experiment, Balsdon and Clifford (2017) found a far weaker repulsive effect of head orientation on perceived gaze direction than has previously been measured. The experiment

Rather, gaze in the leftward oriented head will be perceived as more rightward than gaze in the rightward oriented head.

compared a commonly used single-interval design with a twointerval design, to investigate the use of the two-interval design as a method of eliminating the possibility of response bias from measures of perceived gaze direction. In the single-interval design, a single stimulus is shown to the observer and they are asked to make a judgment about the direction of gaze. In the two-interval design the observer is shown two stimuli (one after the other) and asked which of the two has more direct gaze. Because the order of the stimuli in the two-interval task is counterbalanced, a tendency to make a certain response does not correspond to reporting a particular stimulus, whereas, in the single-interval design observers may easily develop a tendency to report, for example, leftward gaze, when presented with a rightward oriented head. The two-interval design thereby minimizes response bias, and thus would be more appropriate for comparing the effect of head orientation across populations that may differ systematically in their response biases, such as in patients with schizophrenia (Bentall and Slade, 1985; Brébion et al., 1998). The measured effect in the single-interval task was similar to previous measures, with an overall 'weighting' of head orientation of −0.25, corresponding to a repulsive effect. In the two-interval task the measured weighting of head orientation was 0.07 on average, corresponding to a slight overall attractive effect.

The single- and two-interval tasks are proposed to measure the same perceptual effect, and thus any difference between the measurements made across the two tasks would normally be attributed to response bias in the single-interval task. The results of the two-interval task would therefore suggest that the measured repulsive effect in the single-interval task is merely the result of observers adopting a tendency to respond that gaze is oriented in the opposite direction to the head. This suggestion is untenable for two reasons. First, the repulsive effect can be directly observed by examining example stimuli as in **Figure 2**. If the true perceptual effect were actually attractive, then gaze would be perceived more in the same direction as head orientation, yet the reader should find that gaze direction in the leftward oriented head appears more rightward than in the rightward oriented head. Second, previous experiments have used different response types, so if the repulsive effect were entirely due to the tendency to make a particular response, this should differ with the required response. Furthermore, there is no theoretical basis (nor, to our knowledge, empirical evidence) for a systematic response bias to be implemented in tasks that require observers to report the perceived gaze direction on a scale, such as with the pointer judgment used by Otsuka et al. (2016), and the protractor adjustment used by Anstis et al. (1969). The difference between the single- and two-interval tasks must therefore be explained by something other than response bias. The four experiments presented in this manuscript extend and replicate this work by systematically examining the differences between these two tasks and what causes the differences in the behavioral responses.

The first experiment sought to examine whether any difference in the stimulus presentation across the two tasks could account for the differences in the measured effects of head orientation on perceived gaze direction. Of particular concern was the presentation of two oppositely oriented heads in quick succession in the two-interval task. In the single-interval task, the order of stimuli was randomized, and there was a slightly longer duration between stimulus presentations, as observers were responding after each stimulus. In the two-interval task, the presentation of the first head could alter the perception of the second head by means of fast adaptation, such that the second head appears more repulsed from the first: if the perceived orientation of the head were exaggerated in this way, without altering the information within the eye region, then the measured effect on perceived gaze direction would be more attractive, as was observed in Balsdon and Clifford (2017). If this were the case, then presenting two heads of the same orientation would result in the opposite bias as is measured when oppositely oriented heads are presented. Another possible confound is that, when the head is oriented, one eye is closer to the observer than the other, depending on head orientation. Evidence suggests that observers rely more on the information from the closer eye (Noll, 1976), so the presentation of opposing heads may present observers with the problem of having to switch their attention between eyes, or rely on weaker information from the further eye in one of the intervals. These possibilities would also be mitigated in a two-interval design where the same head orientation is presented across both intervals. To further eliminate any possibility of interactions between stimuli, the following experiment also modified the stimulus presentation procedure of Balsdon and Clifford (2017) to include 500 ms of spatially filtered noise after each stimulus presentation, and add jitter to the stimulus presentation location from stimulus to stimulus.

An additional between subjects condition was added to the experimental procedure, in which participants were presented with the same stimuli as in Balsdon and Clifford (2017), but were only shown the eye region of the stimuli. In these stimuli the direct attractive effect of the head is reduced, whilst the indirect repulsive effect is maintained, resulting in an overall greater repulsive effect of head orientation on perceived gaze direction (Otsuka et al., 2014, 2015). Another possible difference between tasks is that, in the single-interval task, observers may be able to mostly ignore the orientation of the head and focus on the relative iris-eccentricity, which would drive a stronger repulsive effect,

whereas, in the two-interval task, observers' attention is drawn to the head because two oppositely oriented heads are presented in quick succession. If the difference between tasks resulted from some difference in the way observers were attending to the surrounding head, then there should be no difference between tasks when observers are presented with only the eye-region of the stimuli.

# EXPERIMENT 1

The methods are fundamentally the same as those presented in Balsdon and Clifford (2017). Changes to these methods are explicitly stated in the full description below.

### Methods

#### Participants

Participants were recruited from the UNSW first year psychology participation scheme after the study was given ethical approval by the UNSW human research ethics committee, which adheres to the declaration of Helsinki. A total of 49 participants were recruited and randomly allocated to one of two conditions (whole-head and eye-region) such that 24 participants completed the whole-head condition, and 25 completed the eye-region. All participants gave written informed consent to participating. After applying the exclusion criteria as detailed in the analysis section, 20 participants in each condition remained in the analysis.

#### Apparatus

Stimuli were presented using MATLAB (Mathworks) and the Psychtoolbox extensions (Brainard, 1997; Pelli, 1997; Kleiner et al., 2007) on a 32<sup>00</sup> Display++ LCD monitor (Cambridge Research Systems, Rochester, United Kingdom) with a refresh rate of 120 Hz and resolution 1920×1080, with a gray background, mean luminance 60 cd/m<sup>2</sup> . Participants sat 57 cm from the screen with their chin on a chin rest.

#### Stimuli

Four gray-scale faces (two male and two female), with cropped hair and neutral expressions, were created with Daz software<sup>1</sup> . Observers were presented with two versions of each face, one rotated 15◦ to the right and the other 15◦ to the left, an example is shown in **Figure 2**. To control for any effect of stimulus asymmetry, on half the trials, a rightward oriented face was presented by vertically flipping the leftward oriented face and similarly, a leftward oriented face was presented by vertically flipping a rightward oriented face. Eye deviation was manipulated by replacing the original eyes with realistic counterparts that could be moved according to precise angular coordinates. The whole-head stimuli were presented to fit in approximately 400×400 pixels, subtending approximately 14×14 degrees of visual angle, and are the same as used in Balsdon and Clifford (2017). In the eye-region condition (not included in Balsdon and Clifford, 2017), the same stimuli were presented, but only the region around the eyes was drawn on the screen, a rectangle of 130×30 pixels centered on the bridge of the nose. Stimuli were presented within a 400 ms temporal window against a gray background, with a 100 ms raised cosine ramp at onset and offset temporally bordering 200 ms at full contrast. To prevent motion cues, the position of the stimuli on the screen was jittered randomly within a region up to 15×15 pixels from the screen center on each presentation. Each stimulus was followed immediately by a 500 ms presentation of spatially filtered noise (where the spatial amplitude spectrum matched that of the face stimuli). These precautions were not in place in Balsdon and Clifford (2017).

#### Procedure

Observers completed two tasks: the single-interval task, and the two-interval task. The single-interval task was conducted exactly as in Balsdon and Clifford (2017), with the exception of the modifications to stimulus presentation outlined above (jitter in the stimulus position, and the additional noise mask following stimuli). On each trial observers were shown a single stimulus and were asked to respond as to whether the gaze of the stimulus was directed to the right or to the left of them. Responses were entered by pressing '1' for left, and '2' for right, on a standard QWERTY keyboard. Observers were presented with 11 eye deviations ranging from −10◦ to +10◦ in steps of 2◦ for each head orientation, with 14 presentations of each eye-deviation/head orientation, making a total of 308 trials.

The two-interval task was modified to contain two conditions: opposite-head and same-head conditions, which were pseudorandomly intermixed within each block. The opposite-head condition was the same as presented in Balsdon and Clifford (2017) (with the exception of the changes to stimulus presentation outlined above). On each trial, observers were presented with two stimuli (of the same face identity) in succession (with each stimulus followed by 500 ms of noise). Observers were asked to respond as to which interval contained gaze that was more directed at them. Responses were entered by pressing '1' for first interval and '2' for second interval. In the opposite-head condition one interval contained a rightward oriented head and the other contained a leftward oriented head (the order of which was chosen at random). Observers were presented with 11 pairs of eye-deviations, which differed by −10◦ to +10◦ , in steps of 2◦ , relative to base deviations of ±5 ◦ (separate trials). With 14 repetitions of each of these trials, there were 308 trials in the opposite head condition. In the same-head condition the head orientations of stimuli in both intervals were either leftward oriented or rightward oriented. The eye-deviations of these stimuli were the same as in the opposite-head condition, except that they were presented relative to a base deviation of −5 ◦ in one stimulus and +5 ◦ in the other of the pair (the order of which was counterbalanced across trials). Again, 14 repetitions of these trials were used, making 308 trials in the same-head condition, totaling 616 intermixed trials in the two-interval task. Both the single- and the two-interval tasks were completed in a single session of less than 1 h. The order of the tasks was randomized across participants.

<sup>1</sup>http://www.daz3d.com/

#### Analysis

The analysis was conducted in the same manner as Balsdon and Clifford (2017).

In the single interval task, logistic functions were fit to each participant's proportion of 'rightward' responses, and the point of subjective equality (PSE, the gaze deviation at which half the fitted responses were rightward) was taken as the gaze deviation corresponding to subjectively direct gaze. The influence of head orientation was then taken as half the difference between the PSEs for the leftward and rightward oriented heads.

In the two-interval task, opposite-head condition, logistic functions were fit to each participant's proportion of trials where the stimulus with a rightward oriented head was chosen. The PSE was the point at which gaze was perceived as equally direct (or equally averted) between the two heads, and the average of the PSEs for base deviations of −5 ◦ and +5 ◦ was taken as the influence of head orientation on perceived gaze direction. A similar analysis was conducted on the same-head condition, except the PSEs were now calculated for pairs of leftward and rightward heads separately, and the influence of head orientation on perceived gaze direction was taken as half the distance between the PSEs for the leftward and rightward head trials.

Participant's data were excluded from further analysis based on two criteria. First, if the inverse slope of the logistic function exceeded the range of deviations tested, indicating that their responses did not vary systematically with the gaze deviations presented. Second, if the calculated PSE was outside the range of deviations presented, since an accurate measure of the PSE would not be possible in this case. Further inspection of the data indicated that some participants excluded for these reasons appeared to be responding to the orientation of the head rather than the direction of gaze, whilst others appeared non-compliant with experimental instructions.

All statistical analyses were carried out on the measures of the influence of head orientation on perceived gaze direction, as calculated from the PSEs. However, we also present the calculated weighting of head orientation for comparison with previous experiments (Otsuka et al., 2014). The relative weighting (w) of head orientation is calculated such that:

$$\bowtie \mathsf{w}H + (1 - \mathsf{w})E = \mathsf{0}$$

Where H is the orientation of the head and E is the gaze deviation corresponding to perceptually direct gaze. Rearranging this equation gives:

$$w = \frac{-E}{H - E}$$

The relative weighting of head orientation is independent of the degree of head rotation (assuming a linear relationship) such that measures can be compared across experiments that employ stimuli of differing head orientations.

#### Results

A 3×2 mixed ANOVA, with task (single-interval, two-interval same-head, and two-interval opposite-head) as a within subjects measure and condition (whole-head and eye-region) as a between subjects measure, showed a significant effect of task [F(1.35,51.31) = 96.63, p < 0.001, η 2 <sup>p</sup> = 0.72] and a significant effect of condition [F(1,38) = 10.62, p = 0.002, η 2 <sup>p</sup> = 0.22], but no significant interaction [F(1.35,51.31) = 0.64, p = 0.473, η 2 <sup>p</sup> = 0.02]. Within subjects comparisons include a Greenhouse-Geisser correction for violating the assumption of sphericity [χ 2 (2) = 24.26, p < 0.001]. The main effect of task was clearly driven by the difference in the measures from the single-interval task compared to the two-interval opposite-head task (mean within-subject difference = 3.75◦ ± 0.68◦ 95%CI) and the two-interval same-head task (mean within-subject difference = 3.44◦ ± 0.71◦ ), compared to which the difference between the same- and opposite-head two-interval tasks was miniscule (mean within-subject difference = −0.31◦ ± 0.34◦ ). These measures, transformed into weightings of head orientation, are shown in **Figure 3A**.

FIGURE 3 | Comparison of measurements across single-interval, two-interval opposite head, and two-interval same-head tasks, from Experiment 1. (A) Distribution of measures of the weighting of head orientation in each task. The width of each 'violin' corresponds to the probability density at each head weighting. The mean is shown by the solid black line and can be compared to 0, which shows veridical perception. The shaded area is the 95% confidence interval. A negative head weighting indicates a net repulsive effect. (B) Individual measurements of the influence of head orientation on perceived gaze direction in the opposite heads two-interval task compared to the same heads two-interval task. The whole-head condition is shown with filled markers and the eye-region condition shown with open markers. (C) Individual measures in the single- and two-interval tasks. The whole-head condition is shown in filled markers and the eye-region condition shown in open markers. Square vs. Circular markers are used for the opposite- and same-head tasks, respectively. The gray lines show equality. Measurements in (B,C) are from the raw PSEs (in units of degrees), and thus a positive value indicates a net repulsive effect.

In order to examine the relationship between measures across tasks, tests of the correlations between the tasks were carried out by fitting a line (y = mx + b) that minimizes the perpendicular distance to each point (the influence of head orientation on perceived gaze direction measured in one condition compared to another), and taking the slope of this line as the correlation. The significance of the correlations was tested using a nonparametric bootstrap analysis, where a sample of data was drawn from the full data (with replacement) to match the original experiment, and assessing the slope of the best fitting line over 1,000 iterations of this procedure. The strongest correlations were found between the two-interval tasks (opposite vs. same heads; whole-head, m = 0.91, p < 0.002; eye-region, m = 1.02, p < 0.002), as can be seen from the closeness of the data to the line of equality in **Figure 3B**. The single-interval and twointerval tasks also showed some correlation (**Figure 3C**) in the whole-head condition (opposite-head m = 0.75, p = 0.024; samehead m = 0.62, p = 0.034), and for the same-head two-interval task in the eye-region condition (m = 0.54, p = 0.022), but was not significant in the opposite-head two-interval task (m = 0.53, p = 0.108).

Measurements in the whole-head condition were very similar to those presented in Balsdon and Clifford (2017), despite the changes to stimulus presentation that eliminated motion cues and the possibility of interactions between stimuli. The measurements from the single-interval eye-region condition are also in line with previous experiments (Otsuka et al., 2014, 2015, 2016). The twointerval eye-region condition measured, on average, an overall repulsive effect of head orientation on perceived gaze direction. A stronger repulsive effect in the eye-region condition is expected under the dual-route model, as the removal of the surrounding head weakens the direct attractive effect of the head, thereby resulting in a stronger net repulsive effect. Although the twointerval task shows measurements that could be suggestive of a very minimal overall effect of head orientation on perceived gaze direction in the whole-head condition, there is a net repulsive effect measured when only the eye-region is shown to observers. The two-interval task therefore displays a similar pattern of measurements as the single-interval task (a stronger repulsive effect in the eye region condition) that is consistent with the dual-route model, the difference is rather that the overall repulsive effect is measured as weaker in the two-interval task (or that the overall attractive effect is measured as stronger) both in the whole-head and in the eye-region conditions. In both the whole-head and the eye-region condition there was little evidence for a difference between the opposite- and same-head trials in the two-interval task, and indeed the measures were tightly correlated. There was a large significant difference between the measurements from the single- and two-interval tasks in both the whole-head and eye-region condition, and although the measures were still correlated (observers who displayed a stronger repulsive effect in one task also tended to display a stronger repulsive effect in the other task), these correlations were weaker, and in one case, not significant. The results of this experiment therefore rule out the possibility that any superficial difference in the nature of stimulus presentation across tasks could account for the differences in measurements.

Another possible difference between the tasks is the specific decision observers are making. In the single-interval task, observers are asked to decide whether the eyes are looking to the left or to the right of them, whereas, in the two-interval task, observers are making a decision concerning how direct gaze is. There is some experimental evidence for a difference between reporting direct gaze compared to making judgments about the relative direction of gaze. For example, Seymour et al. (2017) found that the "direct gaze bias" measured in patients with schizophrenia, where patients report a wider range of gaze deviations as being directed at them when asked if they are "being looked at," is not apparent when patients are asked to judge whether gaze is directed "left, right, or straight ahead." Patients with Schizophrenia therefore had trouble adopting an egocentric perspective, making judgments about gaze direction relative to themselves, but displayed no difference from typical observers when adopting an allocentric perspective.

Experiment 2 was designed to begin addressing this question of whether observers draw on different sensory cues to make different judgments about the direction of another's gaze. A new group of participants were recruited to complete the singleinterval and two-interval tasks, in addition to a new version of the single-interval task where the judgment matched that of the two-interval task – they were asked to respond as to whether gaze was directed at them or not. If the difference between the single- and two-interval tasks is the result of the different types of judgments required, then there will be no difference between the single- and two-interval tasks when observers are asked to make the same type of judgment in each task, that is, a judgment about the directness of gaze.

# EXPERIMENT 2

#### Methods

Methods were the same as in Experiment 1, with the following exceptions.

#### Participants

A new group of participants were drawn from the same pool as Experiment 1. A total of 49 participants completed the experiment, with 24 in the whole-head condition, and 25 in the eye-region condition. All participants gave written informed consent to participating. After applying the same exclusion criteria as Experiment 1, 20 participants remained in the analysis for each condition.

#### Procedure

The two-interval task was altered to contain only the oppositehead trials, as it was originally designed, and participants were asked to judge which interval contained the more direct gaze. Participants were also asked to complete two versions of the single-interval task. The left/right (l/r) version asked for the same judgment as previously, responding '1' if gaze was directed to the left of them, and '2' if gaze was directed to the right of them. The direct yes/no (y/n) version asked participants to judge whether gaze was directed at them or not, and participants were

instructed to press '1' if the eyes were looking directly at them, and '2' otherwise. The stimuli in both single-interval tasks were exactly the same, and all stimuli were presented in the same manner as Experiment 1, with a 500 ms presentation of a noise patch following each stimulus, and the location of the stimulus jittered randomly on each presentation to be within 15×15 pixels of screen center. The three tasks were completed in a single experimental session of less than 1 h, and the order of the tasks was randomized across participants.

#### Analysis

Data processing was conducted in the same manner as Experiment 1 for the single-interval l/r task and the two-interval task. A slightly more complex approach was required for the single-interval y/n task: The proportion of 'direct' responses at each gaze deviation was fit with the difference from two logistic functions (one that would correspond to increasing 'leftward gaze' responses with more leftward gaze, and one that would correspond to increasing 'rightward gaze' responses with more rightward gaze, had the participants been asked to identify the direction of indirect gaze), as shown in **Figure 4**. The logistic functions were constrained to have the same slope and be equidistant from the peak of the proportion of direct responses. Four parameters could therefore describe the proportion of direct responses in both head orientations: the peak of the proportion of direct responses in each head orientation (two parameters), the distance between the means of the two logistic functions, and the slope of the logistic functions. Four logistic functions were defined from these parameters, all sharing the same slope, with the means calculated from the parameters for the peak of the proportion of direct responses and the distance between the means. The difference between pairs of logistic functions was then fit to the proportion of direct responses in each head orientation. The peak of the proportion of direct responses was taken as the gaze deviation perceived to be most direct. Analogous to the single-interval l/r task,

the influence of head orientation in the single-interval y/n task was calculated as half the difference between the gaze deviation perceived to be most direct in the leftward and rightward oriented heads.

#### Results

A 3×2 mixed ANOVA with task (single-interval l/r, singleinterval y/n, and two-interval) as a within subjects factor, and condition (whole-head and eye-region) as a between subjects factor, revealed a significant main effect of task [F(1.35,51.4) = 52.258, p < 0.001, η 2 <sup>p</sup> = 0.58], and a significant effect of head condition [F(1,38) = 12.377, p = 0.001, η 2 <sup>p</sup> = 0.25], but no significant interaction [F(1.35,51.4) = 2.238, p = 0.133, η 2 <sup>p</sup> = 0.06]. Within subjects comparisons include a Greenhouse-Geisser correction for violating the assumption of sphericity [χ 2 (2) = 24.12, p < 0.001]. The difference between the singleinterval y/n and the two-interval tasks was negligible (mean within subjects difference = 0.39 ± 0.39 95% CI) compared to the difference between the single-interval l/r task and the twointerval task (mean within-subjects difference = 3.19 ± 0.85) and with the single-interval y/n task (mean within subjects difference = 2.80 ± 0.77). The overall means, transformed into head weightings, are presented in **Figure 5A**.

Tests of correlations were carried out as in Experiment 1, with the strongest correlations observed between the single-interval y/n and the two-interval tasks (whole-head condition m = 0.89, p < 0.002; eye-region condition m = 1.01, p < 0.002), as can be seen from **Figure 5B**. Correlations between the single-interval l/r task and the other tasks were not significant in the wholehead condition (p = 0.45 and p = 0.65 for the single-interval y/n and two-interval tasks, respectively), but were evident in the eye-region condition (single-interval y/n m = 0.72, p = 0.002; two-interval m = 0.67, p = 0.002), as seen in **Figure 5C**.

The measurements from the single-interval l/r and the twointerval tasks were similar to the measurements from those tasks in Experiment 1, with a stronger repulsive effect in the single-interval l/r task compared to the two-interval task, and a stronger repulsive effect in both tasks in the eye-region condition compared to the whole-head condition. The large difference in measurements between the single- and two-interval tasks virtually disappeared when observers were asked to make a nondirectional judgment in the single-interval task that was similar to the judgment made in the two-interval task. This strongly suggests that the difference between measurements was the result of the type of judgment required by each task: Observers are showing different effects of head orientation on perceived gaze direction depending on whether they are making a directional (left vs. right) or non-directional (concerning whether gaze is directed at them or not) judgment. There are several possible explanations for this; observers may be weighting the same sensory evidence differently across judgments, or they may be integrating different sensory cues to gaze direction according to the task at hand. The weaker correlation between measures across judgments (compared to within the non-directional judgment tasks) could suggest that observers are integrating different evidence across the judgments (as this would increase

shows veridical perception. The shaded area is the 95% confidence interval. A negative head weighting indicates a greater repulsive effect. (B) Comparison of individual measures of the influence of head orientation on perceived gaze direction in the single-interval y/n and the two-interval task. The whole-head condition is shown with filled markers, and the eye-region condition is shown with open markers. (C) Comparison of the single-interval l/r task with the tasks in which a non-directional judgment was made (the single-interval y/n and the two-interval tasks). The whole-head condition is shown in filled markers and the eye-region condition shown in open markers. Square vs. Circular markers show the single-interval y/n and two-interval tasks, respectively. The gray line shows equality. Measurements are from the raw PSEs, and thus a positive value indicates a net repulsive effect.

uncorrelated noise in the judgments). Thus, this possibility was examined in Experiment 3.

The circularity of the pupil/iris offers a non-directional cue to direct gaze. When gaze is directed at the observer (irrespective of the orientation of the head) the shape of the pupil projected to the observer will be circular. As gaze deviates away from the observer, the projected shape of the pupil becomes more elliptical, but equally so in both leftward and rightward gaze directions, as shown in **Figure 6**. Thus, the apparent circularity of the pupil/iris could be used to assess whether gaze is direct or not, but offers no evidence as to whether gaze is deviated in a more rightward or leftward direction. Humans (and monkeys) are especially sensitive to the aspect ratio of ellipses, being capable of discriminating perfect circles from ellipses with an aspect ratio of just 0.98 (Laursen and Rasmussen, 1975). Thresholds for discriminating aspect ratios of ellipses are smaller than those for

discriminating rectangles (Zanker and Quenzer, 1999; Morgan, 2005) and evidence suggests that this hyperacuity for regularity in circles is supported by specialized mechanisms for curvature discrimination (Dobbins et al., 1987; Dumoulin and Hess, 2007). This specialized sensitivity means that the circularity cue could be used to judge whether gaze is direct or averted, despite the fact it offers no evidence for deciding whether gaze is directed left or right.

Experiment 3 tests whether the circularity cue is used in non-directional judgments by comparing directional and nondirectional judgments under two eye conditions: rotated and translated eyes. In natural conditions, when someone averts their eyes from an observer in the horizontal dimension, this causes two sensory transformations to the iris from the observer's perspective. First, as described above, the pupil/iris is rotated, such that the shape it projects to the observer becomes more elliptical. Second, the position of the pupil/iris is translated within the visible eye region, such that it moves closer to the edge of the eye socket. In the rotated eye condition in Experiment 3 the direction of gaze is defined by the rotation of the eyes as in normal stimuli, where the position of the iris is translated and the projected shape becomes more elliptical, such that the circularity of the pupil/iris can be used as a cue for direct gaze (as in the previous experiments). However, in the translated eye condition, eye direction is defined by moving the pupil/iris such that it is centered in the same location as with rotated eyes, but the pupil/iris remains circular. Thus, in the translated eye condition the circularity of the pupil cannot be used to infer averted gaze,

but there are still pupil translation cues to gaze direction. It is predicted that if the use of the circularity cue is driving the difference between directional and non-directional judgments, then there will be no difference between these judgments in the translated eye condition. Furthermore, if the circularity of the pupil/iris is used as a cue to direct gaze, then it is predicted that observers will be more willing to accept gaze as direct in the translated eye condition compared to the rotated eye condition, when making non-directional judgments, even in directly oriented heads.

# EXPERIMENT 3

## Methods

Methods were largely the same as in the previous experiments, with the following exceptions.

#### Participants

Twenty observers were recruited in the same manner as in Experiments 1 and 2. All participants gave written informed consent to participating. There was only one group of participants as only the whole-head stimuli were tested. After applying the same exclusion criteria as previously, 18 observers were included in the analysis.

#### Stimuli

In the rotated eye condition, the stimuli were exactly the same as presented in Experiments 1 and 2. In the translated eye condition, the manipulation of eye deviation was conducted by drawing a circular pupil and iris centered on the same point as in the rotated eye condition (as if filling in the dotted outline in the rotated eyes of **Figure 6**). An additional head orientation condition was added to the procedure, where stimuli with direct facing heads were created in the same manner as already described.

#### Procedure

Participants were asked to complete two tasks: a single-interval y/n task and a single-interval l/r task. On each trial in the singleinterval l/r task, observers were presented with a stimulus and asked to judge whether gaze was directed to the left or to the right of them. Responses were entered by pressing '1' for leftward and '2' for rightward on a high-speed mechanical keyboard, which allowed for the accurate measurement of reaction times. Observers were presented with gaze deviations ranging from −14◦ and 14◦ degrees in steps of 2◦ , and observers made eight responses to each gaze deviation for each of the three head orientations (−15◦ , 0◦ , and 15◦ ) and in each eye condition (translated or rotated), which were presented in pseudo-random order, making a total of 720 trials. These exact same trials were used in the single-interval y/n task, though presented in a different random order. The only difference was the judgment required of the observers: they were asked to press '1' if they thought gaze was not directed at them, or '2' if they thought gaze was directed at them. The order of tasks was randomized across participants, and both tasks were completed in a single session of approximately 1 h.

#### Analysis

Analysis was conducted in the same manner as Experiments 1 and 2 for the rotated and translated eye conditions separately. After calculating the PSEs, two observers were removed from further analysis as they displayed PSEs beyond the range of eye deviations presented in the l/r task, and closer inspection indicated that their responses did not vary with gaze direction, even at the most extreme eye deviations.

The influence of head orientation on perceived gaze direction was then compared across tasks and conditions, along with measurements of subjectively direct gaze in the direct head. Two follow-up exploratory analyses were then conducted. The first examined the effect of eye rotation vs. translation on the tendency of observers to respond that gaze was direct. The other explored measures of participants' reaction times. Reaction times were measured relative to the offset of the stimulus, and for each comparison the median reaction time of each observer was used for further analysis.

# Results

A 2×2 repeated measures ANOVA with task (l/r vs. y/n) and eye condition (rotated vs. translated) as within subjects factors, revealed a significant effect of task [F(1,17) = 51.395, p < 0.001, η 2 <sup>p</sup> = 0.751] on the influence of head orientation on perceived gaze direction. There was no effect of eye condition [F(1,17) = 0.004, p = 0.949, η 2 <sup>p</sup> < 0.001], and no interaction [F(1,17) = 1.723, p = 0.207, η 2 <sup>p</sup> = 0.092]. For comparison with previous experiments, the corresponding head weightings are shown in **Figure 7**. The average measured head weightings are, if anything, less repulsive than previously measured in the left/right task [M(rotated) = −0.27 and M(translated) = −0.25, compared to M = −0.35 in Experiment 2], and in the yes/no [M(rotated) = 0.09 and M(translated) = 0.07, compared to

M = −0.002 in Experiment 2], though no formal comparisons have been made.

In the y/n task, observers accepted a greater range of gaze deviations as being directed toward them in the rotated eye condition compared to the translated eye condition, as measured by the separation between the two logistic functions describing the proportion of direct responses, which is a similar measure to the width of the cone of direct gaze (Mareschal et al., 2013) [mean difference = 0.764, t(17) = 2.983, p = 0.008]. We had hypothesized the effect to be in the opposite direction. A comparable measure in the l/r task is the distance from 25% rightward responses to 75% rightward responses, based on the fitted logistic function. There was no significant difference in this corresponding width measure between translated and rotated eye conditions in observers' l/r responses to the direct head [mean difference = 0.404, t(17) = 0.717, p = 0.483].

Median reaction times were found to be significantly shorter in the l/r task compared to the y/n task [mean difference = −0.1s, t(17) = −2.898, p = 0.01]. The pattern of reaction times across gaze deviations within each task was then examined. In the l/r task reaction times tended to increase with decreasing gaze deviation, whereas, in the y/n task, reaction times were seen to increase with increasing gaze deviation up to about ±6 degrees, and then decrease again with increasing deviations. After separating the median reaction times by head orientation it was found that the distinct patterns of response times was driven by the direct head condition (**Figure 8**). Whilst this pattern may have been washed out by variability in observers' perceived direct gaze in the rotated heads, it is also unlikely that this pattern is driving the overall differences in reaction times between tasks, as the differences within the y/n and l/r tasks are smaller than the differences between them.

If the use of the circularity cue were causing the difference between the directional and non-directional judgments, then this difference would be (at least) diminished in the translated eye condition. We found no evidence for this. Rather, differences in measurements of the influence of head orientation on perceived gaze direction were driven solely by task differences. This suggests that the use of the circularity cue is not driving the difference between the directional and non-directional tasks, and there is little interaction between this cue and the effect of the orientation of the head on perceived gaze direction.

It could be that, by intermixing the translated and rotated eye conditions, observers stopped relying on the circularity cue altogether as it became unreliable. If this were the case, and the use of the circularity cue were driving the task differences, then there would be no difference in the measured influence of head orientation on perceived gaze direction across tasks, but the task differences were just as apparent as in Experiment 2. Furthermore, if observers were not using the circularity cue at all, then there should be no difference in measurements across eye conditions. Yet, there was a significant increase in the width of the cone of direct gaze measured in the y/n task in the rotated eye condition compared to the translated eye condition. It was expected that, if observers were using the circularity of the pupil as a cue to direct gaze, then observers may show a wider cone of direct gaze in the translated eye condition, where the pupil remains circular across gaze deviations. That the opposite effect was observed suggests that the difference was the result of the change in the geometric and luminance cues within the eye region: In the translated eye condition, the pupil/iris was wider than in the rotated eye condition when the eyes were deviated, meaning that the edge of the iris was closer to the edge of the eye when the centers of the pupils were fixed (as shown in **Figure 6**). The distance of the edge of the iris to the edge of the eye, or perhaps more simply, the slight change in the apparent sclera ratio this would create, could be taken as evidence for eye deviation, causing the decrease in the width of the cone of direct gaze.

The difference in reaction times between tasks could be the result of a difference in the accumulation and processing of sensory evidence between tasks. One proposal is that there may be a difference in the processing of information from an allocentric (with respect to 3D space) compared to an egocentric

(with respect to oneself) perspective. For example, Senju and Johnson (2009) have proposed a model in which direct gaze is initially processed by a subcortical route via the superior colliculus, pulvinar, and amygdala. This fast route utilizes mainly low spatial frequencies and is proposed to modulate the cortical processing of more finely tuned gaze direction information that feeds through to the anterior Superior Temporal Sulcus (aSTS). The difference in the processing times of these pathways could be responsible for the difference in reaction times associated with each judgment, though another possibility is that the difference in reaction times emerged at the response mapping stage. These two possibilities were tested in Experiment 4, where the left/right and yes/no tasks were compared over two stimulus presentation duration conditions. By presenting stimuli for very short or relatively long durations the amount of evidence accumulation is restricted and increased, respectively. If the difference between the tasks results from a difference in the amount of evidence accumulation required for each decision, then we will observe an interaction between presentation duration and the difference in the head weighting between the two tasks.

# EXPERIMENT 4

# Methods

Methods were the same as used in the previous experiments with the exceptions detailed below.

#### Participants

Twenty-five participants were recruited in the same manner as before. All participants gave written informed consent to participating. As in Experiment 3, only the whole head stimuli were tested, in a single group of participants. After applying the same exclusion criteria as in previous experiments, 17 participants were included in the final analysis.

#### Procedure

Participants were asked to complete the left/right and yes/no tasks, using the same stimuli and task instructions as in Experiment 2. There were two changes to the experimental design. First, the response keys were changed such that, in the left/right task observers were asked to press the 'j' key for 'left' and the 'k' key for 'right,' and in the yes/no task, observers pressed 'j' for 'direct' and 'k' for 'not direct' (note that the order of these yes/no responses on the keyboard is reversed compared to Experiment 3). Second, two stimulus duration conditions were included: in previous experiments stimuli were presented for 400 ms (including ramp), here, in the 'short duration' condition, stimuli were presented for a total of 150 ms, and in the 'long duration' condition, stimuli were presented for a total of 900 ms, and each of these conditions included a 50 ms cosine ramp at stimulus onset and offset. Observers were encouraged to respond as quickly but as accurately as they could after the stimulus. Observers completed the four experimental conditions (2 tasks × 2 duration conditions) over four separate blocks, in pseudo-randomized order.

# Results

The influence of head orientation on perceived gaze direction was calculated for each task and each stimulus duration condition separately. These measures, transformed into weightings of head orientation, are presented in **Figure 9**. A 2×2 repeated measures ANOVA, with task (l/r vs. y/n) and stimulus duration (150 ms vs. 900 ms) as within subjects factors revealed a significant effect of task [F(1,16) = 34.81, p < 0.001] but no significant effect of stimulus duration [F(1,16) = 0.02, p = 0.89] and no significant interaction [F(1,16) = 0.02, p = 0.88]. **Figure 10** shows the same comparison of reaction times as in Experiment 3, for each stimulus duration: there was clearly no effect of task on reaction time.

These results show that stimulus duration had no significant effect on the measures of the influence of head orientation

FIGURE 10 | Reaction times by task, across head orientations and gaze deviations, from Experiment 4. The yes/no task is shown in the solid line (mean) with the dark shaded region (95% within subject confidence interval), whilst the left/right task is shown in the dotted line (mean) with light shaded region (95% CI).

on perceived gaze direction, suggesting that any differences in the time course of accumulation and processing of sensory evidence do not significantly contribute to the differences between judgments. Furthermore, the difference in reaction times between judgments found in Experiment 3 was not replicated in Experiment 4. Thus, the difference in reaction times was likely a difference at the response mapping stage. In Experiment 3 observers were asked to press '1' if gaze was averted and '2' if gaze was direct, whereas, in Experiment 4, observers were asked to press 'j' if gaze was direct and 'k' if gaze was averted (the reverse order on the keyboard). It may be that this order of judgments feels more natural to the observer and enables them to make yes/no responses as fast as they make the left/right responses. Together, these results indicate that there is little evidence for a difference in sensory evidence accumulation between tasks. It is therefore unlikely that the difference between tasks is the result of observers utilizing different sources of sensory evidence to make their judgments in each task.

# DISCUSSION

In testing a bias minimizing measure of the influence of head orientation on perceived gaze direction, we (Balsdon and Clifford, 2017) found a large difference between the measures of a bias minimizing two-interval task and a single-interval task that was similar to previous tasks. Experiment 1 tested whether the difference between the single- and two-interval tasks could arise from a superficial difference in the way stimuli were presented. Any effect of apparent motion cues (which could have been more prominent in the two-interval compared to the single-interval task) was minimized by jittering the location of the stimuli on the screen, and adding noise after each stimulus presentation. We also tested the possibility that the perceived head orientation of the second stimulus was influenced by the first, by testing a two-interval task where the same head orientation was presented twice. There was no difference in measures from the two-interval task with same- and oppositely oriented heads. Furthermore, recent evidence did not find a sequential effect of head orientation (where the reported head orientation may be influenced by previously presented head orientations; Alais et al., 2018). Together, this evidence suggests that the difference between tasks cannot be reduced to a superficial difference in the way stimuli are presented.

Experiment 2 provided evidence that the difference between tasks resulted from something more enduring than differences in stimulus presentation. When observers judged whether gaze was direct or averted in a single-interval y/n task, the measured influence of head orientation on perceived gaze direction matched that of the two-interval task (both in terms of the group average, and the correlation between individual measures), despite the fact that observers in the single-interval y/n task were presented with exactly the same stimuli, in the same manner, as in the single-interval l/r task. In comparison, measurements between the two types of judgments (directional vs. non-directional) differed systematically, where the singleinterval l/r task revealed a stronger net repulsive effect in both the whole-head and eye-region conditions compared to when observers made a non-directional judgment. Although observers who displayed a stronger repulsive effect in the directional judgment task also tended to display a stronger repulsive effect in the non-directional judgment tasks, the correlations between measures across these types of judgments were far weaker than the correlation between the two tasks with non-directional judgments (the single-interval y/n task and the two-interval task). This could suggest that observers were using different evidence across the two judgments.

The circularity of the pupil/iris could be used to make nondirectional judgments of gaze direction, but not directional judgments. Removing the circularity cue (by keeping the pupil/iris circular and translating the pupil) in Experiment 3 did not affect measurements of the influence of head orientation on perceived gaze direction. There was some effect of the circularity cue on perceived gaze direction, as observers were more likely to respond that gaze was averted in the translated eye condition, suggesting that the experimental manipulation did have some effect on perceived gaze direction, just not on the influence of head orientation on perceived gaze direction. Although this is the only cue (to our knowledge) that could be integrated differently between directional and non-directional judgments, the difference in reaction times between tasks in Experiment 3 suggested that there could still be some difference in the way that observers accumulate and process sensory information for gaze direction across judgments (for example, in utilizing a subcortical route in processing information relevant to whether gaze is direct).

Experiment 4 tested whether the difference in reaction times reflected a difference in stimulus processing between the two judgments, or a difference at the response mapping stage. If a there was a difference in the accumulation and processing of sensory evidence depending on the judgment, and this was causing the difference in the measures of the influence of head orientation on perceived gaze direction, then we would have expected the difference in the measurements between judgments to change with the stimulus duration. There was no effect of stimulus presentation duration on the measures of the influence of head orientation on perceived gaze direction and, importantly, no interaction between stimulus duration and judgment type/task. Rather, when the correspondence between response buttons and response was reversed in the y/n task in Experiment 4, the large difference in reaction times seen in Experiment 3 was no longer evident, suggesting that the reaction time differences seen in Experiment 3 were the result of a difference at the response mapping stage. Thus the results of Experiment 4 do not show any evidence, at the behavioral level, that the difference between measures of the influence of head orientation on perceived gaze direction from directional and non-direction judgments is caused by differences in evidence accumulation and processing, such as the utilization of a fast subcortical route for the prioritized detection of direct gaze. This, together with the evidence of Experiments 1–3, suggests that observers are weighting the same perceptual evidence differently depending on their intention in examining gaze stimuli – whether they are going to make a directional or non-directional judgment.

When asked to make a directional judgment about where someone is looking, the orientation of the head has an overall repulsive effect on the perceived direction of gaze (Anstis et al., 1969; Otsuka et al., 2014, 2015, 2016). This net repulsive effect results from a stronger weighting of the indirect effect of head turn on the eye-region, compared to the direct attractive effect of head orientation (in accordance with the dual-route model, **Figure 1**). When the direct attractive effect is weakened (for example, when showing only the eye-region of stimuli) this repulsive effect increases, and gaze direction is perceived even more in the opposite direction to the orientation of the head.

When observers were asked to make non-directional gaze judgments, their responses suggested that their perception was closer to the veridically presented gaze deviation than in the directional judgment task. It is not the case that there is no influence of head orientation on perceived gaze direction in the non-directional judgment task, as responses in the eyeregion condition still indicated a net repulsive effect. Rather, the incorporation of the head as a direct cue in the whole-head condition appears to balance the repulsive effect of head rotation on the eye-region information, allowing for a high level of gaze constancy (invariance in perceived gaze direction across different head orientations; Otsuka et al., 2015). This balance is lost when the direct cue is weakened in the eye-region condition, as the information about head orientation is much impoverished.

It should be noted that not all previous experiments using a non-directional judgment have found near perfect gaze constancy. Gibson and Pick (1963; replicated by Moors et al., 2016) found a net repulsive effect of head orientation when observers reported whether the gaze of a real human 'looker' was directed at them or not. We calculated the head weighting in Gibson and Pick's data to be around −0.1 (on average, 2.8 degrees gaze deviation in the same direction as head orientation was perceived to be direct in heads rotated 30 degrees), which is a far smaller net repulsive effect compared to that measured by the directional judgment tasks presented here and previously (an average head weighting of −0.25). The main differences between the non-directional judgment tasks presented here and that of Gibson and Pick (1963) are the angle of head rotation (30 degrees compared to the 15 degrees here), the viewing distance (observers were 2 m from the face in Gibson and Pick), and the viewing time (observers viewed the face until they gave a response). This may mean that at greater distances, or with increased head rotation, observers weight the sensory evidence differently when making non-directional judgments, and thus show a different balance of the attractive and repulsive effects of head orientation. These possibilities deserve further investigation. Another possibility, discussed in Moors et al. (2016) is that there can be some inter-individual variability in the effect of head orientation on perceived gaze direction. Gibson and Pick (1963) included just six participants; Moors et al. (2016) included twelve, whereas our experiments show consistent replication with sample sizes of 20. Indeed, the violin plots across all experiments show observers display a distribution of effects that crosses 0 in every task.

The two-interval task was originally tested as a method of minimizing bias in the measurement of the effect of head orientation on perceived gaze direction (Balsdon and Clifford, 2017), as this would be useful in comparing gaze perception across populations that may systematically differ in their response biases, such as in clinical populations. The two-interval task produces measurements significantly different from the single-interval l/r task, and the measured effect in the two-interval task does not seem to reflect the immediate impression that dominates our perception when simply viewing example stimuli, such as those in **Figure 2**. Given that the two-interval task does not measure the same effect of head orientation on perceived gaze direction as 'directional' tasks such as when observers are making left/right categorizations of gaze, a more appropriate task for testing clinical populations may therefore be the use of an estimation judgment, as in Anstis et al. (1969) and Otsuka et al. (2016). Such judgments are likely to be less affected by response bias, but still measure the overall repulsive effect that dominates passive viewing. This is not to say that measurements from the two-interval task, or non-directional judgment tasks per se, do not reflect the perceptual experience of the observer. What the results of these experiments suggest is that when tasked with deciding if they are being looked at, observers weight and integrate sensory evidence differently, in such a way that there is a genuine (though slight) difference in their perceptual experience of gaze direction. Under the dual-route model this can be captured simply by a change in the relative weightings of the direct and indirect effects of head orientation on perceived gaze direction.

Flexibility in the weighting of sensory cues integrated into the perception of high-level visual stimuli, such as gaze direction, has been demonstrated across a number of previous experiments in which the stimulus was directly manipulated. For example, Langton et al. (2004) showed similar effects of head orientation on perceived gaze direction irrespective of whether the orientation of the head was cued by the angle of the nose or the contour of the head, suggesting that observers are capable of extracting the orientation of the head from whichever sensory cue was provided. Noll (1976), and more recently Otsuka and Clifford (2017), found that observers could use sensory information from either eye when the other was not visible but would rely more heavily on information from the further eye when both were visible. This suggests that observers are more than capable of utilizing whatever sensory evidence is available for the perception of another's gaze direction, and integrating this evidence in a flexible manner. However, the current study provides the first evidence that this flexible integration is important not only for dealing with large changes in the quality and availability of sensory evidence, but also for the intention of the observer when accumulating sensory evidence – in this case, whether they are to make a directional or non-directional judgment. This clearly has strong implications for future research into the perception of gaze direction, and high-level vision in general.

One characterization of the difference in the intention of the observer across the directional and non-directional tasks is that observers take an allocentric vs. egocentric perspective, also termed triadic vs. dyadic gaze perception. Egocentric/dyadic gaze perception involves examining where the looker is directing their gaze relative to the observer, whilst allocentric/triadic gaze perception involves examining where the looker is directing

their gaze with respect to the world. Whilst egocentric gaze perception is important for interpersonal relations (Senju and Johnson, 2009; Hamilton, 2016) and has been linked with both neural reward circuitry and threat circuitry (Kampe et al., 2001; Adams et al., 2003), allocentric gaze perception is important for joint attention to objects in the environment and the capture of attention through gaze cueing (Friesen and Kingstone, 1998). These two roles of gaze perception could be thought to require different emphasis on different cues, and indeed, it was the adoption of an egocentric perspective that Seymour et al. (2017) found to cause a difference in gaze processing in patients with schizophrenia. It would be interesting, given the emphasis on local vs. global processing in people with Autism Spectrum Disorder (Simmons and Todorova, 2018), to examine to what extent the weighting of head orientation information changes across allocentric and egocentric perspectives in this population, for example, by comparing responses across directional and nondirectional judgment tasks.

#### CONCLUSION

In these experiments, we showed systematic differences in the measured effect of head orientation on perceived gaze direction, depending on the task of the observer. We showed that these differences were not caused by some artifact of stimulus presentation (Experiment 1), but rather, a difference in the type of judgment being made – whether observers were judging the horizontal direction of gaze, or making a non-directional judgment about whether they were being looked at (Experiment 2). The difference between judgments is not because observers are relying on different sensory evidence depending on the judgment (Experiments 3 and 4), but because observers are integrating the same evidence in a flexible manner; applying a different relative weighting to the direct head cue and information from

### REFERENCES


the eye region according to their intention in sampling evidence of gaze direction. This is important for future research in considering what type of task to implement for testing different hypotheses, especially in research with clinical populations where systematic differences in the ways patients conceptualize tasks may contribute to differences in measures.

# DATA AVAILABILITY

The raw data supporting the conclusions of this manuscript will be made available by the authors, without undue reservation, to any qualified researcher.

#### AUTHOR CONTRIBUTIONS

TB and CC designed the experiments and analyses. TB carried out the experiments, conducted the analyses, and wrote the manuscript. CC contributed to the manuscript and provided equipment. Both authors gave final approval for publication.

# FUNDING

This research was supported by an Australian Research Council Discovery Grant (DP160102239 to CC).

# ACKNOWLEDGMENTS

We would like to thank Yumiko Otsuka and Matt Patten for contributing to the stimulus presentation code, Alysha Nguyen for contributing to data collection, and Yumiko Otsuka, Isabelle Mareschal, and the UNSW Vision group for their helpful comments.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Balsdon and Clifford. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# An Investigation of Attention to Faces and Eyes: Looking Time Is Task-Dependent in Autism Spectrum Disorder

Teresa Del Bianco1,2 \*, Noemi Mazzoni1,3, Arianna Bentenuto<sup>1</sup> and Paola Venuti<sup>1</sup>

<sup>1</sup> ODF Lab, Department of Psychology and Cognitive Science, University of Trento, Rovereto, Italy, <sup>2</sup> Centre for Brain and Cognitive Development, Birkbeck, University of London, London, United Kingdom, <sup>3</sup> Child Psychopathology Unit, Scientific Institute, IRCCS Eugenio Medea, Bosisio Parini, Italy

A defective attention to faces and eyes characterizes autism spectrum disorder (ASD), however, the role of contingent information – such as the task instructions – remains still unclear. Our study aimed to investigate the face-orienting response and the subsequent attentive selection in the presence of varying task instructions in individuals with atypical and typical development. Twenty young adults with ASD and 24 young adults with typical development participated in our eye-tracking study. The participants received one of three different instructions at the beginning of each trial and watched scenes of a social interaction. The instructions asked either to find an object (visual-search, VS), to identify which actor was paying attention to the conversation (gaze-reading, GR), or to simply watch the video (free-viewing, FV). We found that the groups did not differ in terms of proportion of first fixations to the face. Nonetheless, average looking time and proportional looking time to faces differed across groups. Furthermore, proportional looking time to faces was task-dependent in the ASD group only, with maximum proportion in the GR and minimum in the VS condition. This result cannot be explained by a lack of an initial bias to orient to the face, since the face-orienting tendency was similar in the ASD and the control group.

#### Edited by:

Chris Ashwin, University of Bath, United Kingdom

#### Reviewed by:

Jessica Royer, McGill University, Canada Zoë Charlotte Leonora Freeman, University of Bath, United Kingdom

> \*Correspondence: Teresa Del Bianco teresa.delbi@gmail.com

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 19 July 2018 Accepted: 06 December 2018 Published: 18 December 2018

#### Citation:

Del Bianco T, Mazzoni N, Bentenuto A and Venuti P (2018) An Investigation of Attention to Faces and Eyes: Looking Time Is Task-Dependent in Autism Spectrum Disorder. Front. Psychol. 9:2629. doi: 10.3389/fpsyg.2018.02629 Keywords: autism spectrum disorder, social gaze, eye-tracking, attention, atypical development

# INTRODUCTION

The face is the container and the source of rich social information, such as gaze, emotional expressions, and language. Not surprisingly, human adults preferentially orient to and look longer at images of faces compared to other competing stimuli (Palermo and Rhodes, 2007; Shah et al., 2013). This preference is likely regulated by specific properties of the face – primarily, its configuration and contrast polarity, i.e., the reciprocal arrangement of three dark elements, the eyes and the mouth, surrounded by a lighter-colored area (Stein et al., 2011). The same factors influence the orientation to specific features of a face, in particular the eye-gaze (Tipples, 2005). It has been suggested that an innate visual preference for faces and eyes subtend this powerful bias (Gliga et al., 2009). In fact, newborns show a strong visual preference for face configuration (Johnson et al., 1991), and infants prefer to look at faces with open eyes (Farroni et al., 2002). This

bias has been related to the socio-evolutionary value of the face and the eyes, and it may contribute to the protracted development of the sophisticated human face expertise (Morton and Johnson, 1991; Johnson et al., 2015).

Autism spectrum disorder is a condition of atypical neurodevelopment, characterized by difficulties in social interaction and communication, and restricted interests and behaviors (American Psychiatric Association, 2013). It has been documented that adult people with ASD allocate less attention to faces and their internal features, in particular eyes, and even mouths (Guillon et al., 2014; Chita-Tegmark, 2016). The innate attentive bias that we described above might provoke this attentive deficit (Johnson et al., 2015). This view could explain the specific behavioral and cognitive signatures of ASD, such as the diminished face-processing ability, the disregard for gaze-information, the unawareness and/or the variable degree of difficulty with facial and gaze cues. A major consequence of this deficit could be an insufficient exposure to the face and the eyes during development, not sustained by automatic orienting.

However, longitudinal studies seem to oppose this idea. One study showed an equally powerful attentive bias to faces in infants at-risk (i.e., younger siblings of children with ASD) that later developed the condition and non-at-risk infants during the 1st year of life (Elsabbagh et al., 2013b). In addition, attention to faces appears to decrease only during childhood and adulthood in ASD (Guillon et al., 2016; Kleberg et al., 2017). Specific attention to the eye-gaze is present too in infants at risk of ASD, but starts to decrease earlier, between 2 and 6 months of life, in infants that later develop the condition (Jones and Klin, 2013). These results suggest that the primitive attentive bias for faces and eyes might be intact in ASD early in life, but that attention to the face and the eyes might heterogeneously deteriorate across multiple and possibly divergent developmental paths; therefore, alternative hypotheses have been offered to explain the profound difficulties in face- and gaze-processing that characterize ASD.

An interesting hypothesis focuses on general attentive regulation difficulties that might have cascade effects on the social domain as well as the non-social domain (Elsabbagh et al., 2013b). If face-orienting was subtended by an innate bias, it would only be minimally affected by contextual variation; however, if this function was primarily a specialization of a general attentive ability, it would be expected that factors influencing attentive modulation affect face-orienting as well. In favor of this hypothesis, a set of results highlight that the sensory modality of the contextual information attracting the attention to the face and to the eyes is crucial for the characterization of this deficit in ASD. For instance, it has been reported that the visual preference for the face drops in individuals with ASD when a more refined attentive regulation is required, e.g., when the facial information is conveyed by isolated eye-gaze cues compared to more salient cues, such as global head cues (Thorup et al., 2016), or prioritized by motion cues compared to the static presentation of cues in a sequence (Benson et al., 2009; Fletcher-Watson et al., 2009). Other results suggest that the attention to the face may be even more influenced by contextual enrichment. For instance, individuals with ASD showed diminished facelooking time with realistic video clips and verbal content, e.g., an actor greeting and talking to the participant (Chawarska et al., 2013), and did not increase the fixation duration on the face when it was moving, as opposed to a still portrait, differently from individuals with typical development (Rigby et al., 2016). As a conclusion, differences that might shed light on the nature of the face- and eye-orienting impairment in ASD may involve specific contextual cues that integrate with the attentive regulation function.

Even though gaze and motion cues clearly express a focus of difficulty, to date we lack a complete picture of the alteration of face-orienting and gaze-following as scarce information is available on other types of cues, such as explicit cues. The effect of verbal cues – that often take the form of orders and instructions – on the attention to the face and the eye-gaze might offer a crucial insight about specific patterns of visual exploration in ASD. For instance, in an experiment where participants were explicitly instructed to look for objects, individuals with ASD showed overall shorter fixations compared to individuals with TD (Joseph et al., 2009). This result has been related to a higher mastery of individuals with ASD in the visual processing of objects, compared to individuals with TD. Instead, it may be expected that a task involving the processing of faces and eyegaze, such as reading facial expressions, might be associated with longer fixations, due to more difficult processing (Baron-Cohen et al., 1995); however, this effect has not been investigated yet with the eye-tracking technique.

## Aim of the Current Study

This study aimed to gain additional insight into the attentional processes involved in the lack of specific attention to the faces and the eye-gaze in ASD. We believe that it is crucial to clarify whether an innate attentive bias for faces is impaired, or the lack of preference for faces is affected by a generalattentive dysregulation. In fact, the contribution of one of these two factors may give a very different outcome. In order to achieve this goal, we measured the face-orienting tendency and the face-looking time in young adults with and without ASD using realistic, dynamical stimuli representing a simple social interaction. Additionally, we included explicit instructions that required that the participant completed a visual-search and a mentalization task, conditions that have not been directly explored with the eye-tracking technique. In our view, a lack of specific innate bias would affect the attention to the face, while deviances regarding other areas of interest would be nonsignificant. We expected face-orienting to differ between the groups. Regarding the effect of the explicit instructions, we expected that the task instructions might have opposite effects in participants with TD and ASD. Participants with ASD might be less attracted to the face and display shorter fixations' duration compared to participants with TD, even when the instruction focuses on a task that prioritizes the information coming from the face and the eye-gaze. On the other hand, if the alteration lies on a general impairment of attention, we may observe a similar pattern in both conditions simply because the instruction requires a certain degree of attentive regulation. This effect may also be generalized to AOIs other than the face and would regard the visual-search condition too.

# MATERIALS AND METHODS

fpsyg-09-02629 December 14, 2018 Time: 14:38 # 3

# Participants

Twenty-four young adults with typical development (TD) and 20 young adults with a diagnosis of "High Functioning Autism" (13) or "Asperger Syndrome" (7) participated the study. Experienced clinicians established that the participants met the criteria for ASD as specified in the Diagnostic and Statistical Manual IV (DSM-IV; American Psychiatric Association, 1994) or DSM-5 (American Psychiatric Association, 2013), or the Autism Diagnostic Observation Schedule (Lord et al., 2000), or the Autism Diagnostic Interview (Lord et al., 1994). The essential information about the participants is reported in **Table 1**. Wilcoxon tests indicate that participants did not differ in terms of age (W = 228, p-value = 0.58), the Intelligence Quotient (IQ) as measured with the Raven Matrices (W = 90.5, p-value = 0.07) and the Socio-Economical Score (SES; W = 179, p-value = 0.62). Additionally, the IQ of participants with ASD was assessed with Wechsler Scales: as the verbal sub-quotient of the participants with ASD lied within the normative range, we expected an optimal reception of the verbal instructions and included all of them in the analysis. Participants with TD were recruited at the University of Trento; participants with ASD were recruited at the "Laboratory of Diagnosis, Observation, and Education" (ODFLab) of the University of Trento. Written informed consent was obtained from all the participants in accordance with the Declaration of Helsinki.

# Tools

#### Apparatus

We used a Tobii T120 eye-tracker (Tobii Technology, Stockholm), with a sampling rate of 60 Hz. The integrated monitor had a resolution of 1280<sup>∗</sup> 1024 and a size of 1700. The experiment was designed and run through the software Ogama (Vosskühler et al., 2008). For collecting the participant's answers, we used a Python script.

#### Stimuli

The stimuli consisted of 24 10-s videos, displaying 3 actors seated in front of a neutral wall. The central actor was always a female, while in half of the videos the actors on the sides were both males and the other half they were both females. The central model had only her back visible, and she pronounced a predefined sentence in Italian (i.e., "I will go home next Tuesday. I am going to University with the whole family."). When the central actor started to talk, the two models on the sides shifted their gazes either toward/away from the central model. The other two actors were facing-forward and one of them wore a pen on his/her shirt. The position of the facing-forward actors, the direction of the eyegaze toward and away from the central actor, and the position of the pen were counterbalanced across the experiment.

During each block, a 7-s instruction preceded the onset of the video and a 7-s answer screen followed the video. The total duration of one block (including instruction, video and answer screen) was 24 s. The three types of instruction were:


The correct answers consisted in indicating the side of the model wearing the pen on his/her body (VS condition) or that shifted his/her eye-gaze toward the central model (GR condition) by pressing the key A (left) and L (right). The A and the L keys were covered with a white tag.

Each question was repeated on the answer screen above two photographs of the actor, located at the sides of the screen. Each instruction was repeated 8 times in a randomized order, for a total of 24 blocks per participants. Each trial started with one of these instructions, for a total of 8 FV, 8 VS, and 8 GR conditions. For a graphical representation of the stimuli presentation, see **Figure 1**.

#### Procedure

The participant sat in from of the eye-tracker and the keyboard in a homogeneously well-lit room. The experimenter explained the calibration procedure and instructed the participants to follow the instructions before each video and to press one of the two specified keys to choose an answer when displaying the answer screen. The keys were selected to be widely apart (L and A, respectively, at the extreme right and left of the Italian keyboard). The keys were marked by a white label, highly contrasted with the black keyboard (see **Figure 1**). The participant was instructed to press the key corresponding to the position of the actor on the answer screen (right or left). After instructing the participant, the experimenter sat behind a curtain and monitored the participant's gaze.

Before starting the experiment, the participants performed two practicing blocks without recording eye movements.

TABLE 1 | Means and standard deviations of Age, SES and IQ (ND, neurodevelopment; SES, socio-economical score; IQ, intelligence quotient; ♀= female; N = number; M = mean; SD = standard deviation).


<sup>1</sup>Socio-economical score, calculated using the Four-Factor Index of Social Status (Hollingshead, 1975). The sample represented a medium status in the Italian population (Venuti and Senese, 2007).

Subsequently, the experimenter started the 5-points calibration procedure, consisting in a red ball moving between the edges and the center of the screen. The calibration was accepted when all the positions had been sampled (on average, no more than 2 attempts were needed for each participant). The proportion of first fixations landing on the face, the average fixation duration on the face and on the body, the proportional looking time on face, and the percentage of correct responses were calculated.

# Data Analysis

#### Preliminary Analysis

We pre-processed the data using the standard fixation filter of Ogama (distance threshold = 35 pixels, samples minimum value = 10). Total fixation durations were calculated within 6 predefined AOIs, (face and body of the models, central model and background), drawn on the stimuli and aggregated in two groups (faces and bodies). The following preliminary and main analysis were carried out in R (R Core Team, 2015).

The percentage of correct responses given with the keyboard over the total number of responses was calculated for each subject (accuracy hereafter). By comparing the accuracy between groups and conditions, we ensured that participants from both groups understood the task correctly and were able to deliver a correct response. In fact, Wilcoxon Tests did not show any significant difference in accuracy between participants with ASD and TD (general accuracy: ASD = 87.9%, standard deviation = 12.8, TD = 88.8%, standard deviation = 12.4, W = 224, p-value = 0.7. VS: ASD = 81.2%, standard deviation = 24.1, TD = 81.2%, standard deviation = 24.1, W = 230.5, p-value = 0.82. GR: ASD = 88.8%, standard deviation = 16.1, TD = 85.9%, standard deviation = 20.6, W = 225.5, p-value = 0.71). Furthermore, we analyzed a general measure that is negatively correlated with search efficiency, the number of fixations within the AOIs (Holmqvist et al., 2011). The number of fixations on the face across conditions did not differ between groups (FV: W = 178.5, p-value = 0.30; GR: W = 167, p-value = 0.18; VS: W = 150.5, p-value = 0.08) or within groups (ASD: GR vs. VS, W = 112.5, p-value = 0.01 – not resisting to Bonferroni correction -, FV vs. VS, W = 255, p-value = 0.49, GR vs. FV, W = 147, p-value = 0.15), suggesting that the level of processing difficulty of the gaze-information was similar across groups and tasks. However, the groups differed significantly for the number of fixations on body in the VS condition (W = 331.5, p-value = 0.005), that was higher than in the other conditions in the ASD group (VS vs. FV, W = 111.5, p-value = 0.017, VS vs. GR, W = 352, p-value = < 0.001).

The average data loss (as measured by the output "Percentage of Samples Out of the Screen" of the Ogama software) within the duration of the movies was very low, with an average of 0.01 (standard deviation = 0.1) in the TD group and 1.6 (standard deviation = 6.5) in the ASD group. The percentage of data loss was compared through the Wilcoxon Test and did not differ between groups (W = 271, p-value = 0.36). The data loss had inter-subject minimal variation, with a minimum z-score of −0.17 and a maximum z-score of 0.08 across the two groups.

#### Main Analysis

As the variables were not normally distributed, we performed all the analysis with non-parametric tests (Wilcoxon Signed Rank Tests). To test whether the faces would primarily attract the attention of both groups, we calculated the proportion of first fixations landing on the face (FF%). To calculate the FF%, we selected only those trials where the eye position was recorded on the AOI "Center" before the onset of the first gaze shift (see **Table 2**; for mean proportions of FF%, see **Table 3**). Participants who displayed less than 3 valid trials were excluded from the subsequent analysis, thus resulting in a final sample of 39 participants (16 from the ASD group, 23 from the TD group). We divided the total numbers of valid trials where the first fixation landed on the AOI "Face" by the total numbers of valid trials where the first fixation landed on any of the other AOIs, i.e., body, central model and background (FF% means and standard deviations are reported in **Tables 4**, **5**). We then compared the

TABLE 2 | Number of valid trials per condition for each participants' group (ND, neurodevelopment; ASD , autism spectrum disorder; TD, typical development; FV, free-viewing; VS, visual search; GR, gaze-reading; N, number; M, mean; SD, standard deviation).


TABLE 3 | Mean proportion and standard deviations of FF% by group and condition (ND, neurodevelopment; ASD, autism spectrum disorder; TD, typical development; FV, free-viewing; VS, visual search; GR, gaze-reading; N, number; M, mean; SD, standard deviation).


TABLE 4 | output of the mixed model analyzing the number of trials where the first fixation landed on the face in the two groups.


Estimates that are significantly different from 0 are marked with a (<sup>∗</sup> ). The estimates are expressed in the logit scale and can be converted to proportions with the formula exp(logit)/[1 + exp(logit)].

FF% to the probability of hitting the AOI "Face" by chance (1/total N of independent AOIs = 0.2) and performed group comparisons. A proportion significantly higher than chance indicate that a bias to shift the eye-movement from the fixation center to the face exist. An equal proportion in both groups may indicate that participants with and without ASD showed a similar bias to direct their first fixation to the face. Additionally, we compared the number of trials where the first fixation landed on the face compared to the number of trials where the first fixation landed on any other AOI (with a generalized linear mixed model).

For exploring the effect of task instructions, we examined two aggregated measures: the average fixation duration on the face and on the body (FD), and the proportional looking time on face (LT%, calculated as the Total Fixation Duration on the Face divided by the Total Fixation Duration on TABLE 5 | Mean FD and standard deviations in seconds on face and body by group and condition (ND, neurodevelopment; ASD, autism spectrum disorder; TD, typical development; FV, free-viewing; VS, visual search; GR, gaze-reading; N, number; M, mean; SD, standard deviation).


all the AOIs; means and standard deviations of LT% are reported in **Table 6**). We compared the FD on the body and the face, and the LT% on face between and within groups through Wilcoxon Tests. Both variables account for the adaptation of eye-movements to instructions (Holmqvist et al., 2011); furthermore, the proportional looking time accounts also for idiosyncratic scanning differences (Fu et al., 2012).

All the reported comparisons have been selected through Bonferroni Correction (p-value < 0.05/N of comparisons).

# RESULTS

# Proportion of First Looks From Center to the Face (FF%)

Multiple Wilcoxon tests revealed that FF% was above chance in all groups and conditions (FV: ASD: W = 115, p-value = 0.001, TD: W = 216, p-value = 0.009; VS: ASD: W = 108, p-value = 0.003; GR: ASD: W = 134, p-value < 0.001, TD: W = 272, p-value < 0.001). The result of TD participants in Condition 2 was significant but did not resist to Bonferroni Correction (p-value > 0.01). The groups did not differ in terms of FF% across conditions (FV: W = 199.5, p-value = 0.42; VS:

TABLE 6 | Mean LT% on face and standard deviations by group and condition (ND, neurodevelopment; ASD, autism Spectrum disorder; TD, typical development; FV, free-viewing; VS, visual search; GR, gaze-reading; N, number; M, mean; SD, standard deviation).


W = 209.5, p-value = 0.27; GR: W = 156, p-value = 0.43). The comparison between conditions within each group carried out with Kruskal–Wallis Test was not significant for the ASD group (Chi-squared = 1.2, p-value = 0.52); the same test turned out significant for the TD group (Chi-squared = 9.5, p-value = 0.008). Additional Wilcoxon tests revealed that the VS conditions significantly differed from both FV and GR conditions in TD with a lesser proportion of first fixations to the face (VS vs. GR: W = 33, p-value = 0.001; VS vs. FV: W = 5, p-value = 0.0001). The results concerning FF% are displayed in **Figure 2**.

# Number of Trials Where the First Fixation Landed on the Face

Our generalized linear model included group, condition and their interaction as predictors of the number of trials where the first fixation landed on the face. The model allowed for random intercepts for each subject, and no fixed intercept. The model showed that participants pertaining to both groups significantly fixated the face first, compared to other AOIs, in the majority of trials – around 80% (ASD: estimate = 1.41, standard error = 0.35, z-value = 3.98, p-value < 0.001; TD: estimate = 1.49, standard error = 0.31, z-value = 4.67, p-value < 0.001). The model also indicates that the probability of hitting the face was significantly lower in the VS condition. However, the interaction between the condition and the group were not significant (see **Tables 4**, **5** and **Figure 3** for further details), confirming that the groups did not differ across conditions.

# Average Fixation Duration (FD)

Fixation duration (FD) on Body differed significantly between the groups in the VS condition (W = 369, p-value = 0.001), with longer FD in the ASD group. FD on Face was significantly different between the groups in the GR condition (W = 388, p-value < 0.001), with longer FD in the ASD group. We found no significant correlations between the FD and the IQ level of the participants in both groups. The results are displayed in **Figure 4**.

# Proportional Looking Time on Face Compared to the Other AOIs (LT%)

The groups differed in terms of LT% across conditions (FV: W = 87, p-value < 0.001; VS: W = 36, p-value < 0.001; GR: W = 113, p-value = 0.002). Within-group comparisons showed that LT% significantly differed across conditions in the ASD group only (significant alpha-value < 0.01; ASD, FV vs. VS: W = 157, p-value = 0.002, FV vs. GR: ASD: W = 13, p-value < 0.001, VS vs. GR: ASD: W = 13, p-value < 0.001; TD, FV vs. VS: W = 34, p-value = 0.96, FV vs. GR: W = 9, p-value = 0.03, VS vs. GR: W = 8, p-value = 0.09), as shown in **Figure 5**.

# DISCUSSION

In the present study, we investigated the state of face-orienting and face-looking time in young adults with ASD using realistic stimuli and explicit task instructions. In sum, our results show that:


The first set of results is in contrast with the renowned evidence of people with ASD looking less at faces (Klin et al., 2002; Chawarska et al., 2010); this discrepancy might be explained by the fact that our stimuli represented a fairly simple situation (compared for instance to the scenes of the movie "Who is afraid of Virginia Woolf?" used by Klin et al., 2002). Furthermore, all the participants in the ASD group were high functioning and may have developed strategies or compensatory mechanism to obviate face-processing difficulty. Nonetheless, our result is in line with the evidence that face-orienting abilities are not always impaired in individuals with high-functioning ASD (Shah et al., 2013; Elsabbagh et al., 2014). Considering that the face-orienting bias is (1) documented in infants at risk of ASD (Elsabbagh et al., 2013b), (2) heterogeneously impaired in children with ASD (Chawarska et al., 2013), and (3) correlates with face-processing abilities (de Klerk et al., 2014), we may conclude from our result that face-orienting may either deteriorate in certain subgroups of children with ASD or endure a developmental delay, but it possibly recovers and/or establish compensatory mechanisms in adulthood (Belmonte and Yurgelun-Todd, 2003; New et al., 2010; Sheth et al., 2010). A putative mechanism might be the progressive specialization

FIGURE 4 | Average FD within the two groups of AOIs, body and face, across groups and conditions (FD, fixation duration; ASD, autism spectrum disorder; TD, typical development; <sup>∗</sup> , significant comparison).

of the face-sensitive areas, whose specialization is not fully accounted by an innate preparedness (Johnson, 2011). Indeed, according to the Interactive Specialization view (Johnson, 2011), accumulating experience with faces is crucial for the refinement of automatic face-orienting mechanisms. In ASD, while the automatic ability to orient to faces is intact, several other symptoms can impede the massive input of face experience and prevent the development of the deputed neural areas and, hence, the development of refined face-processing skills (Johnson et al., 2005). However, additional cognitive resources, an improved management of symptoms in high functioning individuals, and/or ongoing intensive therapy, can counterbalance the initial lack of experience and start a progressive recovery of faceprocessing skills.

The second set of results is in line with the hypothesis that difficulties in face encoding and in gaze-processing correlates with a prolonged fixation duration on the face in individuals with ASD, as it has been previously reported (Elsabbagh et al., 2013a). However, we did not observe any difference in the accuracy of the responses. Furthermore, the longer fixation duration was not limited to the faces in the GR condition, but involved the body in the VS condition too, thus excluding a face-processing difficulty. Notably, the significant differences were limited to those AOIs that were relevant to the task (Body in VS condition, Face in GR condition).

The longer FD on task-relevant areas and the variable amount of face LT% depending on the instruction could be explained with participants with ASD taking more time to correctly elaborate the stimulus and to extract from the stimuli the task-relevant information or sticking more to the task and being less distracted by the other available AOIs. In the first case, participants with ASD would perform a higher number of longer fixations because they need more time to elaborate the information contained in the stimulus to reach the same level of accuracy of participants with TD. In the second case, participants with ASD would perform longer but an equal number of fixations because they have difficulty to disengage from the AOI brought into focus by the instruction. With regard to the face, we cannot overrule one of these two explanations, as we did not find increased number of fixations; however, the AOI drawn on the face is relatively small, and all the information conveyed by its elements can be explored without increasing the number of eye-movements. On the contrary, the result regarding the body suggests that individuals with ASD sampled a higher number of positions for longer periods of time for succeeding the visual search (see fixation map in **Figure 6**). It is also noticeable that participants with ASD shifted their first fixation on the face anyway in the VS condition, while participants with TD did not prioritize the face in this condition (i.e., FF% is not significantly above chance) – suggesting that participants with ASD might have had less task efficiency. One of the implications of this results may be that, when their attention is explicitly drawn to objects, persons with ASD may take longer for elaborating the stimuli and end up disregarding other visual items – faces and their components included, such as the eyes. This fact may be problematic for people with ASD, as extrinsic events often disturb social interactions and social partners highlight external objects with gestures and utterances. Once an object captivates their attention, a particularly difficult task may arise for people with ASD, as they need time to elaborate and flounder to shift their attention back

to the face and the eyes, with all that this implies. Moreover, this peculiar attentive style may be reconnected to the description of the defective "zooming-out" of the attentional focus and the overly circumscribed orienting tendencies in individuals with ASD (Robertson et al., 2013; Ronconi et al., 2013).

# CONCLUSION

In the current study, we did not record any difference in face-orienting in the presence of explicit instructions and with dynamical, realistic video clips of a social interaction: this result suggests that the orientation to faces and eyes might undergo compensatory mechanisms that are acquired and refined thanks to the accumulation of experience with human social partners. On the other hand, the average looking time was prolonged on task-relevant areas not limited to the face and the eyes in all the experimental conditions – observation that could be explained by general processing difficulties in young adults with ASD. The major implication of this interpretation is that this alteration of a domain-general difficulty might have even more massive cascade effects compared to the alteration of a domain-specific function, such as gaze-processing. In fact, a child that that is not able to "zoom-out" might miss interaction opportunity, overly focus on complex environmental stimuli, and even become upset if the prolonged focus of attention involves distressing stimuli. This picture overlaps with existing reports of the orienting tendencies of individuals with ASD, i.e., less responsive and prone to distress (Zwaigenbaum et al., 2005). Future research could investigate the transversal effects of these defects. We believe that our result (of task-dependent fixation with two tasks that involved completely different goals) might constitute a first example of this case.

# ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the Ethics Committee of the University of Trento with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Ethics Committee of the University of Trento.

# AUTHOR CONTRIBUTIONS

fpsyg-09-02629 December 14, 2018 Time: 14:38 # 10

TDB designed the experiments, recruited the participants and collected the data, carried out the statistical analysis and wrote the first draft of the manuscript, under the supervision of

# REFERENCES


AB and PV. NM and AB helped recruiting the participants and collecting the data. TDB, NM, AB, and PV actively discussed the results and outlined the interpretation together. TDB finalized the draft that was later reviewed by NM, AB, and PV.

Hollingshead, A. B. (1975). Four Factor Index of Social Status. New Haven, CT: Privately Printed.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Del Bianco, Mazzoni, Bentenuto and Venuti. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Direct Gaze Partially Overcomes Hemispatial Neglect and Captures Spatial Attention

Miguel Leal Rato<sup>1</sup> , Inês Mares<sup>2</sup> , Diana Aguiar de Sousa<sup>3</sup> , Atsushi Senju<sup>2</sup> and Isabel Pavão Martins3,4 \*

<sup>1</sup> Hospital Prof. Doutor Fernando Fonseca, Amadora, Portugal, <sup>2</sup> Centre for Brain and Cognitive Development, Department of Psychological Sciences, Birkbeck, University of London, London, United Kingdom, <sup>3</sup> Department of Neurosciences and Mental Health, Neurology, Hospital de Santa Maria, University of Lisbon, Lisbon, Portugal, <sup>4</sup> Language Research Laboratory, Instituto de Medicina Molecular, University of Lisbon, Lisbon, Portugal

Direct gaze has been shown to be a particularly important social cue, being preferentially processed even when unconsciously perceived. Results from several visual search tasks further suggest that direct gaze modulates attention, showing a faster orientation to faces perceived as looking toward us. The present study aimed to analyze putative modulation of spatial attention by eye gaze direction in patients with unilateral neglect. Eight right hemisphere stroke patients with neglect performed a target cancelation paradigm. Patients were instructed to cross all open-eyed pictures amidst closed eyed distractors. Target images were either in direct or averted gaze. Participants performed significantly better when observing targets with direct gaze supporting the hypothesis that this gaze direction captures attention. These findings further suggest that perception of direct gaze is able to diminish the visuospatial impairment seen in neglect patients.

#### Edited by:

Paola Ricciardelli, Università degli Studi Milano-Bicocca, Italy

#### Reviewed by:

Giuseppe Vallar, Università degli Studi Milano-Bicocca, Italy Colin W. G. Clifford, The University of New South Wales, Australia

#### \*Correspondence:

Isabel Pavão Martins ipavaomartins@gmail.com

Received: 15 May 2018 Accepted: 17 December 2018 Published: 15 January 2019

#### Citation:

Leal Rato M, Mares I, Aguiar de Sousa D, Senju A and Martins IP (2019) Direct Gaze Partially Overcomes Hemispatial Neglect and Captures Spatial Attention. Front. Psychol. 9:2702. doi: 10.3389/fpsyg.2018.02702 Keywords: hemispatial neglect, direct gaze, averted gaze, visuospatial attention, unconscious perception

# INTRODUCTION

Detecting salient stimuli in the environment is crucial for survival and adaptation, allowing an individual to orient to possible sources of threat or to relevant objects. Saliency refers to a stimulus' distinctive sensorial properties or behavioral importance in relation to others. In humans, direct gaze could be a social cue of particular saliency due to its significance in communication and nonverbal social interaction (von Grünau and Anston, 1995).

Direct gaze has been shown to modulate several concurrent cognitive processes such as face encoding and retrieval (Conty and Grèzes, 2012), emotion processing (for a review see Rigato and Farroni, 2013), gender discrimination (Macrae et al., 2002; Vuilleumier et al., 2005), retrieval of semantic information (Macrae et al., 2002) and pro-social behavior (Bateson et al., 2006). Furthermore, direct gaze is preferentially detected in visual search tasks when compared with other gaze directions (von Grünau and Anston, 1995; Senju et al., 2005), a preference that occurs even in express saccades as observed in reflexive orienting (Mares et al., 2016). This preference for direct eye gaze is present from birth as observed in newborn babies (Farroni et al., 2002). Privileged perception of direct gaze occurs even under unconscious stimulus presentation, as shown in a paradigm using a form of binocular rivalry, continuous flash suppression

(CFS; Stein et al., 2011). In this paradigm, direct gaze was shown to break suppression faster than averted gaze, with a concomitant modulation of neural responses suggesting that faces with direct gaze are processed effectively under an unconscious condition (Yokoyama et al., 2013). Furthermore, enhanced amygdala activation for direct gaze was found in a patient with complete cortical blindness (Burra et al., 2013). This line of research suggests that direct gaze can be partially processed even when not consciously perceived. In this case, some residual processing of direct gaze could still modulate orienting even in patients with damaged attention mechanisms, as occurs in hemispatial neglect (HSN).

HSN commonly occurs following right hemisphere lesions and is characterized by contralesional spatial defects (such as deficits in saliency coding, spatial attention and visuospatial short-term memory), alongside with non-spatial defects (reorienting, target detection, and arousal/vigilance deficits; Vallar, 1998). The classic, viewer-centered (egocentric), neglect (Medina et al., 2009) occurs mainly after asymmetric or focal brain damage, most frequently caused by stroke in the right hemisphere (Hillis, 2013), particularly in the inferior parietal lobule, superior temporal gyrus and/or inferior frontal gyrus, leading to deficits on the left side of space (Corbetta and Shulman, 2011; Corbetta, 2014). A study by Azouvi et al. (2002) has shown that about 85% of subacute right hemisphere stroke patients presented some degree of unilateral neglect, which was considered as clinically significant (moderate to severe) in 36.2% of cases. The presence of HSN was task dependent, as tasks including a strong visual component were the most sensitive to the spatial defect of HSN, and the automatic rightward orientation bias "(i.e., the spontaneous tendency to orient toward the right hemifield)" seemed to be the best indicator of unilateral neglect.

The neural mechanisms underlying HSN spatial deficit can be dynamically modulated by either endogenous or exogenous signals, creating a complex interaction between attention, movement and arousal (Corbetta, 2014; Duclos et al., 2014). Interestingly, stimuli do not need to be consciously perceived by the HSN patient, who can use the information provided by peripheral cues to orient their attention toward the neglected space, even without conscious awareness of the presented stimuli (Wansard et al., 2015). This has been often observed for instance with emotional stimuli (see Domínguez-Borràs et al., 2012 for a review). Gaze in particular has been shown to be able to modulate attention in neglect patients when presented centrally (Maravita et al., 2007), decreasing peripheral target detection on the contralateral hemispace.

Despite this, no study to our knowledge has shown a modulatory effect of direct gaze on attention throughout the visual space in patients with HSN. The presence of such an effect would be supported by behavioral studies that have shown that perception of direct gaze still occurs when attention is diminished in typical participants (Yokoyama et al., 2014). Thus, we hypothesize that direct gaze can be processed even when individuals are unable to direct their attention toward it. To test this hypothesis, we investigated direct gaze detection in patients with HSN. We predict that direct gaze processing can take place even when the HSN patients are unable to attend to it, which should result in a better performance in detecting direct gaze than averted gaze.

# MATERIALS AND METHODS

# Patients

Nineteen patients (7 females; age, range: 41–84, M = 64.32 ± 12.96 years old; education, range: 4–13, M = 6.68 ± 3.49 years) with right hemisphere acute stroke admitted to a Stroke Unit were included for the initial assessment. Patients were excluded if they presented with significantly altered mental state as assessed by clinical evaluation, any major medical comorbidity, a significant speech or comprehension impairment or if they were in a state of non-cooperation. All patients signed an informed consent previously to participation and this study was reviewed and given authorization to start by the local Ethics Committee (Centro Hospitalar Lisboa Norte – Faculdade de Medicina de Lisboa ).

# HSN Assessment

All patients were assessed with a bedside HSN test battery including tasks of star cancelation, line crossing, figure copying, and menu reading adapted from the Behavioral Inattention Test (BIT, Wilson et al., 1987), presented sequentially. HSN was defined as any left skewed visuospatial defect identified in either the star cancelation or line crossing tasks.

# Experimental Procedure

Patients with HSN performed a cancelation task consisting of an array with images of open and closed eyes. Patients were asked to search and mark by crossing over the stimulus all open-eyed stimuli (either in direct or averted gaze) amongst closed-eyed distractors. Fourteen targets and distractors were distributed across a standard white horizontal 21 × 29.7 cm (A4) sheet of paper, centered relative to each patient's body midline, in an unstructured pseudo-random array, in order to increase the sensitivity of the tasks (Azouvi et al., 2002). All faces were laterally oriented to the right or left, to avoid low level visual confounds such as face symmetry (George et al., 2001). Each participant performed four randomized trials corresponding to four different conditions varying target gaze direction (direct and averted gaze) and overall face orientation (right and left).

The sheets were randomized by an outside person by printing in random order and manually shuffling the sheets before participant inclusion. Allocation was secured by keeping each set of sheets in an opaque envelope until bedside examination.

Meaning, a patient would be shown a random sequence of 4 trials, one with direct gaze and right face orientation, another with direct gaze and left face orientation and the corresponding for averted gaze (**Figure 1**). Participants had no time limit to complete each task.

Patients were free to move their heads in relation to the presented sheets. Although this could possibly be a confounding factor (as the side of the sheet that is in the left hemispace will no

longer necessarily be in the left hemifield), we know that patients with neglect tend to avoid exploring the left hemispace, due to an ipsilesional gaze bias (Fruhmann Berger et al., 2008). Thus we expect an overall neglect of the left hemispace irrespective of hemifield.

Trials with oriented faces to the right and left were merged for each gaze condition. Accuracy was assessed as the total number of open-eyed stimuli identified, with a larger number indicating a smaller spatial defect.

#### Statistical Analysis

To analyze the effect of gaze direction across the visual field, targets were divided into four vertical areas of equal size. Two way repeated measures analysis of variance (ANOVA) were performed, with gaze direction (direct and averted) and position (four positions from the left to the right) as factors for hits, false alarms and d'. d' is a measure of accuracy, with hits defined as correctly crossed open eyes, and false alarms defined as incorrectly crossed closed eyes. Hit and false alarm rates of one were corrected by calculating 1−1/(2∗number of possible hits/false alarms) and zero by calculating 1/(2<sup>∗</sup> number of possible hits/false alarms). Additional post hoc analyses (twotailed t-tests) were performed as required. A possible effect of gaze cueing was also analyzed for targets with averted gaze using a two-tailed t-test. Effect sizes for dependent t-tests were calculated using the formula proposed in Eq. (3) of Dunlap et al. (1996).

# RESULTS

Eight of the 19 patients assessed were found to have HSN (3 females; age, range: 53–80, M = 65.75 ± 8.61 years old; education, range: 4–12, M = 5.37 ± 2.77 years). There was no significant difference between non-HSN and HSN patients regarding age, gender distribution, years of education, or time from stroke until assessment.

Demographic characteristics and clinical data of the studied HSN population are shown in **Table 1** and CT scan results in **Figure 2**.

A positive correlation was found between the scores on the line cancelation task, used to assess HSN, and the number of targets identified in direct and averted gaze (r = 0.84, n = 8, p = 0.009 and r = 0.88, n = 8, p = 0.004, respectively). Furthermore, the difference between accuracy in direct and averted gaze correlated negatively with the time from stroke until assessment (r = −0.79, n = 8, p = 0.019).

Hit rates were higher for direct gaze (M = 0.52 ± 0.24) than for averted gaze [M = 0.45 ± 0.25; F(3,21) = 8.61, p = 0.022, n 2 <sup>p</sup> = 0.55]. A main effect of position was also significant

of assessment.

fpsyg-09-02702 January 12, 2019 Time: 17:4 # 4


TABLE 1 | Demographic characteristics and clinical data of the studied neglect population and main results (I = Ischaemic, H = Hemorrhagic, AG = Averted Gaze, DG = Direct Gaze).

[F(3,21) = 11.44, p < 0.001, n 2 <sup>p</sup> = 0.62], with decreasing hit rates toward the left hemispace (p < 0.043), with exception of the two most leftward areas which did not differ (p = 0.028; from right to left, 1st, M = 0.82 ± 0.16; 2nd, M = 0.50 ± 0.37; 3rd, M = 0.38 ± 0.38; 4th, M = 0.23 ± 0.24). As before no interaction between stimuli position and gaze direction was found [F(3,21) = 1.45, p = 0.26,n 2 <sup>p</sup> = 0.17]. **Figures 3A,B** show the hit rate and the false alarm rate for averted and direct gaze, respectively. A version of this graph (**Supplementary Figure 1**) showing the hit rate for averted and direct gaze divided by left or right head orientation of the stimuli is available as **Supplementary Material**.

Regarding false-alarms, there was an interaction between gaze direction and stimuli position [F(3,21) = 5.33, p = 0.007,n 2 <sup>p</sup> = 0.43]. There was a trend for more false alarms for averted gaze trials (M = 0.30 ± 0.23) compared with direct gaze (M = 0.16 ± 0.12) in the most rightward position [t(7) = 2.32, p = 0.05, Cohen's d = 0.59], with no difference for the remaining positions (p > 0.17). Note that false alarms were not higher for direct than for averted gaze, which shows that the higher hit rates for direct gaze condition is not based on overall response bias (i.e., to cross out any stimuli).

For accuracy, as assessed with d', we observed a significant main effect of position [F(3,21) = 6.05, p = 0.004, n 2 <sup>p</sup> = 0.464], with a difference between the most extreme right area (M = 1.92 ± 1.05), and marginally the second (p = 0.07, M = 1.18 ± 1.45), and significantly the remaining most leftward areas (p < 0.04; M = 1.00 ± 1.38, M = 0.371 ± 0.81, respectively) (see **Figure 3C** ). We also observed a trend for

larger accuracy for direct gaze (M = 1.26 ± 1.04), compared to averted gaze (M = 0.98 ± 1.03), which did not reach significance [F(1,7) = 4.39, p = 0.074, n 2 <sup>p</sup> = 0.39]. Interaction between position and gaze direction was not significant either [F(3,21) = 0.55, p = 0.65,n 2 <sup>p</sup> = 0.07].

No effect of gaze cueing toward the neglected hemispace was observed for d' [t(7) = 0.77, p = 0.47, Cohen's d = 0.20], with no differences in accuracy found between faces in averted gaze toward the left (M = 1.29 ± 1.18%) or the right (M = 1.02 ± 1.44%). Similarly, no difference were observed when using hit rates [t(7) = −0.18, p = 0.86, Cohen's d = −0.03]. Main results are summarized in **Table 1**.

# DISCUSSION

The current study tested the hypothesis that residual processing of direct gaze is sufficient to lead to an increased detection of targets that would otherwise be unattended. The current findings support this hypothesis as participants detected a higher proportion of faces with direct gaze than faces with averted gaze. This result reflects differences in attentional capture.

This replicates previous studies that have shown that residual abilities in sensorial processing can modulate attention toward the contralesional stimuli if the perceived stimuli are salient. Socially or biologically relevant stimuli such as faces have been shown to be partially analyzed without attention (Vuilleumier, 2000). Using fMRI and ERP methodologies, unconscious face perception in HSN patients has further been shown to elicit V1 activations and N170 face specific components suggesting that visual processing still occurs in the absence of awareness (Vuilleumier et al., 2001). Furthermore, emotional expressions have been found to modulate spatial attention and orienting in HSN patients (for a review see Domínguez-Borràs et al., 2012). Direct gaze in particular has been shown to be partially processed in the absence of conscious perception, both in a patient with blindsight (Burra et al., 2013) and in paradigms (e.g., CFS) that use binocular rivalry to present stimuli unconsciously (Stein et al., 2011; Yokoyama et al., 2013).

Nonetheless, previous studies failed to demonstrate an advantageous effect of direct gaze in HSN. Vuilleumier (2002) tested the effect of gaze direction on attention in HSN patients by using an extinction paradigm, where patients' ability to detect contralesional stimuli is diminished by a simultaneous display on the ipsilesional field. Vuilleumier's (2002) study did not find an attention capture effect of direct gaze when displayed on the left visual field. On the other hand, a congruent gaze direction, meaning, gaze directed toward the left, was shown to cue attention to the contralesional spatial hemifield, leading the authors to conclude that gaze information is not extracted unconsciously or "preattentively" in HSN. Results from the present study seem to oppose both these findings with a clear attention capture of direct gaze occurring throughout the visual space, and in particular in the left hemispace, in the absence of a possible cueing effect of faces with averted gaze toward the left. The use of an extinction paradigm in Vuilleumier's (2002) study could account for the different results. It is possible that while a gaze cueing effect in HSN might be specific to the extinction symptoms, a cancelation task might be more sensitive to evaluate the effect of direct gaze in the broader neglect deficit.

Our study is the first to demonstrate that direct gaze enhances target detection in the HSN patients, diminishing the visuospatial impairment associated with HSN. These unconscious effects of direct gaze have been proposed to result from a 'fast' pathway, mediated by subcortical structures including the superior colliculus, pulvinar, and amygdala (Senju and Johnson, 2009).

Individuals with lesions involving the medial corticosubcortical networks may suffer from more severe HSN symptoms and are more likely to develop chronic spatial neglect. Chen et al. (2012) found that this might explain the degree of efficacy of HSN treatments, such as prism adaptation treatment. These medial temporal regions might even provide critical support for neural or chemical plasticity in spontaneous recovery. Moreover, due to the importance of the pulvinar to certain core mechanisms of attention, for instance the unconscious processing of salient stimuli such as faces (Troiani and Schultz, 2013), it may play an important role in this syndrome and its integrity might be related to a better prognosis and be predictive of treatment success.

The presence of a "preattentive" effect of eye contact on patients with HSN might be mediated by this pathway, in which case a lesion in one of its core structures might abolish such an effect. Future studies analyzing the effect of direct gaze in HSN patients with neglect due to pulvinar damage could clarify the role of this pathway in gaze processing.

As for limitations, we did not include a control group, as we expected a task ceiling effect in participants without neglect. Nevertheless, we recognize that this is a potential flaw in design. It would need to be addressed in a future study, ideally including a control group without neglect, but with whom a cancelation task will measure a non-ceiling and meaningful effect. Additionally, as patients did not have a time limit to complete our tasks, latency or speed of stimuli completion was not evaluated. Taking into account the small sample size, future studies are needed to replicate these preliminary findings in a larger sample of patients with neglect.

# CONCLUSION

The current study suggests that direct gaze is processed preattentively, helping to partially overcome spatial deficits in HSN. Furthermore, it provides converging evidence of the saliency of direct gaze when compared with other gaze directions by using a common organic lesion paradigm. The benefit of direct gaze (putatively due to its inherent saliency) was not specific for the left hemifield. Further studies will be beneficial to understand the underlying neural structures of the effect of direct gaze on visuospatial attention or how it can improve the management of chronic HSN patients.

# ETHICS STATEMENT

fpsyg-09-02702 January 12, 2019 Time: 17:4 # 6

The project and its protocol were approved by the Ethics Committee of the Hospital de Santa Maria, Centro Hospitalar Lisboa Norte. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

# AUTHOR CONTRIBUTIONS

MLR and IM revised the literature. MLR, IM, and IPM conceived and designed the experiments. MLR and DAdS collected the data. MLR, IM, and AS performed the analysis. MLR and IM wrote a first draft of the manuscript that was revised by DAdS, AS, and IPM. All authors read and approved the final manuscript.

# REFERENCES


# FUNDING

The project was supported by Programa Educação pela Ciência, GAPIC/FMUL University of Lisbon (20140014). IM was supported by Fundação para a Ciência e a Tecnologia (SFRH/BD/84737/2012) and AS was supported by a United Kingdom Medical Research Council Career Development Award (G1100252).

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2018.02702/full#supplementary-material


study. Proc. Natl. Acad. Sci. U.S.A. 13, 3495–3500. doi: 10.1073/pnas.0514 36898


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer GV and handling Editor declared their shared affiliation at the time of review.

Copyright © 2019 Leal Rato, Mares, Aguiar de Sousa, Senju and Martins. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Verbal Descriptions of Cue Direction Affect Object Desirability

Jason Tipples<sup>1</sup> \*, Mike Dodd<sup>2</sup> , Jordan Grubaugh<sup>3</sup> and Alan Kingstone<sup>4</sup>

<sup>1</sup> School of Social Sciences, Psychology, Leeds Beckett University, Leeds, United Kingdom, <sup>2</sup> Department of Psychology, University of Nebraska – Lincoln, Lincoln, NE, United States, <sup>3</sup> National Institutes of Health, Bethesda, MD, United States, <sup>4</sup> Department of Psychology, University of British Columbia, Vancouver, BC, Canada

Approach-avoidance behaviors are observed across a broad range of species. For humans, we tend move toward things we like, and away from things we dislike. Previous research tested whether repeatedly shifting visuo-spatial attention toward an object in response to eye gaze cues can increase liking for that object. Here, we tested whether a gaze-liking effect can occur for verbal descriptions of looking behavior without shifts of attention. Also, we tested the gaze specificity hypothesis – that the liking effect is specific to gaze cues – by comparing the effect of different types of cue (pointing gestures and arrow cues). In Experiment 1, participants (N = 205) were split into 5 groups according to the type of cue that was described as directed either toward or away from an object. The results show that (1) attention is not necessary; the liking effect was recorded for verbal descriptions of looking, (2) the effect also occurs for descriptions of pointing and arrows, and (3) the liking effect is enhanced for gaze cues compared to arrows, consistent with the gaze specificity hypothesis. Results from a further experiment suggest that the effect is not due to demand compliance. We conclude that the gaze-liking effect occurs for verbal descriptions of eye gaze. Indeed, because our method bypasses altogether the use of visual cues, objects, and shifts in visual selective attention, our paradigm appears to be more sensitive at tapping into the fundamental approach-avoidance response that mediate the implicit liking effect. As such, it offers new opportunities for research investigations in the future.

Keywords: gaze, arrows, liking, attention, cue

# INTRODUCTION

Approach-avoidance reactions are a fundamental aspect of human behavior – we move away from things we dislike and toward things we like. Because people have the capacity to infer mental states from other people, it follows that that people might use other people's approach-avoidance behavior as a basis for their own decisions and preferences. If I see you move toward an object, then that may indicate that I will also like that object. Similarly, if I see your eyes gaze toward an object then I might infer that the object is desirable.

To study the effects of gaze direction on object liking one innovative study (Bayliss et al., 2006) used an implicit learning task, in which shifts of attention were expected to increase object desirability. During the task, participants were asked to classify an object that appeared to the left or right of a face. The objects were presented in different colors. Across trials, the face looked consistently either toward or away from one of the colored versions of the object. In a final block of trials, participants both classified the object and rated the extent to which they liked the object. The key finding was increased liking for gazed-at objects – people rated objects as more likeable if they saw people gaze repeatedly toward rather than away from the objects.

#### Edited by:

Paola Ricciardelli, University of Milano-Bicocca, Italy

#### Reviewed by:

Christopher Striemer, MacEwan University, Canada Pablo De Gracia, Midwestern University, United States

#### \*Correspondence:

Jason Tipples w.tipples@leedsbeckett.ac.uk

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 02 November 2018 Accepted: 18 February 2019 Published: 11 March 2019

#### Citation:

Tipples J, Dodd M, Grubaugh J and Kingstone A (2019) Verbal Descriptions of Cue Direction Affect Object Desirability. Front. Psychol. 10:471. doi: 10.3389/fpsyg.2019.00471

**75**

This effect appeared to reflect an implicit process because participants did not report being aware that gaze direction had been deliberately manipulated to look toward certain objects, and away from other objects. Support for the idea that people had shifted attention in response to the gaze cues came from analyses of reaction times – reaction times were faster when the eyes appeared to look toward the location of a target (congruent trials) compared to away from the target (incongruent trials). Finally, and in keeping with the idea of a specialized social perception process, the effect of cue direction on liking did not occur when the gaze cues were replaced by arrows (Bayliss et al; Experiment 2). Other studies have extended this work by, for example, varying attributes of the face that are relevant to social interaction including expression (Bayliss et al., 2007), trustworthiness (King et al., 2011; Treinen et al., 2012) and attractiveness (Strick et al., 2008).

Despite the intuitive plausibility of the gaze-liking effect, a recent replication attempt (Tipples and Pecchinenda, 2018) of the gaze-liking effect indicated that the effect is much smaller (d<sup>z</sup> = 0.02) than originally thought (d<sup>z</sup> = 0.94). One possible reason for the small effect size is that the implicit learning task used in previous research does not directly assess the question of interest "Do participants prefer objects that are looked at by other people?" Instead, the task requires (implicit) learning of a relationship between gaze direction and objects, via shifts in attention. Some participants may fail to learn the gaze-object relationship – perhaps because they do not shift and maintain their attention to the object on some trials – and consequently, liking does not increase for gazed-at objects.

Our hypothesis for the current research is that the method used to increase liking – learning the gaze-object relationship via shifts in attention – is not necessary for gaze and other cues to affect liking for objects. Instead, all that is required is to present cue information and ask for ratings of likeability. We tested this idea across 5 groups of participants in which we varied the type of cue information. To test for a relation between cue direction and object liking we used verbal descriptions of gaze and other types of cue information. Our reason for using verbal descriptions was that mental states can be described by verbal communication (for a review see; Saxe, 2006) and language is related the development of mental state attribution (Dunn and Brophy, 2005). Our second hypothesis relates to the type of cue used to direct attention. Following previous research, we tested the gaze specificity hypothesis; that the gaze liking effect is unique to eye gaze cues. Specifically, cue type was varied between participants by allocating participants to one of five different groups. These are described below in detail.

Participants in the first group (LOOK – IMAGE) were asked to rate images of household objects that were paired with verbal descriptions of actors gazing toward ("Michael looked toward") or away from ("Michael looked away from") an image of an object. For the second group, (LOOK – WORD), the actors were described gazing toward or away from a verbal description of the object (e.g., "a screwdriver") rather than a picture of the object. For the third group (POINT) the word "pointed" replaced the word "looked" ("Michael pointed away from"). A fourth group (ARROW) received verbal descriptions of an arrow pointing toward ("An arrow pointed toward") or away from ("An arrow pointed away from") the objects. Given the findings of previous research (Bayliss et al., 2006) we do not expect arrow cues to increase liking but rather expect the liking effect to be restricted to either gaze cues (the LOOK – IMAGE and LOOK – WORD groups). To test for the overall effect of presenting a directional cue, a further control condition (WORD – ONLY) was included in which any mention of the cue was removed and participants were presented with either the word "toward" or "away." Finally, in Experiment 2 we tested for possible demand compliance.

# EXPERIMENT 1

## Materials and Methods Participants

Participants were 205 students (125 female, 80 male) from the University of British Columbia. The mean age (and standard deviation) of the male and female participants in each experimental group are displayed in **Table 1**. The experiment was carried out in accordance with the ethical standards of the American Psychological Association and with the 1964 Helsinki declaration and its later amendments. The study was approved by the Psychology Ethics Committee at University of British Columbia, Vancouver. Written, informed consent was obtained from all participants in the study.

#### Stimuli and Procedure

For the first LOOK – IMAGE group, the stimuli consisted of the images of 16 objects typically found in a garage (e.g., pliers and rake) and 16 objects typically found in a kitchen (e.g., electric whisk and saucepan). For the remaining groups (LOOK –WORD, POINT, ARROW, and WORD ONLY) words replaced the images used in the LOOK – IMAGE group. The images of objects were selected by the first author on the basis that they were easily recognizable as objects that might typically be found in either the garage or kitchen. For all subsequent conditions (LOOK – WORD, POINT, ARROW, and WORD – ONLY), the images were replaced with verbal descriptions of the objects (e.g., "Pliers"). All objects or verbal descriptions of objects were rated on a single sheet of paper (see **Supplementary Material** for the sheets used for the LOOK – IMAGE and LOOK – WORD groups). On one

TABLE 1 | The number (N), mean (M) and standard deviation (SD) of the age of male and female participants as a function of group (LOOK – IMAGE, LOOK – WORD, POINT, ARROW, and WORD ONLY).




∗ In the LOOK – IMAGE experiment participants rated pictures of objects. All other groups read descriptions of objects.

side of the paper a person was described as looking toward the object (e.g., "Michael looked toward the"). On the other side the person was described as looking away from the same object (e.g., "Michael looked away from the"). The text appeared above the object name. On each side, object names appeared in a fixed location in one of four columns. Each column contained one specific type of object. From left to right, the columns contained descriptions garage objects followed by 2 columns of kitchen objects and finally, the remaining 8 garage objects.

#### Procedure

Participants were randomly allocated to one of five groups (LOOK – IMAGE, LOOK –WORD, POINT, ARROW, and WORD ONLY). Testing took place in large classroom in groups of approximately 20–30 participants. The order in which a specific side of the sheet of paper was completed was counterbalanced across participants. For example, half of the participants in the LOOK group received the description of a person looking toward a specific object first, followed by a description of the same person looking away from the same objects (on the reverse of the sheet) whereas the remaining participants received the description in the reverse order. Participants were asked to rate the degree to which they liked the objects using the following 9-point scale: 1, do not like at all; 2, 3, 4, 5, 6, 7, 8, 9, like very much.

Participants were informed that there were no correct or incorrect answers. The scale appeared beneath each object name. In the POINT condition, the word "looked" was replaced by the word "pointed." In the ARROW group the words "an arrow" replaced the names given to the actors and the word "pointed" was also used (e.g., "an arrow pointed toward the"). Finally, in the WORD ONLY condition, the description of the person or arrow looking or pointing was removed – participants received either the word "toward" or "away" above the object name.

### Results and Discussion

The mean liking ratings and standard deviations for each combination of cue direction and group are shown in **Table 2**. The mean liking ratings were analyzed in a cue direction (toward, away) X group (LOOK – IMAGE, LOOK – WORD, POINT, ARROW, and WORD ONLY) mixed ANOVA with cue direction as the within subjects variable. There was a main effect of cue direction, F(1,203) = 47.60, p < 0.0001 and a cue direction X group interaction, F(4,203) = 4.74, p < 0.005. The main effect of group was not significant, F(4,203) = 0.55, p = 0.70. Our prediction was that the cueing effect would be restricted to descriptions of eye gaze behavior (the LOOK groups) and therefore, we analyzed the interaction by testing for the simple main effect of cue direction (toward vs. away) for each group separately. Mean liking ratings for objects were higher in the toward condition compared to the away condition for the LOOK – IMAGE, F(1,43) = 15.32, p < 0.0001, LOOK – WORD, F(1,43) = 19.05, p < 0.0001, POINT, F(1,38) = 14.05, p < 0.0001 and ARROW group, F(1,43) = 5.95, p < 0.05 but not the WORD ONLY group, F(1,36) = 0.33, p = 0.57.

In addition, as can be seen in **Figure 1**, there were relative differences in the magnitude of the cue liking effect (mean liking ratings in the toward condition minus mean liking ratings in the away condition) as a function of group. Specifically, post hoc analyses of the cue liking effect (mean liking ratings toward minus mean liking ratings away) using the Tukey-Kramer adjustment for multiple pairwise comparisons showed that the magnitude of the liking effect was higher in the LOOK – IMAGE condition compared to the WORD ONLY group [difference = 0.71; 95% CIadjusted (0.09–1.34), padjusted = 0.015] and also, higher in the LOOK – WORD group compared to both the ARROW [difference = 0.61; 95% CIadjusted (0.01–1.21), padjusted = 0.041] and WORD ONLY [difference = 0.84; 95% CIadjusted (0.21–1.46), padjusted = 0.002] groups.

# EXPERIMENT 2

We have shown that a variety of cues affect object liking, and moreover, the effect occurs for verbal descriptions of both the cue and the target object. Our interpretation is that individuals base their preference on previous experience – they know that people often look toward objects they like, and look away from objects they dislike, and people use this knowledge to help decide whether they like or dislike an object. An alternative interpretation is that the apparent effects are due to demand compliance – participants reported higher ratings for cued objects simply because they thought they were supposed to. The lack of an effect in the WORD ONLY group seems to rule out this possibility. However, it is possible that participants in this condition may not have had sufficient information to understand the implications of the word in isolation and consequently failed to rate the objects as liked more in the toward condition vs. the away condition.

To address the issue of demand compliance Bayliss et al. (2006) asked participants (during a post-experimental debrief) what they felt had influenced their ratings. In that study, participants never mentioned the cue, and appropriately the investigators concluded that participants did not work out the purpose of the study. Therefore, we attempted to replicate the LOOK – IMAGE in a new sample of participants who were asked (at debriefing) to list any factors that they thought may have may have influenced their ratings of items. The stimuli were identical to those used in the LOOK – IMAGE condition and also, the order in which a specific side of the sheet of paper was completed was counterbalanced across participants.

# Materials and Methods

#### Participants

There 74 participants (mean age = 20.43; SD = 4.04; 61 females and 13 males). The experiment was carried out in accordance with the ethical standards of the American Psychological Association and with the 1964 Helsinki declaration and its later amendments. The study was approved by the Psychology Ethics Committee at University of British Columbia, Vancouver. Written, informed consent was obtained from all participants in the study.

#### Design and Procedure

The design and procedure were identical to the LOOK – IMAGE condition in Experiment 1 except that participants were asked (at debriefing) to list any factors that they thought may have may have influenced their ratings of items.

# Results and Discussion

Following Bayliss et al. (2006) none of the participants mentioned direction of cue as having influenced their ratings but instead, reported basing their ratings on various aspects (e.g., usefulness) of the target objects. A paired samples t-test, showed that participants gave higher mean liking ratings in the toward condition (M = 4.90; SD = 1.07) compared to the away condition (M = 4.32; SD = 0.99), t(73) = 4.43, p < 0.001 [difference = 0.58; 95% CI (0.32–0.84)].

# General Discussion

The results show that increased liking for gazed-at objects is also found for verbal descriptions of gaze direction. Specifically, participants rated both images and verbal descriptions of objects as more likeable after reading a written description of a person looking toward rather than away from the objects. Although the results showed that the cue direction effect was not specific to eye gaze – the gaze liking effect was recorded for descriptions of pointing and also, for an arrow cue – the effect was largest in magnitude for descriptions of gaze behavior. In other words, the results offer partial support for the gaze specificity hypothesis because descriptions of individuals looking at objects produced the largest effect – eye gaze may not be unique but it is a highly effective cue.

The finding that non-social cues also exert influence contrasts with the lack of an effect for arrows reported previously in a study using an implicit measure of influence (Bayliss et al., 2006). As noted earlier this may because the original gaze-liking effect is much smaller than previously thought, and consequently, detecting such effect for arrow cues will be difficult unless the sample size is very large (N > 500; see Tipples and Pecchinenda, 2018). In contrast, in the current research, the effect was replicable and therefore, the current task is more suitable for testing for differences in magnitude across different cue types. Indeed, as already noted, the current data go some way to supporting the existence of a mechanism that is more sensitive

to social vs. non-social cue differences because the effect of gaze on liking (in the LOOK WORD condition) was relatively greater than the arrow cue condition. This increased sensitivity for gaze cues may reflect a specifically social process that evolved to respond to gaze cues but has been co-opted to respond to other cue types.

In keeping with research designed to investigate mental state attribution, the effect of gaze on liking generalized to verbal stimuli, i.e., the effects do not depend on actual visual observation of either the cue or object stimuli. One interpretation of this finding is that the effects of cue stimuli on liking reflect existing knowledge. When forced to make a liking decision based on limited information participants rely on what they know about the relationship between cue direction and the likeability of objects. What knowledge is relevant to direction cues and object evaluation? Individuals may rely on knowledge of human motivation to either approach a rewarding object or avoid an unpleasant object when they read descriptions of individuals looking toward or away from an object. Put differently, participants may know that gazing-away from an object typically means that the object is not rewarding and therefore, they give the object a lower rating. A reliance on existing knowledge does not necessarily mean that the gaze-induced-liking is not rooted in a biologically based process. The tendency to use gaze and facial expressions as a basis for evaluating objects may still have a biological basis that is present in the early stages of human development.

Comparison of the WORD ONLY condition with the words "TOWARD" and "AWAY" with "LOOK – WORD" condition indicates that simply presenting motivational relevant words is not sufficient to record a gaze liking effect. Instead, the current findings suggest that sentence frame in which an actor is described as acting toward an object is needed to record the gaze-liking effect. One interpretation of this finding is that the gaze-liking effect requires propositional knowledge – a statement about the world and more specifically a statement about relationships in the world ("Michael looked toward..."). This finding accords with a growing recognition that propositional knowledge contributes to implicit evaluation processes (De Houwer, 2014). Specifically, according to the propositional model of implicit evaluation (De Houwer, 2014) automatic evaluation depends on the formation or activation of propositional knowledge. The current results are consistent with such an account.

Were our cue effects due to demand compliance? Put differently, did participants work out the purpose of the study and then respond in the manner expected by the experimenter? Following the methodology of Bayliss et al. (2006), our participants in Experiment 2 did not report using the cue to make their likeability judgments. Bayliss et al. (2006)

#### REFERENCES

concluded that the lack of self-reported knowledge indicated that their participants had not worked out the purpose of the study. Against this high standard we also conclude that participants in our investigation had not worked out the purpose of the study, that is, the effects were not due to demand compliance.

In conclusion we have shown that shifting attention to an object is not necessary for eye gaze cues to influence object liking – the effect occurs for verbal descriptions of eye gaze. Furthermore, a variety of cues can influence object liking – the effect is not restricted to eye gaze cues – and moreover, the effects were not due to demand compliance. In sum, our results suggest a separate, conceptually based route, by which cue direction can affect object evaluation. Whether this route is also responsible for previous findings of liking that employed the gaze cue paradigm is an exciting issue for future investigation. One thing is very clear, while the conceptually based route may explain past findings of liking with the gaze cue paradigm, those past findings with the gaze cue paradigm are unable to account for the present data insofar as they are grounded on the assumptions that an external gaze cue and a shift of visuo-spatial attention toward or away from an object are necessary to trigger a liking effect.

# DATA AVAILABILITY

The datasets generated for this study are available on request to the corresponding author.

# AUTHOR CONTRIBUTIONS

JT and AK discussed the design of the experiments. MD arranged for data collection. JG collected the data. JT analyzed the data and wrote the manuscript. AK and MD commented on various drafts.

# FUNDING

This research was supported by an NSF EPSCoR Research Infrastructure Award 1632849 to MD and a Discovery Grant to AK.

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2019.00471/full#supplementary-material


Bayliss, A. P., Frischen, A., Fenske, M. J., and Tipper, S. P. (2007). Affective evaluations of objects are influenced by observed gaze direction and emotional expression. Cognition 104, 644–653. doi: 10.1016/j.cognition.2006.07.012


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Tipples, Dodd, Grubaugh and Kingstone. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Role of Eye Gaze During Natural Social Interactions in Typical and Autistic People

#### Roser Cañigueral\* and Antonia F. de C. Hamilton

Institute of Cognitive Neuroscience, Division of Psychology and Language Sciences, University College London, London, United Kingdom

Social interactions involve complex exchanges of a variety of social signals, such as gaze, facial expressions, speech and gestures. Focusing on the dual function of eye gaze, this review explores how the presence of an audience, communicative purpose and temporal dynamics of gaze allow interacting partners to achieve successful communication. First, we focus on how being watched modulates social cognition and behavior. We then show that the study of interpersonal gaze processing, particularly gaze temporal dynamics, can provide valuable understanding of social behavior in real interactions. We propose that the Interpersonal Gaze Processing model, which combines both sensing and signaling functions of eye gaze, provides a framework to make sense of gaze patterns in live interactions. Finally, we discuss how autistic individuals process the belief in being watched and interpersonal dynamics of gaze, and suggest that systematic manipulation of factors modulating gaze signaling can reveal which aspects of social eye gaze are challenging in autism.

#### Edited by:

Chris Ashwin, University of Bath, United Kingdom

#### Reviewed by:

Hélène Cochet, Université Toulouse – Jean Jaurès, France Ouriel Grynszpan, Université Pierre et Marie Curie, France

#### \*Correspondence:

Roser Cañigueral roser.canigueral.15@ucl.ac.uk

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 19 November 2018 Accepted: 28 February 2019 Published: 15 March 2019

#### Citation:

Cañigueral R and Hamilton AFdC (2019) The Role of Eye Gaze During Natural Social Interactions in Typical and Autistic People. Front. Psychol. 10:560. doi: 10.3389/fpsyg.2019.00560 Keywords: eye gaze, audience effect, interpersonal dynamics, social interaction, autism

# INTRODUCTION

In any face-to-face interaction between two people, both agents are continuously exchanging a variety of social signals, such as gaze, gestures or facial expressions. This two-way exchange of social information is possible because they are able to see each other, and consequently both agents can gather and communicate information. Although traditional cognitive research has largely ignored this interactive nature of social encounters, an increasing number of studies are looking at how social behavior changes in a live interaction, as well as how eye gaze of two individuals coordinates to achieve successful communication, that is, to accurately process incoming signals and send back meaningful signals at a suitable pace.

In the present paper, we explore gaze as a communicative signal in a two-person interaction, considering both patterns of gaze to/from the other person and the interpersonal dynamics of gaze in relation to other behaviors. To explore these issues, we first introduce the dual function of eye gaze and describe two cognitive theories that explain changes in behavior when being watched. We then consider gaze exchanges during communicative situations, and propose the Interpersonal Gaze Processing model as a framework to study the dynamics of gaze in face-toface interactions. Finally, we look into the case of autism to discuss how studies on the audience effect and interpersonal dynamics of gaze can shed light on why autistic people find social communication challenging.

**81**

# THE DUAL FUNCTION OF EYE GAZE

Eye gaze has a dual function in human social interaction – we can both perceive information from others and use our gaze to signal to others (Argyle and Cook, 1976; Gobel et al., 2015; Risko et al., 2016). Simmel (1921) already stated that "the eye cannot take unless at the same time it gives." This contrasts with the auditory modality, where we use our ears to hear, but our mouth to speak. This makes our eyes a powerful tool for social interactions, with a "uniquely sociological function" (Simmel, 1921). For instance, when we see a pair of eyes we can gather information about what other people are looking at (Frischen et al., 2007), and how they feel or think (Baron-Cohen et al., 1997). At the same time, we can use our eyes to strategically cue another's attention (Kuhn et al., 2009). Depending on the duration and direction of our gaze, we are also able to perceive and signal a variety of meanings, such as desire to communicate (Ho et al., 2015), threat and dominance (Ellyson et al., 1981; Emery, 2000), attractiveness (Argyle and Dean, 1965; Georgescu et al., 2013), or seeking for approval (Efran and Broughton, 1966; Efran, 1968).

The dual function of the eyes has often been ignored in cognitive research studying social interactions. In typical lab studies, participants interact with a monitor that displays pictures or videos of other people, while their gaze or other behavior is recorded (see Risko et al., 2012 for a review). In these experimental settings signals are sent only one-way (from the picture to the participant) and the dual function of gaze is completely lost. Although these traditional approaches allow good experimental control, they are not interactive (Schilbach et al., 2013; Gobel et al., 2015; Risko et al., 2016). Recent research has implemented more ecologically valid approaches that can restore the dual function of gaze. The belief that someone can see us, intrinsic to live interactions, is thought to recruit a range of social cognitive processes that are missing when participants interact with videos or pictures (Risko et al., 2012, 2016; Schilbach et al., 2013). Moreover, in face-to-face interactions communication is multimodal (Vigliocco et al., 2014): information is exchanged through eye gaze, but also through gestures, facial expressions or speech, and all these signals need to be integrated over time and across agents (Jack and Schyns, 2015; Hirai and Kanakogi, 2018; Holler et al., 2018).

In the following, we first describe two cognitive theories that explain changes in behavior when being watched. Then, we discuss why interpersonal dynamics are relevant when studying social eye gaze.

# COGNITIVE THEORIES OF THE AUDIENCE EFFECT

We behave differently when we are alone or in the presence of others. For instance, when we are with other people our actions become more prosocial (Izuma et al., 2009; Izuma et al., 2011), our memory improves (Fullwood and Doherty-Sneddon, 2006), and we smile more (Fridlund, 1991). Triplett first introduced this idea 120 years ago, when he showed that cyclists were faster when competing against each other than against a clock (Triplett, 1898). To explain this effect, he suggested that the "bodily presence of another" causes changes in the behavior of participants, which makes them more competitive when racing against others. However, previous research has shown that there is more than one way in which the presence of another person can change our behavior.

On the one hand, social facilitation refers to a change in behavior caused by the presence of a conspecific that may or may not be watching us (Zajonc, 1965). This effect is present in humans but also in a wide range of species (e.g., cockroaches, rats and monkeys), suggesting that it relies on a simple mechanism like arousal. Zajonc further claimed that an increase in arousal in the presence of others would facilitate dominant behaviors (i.e., responses that are elicited most quickly by a stimulus). For instance in an easy task the dominant response is usually the correct one, while in a difficult task the dominant response is usually the incorrect one. Zajonc and Sales (1966) found that, in the presence of a conspecific, participants performed better on a verbal recognition task with familiar items (easy task), and worse on the same task with unfamiliar items (hard task). This effect has been found in a range of tests on both mental (Geen, 1985) and physical skills (Strauss, 2002). Blascovich et al. (1999) replicated these findings and also showed that, in the presence of others, the cardiovascular system is differently triggered depending on the task: in a difficult task the cardiovascular response fits a threatlike pattern, whereas in an easy task the cardiovascular response fits a challenge-like pattern. This suggests that the facilitation of different dominant responses in the presence of others is mediated by different arousal patterns.

On the other hand, the audience effect is a change in behavior specifically caused by the belief that someone else is watching me. It builds on mechanisms which process the perceptual state of the other, known as perceptual mentalising (Teufel et al., 2010b). Perceptual mentalising modulates the processing of social information from the eyes in a variety of ways. For example, seeing a live-feed of a person with transparent glasses (who can see) leads to a larger gaze cuing effect than a matched stimulus of a person with opaque glasses (who cannot see) (Nuku and Bekkering, 2008; Teufel et al., 2010a), and similar results are seen in tests of visual perspective taking (Furlanetto et al., 2016). This demonstrates that even basic social processing is influenced by the knowledge that another person can see something. The audience effect takes this one step further, considering how our social cognition is affected by the knowledge that another person can see us.

Audience effects differ from social facilitation in that social facilitation could occur if another person is present but looking away, whereas audience effects are specific to the case when another person is believed to be watching (even from another location). When people believe they are being watched, they typically change their behavior to maintain a positive public image. This has been described in terms of self-presentation theory (Bond, 1982), which claims that people modulate their performance in front of others to maintain a good public image and increase their self-esteem. Bond (1982) further showed that making errors while being observed translates into decreased self-esteem and poor performance, regardless of task difficulty.

The audience effect and the dual function of gaze are closely linked in that both require someone who can see us. In line with this, recent evidence suggests that being watched modulates gaze patterns directed at the face of the observer, because in this context direct gaze acquires a social meaning that an individual may or may not wish to signal to someone else. These studies show that in a live interaction people look less to the other person than in a pre-recorded interaction (Laidlaw et al., 2011; Gobel et al., 2015). This change in gaze patterns is further modulated by several factors, such as the observer's social status (high rank or low rank; Gobel et al., 2015) or role in the interaction (speaker or listener; Freeth et al., 2013; Ho et al., 2015). Thus, when being watched eye gaze is adjusted to send appropriate signals to the observer, rather than to only gather information from the environment.

In the following, we strictly focus on changes in social behavior that derive from audience effects, that is, from the belief in being watched. To explain these changes, two main cognitive theories have been proposed: the Watching Eyes model (Conty et al., 2016) and reputation management theory (Emler, 1990; Resnick et al., 2006; Tennie et al., 2010). Both theories give plausible explanations about the relationship between an individual and an observer, but they have different focus. The Watching Eyes model concentrates on how an observer influences cognitive processing within individuals (self-focus), beyond self-esteem effects proposed by self-presentation theory. Reputation management theory explains how individuals manipulate the observer's beliefs to their advantage (other-focus) in an updated version of the selfpresentation theory. Below we describe each of these theories in more detail.

### Watching Eyes Model

A pair of eyes watching us is an ostensive communicative cue (Csibra and Gergely, 2009) that rapidly captures our attention (Senju and Hasegawa, 2005). Early work on gaze processing proposed various mechanisms how direct gaze modulates our attention and behavior. For instance, Baron-Cohen (1995) suggested that there is a specialized Eye Direction Detector module in the brain. This module rapidly identifies whether we are the target of someone else's attention by processing the direction of other people's eyes relative to us. The detection of direct gaze will in turn trigger mentalising processes that allow us to interpret the other person's mental states (Baron-Cohen and Cross, 1992; Baron-Cohen et al., 1997). Later, Senju and Johnson (2009) coined the term "eye contact effect" to describe changes in cognitive processing following perception of direct gaze, and introduced the Fast-track Modulator model of gaze processing. This model suggests that detection of direct gaze is implemented by a fast subcortical route involving the pulvinar and amygdala, and is modulated by higher cortical regions that depend on social context and task demands. The recently proposed Watching Eyes model (Conty et al., 2016) builds up on these models and suggests that audience effects are due to the "self-referential power of direct gaze."

Similar to the Fast-track Modulator model by Senju and Johnson (2009), the Watching Eyes model proposes two stages in the processing of direct gaze. In the first stage, direct gaze captures the beholder's attention by a subcortical route. This seems to be an automatic effect of direct gaze (Senju and Hasegawa, 2005), and is thought to be triggered by the detection of lowlevel visual cues in eye gaze (e.g., luminance distribution in the eye; von Grünau and Anston, 1995; Kobayashi and Kohshima, 2001). Then, the subcortical route engages mentalizing brain areas (medial prefrontal cortex and temporo-parietal junction) that process the perceptual state of the observer, that is, the belief that s/he is or is not watching us. In the second stage, if the observer can see us, then direct gaze will elicit selfreferential processing, and the sense of self-involvement in the interaction will increase. This will lead to the Watching Eyes effects, causing a change in behavior in various ways, such as enhancement of self-awareness (Pönkänen et al., 2011; Baltazar et al., 2014; Hazem et al., 2017) or promotion of prosocial actions (Izuma et al., 2011, 2009).

Recently, Hietanen and Hietanen (2017) have directly tested the Watching Eyes model of self-referential processing. To measure self-referential processing they used the foreignlanguage task, where participants read sentences in a language that they do not understand and need to match underlined words with pronouns in their native language. In this task, more use of first person singular pronouns is thought to be related to more self-referential processing. Participants completed this task but they watched a video-clip of a person with direct or averted gaze before each sentence was presented. Results showed no effect of eye gaze direction on the pronouns used. Then, a second group of participants completed the same task while they watched live faces with direct or averted face. They found that participants in the direct gaze group used more first person singular pronouns than the averted gaze group. In line with this, a recent study on bodily self-awareness (Hazem et al., 2017) has found that participants are more accurate in rating the intensity of a physiological signal when they believe they are in online connection with someone wearing clear sunglasses (the observer can see through) rather than someone wearing opaque sunglasses (the observer cannot see through). Taken together, these findings show evidence in favor of the Watching Eyes model: to trigger self-reference and self-awareness it is not enough to see a pair of eyes directly gazing at us – the belief that these pair of eyes can see us is also required.

Yet, it is important to consider that different tasks measure different forms of self-reference and self-awareness. This means that different tasks are likely to engage different self-related cognitive processes, which might have different sensitivity to the belief in being watched. For instance, the pronoun-selection task used by Hietanen and Hietanen (2017) is rather intuitive and has been shown to be sensitive to manipulations of selfawareness (Davis and Brock, 1975). However, it could be that other tasks which elicit more complex self-referential cognitive processes (e.g., self-referential effect memory task; Craik and Tulving, 1975; Lombardo et al., 2007) are not as sensitive to this top-down modulation. It is equally important to distinguish between different forms of self-awareness, such as bodily selfawareness (accuracy in reporting physiological signals; Cameron, 2001) and metacognitive self-awareness (accuracy in judging

performance in a task; Fleming and Dolan, 2012). Thus, it remains to be seen whether direct gaze and the belief in being watched modulate all forms of self-referential processing and self-awareness or not.

### Reputation Management Theory

Reputation is a social construct that emerges from the desire to cultivate good self-impressions in others (Silver and Shaw, 2018). It is based on how we think others see us, and it changes over time depending on our actions (Izuma, 2012; Cage, 2015). People can gain approval from others and increase their own reputation in various ways, such as acting for the benefit of other people or behaving according to social norms. To maintain or manage reputation, individuals need to think about what others think of them, care about how others see them, and have the desire to foster positive impressions in others (Izuma, 2012; Cage, 2015). Thus, mentalizing and social motivation have a central function in reputation management (Saito et al., 2010; Tennie et al., 2010; Izuma, 2012; Cage, 2015). In line with this, neuroimaging studies have shown that mentalizing and reward brain areas are engaged during different phases of reputation management, such as processing what others think of them (e.g., medial prefrontal cortex; Frith and Frith, 2006; Izuma et al., 2010) or anticipating positive reputation (e.g., ventral striatum; Izuma et al., 2009, 2010) respectively.

One strategy that people use to maintain a good reputation in front of others is to behave in a more prosocial fashion (Smith and Bird, 2000; Bradley et al., 2018). A way to measure prosocial behavior in the lab is by using economic games. Because they usually have repeated trials, this facilitates reputation building between participants in the game (Pfeiffer and Nowak, 2006; Bradley et al., 2018). For instance, Filiz-Ozbay and Ozbay (2014) used the Public Goods game and found that people invest more effort to contribute to public, but not private, goods when someone is observing them. Izuma et al. (2011) used the Dictator game (Kahneman et al., 1986; Guala and Mittone, 2010) as a donation task, where participants receive a sum of money and must decide on repeated trials whether to accept a proposal to share the money with a charity, or reject it and keep all the money. Results showed that in the presence of a confederate who pretended to monitor the answers, participants decided to accept the proposed sharing more often than when they were alone in the room. These findings clearly illustrate how participants manipulate the beliefs of the observer to maintain their good reputation.

Several factors modulate how strong the audience effect is on prosocial behavior (Bradley et al., 2018), such as the identity of the observer (experimenter, other participants, stranger) or whether decisions of participants are consequential. For instance, Cage et al. (2013) also used the Dictator game in the presence and absence of a confederate, but additionally contrasted two conditions: one in which participants believed the recipient of the sharing arrangement was an individual who could later reciprocate (consequential decision), and one in which the recipient could not reciprocate (non-consequential decision). They found that participants accepted the chance to share money most frequently in the presence of a confederate and when the confederate could later reciprocate. This shows that the context associated with the observer (e.g., can s/he reciprocate or not?) also modulates the extent to which being watched affects behavior.

Another strategy used to maintain reputation is to behave according to social norms. Social norms can be of various kinds, such as saying thank you or holding a door for someone after you. A more subtle type of social norm is civil inattention (Goffman, 1963), which proposes that the amount of gaze directed to strangers "should be enough to acknowledge their presence but not so much as to indicate that they are of special interest." Multiple studies have used eye-tracking to test if social attention is modulated according to social norms of eye gaze. For instance, Laidlaw et al. (2011) found that participants sitting in a waiting room would look more to a confederate in a video-clip than to the same confederate present in the room. The authors claimed that this change in gaze patterns is due to a social norm whereby it is not polite to stare at someone, which in turn translates into active disengagement.

Some of these studies also show that gaze patterns in live contexts are modulated by a number of factors that do not have any effect when participants watch video-clips. Gobel et al. (2015) found that participants spend more time gazing at video-clips of a low rank confederate and less time gazing at video-clips of a high rank confederate, but only when they believe the confederate will later see their gaze recording. These two gaze behaviors, direct and averted gaze, have been associated with signalling of dominance and submission, respectively (Ellyson et al., 1981; Emery, 2000). In another study, Foulsham et al., 2011 showed that participants gaze less to close pedestrians than distant pedestrians to avoid appearing as an interaction partner to strangers (see also Argyle and Dean, 1965; Gallup et al., 2012). These studies indicate that, when an observer is watching, eye gaze acquires a signaling function and this will subtly modulate gaze patterns to send appropriate signals to the observer. Moreover, the social skills of participants and their looking behavior are correlated in live but not lab settings (Laidlaw et al., 2011). This suggests that individuals who successfully interact with other people are those who can modulate social behavior according to requirements of the social context.

So far, we have discussed how the presence of an observer modulates an individual's cognitive processing, both selffocused (Watching Eyes model) and other-focused (reputation management theory). However, the studies presented above have a major limitation: confederate and participant are not expected (and do not intend) to interact, verbally or physically, with each other. This means that there is no explicit communicative exchange between them. In the same way that social behavior changes when participants watch a video-recorded person or a live person, it could be that it also differs between a situation where there is potential for an interaction and a situation where there is an actual interaction with explicit communicative exchanges (henceforth communicative encounter; Foulsham et al., 2011; Wu et al., 2013; Macdonald and Tatler, 2018). Focusing on the particular case of eye gaze, in the next section we argue that interpersonal gaze dynamics have a key role in modulating social behavior during communicative encounters.

# INTERPERSONAL DYNAMICS OF EYE GAZE

Original studies about the role of eye gaze during communicative encounters date back to the 60 s, when Argyle and colleagues (Argyle and Cook, 1976; Argyle and Dean, 1965) put forward the intimacy equilibrium model, which is the first account on the relationship between "looking and liking:" they showed that gaze directed at other people serves to control the level of intimacy or affiliation with the partner, and that it compensates with other behaviors (e.g., physical proximity) to achieve an equilibrium level of intimacy (see also Loeb, 1972). Furthermore, Watzlawick et al. (1967) proposed the idea that "one cannot not communicate," since the lack of response is a response in itself (e.g., not looking at someone signals lack of interest in the interaction; Goffman, 1963).

Recent studies show that direct gaze can act as an ostensive communicative signal (Csibra and Gergely, 2009). During faceto-face interactions, where individuals exchange information with communicative purpose through a variety of channels (e.g., gaze, gestures, facial expressions, speech), direct gaze helps to integrate and coordinate auditory and visual signals (Bavelas et al., 2002). Moreover, it has been shown that to successfully produce and detect gestures with communicative purpose, information conveyed by gaze signals (e.g., direct gaze) is preferentially used over information conveyed by kinematics of the gesture (Trujillo et al., 2018). Thus, eye gaze has a core function in leading social interactions up to successful communicative exchanges, where there is efficient transmission of information between sender and receiver.

In the studies presented in the previous section, the authors claim that changes in eye gaze when participants are being watched respond to demands of social norms (Foulsham et al., 2011; Laidlaw et al., 2011; Gobel et al., 2015). The context of those studies does not require participants to explicitly communicate with the confederate, but only look (or not) at each other. Moreover, the confederate is usually a complete stranger to the participant. It is therefore not surprising that this awkward interaction without communicative purpose leads participants to modulate eye gaze in compliance with social norms (Wu et al., 2013). However, in communicative encounters (e.g., conversation) gaze patterns need to coordinate with other verbal and non-verbal signals to successfully receive and send signals (Bavelas et al., 2002; Trujillo et al., 2018). In studying such communications, we must consider not just the average pattern of gaze (toward/away from the face) but also the dynamics of gaze behavior in relation to other social events (speech, turn taking, facial expressions, etc.). This means that to succeed during communicative exchanges, eye gaze needs not only modulation by social norms, but also constant adjustments to keep pace with interpersonal dynamics that emerge as the interaction develops.

In the following, we first describe the main social functions that eye gaze has during communicative interactions. Then, we focus on the temporal dynamics of gaze as a key mechanism that enables meaningful interpersonal exchanges during communication, as well as successful progression of the interaction.

# Social Functions of Eye Gaze During Conversation

During communicative encounters, such as conversations, the eyes of both agents are generally very active. In a seminal study on gaze direction during conversation, Kendon identified asymmetrical gaze behavior between speakers and listeners (Kendon, 1967): while listeners gazed at speakers most of the time, speakers shifted their gaze toward and away from listeners. More recently, Rogers et al. (2018) found that during a 4 min conversation participants spent on average 60% of the time directing their gaze toward the face of the other person (only 10% of the time it was directed specifically to the eyes), and that these events were approximately 2.2 s long (for direct eye contact events were 0.36 s long). The brief duration of these events supports Kendon's original findings, because it indicates that participants are constantly alternating their gaze between face or eyes of their partner and other regions. There has been much debate about the meaning of these rapid and subtle changes in eye gaze direction and duration. Kendon (1967) originally suggested that they give rise to three main social functions of gaze. Note that, although the gaze patterns described below allow us to send signals to another person, these signals are sent implicitly and without awareness.

First, he proposed that eye gaze has a regulatory function during conversation, because it allows individuals to modulate transitions between speaker and listener states (i.e., turn-taking). In line with this, it has been found that speakers use averted gaze when they begin to talk and during hesitation (probably to indicate that they want to retain their role as speakers), but they use direct gaze to the listener when they are about to end an utterance (probably to signal that their turn is ending and that the listener can take the floor) (Kendon, 1967; Duncan and Fiske, 1977;Cummins, 2012; Sandgren et al., 2012; Ho et al., 2015). However, as noted by Ho et al. (2015) conversation is a twoway process and this means that the listener is also responsible to regulate in turn-taking. For instance, it has been shown that listeners make more gestures, head shifts and gaze shifts before speaking, probably to indicate to the speaker that they want to take the turn (Harrigan, 1985).

Second, Kendon suggested that eye gaze has a monitoring function: it allows each participant to track attentional states and facial displays of the partner to ensure mutual understanding and seek social approval from others (Efran and Broughton, 1966; Efran, 1968; Kleinke, 1986). Indeed, speakers try to gain more information about what listeners think by engaging in brief periods of mutual eye gaze, which elicit back-channeling (i.e., listener's brief responses showing comprehension of what the speaker is saying) (Bavelas et al., 2002). Rogers et al. (2018) have also proposed that brief and rapid gaze shifting between gaze directed to the eyes and to other facial regions (e.g., mouth, eyebrows) may serve to scan facial features and pick subtle cues that help interpreting the meaning of what is being said. The monitoring function of gaze can also have high cognitive costs. For instance, when participants are asked to look at the face of the experimenter, they perform worse than participants who can avert their gaze naturally (Beattie, 1981), or who are asked to

fixate on other static or dynamic stimuli (Markson and Paterson, 2009). Thus, Kendon also claimed that speakers avert their gaze partly to reduce the costs associated with monitoring a face.

Third, Kendon proposed that eye gaze has an expressive function, which allows participants to regulate the level of arousal in the interaction. He found that some participants tended to avert their gaze at moments of high emotion, and that the amount of eye contact was inversely related to the frequency of smiling. He suggested that averting gaze at these highly emotional moments could be interpreted as a "cut off " act to express embarrassment and reduce arousal. Moreover, the expressive function of mutual eye gaze has been associated with affiliation and attraction (Argyle and Dean, 1965; Argyle and Cook, 1976; Georgescu et al., 2013), with dominance and power (Ellyson et al., 1981; Emery, 2000; Gobel et al., 2015), and more recently with expressing response preference to polar questions (Kendrick and Holler, 2017).

It is important to bear in mind that the social functions of gaze are only meaningful during face-to-face interactions, where both partners can see each other. It is only in this context that eye gaze has a dual function and both agents can perceive and signal information (Gobel et al., 2015; Risko et al., 2016). Moreover, gaze signals are not isolated: speakers need to shift their gaze toward or away from the listener at specific time points during speech, listeners need to coordinate gaze direction with facial expressions to indicate preference or reduce arousal, and speakers and listeners need to engage in brief mutual gaze periods to exchange turns or elicit back-channeling. Thus, to succeed in communicative encounters social signals need to be coordinated within and across conversation partners over time.

# Temporal Dynamics of Gaze

Successful communication requires that both agents involved in the interaction process incoming signals and send back meaningful signals at a suitable pace. Since these signaling exchanges (specially for eye gaze) happen very quickly, timing becomes a critical factor to enable successful progression of the interaction. The need for timed coordination gives rise to patterns of gaze behavior, that is, temporal dependencies that emerge between gaze and other social signals. For instance, using gaze cueing paradigms (e.g., Posner's paradigm; Posner, 1980) it has been shown that averted gaze results in reflexive gaze following behavior, which is key to build joint attention (Pfeiffer et al., 2013). Similarly, there could be a systematic relationship between gaze and speech within an individual (e.g., direct gaze at others when finishing an utterance, but avert gaze when hesitating; Ho et al., 2015), or between the gaze direction of two conversation partners (e.g., establish mutual eye gaze to elicit back-channeling; Bavelas et al., 2002). The presence and direction of these temporal dependencies at different time points can contribute to identifying which social cognitive processes modulate gaze behavior in the course of the interaction.

Experimentally manipulating temporal dynamics of eye gaze in the lab can be challenging, because it requires some degree of control over gaze patterns for at least one of the agents. Virtual reality and humanoid robot avatars offer an efficient alternative to this issue, because their behavior can be meticulously controlled while participants respond with comparable social behaviors as in interactions with real human beings (Pfeiffer et al., 2013).With the aim of studying interactions in a truly reciprocal context, Wilms et al. (2010) created the now widely used gaze-contingent eyetracking paradigm (see also Bayliss et al., 2012; Kim and Mundy, 2012; Edwards et al., 2015). In this paradigm, participants wearing an eye-tracker interact with an avatar whose gaze is controlled by the real-time gaze data collected from the participant. Thus, the avatar becomes a gaze-contingent stimulus that responds to the participant's gaze behavior. Using this paradigm in the context of joint attention, it has been shown that avatars are perceived as more human-like (Pfeiffer et al., 2011) and more likeable (Grynszpan et al., 2017; Willemse et al., 2018) if they follow the gaze of participants to achieve joint attention. Another study has shown that participants are quicker to assume that the avatar understands their instructions when there is contingent gaze following (Frädrich et al., 2018). At the neural level, joint attention has been linked to activation in brain areas related to gaze direction (superior temporal sulcus), processing of reward (ventral striatum) and mental states (medial prefrontal cortex, temporo-parietal junction) (Pelphrey et al., 2004; Schilbach et al., 2010; Pfeiffer et al., 2013; Caruana et al., 2015).

Some attempts have also been made to study the nature of temporal dynamics of gaze in real human-to-human interactions. For instance, Lachat et al. (2012) designed a joint attention task where dyads of participants engaged in joint and nojoint attention periods, respectively. They found that during joint attention periods mu rhythms in centro-parietal regions were suppressed for both leaders and followers, which has been previously associated with interpersonal coordination processes (Naeem et al., 2012). In another study, participants completed a structured interview with a pre-recorded or live confederate, whose gaze was directed at them or averted (Freeth et al., 2013). They found that participants gazed more to the confederate's face if her gaze was directed at them than if her gaze was averted, but only in the live condition. This means that participants' gaze was adjusted according to the looking behavior of the confederate only when their gaze acquired a signaling function (i.e., they were in a live interaction), thus creating a reciprocal social signal. Recently, a dual eye-tracking study (Macdonald and Tatler, 2018) has also shown that pairs of participants who are given specific social roles in a collaborative task align their gaze quicker than pairs who have no social role. This indicates that eye gaze adjusts to the communicative purpose embedded in different social contexts.

Gaze dynamics are fundamental to efficiently communicate with other people, that is, to enable information transfer between individuals. It has recently been suggested that brainto-brain coherence (i.e., synchronization of neural activity between two brains) provides a marker of the success of a communication between two people (Hasson et al., 2012), and several hyperscanning studies show that mutual gaze triggers neural coherence between partners. For instance, mutual gaze mediates neural coupling between parents and infants, which has been associated with appropriate use of communicative signals according to each social context later in development (Piazza et al., 2018). Neural coherence between parents and

infants has been shown to be stronger in live versus prerecorded interactions (Leong et al., 2017). Moreover, in a joint attention task through a video-feed, moment of eye contact was characterized by increased synchronization of frontal brain activity between participants (Saito et al., 2010). Hirsch et al. (2017) have also shown that only when partners in a dyad make eye contact (compared to when both partners look at a photograph of a face) brain-to-brain coherence between partners' increases in regions associated with processing of social information (temporo-parietal and frontal regions). These findings suggest that direct gaze acts as a signal that enhances the temporal alignment of two brains (Hasson et al., 2012; Gallotti et al., 2017), thus facilitating the sharing of information.

All these studies show that temporal coordination of gaze patterns are characteristic of human interactions (Pfeiffer et al., 2011; Willemse et al., 2018), and that they have beneficial effects for the interacting partners, such as increasing the reward value of the interaction (Schilbach et al., 2010), or facilitating social coordination (Lachat et al., 2012; Freeth et al., 2013; Frädrich et al., 2018) and information transfer (Saito et al., 2010; Hirsch et al., 2017; Leong et al., 2017). They also highlight that gaze is a dynamic and interpersonal signal which changes over time depending on the social situation and communicative purpose. However, there is no cognitive model of gaze processing that takes into account these interactive factors. We believe that in the current context of social cognitive research, which has a strong focus on ecologically valid approaches (Schilbach et al., 2013; Risko et al., 2016), there is an urgent need to build up a cognitive model of eye gaze in live interactions. With this aim, in the next section we introduce the Interpersonal Gaze Processing model, which tries to makes sense of gaze dynamics during face-to-face interactions.

# INTERPERSONAL GAZE PROCESSING MODEL: ACTIVE SENSING AND SOCIAL SIGNALING

The dual function of the eyes means that our gaze both gains information form the environment and signals information to others. Early cognitive research already described how the visual system gains information from the environment in non-social contexts (Koch and Ullman, 1985; Itti and Koch, 2001). However, to our knowledge there is no cognitive model of gaze processing in social contexts. Here we draw on two distinct frameworks, from motor control (active sensing; Yang et al., 2016) and from animal communication (signaling theory; Zahavi, 1975; Grafen, 1990), to introduce the Interpersonal Gaze Processing model. This model considers how these two frameworks can be combined in the domain of social gaze to take into account both its sensing and signaling functions. In the following, we describe how active sensing and signaling theory are useful to explain gaze behavior.

#### Active Sensing in Eye Gaze

Active sensing is a key process in our interaction with the world, since it allows our sensors to be directed to the environment in order to extract relevant information (Yang et al., 2016). Gaze behavior (i.e., deciding where to look) can be considered a form of active sensing in that we choose to move our eyes to specific locations to sample useful information from a visual scene. Since our visual system only gains high-resolution information for items falling in the fovea, the motor system needs to move our eyes to orient the fovea to different locations of interest. Thus, our motor actions shape the quality of the sensory information we sample (Yang et al., 2016).

The active sensing framework provides a mathematical account of how we can sample the world with our eyes to get useful information. Because we can only direct our eyes at one location at a time, each eye movement (i.e., saccade) comes at some opportunity cost. For instance, in **Figure 1A**, looking at the woman and child on the bottom means we might lose the chance to get information about the house in the center or the woman and child on the left. Similarly, in **Figure 1B**, looking at the landscape on the right means we will lose information about the blue car on the left or the speedometer. Active sensing suggests that saccades are planned to maximize the information we sample depending on the goal of the task at hand.

To understand how sampled information is maximized it is useful to consider the concept of saliency maps. A saliency map is "an explicit two-dimensional topographical map that encodes stimulus conspicuity, or saliency, at every location in the visual scene" (Itti and Koch, 2001). It results from the combination of different topographical or feature maps, each representing a single visual feature (Koch and Ullman, 1985; Itti et al., 1998; Veale et al., 2017), such as intensity or color. A saliency map is a pre-attentive computation, in the sense that at this stage all locations are competing for representation in the visual cortex (Itti and Koch, 2001). Only the location that is most salient will gain further access in downstream visual areas and the oculomotor nerve, and guide the next eye movement so as to deploy attention in that specific location (Koch and Ullman, 1985; Kastner and Ungerleider, 2000; Itti and Koch, 2001; Veale et al., 2017) (see **Figure 1C**).

Early models of saliency maps only included static features of visual scenes (e.g., color, orientation, intensity, center-surround difference; Koch and Ullman, 1985; Itti and Koch, 2001), but later proposals have suggested saliency maps that also integrate dynamic features (Milanese et al., 1995; Jeong et al., 2008). For instance, the integrated saliency map by Jeong et al. (2008) considers dynamic features such as rotation, expansion, contraction or planar motion. These dynamic features are especially effective in attracting visual attention, and have been associated with an alerting mechanism that rapidly detects moving objects (Milanese et al., 1995). Both static and dynamic features generate a bottom-up bias on the saliency map.

However, saliency maps can also be modeled by a top-down bias emerging from affective features (e.g., preference or dislike for the visual stimuli; Olshausen et al., 1993; Tsotsos et al., 1995; Itti et al., 1998; Veale et al., 2017; Jeong et al., 2008) (see **Figure 1C**). Affective features are mainly associated with the goal of the task at hand, and are integrated with bottom-up information in associative visual areas (extrastriate cortex) (Veale et al., 2017). For instance, as shown on **Figure 1D**, different

search goals will model different priority maps derived from the same saliency map. Recent evidence has also found that when participants view social naturalistic scenes low-level salient features are less important, and participants primarily fixate on the faces and eyes of people in the scene (Nasiopoulos et al., 2015; End and Gamer, 2017; Rubo and Gamer, 2018). This suggests that there is an implicit preferential bias to attend to others in social scenes to obtain information about them (Nasiopoulos et al., 2015). In the same way that non-social task goals (e.g., search for the cell phone) model different priority maps, implicit social task goals (e.g., identify feelings of an actress in a movie) will model different sensing maps. This top-down bias is particularly important in the context of active sensing, since the task goal will modify the reward value of each location in the visual scene and, in turn, determine which information needs to be maximized (Jeong et al., 2008; Yang et al., 2016).

Active sensing provides a useful framework to understand how eye movements are planned to process non-social stimuli (e.g., objects or landscapes), as well as social stimuli in pictures or videos. In both cases, the saccade planner combines bottom-up and top-down features in a priority or sensing map to maximize information relevant for the task and decide where gaze is next directed (Yang et al., 2016). However, in the case of face-toface interactions, our gaze not only needs to maximize the information gained but also send signals to another person (i.e., dual function of eyes; Argyle and Cook, 1976; Gobel et al., 2015; Risko et al., 2016).

# Social Signaling and Eye Gaze

Research on animal communication has explored in detail the question of what behavior counts as a social signal and what message (if any) is sent (Stegmann, 2013). A cue is a behavior or feature that can be used by another creature to guide its behavior; for example, mosquitoes use the increased carbon dioxide in exhaled air as a cue to find people to bite, but there is no benefit here to those sending the cue. In contrast, the mating call of a

bird that attracts a mate acts as a signal because it benefits both sender and receiver (Stegmann, 2013). A key way to distinguish between these is that signals are sent with the purpose of having an effect on another individual, which means they are more likely to be sent when they can be received. In the context of human interaction, signals are sent when another person is present (an audience effect) but should not be sent when a person acts alone. A stronger definition of explicit and deliberate signaling might require sending a signal repeatedly or elaborating on the signal until it is received. However, based on animal communication models (Stegmann, 2013), we will use a minimal definition of communication where signals are sent implicitly.

As described above, our eyes can act both as a cue to our current thoughts (e.g., if I am looking at my watch, I want to know the time) and as a signal to another person (e.g., I ostentatiously stare at my watch to signal to my friend that we must leave the party) (Argyle and Cook, 1976; Gobel et al., 2015; Risko et al., 2016). As Watzlavick's axiom "one cannot not communicate" (Watzlawick et al., 1967) suggests, even in a waiting room where two people are not intended to communicate and avoid engaging in eye contact, they are sending a signal that means "I do not want to interact with you" (Foulsham et al., 2011). This means that, in line with signaling theory, in face-to-face interactions our eye movements are constantly planned so as to send signals to others, and not just to gain information from the world. We propose that the signaling function of gaze creates a signaling map in the brain equivalent to the sensing map generated by the sensing function. In the same way that sensing maps show where to look to gain information, we hypothesize that signaling maps are computed in the brain to show where to look to send an appropriate signal to another person. In the following, we argue that the signaling map is computed by taking into account three key factors: communicative purpose, other's gaze direction, and coordination with other social signals.

First, the value of each gaze target in the signaling map will vary depending on the communicative purpose, that is, the type of message we wish to send. Just as saliency maps incorporate the task goal to create priority or sensing maps of visual attention, signaling maps need to take into account the communicative purpose. Imagine a waiting room with two people, where one person (A) wants to engage in an interaction, but the other person (B) does not. For person A, the optimal signaling behavior is to direct gaze to person B in order to send the message "I want to engage in an interaction with you." However, person B should avert gaze to efficiently signal "I do not want to interact with you." Thus, the signaling map will be different for person A and B, depending on the message they want to send.

Second, the signaling map will change according to the direction of the other person's gaze. The relationship between other's gaze direction and the signaling map lies in the fact that my signal will be received depending on whether the other person is gazing at us or not. Let's go back to the case of the waiting room with person A and B. For person A, who wishes to interact with person B, the optimal signaling behavior is to direct her gaze when person B is also looking at her, in order to disclose interest in the interaction. Directing her gaze when B is not looking has little benefit, because the signal will not be received. Equally, for person B the optimal signaling behavior is to avert gaze specifically when A is looking at her. This illustrates how the values associated with each location in the signaling map changes on a moment-by-moment basis, contingent on the gaze direction of the other person and in relation to communicative purpose.

Finally, the signaling map depends on the need to coordinate with other social signals that are sent in multimodal communication, such as speech or gestures (Vigliocco et al., 2014; Ho et al., 2015; Jack and Schyns, 2015; Hirai and Kanakogi, 2018; Holler et al., 2018; Trujillo et al., 2018). This is particularly relevant for explicit communicative encounters. Imagine that person A and B in the waiting room are now engaged in a lively conversation: to signal interest in keeping the conversation going, the choice of direct or averted gaze will vary depending on the role of each partner in the conversation, as well as the time-course of speech itself. For instance, when person A starts speaking, she may avert gaze every now and then to signal she still has more things to say (Kendon, 1967; Ho et al., 2015). While person B is listening, her gaze may be directed toward person A in order to signal interest in what A is saying (Kendon, 1967; Ho et al., 2015). However, when person A is finishing the utterance, she may look toward person B to signal that she can take the floor (Kendon, 1967; Ho et al., 2015). Thus, the coordination with other social signals also modulates the optimal location in the signaling map on a moment-by-moment basis.

Signaling theory provides a framework to understand how the communicative function of gaze shapes the planning of eye movements during face-to-face interactions. In the following, we propose a model where both active sensing and social signaling are combined to make sense of gaze patterns in human-tohuman communication.

# The Interpersonal Gaze Processing Model

The Interpersonal Gaze Processing model considers how gaze transitions from one state to the other (i.e., how eye movements are planned) when presented with social stimuli (**Figures 2**,**3**). This model distinguishes between two situations that differ in the belief in being watched: one where the social stimulus is a picture or video (i.e., cannot see us), and one where the social stimulus is a real person in front of us (i.e., can see us).

In the first case, where the stimulus is a picture or video of another person, there is no need to send a signal because it will not be perceived. Thus, the planning of eye movements only responds to active sensing, which aims to gain maximal information from the stimulus (Yang et al., 2016). The Interpersonal Gaze Processing model considers that gaze patterns derived from active sensing correspond to baseline gaze behavior. When the goal is to get social information from the picture or video (e.g., what is the man in the picture feeling?) gaze patterns will be mostly influenced by sensing maps (see **Figures 2**,**3A**). This baseline sensing map reveals how people use gaze to gain different types of social information during interactions. For example, in a noisy environment where it is hard to hear, they will look more to the center of the face to help with speech comprehension; conversely, to recognize emotions they

will look more to the eyes (Buchan et al., 2007, 2008; Lewkowicz and Hansen-Tift, 2012). This also demonstrates how task goals (e.g., speech comprehension or emotion recognition) translate in different eye movements depending on the information that needs to be maximized.

In the second case, where the stimulus is a real person in front of us, our eyes will be sending a signal to the other person. Here, the Interpersonal Gaze Processing model proposes that gaze patterns result from a trade-off between sensing maps and signaling maps (see **Figures 2**,**3B**). This means that the planning of eye movements combines the maximal gain of information from a particular location in the sensing map (e.g., eyes of the other person), together with the optimal benefit of gazing to that location in the signaling map. **Figure 4** illustrates how different possible gaze targets on the face of the man can provide various types of information to the woman (sensing function), but also can send different signals to the man (signaling function). Comparing baseline gaze behavior in a video to gaze behavior in a matched real-life interaction, can provide a measure of the signaling components of eye gaze. For example, some studies show that people direct gaze to the eyes of a stranger in a video, but not to the eyes of a live stranger: this indicates that averting gaze from the real person has a meaningful signaling value, since it expresses no desire to affiliate with the stranger and reduces the intensity of the interaction (Argyle and Dean, 1965; Foulsham et al., 2011; Laidlaw et al., 2011). This example considers the case of watching a stranger with a rather neutral face, but another interesting situation is that where partners show emotional facial expressions. Although this scenario has not yet been tested, it would give further insight on how sensing maps and signaling maps are integrated during gaze planning. Moreover, we acknowledge there may also be changes in arousal in association with being watched by a live person (Zajonc, 1965; Myllyneva and Hietanen, 2015; Lyyra et al., 2018), but these effects are not included in our model because of non-specific predictions on sensing and signaling maps.

Thus, the Interpersonal Gaze Processing model proposes that, moment-by-moment, the gaze control systems in the brain must evaluate both the information gained and the signaling potential of a saccade, to determine where to look next. This model and other theories of the audience effect (i.e., Watching Eyes model and reputation management theory) are linked because they are all modulated by the belief in being watched. The Watching Eyes model and reputation management theory explain how the presence of an observer modulates an individual's self- and other-focused cognitive processing, but they do not attempt to explain the dynamics of eye gaze in live communicative exchanges.

obtained with Saliency Toolbox for Matlab (Walther and Koch, 2006).


FIGURE 4 | Different sensing and signaling maps may be used in different contexts. Original image published by Max Pixel under the Creative Commons CC0 License. Original maps were obtained with Saliency Toolbox for Matlab (Walther and Koch, 2006).

By contrast, the Interpersonal Gaze Processing model places special emphasis on communicative purpose and coordination with other social signals (e.g., other's gaze direction, speech, facial expressions): while communicative purpose (together with the belief in being watched) is key to define the signaling map, the coordination with other social signals modulates this map on

a moment-by-moment basis. Future studies on gaze processing should try to elucidate how each of these factors modulates gaze sensing and signaling during communication, as well as if and how these maps are computed and integrated in the brain.

# GAZE PROCESSING IN AUTISM

Autism Spectrum Condition (ASC) is a developmental condition characterized by difficulties in interpersonal interaction and communication, as well as the presence of restricted and repetitive patterns of behavior (American Psychological Association, 2013). Since eye gaze has a critical role in regulating social interactions and enabling successful communicative exchanges, it is not surprising that the presence of abnormal gaze patterns is one of the most used diagnostic criteria for ASC from early infancy (Zwaigenbaum et al., 2005). Although research into gaze behavior in autistic adults has identified some general patterns, it has also yielded some inconsistent findings: some studies using pictures and videos suggest that they avoid looking at the eyes, whereas others indicate that they have typical gaze patterns (Falck-Ytter and Von Hofsten, 2011; Chita-Tegmark, 2016; Frazier et al., 2017). Some of these discrepancies may be a consequence of the wide spectrum in autistic individuals, but in line with the second-person neuroscience framework (Schilbach et al., 2013; Schilbach, 2016), it has been suggested that they could also be a consequence of the lack of experimental paradigms for studying eye gaze in real social interactions (Chevallier et al., 2015; Von dem Hagen and Bright, 2017; Drysdale et al., 2018). Moreover, a recent qualitative study highlights that self-declared autistic adolescents and adults struggle with the appropriate use and timing of eye gaze during face-to-face interactions (Trevisan et al., 2017). These findings suggest that to fully understand autistic social cognition it is necessary to examine how they process social signals in real dynamic interactions.

# Audience Effects in Autism

We have previously presented two distinct cognitive theories to explain audience effects: the Watching Eyes model (Conty et al., 2016) and reputation management theory (Emler, 1990; Resnick et al., 2006; Tennie et al., 2010). Both theories involve mentalizing and distinction between self-beliefs and otherbeliefs, either to process the perceptual state of the observer (Teufel et al., 2010b) or to further infer what the observer thinks of us (Izuma et al., 2010; Cage, 2015). This means that mentalizing is a key cognitive component of audience effects (Hamilton and Lind, 2016). Difficulties in processing mental states of others is one of the hallmarks of autism: they have trouble inferring beliefs and intentions of other people (Happé,, 1994; White et al., 2009), as well as attributing a social meaning to eye gaze (Baron-Cohen et al., 1997), especially when they need to do so spontaneously (Senju et al., 2009). Thus, impaired mentalizing in autistic people implies that being watched will elicit less self-related processing and reputation management, and they will show reduced audience effects (Hamilton and Lind, 2016).

To our knowledge, no studies have directly tested the Watching Eyes model on autistic individuals, but instead have looked at differences in self-referential processing between typical and autistic populations. Lombardo et al. (2007) used a task measuring self-referential memory and found that high-functioning autistic individuals as well as with Asperger Syndrome (two similar subgroups within the autism spectrum) had smaller self-referential bias compared to typical individuals. Moreover, from early infancy autistic individuals' show reduced orienting to their name, which is a salient stimulus uniquely related to oneself (Werner et al., 2000; Nadig et al., 2010). These studies suggest that autistic people have a general impairment in processing self-related information as distinct from otherrelated information, already when they are in a non-interactive environment. Interestingly, it has recently been suggested that autistic people might have a narrower cone of direct gaze (i.e., the range of gaze directions that an individual judges as being directed to oneself), which means that they might be less likely to perceive that an observer is watching them (Gianotti et al., 2018). Thus, a plausible prediction is that autistic individuals will fail to process self-relevant signals in interactive environments, such as the belief in being watched (Conty et al., 2016). Studies directly testing effects of being watched on self-referential processing will be needed to clarify this question.

In contrast, a body of research has investigated reputation management in autism. Using the donation task, it has been found that the frequency of donations of autistic participants is not affected by the presence or absence of a confederate who is watching them (Izuma et al., 2011; Cage et al., 2013). It is worth noting that Izuma et al. (2011) found a social facilitation effect in autistic participants on a perceptual task, which indicates that autistic people have specific difficulties with reputation management processes. Cage et al. (2013) further showed that, while typical participants donated more frequently when the observer could reciprocate, autistic participants had reduced expectation of reciprocity. Moreover, autistic children do not engage in flattery behavior toward others (Chevallier et al., 2012) and do not use strategic self-promotion when describing themselves in front of an audience (Scheeren et al., 2010). These findings demonstrate that autistic people are less inclined to manipulate beliefs of observers to maintain their reputation, either due to mentalizing impairments (Frith, 2012) or to social motivation deficits (Chevallier et al., 2013).

However, it is not clear how social norms of eye gaze (i.e., civil inattention; (Goffman, 1963) are implemented in autism, since no study has directly contrasted gaze patterns of autistic individuals in live versus pre-recorded non-communicative interactions. A study by von dem Hagen and colleagues approached this question in typical individuals with high and low autistic traits (Von dem Hagen and Bright, 2017, Experiment 1). Participants were shown videos of a confederate and were deceived to believe that the videos were either pre-recorded or a live video-feed. They found that people with low autistic traits decreased the amount of gaze directed to the face of the confederate in the live video-feed condition, but no reduction was found in the group with higher autism traits. This finding indicates some degree of insensitivity to the belief in being watched and, consequently, to

social norms associated with social behavior toward strangers. However, it remains to be seen whether these findings are true for individuals with an ASC diagnosis.

# Interpersonal Dynamics of Gaze in Autism

Few studies have looked at how gaze patterns differ between typical and autistic groups during interactions with communicative purpose, and the evidence is mixed. For instance, when asked to actively engage in an interaction (QandA task) over a video-feed, individuals with high autistic traits looked less toward the face of the confederate than individuals with low autistic traits (Von dem Hagen and Bright, 2017, Experiment 2). Using a similar QandA task in a face-to-face interaction, it was found that high amount of autistic traits was not associated to reduced looking to the face, but to reduced visual exploration (Vabalas and Freeth, 2016). However, in a study testing a sample with autism diagnosis, no differences in visual exploration were found between typical and autistic groups (Freeth and Bugembe, 2018). It is worth noting that in all these studies they found no between-group differences in gaze patterns during speaking and listening periods (i.e., typical/low autistic traits and high autistic traits behave equally), which suggests that to some extent social functions of gaze are preserved in autism (e.g., regulating turn-taking during conversation).

We previously argued that in communicative encounters (direct) eye gaze needs to coordinate with other verbal and non-verbal signals, within and between agents, to successfully exchange information (Bavelas et al., 2002; Trujillo et al., 2018). Several studies indicate that autistic individuals do not use direct gaze as a signal to coordinate intra- and inter-personal social behavior in the same way that typical participants do. Using non-interactive stimuli, it has been shown that autistic adults do not follow gaze after eye contact as much as typical participants (Böckler et al., 2014). Moreover, while in typical individuals direct gaze reduces reaction times to generate an action (Schilbach et al., 2012) or to mimic an action (Forbes et al., 2017), this effect is not found in ASC. Similarly, when participants interact with a virtual avatar that displays contingent gaze patterns, autistic children show less gaze following (Little et al., 2017) and individuals with high autistic traits engage in less facial mimicry following joint attention than individuals with low autistic traits (Neufeld et al., 2016). These findings suggest that reduced coordination between eye gaze and other social behavior may have an impact on the successful progression of the interaction.

A reason why autistic people show poor coordination of social behavior could stem from difficulties in appropriately adjusting gaze to the dynamics of communication. It has been found that infants at high risk for ASC alternate less between initiating and responding to joint attention compared to infants at low risk (Thorup et al., 2018), and that they preferentially orient toward a person that always responds in the same way over a person that can show variable responses (Vernetti et al., 2017). This means that, since early infancy, individuals at high risk for ASC experience less dynamic social contexts and less variety in gaze-contingent events. Using a gaze-contingent eye-tracking paradigm with virtual avatars, Caruana et al. (2017) have found that autistic adults are less accurate and take a longer time than typical adults to respond to joint attention. In line with this, Freeth and Bugembe (2018) have found that when a confederate directly gazes at participants during a QandA task, autistic adults look less at the confederate's face than typical adults. These findings suggest that difficulties in adjusting eye contact make it hard for autistic individuals to keep pace with rapid and spontaneous face-to-face interactions.

It has been suggested that a lack of exposure to contingent eye gaze in infancy can impact the specialization of brain areas related to gaze processing (Vernetti et al., 2018). Indeed, a study using live video-feed found that some regions in the social neural network (superior temporal sulcus and dorsomedial prefrontal cortex) are equally engaged during periods of joint attention and periods of no joint attention in ASC (Redcay et al., 2012). This is corroborated by previous studies using non-interactive stimuli, where they found abnormal activation of the social neural network (e.g., superior temporal sulcus, right temporo-parietal junction) when autistic adults processed social information conveyed by eye gaze (Pelphrey et al., 2005; Zilbovicius et al., 2006; Philip et al., 2012; Georgescu et al., 2013). Moreover, a hyperscanning study using live video-feed (Tanabe et al., 2012) found that inter-brain coherence (in frontal regions) during eye contact was lower in autistic-typical dyads compared to typicaltypical dyads, which might reflect difficulties in processing and integrating social signals in ASC. Thus, these studies suggest that atypical intra- and inter-individual patterns of neural activity in response to direct gaze may underlie difficulties in detecting, processing and sending social signals in autism.

Overall, these findings indicate that autistic individuals have difficulties with social dynamics of gaze in real interactions. However, current research is not enough to clearly distinguish which cognitive components of eye gaze processing are most disrupted in autism. In this sense, the Interpersonal Gaze Processing model (**Figures 2**,**3**) provides common ground where studies manipulating various gaze-related factors can come together. We previously suggested that comparing gaze patterns in a video versus a matched real-life interaction provides a measure of the signaling components of eye gaze. If autistic people do not engage in social signaling, the Interpersonal Gaze Processing model predicts that their gaze patterns in live and video conditions should be similar, which is in line with recent evidence (Von dem Hagen and Bright, 2017). Future research should try to systematically study which factors modulating gaze signaling make interpersonal gaze processing challenging in autism.

# CONCLUSION

Natural social interactions are characterized by complex exchanges of social signals, so achieving successful communication can be challenging. This paper aimed to review research manipulating three key factors that modulate eye gaze processing during social interactions: the presence of an interacting partner who can perceive me, the existence of

communicative purpose, and the development of interpersonal and temporal dynamics.

Current findings indicate that the belief in being watched has a strong impact on other-focused social cognition (both on prosocial behavior and social norms of eye gaze), but evidence is less clear for self-focused cognition: future studies should clarify to what extent being watched affects different forms of self-related processes. We also find that, to achieve successful communication, eye gaze needs to coordinate with verbal and non-verbal social signals, both within and between interacting partners. We propose the Interpersonal Gaze Processing model as a framework where gaze sensing and signaling are combined to determine where the eyes will look next in a live interaction. In this model, the belief in being watched and the communicative purpose of the interaction are key to define the gaze signaling map, while the contingencies between different signaling modalities (e.g., gaze, speech) are critical in changing this map on a moment-by-moment basis. Systematic manipulation of these factors could help elucidate how they relate to each other to enable successful communicative encounters, as well as how signaling maps are computed in the brain.

Finally, research on autistic individuals reveals that they are less sensitive to the belief in being watched, but more

### REFERENCES


studies are needed to clarify how the presence of an audience impacts self-related processing in autism. Although evidence on interpersonal dynamics is mixed, it is agreed that autistic individuals have difficulties with social dynamics of eye gaze during real interactions. We argue that the Interpersonal Gaze Processing model provides a framework for future studies to systematically characterize which aspects of gaze communication are most challenging for autistic people.

### AUTHOR CONTRIBUTIONS

RC wrote the initial draft of the manuscript and prepared the figures. AH made critical revisions to the original draft. RC and AH approved the final version of the manuscript.

# FUNDING

This work was supported by University College London Open Access Team. RC acknowledges financial support from "la Caixa" Foundation (ID 100010434, Grant code LCF/BQ/EU16/11560039).






**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Cañigueral and Hamilton. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Gaze Cuing Effects in Peripheral Vision

#### *Takemasa Yokoyama1 \* and Yuji Takeda1,2*

*1 Automotive Human Factors Research Center, National Institute of Advanced Industrial Science and Technology, Tsukuba, Japan, 2 Faculty of Human Science, University of Tsukuba, Tsukuba, Japan*

When we see another person's gaze, spatial attention shifts toward the gaze direction. Thus, a gaze perceiver can more quickly respond to a forthcoming target when it appears in a direction of a gaze giver than when it does not. This phenomenon is termed the *gaze cuing effect*. Previous studies have investigated the gaze cuing effect only in foveal vision; hence, it remains unclear whether the gaze cuing effect is induced when a face is presented in peripheral vision. This is an important issue because in our daily lives we communicate not only with people in front of us but also with those in our periphery. To tackle this question, we manipulated vertically aligned locations of a facial stimulus (i.e., a face stimulus appeared above or below the center fixation) and tested the extent to which a gaze cuing effect, conveyed by gaze shifts of another, is observed in the periphery. The facial stimulus was located 0, ±2.5, ±5.0, and ±7.5° of the visual angle from the center of the display, and a target was presented 5.6° to the left or right of the center of the display. In Experiment 1, when participants responded to the location of an abrupt onset of a target (i.e., localization task), we observed significant gaze cuing effects when a facial stimulus was located 0, ±2.5, and ±5.0°, but not ±7.5°. In Experiment 2, we replicated the findings in Experiment 1 if participants pressed a key only when a target appeared (i.e., detection task). In Experiment 3, we used adjusted sizes of facial images based on the cortical representations and manipulated eye directions of the facial images oriented toward the possible target locations; it resulted in enlarged effective field of view for gaze cuing effects. The study reveals that gaze cuing effects can appear even in peripheral vision and within a vertical distance of 5.0° of the visual angles, but the effective field of view is expanded when the facial image is adjusted based on the cortical representations, and eye gaze directly looks at the possible target locations.

Keywords: gaze cuing effect, peripheral vision, attention, gaze perception, visual angle

# INTRODUCTION

People receive a wealth of social information from the gaze of others, information that we use to facilitate social interactions. Furthermore, people can infer what others favor and where their interest lies. Therefore, people habitually pay attention to another's gaze in order to anticipate the mental state of a gaze givers, i.e., his/her thoughts and desires (Baron-Cohen, 1995). Because the information from the gaze of another person allows us to determine appropriate

#### *Edited by:*

*Chris Ashwin, University of Bath, United Kingdom*

#### *Reviewed by:*

*Colin W. G. Clifford, University of New South Wales, Australia Keith Schneider, University of Delaware, United States*

#### *\*Correspondence:*

*Takemasa Yokoyama yokoyama@lit.kobe-u.ac.jp; yokoyama.takemasa@aist.go.jp*

#### *Specialty section:*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology*

*Received: 12 October 2018 Accepted: 13 March 2019 Published: 04 April 2019*

#### *Citation:*

*Yokoyama T and Takeda Y (2019) Gaze Cuing Effects in Peripheral Vision. Front. Psychol. 10:708. doi: 10.3389/fpsyg.2019.00708*

**99**

behavior in relation to the gaze giver, such social abilities facilitate social interactions. Thus, gaze perception plays a critical role in social situations.

Gaze perception facilitates cognitive processes, such as attention as well as social interactions. For example, if the gaze direction of another indicates a rightward direction, then the spatial attention of the gaze perceiver typically shifts to align with the gaze direction of the gaze giver (Friesen and Kingstone, 1998; Driver et al., 1999; Langton et al., 2000). This phenomenon is termed a gaze cuing effect (Frischen et al., 2007; Birmingham and Kingstone, 2009). To examine this phenomenon, a modified Posner paradigm (i.e., gaze cuing paradigm) is usually enlisted. Thus, if a target is presented in the direction indicated by gaze, participants can make a response to the target more quickly. The gaze cuing effect is induced even when a gaze direction does not accurately predict a target location (Friesen and Kingstone, 1998). In addition, this automatic attentional orienting is intact even when other functions, such as working memory, also consume resources for spatial representations (Yokoyama et al., 2019). Because the gaze of another individual modulates automatic attentional orienting despite adverse situations, the gaze cue is a powerful visual stimulus to modulate spatial attention.

Although there are numerous studies of gaze perception involving foveal vision, very little is known about gaze perception involving peripheral vision. In a gaze discrimination task, Loomis et al. (2008) manipulated locations of a facial stimulus and found that participants could discriminate gaze directions of the facial stimulus to the extent of 4° visual angles. Also, studies of Palanica and Itier (2014, 2017) revealed that accurate discrimination of gaze directions (e.g., left, right, up, down) covers up to 6° of horizontal and vertical eccentricities. In general, peripheral vision decreases spatial resolution of a visual stimulus (Chung et al., 1998; Nasanen and O'Leary, 1998), but Florey et al. (2015) found that poor gaze discrimination in peripheral vision is not the result of decreased spatial resolution, but rather it is due to other factors, such as crowding and prior information for gaze or head directions. In our previous study, participants could discriminate between direct and averted gaze, but not between leftward and rightward gaze, when their attention was allocated to the central letter discrimination task (Yokoyama et al., 2014). Although some studies have addressed gaze discrimination in peripheral vision, it remains unclear whether a gaze cuing effect occurs beyond foveal vision. When humans interact within a group, they must rely on peripheral vision for perceiving gaze of others, i.e., all group members are not always directly in front of a viewer. Hence, it is conceivable that the human visual system enables accurate gaze discrimination in peripheral vision. Similar to gaze discrimination, gaze cuing effects are also important in social interactions in which people provide information on future desires and interests through their eye movements.

The aim of this study was to ascertain whether gaze cuing effects would occur beyond foveal vision. To tackle this issue, we quantitatively manipulated location eccentricities of a facial

Yokoyama and Takeda Gaze Cuing Effects

image to measure the gaze cuing effect. Because a target appears at the side of a facial stimulus in a gaze cuing task (Friesen and Kingstone, 1998; Driver et al., 1999; Yokoyama et al., 2012), we manipulated vertical eccentricities. We prepared ±7.5, ±5.0, ±2.5°, or 0° (the center) from the center of the display (+ means upward; − means downward), and the facial stimulus was located at one of the seven possible locations. In Experiment 1, we conducted a localization task; participants pressed one of two keys corresponding to the location of an abrupt onset of a target. In Experiment 2, we conducted a detection task; in this case, participants pressed a key only when a target appeared. The latter aimed to replicate the Experiment 1 findings without contamination from the stimulus-response compatibility effect. In Experiment 3, we examined two additional factors: the effective field of view regarding the gaze cuing effect and the effect of accurate gaze directions to a target. To this end, the facial stimuli were adjusted based on the cortical representations, and eye gaze of the face stimuli directly looked at the possible target location.

## EXPERIMENT 1

# Method

#### Participants

Twenty-three paid volunteers (12 female, age range: 18–32) participated in Experiment 1. They provided written informed consent, as approved by the institutional review board of the National Institute of Advanced Industrial Science and Technology. All participants had normal or corrected-to-normal acuity of vision, and all were naïve to the purposes of this experiment.

#### Apparatus

Visual stimuli were displayed on the ASUS ROG SWIFT PG258Q LCD display of 1,920 × 1,080 pixels (refresh rate was 120 Hz). Visual display and data collection were controlled using the Psychophysics Toolbox of MATLAB (Brainard, 1997; Pelli, 1997) on Microsoft Windows 7. Participants were tested individually in a darkened room, and the viewing distance was approximately 57 cm.

# Visual Stimuli and Procedure

We used a schematic face (2.3° × 2.3°); an example of the stimulus is shown in **Figure 1**. The central fixation was presented in 500 ± 100 ms, followed by a schematic face with direct gaze lasting 700 ms. Positions of the facial stimulus were located at ±7.5, ±5.0, ±2.5°, or 0° from the center of the display (0° was display center, and plus refers a locus above center with a minus signal representing below this center). Because eyes were positioned at a top of a facial image, those positions (±7.5, ±5.0, ±2.5, and 0°) matched the eye position. After eye gaze of the facial stimulus indicated a left or right for 300 ms (i.e., gaze cue), a target Gabor patch (size: 1.4° × 1.4°, spatial

frequency: 3.2 c/d, Michelson contrast: 0.625, orientation: vertical) was presented 5.6° to the left or right of the center of the display until a response was made. Gaze direction of the face did not predict the target location, and participants were instructed about the manipulation before starting the experiment. Participants pressed the 1 (left) or 2 (right) key with their right hand as quickly and accurately as possible to judge the target location (i.e., a localization task) while they tried to ignore the gaze direction of the facial stimulus. Congruent (gaze direction and target location were identical) and incongruent (gaze direction and target location differed) conditions were randomly intermixed in a block. Also, the seven locations of the facial stimulus were randomly intermixed in a block. Twelve blocks of 84 trials were employed [total 1,008 trials: 72 trials × congruency (2) × location (7)], preceded by 14 practice trials.

#### Results

Error trials were removed from further analysis (Total: 1.56%, congruent trials: 1.17%, incongruent trials: 1.95%). We used the median of collected reaction time data to examine gaze cuing effects as computed for every participant in each condition from the remaining data (Driver et al., 1999; Ristic et al., 2002; Burton et al., 2009; Ivanoff and Saoud, 2009; Marotta et al., 2013; Bobak and Langton, 2015) and computed for every participant in each condition from the remaining data. **Figure 2A** shows RTs in the gaze cuing task, and **Figure 2B** shows gaze cuing effects (RTs in the incongruent-congruent conditions) on RTs. We performed 2 *Congruency* (congruent, incongruent) × 7 *Location* (+7.5, +5.0, 2.5, 0, −2.5, −5.0, −7.5°) repeated measures analysis of variance (ANOVA). We found the main effect of *Congruency* [*F*(1, 22) = 30.971, *p* < 0.001, h*p* <sup>2</sup> = 0.421] and *Location* [*F*(1, 22) = 5.076, *p* < 0.001, h*p* <sup>2</sup> = 0.203]. There was also a significant interaction between *Congruency* and *Location* [*F*(1, 22) = 5.037, *p* < 0.001, h*p* <sup>2</sup> = 0.213]. To further assess the interaction between *Congruency* and *Location*, a simple main effect analysis was performed with a Bonferroni correction. This yielded significant differences between congruent and incongruent conditions in +5.0° (*t*22 = 2.272, *p* < 0.05, *d* = 0.16), +2.5° (*t*22 = 5.579, *p* < 0.001, *d* = 0.40), 0° (*t*22 = 3.36, *p* < 0.005, *d* = 0.36), −2.5° (*t*22 = 4.011, *p* < 0.001, *d* = 0.34), and −5° conditions (*t*22 = 3.721, *p* < 0.005, *d* = 0.21). In addition, we found a simple main effect of *Location* in the congruent condition [*F*(1, 6) = 11.667, *p* < 0.001, h*p* <sup>2</sup> = 0.347] but not in the incongruent conditions [*F*(1, 6) = 0.458, *p* = 0.839, h*p* <sup>2</sup> = 0.020].

#### Discussion

We found that the congruent RTs were significantly faster than the incongruent RTs (i.e., consistent with the gaze cuing effect) beyond foveal vision. In addition, the gaze cuing effect persisted when a face was located a ±5.0° visual angle. Furthermore, we found face location effects only in the congruent condition.

In Experiment 1, we conducted the localization task in which participants were required to press one of two keys

corresponding to the target location. Because cue directions and responses were consistent in the congruent condition, the stimulus-response compatibility might have been confounded in the results in Experiment 1. To rule out the possibility, in Experiment 2, we used a detection task in which participants were required to press a key only when a target appeared irrespective of the target location.

## EXPERIMENT 2

#### Method

The method in Experiment 2 was identical to Experiment 1, with the exception of the following details.

#### Participants and Procedure

Twenty-one paid volunteers (8 female, age range: 20–35) participated in Experiment 2. We used the detection task in Experiment 2, in which catch trials (no target trials) were included. Each catch trial ended 1 s after eye gaze moved to left/right. A total of 1,134 trials included 126 catch trials (about 11.1%). Participants were instructed to press the space key only when the target Gabor was presented. If participants did not press a key in 1 s after a target appeared, such trials were considered miss trials.

#### Results

Miss trials were removed from further analyses (Total: 4.82%, congruent trials: 5.02%, incongruent trials: 4.61%). False alarm rates in the catch trials were 8.6%. **Figure 3A** shows RTs in the gaze cuing task, and **Figure 3B** shows gaze cuing effects (RTs in the incongruent-congruent conditions) on RTs. We performed a repeated measures ANOVA based on a 2 *Congruency* (congruent, incongruent) × 7 *Location* (+7.5, +5.0, 2.5, 0, −2.5, −5.0, −7.5°) design. The main effects of both *Congruency* [*F*(1, 20) = 24.032, *p* < 0.001, h*p* <sup>2</sup> = 0.545] and *Location* [*F*(1, 20) = 9.115, *p* < 0.001, h*p* <sup>2</sup> = 0.313] emerged. Also, a significant interaction between *Congruency* and *Location* [*F*(1, 20) = 2.561, *p* < 0.05, h*p* <sup>2</sup> = 0.113] was evident. To further assess this interaction between *Congruency* and *Location*, a simple main effect analysis was performed with a Bonferroni correction. This analysis yielded significant differences between congruent and incongruent conditions in the +5.0° (*t*20 = 2.635, *p* < 0.05, *d* = 0.24), +2.5° (*t*20 = 2.821, *p* < 0.05, *d* = 0.37), 0° (*t*20 = 2.841, *p* < 0.05, *d* = 0.29), −2.5° (*t*20 = 2.831, *p* < 0.05, *d* = 0.28), and −5° conditions

(*t*20 = 2.165, *p* < 0.05, *d* = 0.23). Furthermore, we found a simple main effect of *Location* in the congruent condition [*F*(1, 6) = 8.535, *p* < 0.001, h*p* <sup>2</sup> = 0.299] and incongruent conditions [*F*(1, 6) = 3.104, *p* < 0.01, h*p* <sup>2</sup> = 0.13].

RT-congruent RT). The error bars represent standard error of mean.

#### Discussion

In Experiment 2, using the detection task, we observed results that were qualitatively similar to results in Experiment 1. Thus, the results in Experiment 1 should not be attributed to the occurrence of stimulus-response compatibility.

In Experiments 1 and 2, we did not find gaze cuing effects in the ±7.5° conditions. Considering the effective field of view regarding gaze cuing effects, two factors should be examined: one involves spatial resolutions in peripheral vision, and the other concerns gaps between gaze directions of the facial image and target locations. Spatial resolutions decrease as eccentricities increase (Rosenholtz, 2016), and the cortical representations are larger in foveal than in peripheral vision (Rovamo and Virsu, 1979). Therefore, if the size of the facial image is adjusted to reflect cortical representations, then the effective field of view regarding gaze cuing effects might be enlarged for a participant. Concerning gaps between gaze directions and target locations, the eyes of facial images in Experiments 1 and 2 were not oriented toward possible target locations precisely when the facial image was presented in peripherally. For example, although a target appeared at an obliquely downward (upward) location from a face image in the congruent with +7.5° (−7.5°) condition, the eye gaze of the image was directed to horizontal locations. Thus, if the facial image accurately looked at the possible target location, gaze cuing effects might be stronger. In Experiment 3, we manipulated the image size and gaze directions of the facial images based on eccentricities. The goal was to assess whether the effective field of views regarding gaze cuing effects would be enlarged.

# EXPERIMENT 3

We conducted two experiments in Experiment 3. These involved localization and detection tasks in Experiments 3a and 3b, respectively.

#### Method

The method in Experiments 3a (localization task) and 3b (detection task) was identical to Experiments 1 and 2, respectively, with the exception of the following details.

#### Participants and Visual Stimuli

Nineteen paid volunteers participated in Experiment 3a (6 female, age range: 20–33); the same number of paid volunteers participated in Experiment 3b (10 female, age range: 20–29). We calculated the M (a cortical magnification factor) value based on the study of Rovamo and Virsu (1979) to adjust the facial images in peripheral vision. As a result, size of the facial images was 4.08° × 4.08°, 5.87° × 5.87°, and 7.68° × 7.68° in the ±2.5, ±5.0, and ±7.5° conditions, respectively (the image size of the 0° condition was 2.3° × 2.3°). In addition, eye gaze of the facial images was oriented toward the possible target locations in all location conditions (**Figure 4**).

FIGURE 4 | Examples of facial stimuli in Experiment 3. Eyes of the facial image accurately looked at the possible target locations. (A) An example of the +2.5° condition. (B) An example of the +5.0° condition. (C) An example of the +7.5° condition.

# Results

#### Experiment 3a (Localization Task)

Error trials were removed from further analysis (Total: 2.56%, congruent trials: 1.85%, incongruent trials: 3.27%). **Figure 5A** shows RTs in the gaze cuing task, and **Figure 5B** shows gaze cuing effects (RTs in the incongruent-congruent conditions) on RTs. We performed 2 *Congruency* (congruent, incongruent) × 7 *Location* (+7.5, +5.0, 2.5, 0, −2.5, −5.0, −7.5°) repeated measures ANOVA. We found the main effect of *Congruency* [*F*(1, 18) = 31.036, *p* < 0.001, h*p* <sup>2</sup> = 0.633]. However, we did not observe a main effect of *Location* [*F*(1, 18) = 1.862, *p* = 0.094, h*p* <sup>2</sup> = 0.093]; also, no significant interaction between *Congruency* and *Location* was observed [*F*(1, 18) = 0.790, *p* = 0.579, h*p* <sup>2</sup> = 0.042].

#### Experiment 3b (Detection Task)

Miss trials were removed from further analysis (Total: 1.38%, congruent trials: 1.54%, incongruent trials: 1.23%). False alarm rates in the catch trials were 6.3%. **Figure 6A** shows RTs in the gaze cuing task, and **Figure 6B** shows gaze cuing effects (RTs in the incongruent-congruent conditions) on RTs. We performed a repeated measures ANOVA with 2 *Congruency*

(congruent, incongruent) × 7 *Location* (+7.5, +5.0, +2.5, 0, −2.5, −5.0, −7.5°). We found a significant main effect of *Congruency* [*F*(1, 18) = 53.369, *p* < 0.001, h*p* <sup>2</sup> = 0.748]. However, a main effect of *Location* was not significant [*F*(1, 18) = 0.712, *p* = 0.640, h*p* <sup>2</sup> = 0.038]; also, no significant interaction between *Congruency* and *Location* was observed [*F*(1, 18) = 0.936, *p* = 0.472, h*p* <sup>2</sup> = 0.049].

#### Discussion

In Experiment 3, we manipulated size of the facial images based on the cortical representations and accurate gaze directions of facial images. In the both experiments, we observed gaze cuing effects in all the location conditions. However, we did not find interaction between congruency and location that we observed in Experiments 1 and 2. Hence, our manipulations in Experiment 3 enlarged the effective field of view regarding gaze cuing effects.

### GENERAL DISCUSSION

The present study provides the first evidence of gaze cuing effects in peripheral vision. We also found that the gaze

cuing effects persist up to ±5° of vertical eccentricity when we did not manipulate size of facial stimuli and gaze directions (eye gaze directed horizontally), regardless of eccentricities (Experiments 1 and 2). Previous studies have indicated that accurate gaze discrimination is up to 4–6° of the visual angles (Loomis et al., 2008; Palanica and Itier, 2014, 2017). Note that, in vertical eccentricity, Palanica and Itier (2017) reported that accurate gaze discrimination can occur up to ±6° of the visual angle. Our results are in a range of these findings in Palanica and Itier (2017). Thus, gaze perception, including gaze cuing effects, should occur up to approximately ±6° of vertical eccentricities (but see Florey et al., 2015).

Based on the results in Experiments 1 and 2, peripheral vision is more likely to influence the congruent condition rather than the incongruent condition. We observed a simple main effect of *Location* in the congruent condition in both Experiments 1 and 2. Although we observed a simple main effect of *Location* in the incongruent conditions in Experiment 2 (but not in Experiment 1), the effect size of the incongruent condition was much smaller than that in the congruent condition (congruent condition: h*p* <sup>2</sup> = 0.299, incongruent condition: h*p* <sup>2</sup> = 0.13). Hence, periphery appears to impact attentional benefits (rather than attentional costs) by perception of gaze direction. Although gaze cuing effects exist in the peripheral vision, attentional benefits gained by perception of gaze direction are stronger in the foveal vision than in peripheral vision.

If image size changed corresponding to the cortical representations and its eyes, i.e., in a relevant facial image, are appropriately oriented toward the possible target location (Experiment 3), then the effective field of view regarding gaze cuing effects is enlarged. In addition, because we did not find an interaction between congruency and face locations in the experiment, it is plausible that certain eccentricities, namely those with visual angles up to ±7.5°, do not influence gaze cuing effects given such manipulations. It should be noted that Florey et al. (2015) have proposed that reduced spatial resolution in peripheral vision might not lead to a decrease

#### REFERENCES


of accuracy in a gaze discrimination task when participants are required to identify the gaze directions of face stimuli. This may be inconsistent with the present findings; that is, manipulation of face size in peripheral conditions could enlarge the effective field of view regarding gaze cuing effects. Although speculative, this inconsistency may be caused by the differences in cognitive processes. When gaze cuing effects are induced, participants have to perceive gaze direction of the facial image and then shift their attention to the location implicated the eye gaze. On the other hand, the gaze discrimination task required only the perception of gaze directions. Thus, it is possible that the lack of a gaze cuing effect in the ±7.5° condition of Experiments 1 and 2 is caused by a dysfunction of attentional shifts induced by lower accuracy of face/gaze representations. Further studies are needed to clarify this issue.

In conclusion, we have shown gaze cuing effects exist outside foveal vision. Gaze perception plays an important role in social interactions, and when we interact within a group, gaze perception beyond foveal vision is critical. Gaze cuing effects occur in periphery, and this social ability helps us to engage in smooth social interactions with others.

#### AUTHOR CONTRIBUTIONS

TY conceived the experiments. TY and YT designed the experiments. TY performed the experiments. TY and YT analyzed the data, contributed reagents, materials and analysis tools, and wrote the main manuscript text. TY prepared figures.

#### FUNDING

This study was supported by a grant from the JSPS Grant-in-Aid for Young Scientists [18K18350] to TY and from the National Institute of Advanced Industrial Science and Technology to YT [AAZ202761C1].

normal peripheral vision. *Vis. Res.* 38, 2949–2962. doi: 10.1016/ S0042-6989(98)00072-8


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Yokoyama and Takeda. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Turning Away From Averted Gazes: The Effect of Social Exclusion on Gaze Cueing

*Roberta Capellini 1 \* , Paolo Riva1,2 , Paola Ricciardelli1,2 and Simona Sacchi 1*

*1 Department of Psychology, University of Milano-Bicocca, Milan, Italy, 2NeuroMI, Milan Center for Neuroscience, Milan, Italy*

Past studies showed increased sensitivity to other people's gaze after social exclusion. In the present research, across two studies, we tested whether social exclusion could affect the basic cognitive phenomenon of gaze-cueing effect, namely, the tendency to redirect visual attention to the same location that other people are looking at. To this purpose, participants were socially excluded or included using the *Cyberball* manipulation. In Study 1, after the manipulation, participants performed a gaze-cueing task in which an individual's gaze, oriented rightward or leftward, preceded a peripheral target stimulus requiring a simple categorization response. The gaze direction could be congruent or incongruent with the location of the target. Results revealed a reduced gaze-cueing effect for socially excluded than for socially included participants. In Study 2, where human gazes were replaced by arrow cues, such an interaction between social exclusion and trial congruency disappeared, indicating a specific effect of social stimuli. We interpreted these findings with the notion that excluded participants can perceive an averted gaze as a further sign of social exclusion, thus showing a reduced gaze-cueing effect.

#### *Edited by:*

*Britt Anderson, University of Waterloo, Canada*

#### *Reviewed by:*

*Mario Dalmaso, University of Padua, Italy Dirk Kerzel, Université de Genève, Switzerland*

#### *\*Correspondence:*

*Roberta Capellini r.capellini@campus.unimib.it; roberta.capellini@unimib.it*

#### *Specialty section:*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology*

*Received: 05 November 2018 Accepted: 15 April 2019 Published: 16 May 2019*

#### *Citation:*

*Capellini R, Riva P, Ricciardelli P and Sacchi S (2019) Turning Away From Averted Gazes: The Effect of Social Exclusion on Gaze Cueing. Front. Psychol. 10:1000. doi: 10.3389/fpsyg.2019.01000*

Keywords: social exclusion, averted gaze, social attention, reaffiliation, cueing effect

Social exclusion has been defined as the experience of being kept apart from others physically (e.g., social isolation) or emotionally (e.g., being ignored or told one is not wanted; Riva and Eck, 2016). As a common phenomenon, social exclusion can take many forms, such as ostracism (Williams, 2007), rejection (e.g., Maner et al., 2007), and discrimination (e.g., Richman et al., 2016). In our everyday life, the experience of being socially excluded can occur in many occasions; from ostracism on the playground to being bullied in the classroom, discriminated at the workplace, and isolated in the later stage of life.

Considering that social exclusion threatens a fundamental psychological need (i.e., the need to belong; Baumeister and Leary, 1995), it has a profound impact on the human mind, affecting the emotional, behavioral, and cognitive levels (Williams, 2009; Riva and Eck, 2016; for a meta-analysis, see Gerber and Wheeler, 2009). From an emotional perspective, instances of social exclusion have been linked with negative affect, including a cluster of emotions such as sadness, anger, anxiety, and feelings of depression (Buckley et al., 2004; Riva et al., 2017). In behavioral terms, previous research suggested that excluded individuals are more prone to self-defeating behaviors including aggression (Twenge et al., 2001), risk-taking (Twenge et al., 2002; Peake et al., 2013; Svetieva et al., 2016; Buelow and Wirth, 2017), and gambling (Pancani et al., 2018). At a cognitive level, social exclusion negatively influences performance on intelligence tests (Baumeister et al., 2002). Moreover, it has been shown that social exclusion can undermine self-regulation abilities (Baumeister et al., 2005; Stenseng et al., 2015) and impair inhibitory control (Leary et al., 2006). Since the self-regulation of exclusion-related distress deploys attentional resources (Chester and DeWall, 2014; Riva et al., 2014a) leaving limited resources for effective inhibitory control (Lurquin et al., 2014), it could lead to more impulsive or prepotent behaviors. For instance, social exclusion can increase aggression and decrease prosocial behavior (Twenge et al., 2001, 2007) and foster consumption of chocolate chip cookies (Baumeister et al., 2005). Importantly, social exclusion seems to affect not only late-stage cognitive processes but also early-stage cognitive processes, such as selective attention (Xu et al., 2017), which represents an important component of inhibitory control (Friedman and Miyake, 2004; Diamond, 2013). Recently, Xu et al. (2017) showed that exclusion influences selective attention by impairing distractor suppression; in a visual search task, they presented target (as inclusion-related cues) and distractor (as exclusion-related cues) stimuli and found behavioral and neural evidence that exclusion exerted different impacts on target and distractor processing. Only excluded participants reported smaller distractor-positivity amplitudes, as a reflection of distractor suppression, whereas both excluded and included reported similar target-negativity amplitudes as a reflection of target enhancement. Thus, the influence of social exclusion on selective attention was driven by distractor suppression but not by target enhancement.

In parallel, the experience of being excluded may also enhance sensitivity to affiliative social cues (Pickett et al., 2004; see also Shilling and Brown, 2016). Prior study illustrated that ostracized participants, compared to included and control participants, were more accurate at discriminating between genuine and fake smiles (Bernstein et al., 2008) and better able to differentiate between happy and angry faces relative to their ability to differentiate within happy and angry face categories (Sacco et al., 2011). DeWall et al. (2009) showed that socially excluded individuals displayed an attentional bias toward affiliative cues. For instance, they found that participants expecting exclusion were more sensitive to searching for emotional faces in a crowd of neutral faces, more fixated on smiling faces, and slower at disengaging from smiling faces. A similar pattern has been revealed in participants excluded from a virtual ball-tossing game (Xu et al., 2015). Also, Chen et al. (2017b) demonstrated that social exclusion increases attention toward social cues, especially positive social cues.

Accordingly, a great deal of research suggested that excluded individuals have stronger motivation in forming new relationships and are more willing to affiliate with those who may signal approachable intentions toward them (Maner et al., 2007); thus, they may imitate others' movements (Lakin et al., 2008), conform (Williams et al., 2000), and obey (Riva et al., 2014b) to a greater extent than non-excluded individuals. Furthermore, successful reaffiliation can reduce the unpleasant effects of ostracism (DeWall et al., 2010). On the other hand, when reconnection is not possible, people might be—as recovery strategies from social exclusion—motivated to stop their negative emotional state (Buckley et al., 2004) through emotion regulation processes (Riva, 2016), for instance, by turning attention away from their exclusionary situation (i.e., distraction; Wesselmann et al., 2013). Therefore, when there are no reaffiliation opportunities, excluded individuals may sometimes withdraw from interactions and search for self-isolation (Romero-Canyas et al., 2010; Cuadrado et al., 2015), and start to perceive other people as particularly unfriendly and unapproachable (Richman et al., 2016).

# THE ROLE OF DIRECT AND AVERTED GAZE IN SOCIALLY EXCLUDED INDIVIDUALS

Extensive research over the past decade suggests that the eyes convey a wealth of personal information and about their direction of attention to specific people, places, and objects (Birmingham and Kingstone, 2009). Eye gaze, especially direct gaze (i.e., when the gaze of another individual is directed at the observer), is used to indicate interest, express closeness, regulate interactions (Conty et al., 2016), and conveys social connection (Wirth et al., 2010; Wesselmann et al., 2012).

Moreover, eye contact has powerful effects on the receiver (for reviews, see Kleinke, 1986; Hietanen, 2018). Seeing another person with direct gaze automatically elicits positive affective reactions in the observer (Chen et al., 2016, 2017a), increases autonomic arousal (Helminen et al., 2011), and activates brain responses indicative of a tendency to approach (Hietanen et al., 2008). Interestingly, such effects of eye gaze do not have to be conscious (Burra et al., 2013). On the other hand, an averted eye gaze has been deemed to represent a common sign of social exclusion (Williams et al., 1998). Being denied others' direct gaze can elicit brain mechanisms related to avoiding motivation (Hietanen et al., 2008; Abeles and Yuval-Greenberg, 2017) and increase the willingness to act aggressively toward the interaction partner (Wirth et al., 2010). Supporting this notion, a study found that receiving averted gaze, compared to receiving direct gaze, leads participants to feel ostracized (Wirth et al., 2010). In this study, participants briefly watched a face on a computer screen portraying either direct (i.e., by looking at the participant) or averted (i.e., by looking left and right, but not at the participant) gaze. Results showed that even briefly exposing participants to averted gaze, relative to direct gaze, lowers mood and satisfaction of basic needs of belonging, control, self-esteem, and meaningful existence.

Thus, it is not surprising that people are highly sensitive to being attended to (i.e., looked at) by other people, especially those who have been excluded. Past research has shown that ostracized participants look more to the eyes of the interaction partner who had the power to reintegrate them, suggesting that they attempted to make eye contact to get involved into the interaction (Böckler et al., 2014). More recently, excluded participants, compared to included participants, recognized a wider range of gaze directions as being directed at them, possibly because observing direct gaze could make them feel reaffiliated (Lyyra et al., 2017).

However, as already discussed, without the perception of an opportunity for reaffiliation, ostracized individuals may begin to view other people as particularly unfriendly and start to disengage from interactions. For instance, excluded participants, compared to an inclusion group and a nonsocial control group, accepted a smaller range of gaze directions as being directed at them, probably because they do not perceive any opportunity of reconnection with others (Syrjämäki et al., 2018).

A common approach to investigate such a human predisposition to the detection of the eye gaze of others is to study attentional shifts in response to observed eye gaze direction, namely social attention. Social attention refers to the ability to orient attentional resources after observing others' directional behavior (e.g., Friesen and Kingstone, 1998; Langton and Bruce, 1999; Nummenmaa and Calder, 2009). Research on social attention has primarily focused on the role of gaze (e.g., Frischen et al., 2007) by adopting modifications of the classical Posner's spatial cueing paradigm (Posner et al., 1978; Posner, 1980; Posner and Cohen, 1984), namely the gaze-cueing paradigm (for a review, see Frischen et al., 2007). In a typical gaze-cueing task, an onscreen face is presented and displays averted gaze to the left or right. After a given period of time (Stimulus Onset Asynchrony, SOA), a lateral target appears at the looked at or non-looked at location. The standard approach compares detection speed for visual targets and consistently shows the gaze-cueing effect, an attentional shift toward the cued location revealed by faster responses when the target appears to the side of space that was prior cued by the gaze (Friesen and Kingstone, 1998; Driver et al., 1999; Frischen et al., 2007).

The orientation of attention in response to eye gaze provided by others appears to be rapid (e.g., Friesen and Kingstone, 1998) and reflexive (e.g., Driver et al., 1999). However, this seemingly robust and reflexive orienting response can also be sensitive to social modulators, including physical and social characteristics of the target, such as age (Ciardo et al., 2014), social status (e.g., Foulsham et al., 2010; Dalmaso et al., 2014), in-group membership and ethnicity (Pavan et al., 2011; Dalmaso et al., 2016). Moreover, individual differences of the responders can also play a role (Bayliss et al., 2005; Fox et al., 2007; Wilkowski et al., 2009). In particular, Wilkowski et al. (2009) found that individuals low in self-esteem exhibited more pronounced gaze-cueing effect than individuals high in selfesteem, and such an effect was specific to social cues. Thus, it appears that the typical tendency to orient attention in accordance with another individual's eye gaze was enhanced under conditions of low belongingness.

# OVERVIEW OF THE PRESENT RESEARCH

The present work tested whether social exclusion affects a basic cognitive phenomenon such as the gaze-cueing effect. Specifically, we investigated whether socially excluded individuals would behave differently from included ones in a gaze-cueing task.

As discussed, the social psychology literature shows multifaceted and complex responses to social exclusion. In some cases, humans show an affiliative response to exclusion (e.g., DeWall et al., 2009; Xu et al., 2015; Lyyra et al., 2017), especially when they perceive an opportunity for reaffiliation. In others, when opportunities for affiliation are not foreseen, humans may withdraw from the interactions (Romero-Canyas et al., 2010; Cuadrado et al., 2015). Thus, it is worth to address how situational factors can influence the effects of exclusion.

The gaze-cueing paradigm allows disentangling between these two competing dynamics in the context of gaze following. Thus, whether the reconnection motivation prevails, one might expect to find a larger gaze-cueing effect for socially excluded compared to socially included participants; in other words, socially excluded individuals would follow more the agent's gaze (*Hypothesis 1a)*. On the other hand, as averted gaze may represent a further sign of social exclusion for rejected participants (Williams et al., 1998; Wirth et al., 2010) when observing faces portraying averted gaze, individuals may consider the context as lack of affiliative opportunity and wish to disengage from it. Hence, following this rationale, one might expect that excluded participants would show a reduced gaze-cuing effect than the socially included ones (*Hypothesis 1b*).

To test our hypotheses, we carried out two experimental studies in which participants were either socially excluded or included using the *Cyberball manipulation* (Williams and Blair, 2006), a paradigm in which participants engage in a balltossing game with virtual avatars including or excluding them from the game. In Study 1, after the Cyberball manipulation, participants performed a gaze-cueing task in which an individual's gaze, oriented rightward or leftward, preceded a peripheral target stimulus requiring a simple categorization response. The gaze direction could be congruent or incongruent with the location of the target. In Study 2, human gazes were replaced by arrows, well-known directional nonsocial cues typically used as control for gaze in many social attention studies (e.g., Ristic et al., 2002; Friesen et al., 2004).

The procedures were approved by the Ethical Committee of the University of Milano-Bicocca, and were in accordance with the ethical standards of the 1964 Declaration of Helsinki and with the ethical standards recommended by the Italian Association of Psychology (AIP).

# STUDY 1

# Participants

An *a priori* power analysis was conducted for sample size estimation (using G Power 3.1; Faul et al., 2007). With an *α* = 0.05 and power = 0.95, the projected sample size needed to detect a medium effect size (*f* = 0.25) is *N* = 54 for a mixed repeated-measures ANOVA. We advertised the study, and we enrolled all the individuals who answered the call and volunteered to participate even if the final number of participants exceeded the number suggested by the G-Power analysis. Thus, 81 participants (*M*age = 22.90, *SD*age *=* 1.94, range = 19–31 years, 41 females), naive to the purpose of the study, took part in the study. All participants were Italian citizens except for one Ecuadorian and one Chinese with native knowledge of the Italian language. All participants signed a form of informed consent.

#### Procedure

Upon arrival at the lab, participants provided written informed consent and were asked sociodemographic information (gender, age, nationality). Next, participants were told that the study was composed of two parts, apparently unrelated to each other. In the first part, they were told that they would engage in a mental visualization task (for a similar procedure, see Williams et al., 2000). Actually, they were involved in a standard manipulation of inclusionary status. Participants played a virtual online ball-tossing game with virtual avatars, namely the Cyberball (Williams and Blair, 2006). They were told they were playing with two other players, allegedly real participants that were playing Cyberball in another lab. The three of them would take turns tossing a ball to each other. In reality, the two computer avatars were pre-programmed agents randomly assigned to either include or exclude the real participant from the balltossing game. In the exclusion condition, after two passes, the two computer players stopped tossing the ball to the real participant for the rest of the game. In the inclusion condition, the computer players threw the ball to the actual participant for 10 of the 30 total tosses (Williams et al., 2000).

After playing Cyberball, as a manipulation check, all participants in all conditions were asked how often (0–100% of the time) they received the ball and to report how excluded ("*I felt excluded*") and ignored ("*I felt ignored*") they experienced during the mental visualization task (i.e., playing Cyberball). Afterward, participants completed the Need-Threat Scale (Williams, 2009), which assessed participants' satisfaction levels for belongingness (e.g., "*I felt rejected*"), self-esteem (e.g., "*I felt liked*"), control (e.g., "*I felt powerful*"), and meaningful existence (e.g., "*I felt invisible*"). All items were rated on 10-point scales (1 = *not at all* to 10 = *extremely*). We averaged the 20 items and created an overall index of basic needs satisfaction (*α* = 0.91). Finally, they were asked to report their current emotional state (the Rejection-related Emotions Scale, RES; Buckley et al., 2004; *α* = 0.90). The scale includes 24 items assessing 6 clusters of emotions: anger, anxiety, sadness, hurt, rejection, and happiness. Items ranged from 1 = *not at all* to 10 = *very much*. We averaged the 24 items and created an overall index of negative emotions (after reversing happiness scores; *α* = 0.90).

The second part of the study consisted of a gaze-cueing task. The experiment was carried out in a dimly illuminated room. Participants sat approximately 76 cm away from a 22-inch LCD monitor (Asus VW226TL; resolution: 1,680 × 1,050 pixels; refresh rate: 59 Hz; horizontal screen angle: 35°31′0.82″; vertical screen angle: 20°26′0.23″) interfaced with a PC (Pentium 4). The stimuli used consisted of pictures (14.25° × 16.47°) on a gray background of unfamiliar faces gazing at different positions (about 30° left, 30° right or straight ahead—**Figure 1**). The distance between the two outer corners of the eyes was about 8.28°.

A trail started with a centrally presented fixation cross for 900 ms, which participants fixated, and then a face with direct gaze appeared in the center of the screen. After 900 ms, the eyes moved to the left or right, and 200 ms later, the target appeared either on the left or on the right of the screen, namely in a spatially congruent or spatially incongruent position with respect to the gaze direction (**Figure 2**).

The target remained visible until a response was made. Note that gaze direction was randomized and non-predictive of the target location. Indeed, in 50% of the trials, the target appeared in the gazed-at location, and in the other 50% of trials, the target appeared in the opposite gazed-at location. In the same vein, the target had the same probability of appearing on the right or on the left throughout each block.

Participants were asked to make speeded categorization of the target, pressing with the thumb of their (right or left) dominant hand and the forefinger (index) of the same hand the "h" key when an "L" appeared on the screen or the space-bar key when a "T" appeared. Since the "h" key is directly above the space bar, such up/down response was orthogonal to the left/right target location. Responses were allowed after the letter appearance and reaction times (RTs) were recorded.

FIGURE 1 | Example of a cue stimulus in Study 1. Three versions of an unfamiliar face were produced, one with gaze straight ahead (A), one with the pupils averted leftward (B), and another with the pupils averted rightward (C). Written informed consent was obtained from the depicted individuals for the publication of their identifiable images.

The experimental session was composed of 8 training trials followed by 1 block of 128 experimental trials. The order of trials was randomized. At the end of the gaze-cueing task, participants were fully debriefed.

Hence, the experimental design consisted of a 2 (social exclusion manipulation: excluded vs. included) × 2 (cue-target spatial congruency: congruent vs. incongruent) factorial design, with the first factor varying between participants and the second factor varying within participants.

#### Results

#### Inclusionary Status Manipulation Checks

A series of independent-samples *t*-tests revealed that excluded participants reported receiving fewer tosses (*M* = 5.05, *SD* = 3.23) than included participants (*M* = 29.92, *SD* = 9.58)*, t*(79) = 16.04, *p* < 0.001, *d* = 3.57. Moreover, participants in the social exclusion condition felt to be more excluded (*M* = 7.07, *SD* = 3.10) than included participants did (*M* = 2.00, *SD* = 1.58), *t*(79) = −9.08, *p* < 0.001, *d* = −2.02, and more ignored (*M* = 7.79, *SD* = 2.62) than included participants did (*M* = 2.13, *SD* = 1.74), *t*(79) = −11.30, *p* < 0.001, *d* = −2.52.

Additionally, participants who were excluded reported higher level of rejection-related emotions (*M* = 4.12, *SD* = 1.44) than included participants (*M* = 2.90, *SD* = 0.90), *t*(79) = −4.50, *p* < 0.001, *d* = −1.00, and lower level of basic needs satisfaction (*M* = 4.08, *SD* = 1.40) than those who were included (*M* = 5.99, *SD* = 1.22), *t*(79) = 6.50, *p* < 0 .001, *d* = 1.45.

#### Gaze-Cueing Task

We excluded from the analysis training trials (5.88% of trials) and errors (e.g., pressing the "h" key when a "T" appeared; 4.35% of trials). Mean and RTs are reported in millisecond (ms).

First, we conducted an error analysis. Thus, a 2 (cue-target spatial congruency: congruent vs. incongruent) × 2 (social exclusion manipulation: excluded vs. included) mixed-factors ANOVA was carried out on the percentage of the errors. Neither main effects nor interaction effects were significant, *p >* 0.29.

Then, a 2 (cue-target spatial congruency: congruent vs. incongruent) x 2 (social exclusion manipulation: excluded vs. included) mixed-factors ANOVA was carried out on the average RTs. As expected, the analysis showed a significant main effect of cue-target spatial congruency, *F*(1,79) = 32.28, *p <* 0.001, *η*<sup>p</sup> 2 = 0.29. RTs for congruent trials (*M* = 642.99, *SE* = 14.28) were faster than RTs for incongruent trials (*M* = 674.82, *SD* = 15.76). The main effect of the social exclusion manipulation did not reach the level of significance, *F*(1,79) = 3.16, *p =* 0.08, *η*<sup>p</sup> 2 = 0.04. Importantly, the interaction effect between social exclusion manipulation and cue-target spatial congruency was significant, *F*(1,79) = 6.30, *p =* 0.014, *η*p 2 = 0.07. The *post hoc* analyses revealed, in the inclusion condition, significant differences between RTs in congruent (*M* = 662.20 ms, *SE* = 20.81 ms) and incongruent trials (*M* = 708.10 ms, *SE* = 22.96 ms), *p* < 0.001. The same pattern emerged in the exclusion condition: RTs in congruent trials (*M* = 623.79 ms, *SE* = 19.56 ms) were lower than RTs in incongruent trials (*M* = 641.55 ms, *SE* = 21.58 ms), *p* = 0.023 (**Figure 3**).

#### Discussion

Results of Study 1 showed faster responses in congruent trails than in incongruent trials. Consistent with the literature in the gaze-cueing effect (for a review, see Frischen et al., 2007), our findings suggest that eye gaze is likely to shift an individual's attention.

Importantly, with regard to our hypothesis, we found a significant interaction between social exclusion manipulation and the cue-target spatial congruency. Specifically, socially excluded, compared to included, participants showed a reduced attentional shift due to averted eye-gaze detection.

Study 2 aimed to explore whether such a moderating role of social exclusion on social attention could be found also with symbolic cues. To this aim, in Study 2, we adopted a procedure similar to Study 1 with one exception. Here, we replaced human gazes with arrow cues, well-known nonsocial cues typically used as a control for gaze in social attention studies (e.g., Ristic et al., 2002; Friesen et al., 2004).

# STUDY 2

# Participants

As for Study 1, we advertised the study, and we enrolled all the volunteers who answered the call even if the final number of participants exceeded the number suggested by the G-Power analysis. Thus, 80 participants (*M*age = 25.00, *SD*age *=* 5.10, range = 19–52 years, 58 females), naive to the purpose of the study, took part in the study. All participants were Italian citizens except for one Romanian, one Ukrainian, one French, and one Italian-French with native knowledge of the Italian language. All participants provided a written informed consent.

# Procedure

The stimuli used in Study 2 consisted of pictures (14.25° × 16.47°) of an oval of the same size as the face used in Study 1 (10.90° × 6.63°—**Figure 4A**) framing either a black segment with two Xs, one at each end or with a black arrow pointing either to the right or to the left. The two pointing arrows were obtained by removing half the segments of each X-endpoint (**Figures 4B,C**). The distance between the endpoints measured about 8.28°. The experimental procedure of Study 2 was identical to that used in the previous study (**Figure 5**).

Hence, the experimental design consisted of a 2 (social exclusion manipulation: excluded vs. included) × 2 (cue-target spatial congruency: congruent vs. incongruent) factorial design, with the first factor varying between participants and the second factor varying within participants.

## Results

#### Inclusionary Status Manipulation Checks

A series of independent-samples *t*-tests revealed that excluded participants reported receiving fewer tosses (*M* = 8.45, *SD* = 11.18) than included participants (*M* = 32.68, *SD* = 8.48)*, t*(78) = 10.92, *p <* 0.001, *d* = −2.07. Moreover, participants in the social exclusion condition felt to be more excluded (*M* = 7.83, *SD* = 2.28) than included participants did (*M* = 1.48, *SD* = 1.01), *t*(78) = −16.13, *p* < 0.001, *d* = −3.61, and more ignored (*M* = 7.90, *SD* = 1.88) than included participants did (*M* = 1.58, *SD* = 1.13), *t*(78) = −18.25, *p* < 0.001, *d* = −4.08.

Additionally, participants who were excluded reported higher level of rejection-related emotions (*M* = 4.71, *SD* = 1.30) than included participants (*M* = 2.57, *SD* = 0.68), *t*(78) = −9.26, *p* < 0.001, *d* = 2.44, and lower level of basic needs satisfaction

(*M* = 3.60, *SD* = 0.91) than those who were included (*M* = 6.90, *SD* = 1.28), *t*(78) = 13.28, *p* < 0.001, *d* = 2.97.

#### Cueing Task

As in Study 1, we excluded from the analysis training trials (5.88% of trials) and errors (e.g., pressing the "h" key when a "T" appeared; 5.48% of trials). Mean and RTs are reported in millisecond (ms).

First, we conducted an error analysis. Thus, a 2 (cue-target spatial congruency: congruent vs. incongruent) × 2 (social exclusion manipulation: excluded vs. included) mixed-factors ANOVA was carried out on the percentage of the errors. Neither main effects nor interaction effects were significant, *p >* 0.43.

Then, a 2 (cue-target spatial congruency: congruent vs. incongruent) × 2 (social exclusion manipulation: excluded vs. included) mixed-factors ANOVA was carried out on the average RTs. The analysis showed a significant main effect of cue-target spatial congruency, *F*(1,78) = 4.9, *p =* 0.03, *η*<sup>p</sup> 2 = 0.06: RTs for congruent trials (*M* = 642.73, *SE* = 11.36) were faster than RTs for incongruent trials (*M* = 655.10, *SD* = 12.25). Neither the main effect of social exclusion manipulation, *F*(1,78) = 0.31, *p =* 0.58, *η*<sup>p</sup> 2 = 0.004, nor the interaction between social exclusion manipulation and cue-target spatial congruency were significant*, F*(1,78) = 0.66, *p =* 0.42, *η*<sup>p</sup> 2 = 0.008 (**Figure 6**).

#### Discussion

Our data revealed a cueing effect for arrow cues, namely faster RT in congruent trials than in incongruent trials. This result is in line with previous studies (e.g., Ristic et al., 2002; Tipples, 2002; Friesen et al., 2004; Quadflieg et al., 2004; Kuhn and Kingstone, 2009; Galfano et al., 2012) suggesting that, similar to gaze, arrow cues are likely to elicit strongly automatic shifts of attention.

However, results of Study 2 did not reveal any interaction effect between social exclusion manipulation and cue-target spatial congruency.

#### GENERAL DISCUSSION

The present work aimed at investigating whether social exclusion could affect the basic cognitive phenomenon of gaze-cueing effect, namely, the tendency to reorient attentional resources to the same location that other people are looking at. Specifically, we tested whether socially excluded individuals would behave differently from included individuals in a gaze-cueing task. To this end, socially excluded or included (through the *Cyberball* manipulation) participants performed a cueing task. In Study 1, they observed human faces portraying averted eye gaze; in Study 2, human gazes were replaced by arrow cues.

As previously discussed, humans respond to social exclusion in multifaceted ways. They can show affiliative response to exclusion (e.g., DeWall et al., 2009; Xu et al., 2015; Lyyra et al., 2017), especially when they perceive an opportunity for reaffiliation, or, without any opportunity for reaffiliation, they may disengage from the interactions (Romero-Canyas et al., 2010; Cuadrado et al., 2015). The paradigm adopted in the current work allowed disentangling between these two competing dynamics. Thus, if the tendency for reaffiliation prevails, one might expect to find a larger gaze-cueing effect (i.e., higher difference between RTs in congruent trials and RTs in incongruent trials) for socially excluded compared to socially included participants. That is, socially excluded individuals should follow the agent's gaze more. On the other hand, as averted gaze may represent a further sign of social exclusion (Williams et al., 1998), individuals may consider

faces portraying averted gaze as lacking affiliative opportunity. Hence, if the attentional disengagement overcomes as a reaction to exclusion, one might expect that excluded participants would show a reduced gaze-cueing effect (i.e., the lower difference between RTs in congruent trials and RTs in incongruent trials) than socially included participants.

Firstly, corroborating the idea that eye gaze (for a review, see Frischen et al., 2007) and well-known symbolic cues (e.g., Ristic et al., 2002; Tipples, 2002; Friesen et al., 2004; Quadflieg et al., 2004; Kuhn and Kingstone, 2009; Galfano et al., 2012) are capable of orienting the individual's attention, we found faster responses in congruent trails than in incongruent trials, both when observing social (i.e., eye gaze) and symbolic cues (i.e., arrows).

Interestingly however, our data revealed a reduced attentional shift for excluded than for included participants, and such an effect was specific for social cues. Importantly, the interaction between social exclusion and cue-target spatial congruency disappeared with symbolic cues (i.e., arrows; Study 2).

Our findings are in line with the recent literature on social exclusion and eye gaze direction. For instance, Syrjämäki and Hietanen (2018) investigated whether the attentional holding effect of direct gaze would be increased for socially excluded (or included) individual. In this research, participants took part in a *Cyberball* manipulation, and, afterward, they performed an attentional disengagement task, in which they were required to identify peripheral stimuli shown to the left or right of centrally presented faces portraying direct or downward eye gaze. Results revealed that included individuals, compared to excluded individuals, disengaged slower from direct-gaze faces. Thus, social inclusion might have activated affiliation-related cognitive processes causing delayed disengagement of attention from faces cueing affiliation. In the same vein, our findings suggest that an averted gaze may represent a further sign of social exclusion, and, as a consequence, excluded participants showed a reduced cueing effect toward faces communicating a lack of affiliative opportunity.

In Study 2, participants observed symbolic cues. As in the previous study, we found the typical cueing effect, but, in this case, no interaction effect has emerged. Taken together, it seems that the moderating role of social exclusion on social attention is specific for social cues, as shown in Study 1 with eye gaze, and not generalizable to symbolic cues, as in Study 2 with arrows. Thus, given that most of the cognitive psychology literature has established the primacy of socially relevant stimuli, such as eye gaze (Langton et al., 2000), and that socially excluded individuals are particularly sensitive to social cues (e.g., Pickett et al., 2004), it appears that social exclusion may moderate social attention only when social cues are available.

Our findings are theoretically relevant for several reasons. As reported above, past research showed increased sensitivity to other people's gaze after social exclusion. Direct gaze, for instance, indicates that another's attention is directed to the self (Conty et al., 2016), so it may be especially significant for individuals who have been excluded. Indeed, previous studies have shown that ostracized participants, when trying to reconnect with others, seek for inclusive cues, and direct gaze represents an important social cue to this purpose.

Furthermore, because negative attention is preferred to being ignored (e.g., O'Reilly et al., 2014), even when direct gaze could be perceived as a sign of threat (e.g., when it is accompanied by an angry facial expression; Adams and Kleck, 2005), it might reduce the unpleasant effects of exclusion (Rudert et al., 2017; but see also Syrjämäki et al., 2017). Averted gaze represents a primary cue for communicating ostracism and no opportunity for reconnection with others (Williams et al., 1998) although it can signal a danger approaching or the location of an interesting object. Both laboratory and field studies (in social psychology) demonstrate that averted eye gaze from live or virtual confederates can induce feelings of ostracism and basic need threat similar to traditional social exclusion manipulations (e.g., Wirth et al., 2010; Wesselmann et al., 2012; Böckler et al., 2014). In our gaze-cueing paradigm, the observed agents, after a brief initial period of direct eye gaze, consistently portrayed averted gazes, both in congruent trails than in incongruent trials. Hence, our excluded participants can have perceived the averted gaze as a further sign of social exclusion. As noted, socially excluded people desire to make new social connections, but they also wish to make sure that they will not suffer rejection again (Park and Baumeister, 2015), so that when there is no opportunity for reaffiliation, they may withdraw from the interaction (Romero-Canyas et al., 2010; Cuadrado et al., 2015). Thus, it is plausible that, to cope with exclusion, our participants showed reduced cueing effect toward such faces signaling a lack of affiliative opportunity.

Moreover, our findings allow us to disentangle between different alternative explanations. As mentioned, prior research showed that social exclusion deploys cognitive resources (Chester and DeWall, 2014; Lurquin et al., 2014; Riva et al., 2014a). Hence, whether the moderating role of social exclusion on social attention was due to the limited cognitive resources, we would have found similar results either when observing social cues (Study 1) and nonsocial cues (Study 2). Interestingly however, we did not find any interaction between congruency and social exclusion with symbolic cues (Study 2). Thus, we could interpret our findings from a motivational perspective; for socially excluded participants, symbolic cues could have served as informative cues, but not useful for the benefit of the interaction. However, since we did not manipulate the type of cue (social vs. symbolic) in the same experiment, the speculation on these different patterns requires caution and further empirical investigation.

In addition, the present work also extends the existing literature from a methodological point of view. Our results demonstrate that the gaze-cueing paradigm is an appropriate task that helps to shed light on two competing dynamics occurring during social exclusion, namely the reaffiliation and the disengagement strategies.

There are some limitations to the present research to be considered that can be addressed in future research. It is worth noting that the stimuli adopted here were much simpler than situations we face in everyday life. Indeed, the stimuli used in the current experiments were static photographs of an individual appearing on a computer screen. We speculate that in more ecological contexts (e.g., observations of people in real social interactions), social exclusion could even enhance the effect on social attention, probably because real averted gaze may represent stronger signs of rejection. Hence, further research should explore the relationship between social attention and social exclusion using more ecological paradigms, for instance showing real interactions or dynamic displays of eye gaze.

# REFERENCES


From a methodological perspective, here we conducted two different studies, the first focused on the role of eye gaze, and the second focused on the role of arrow. Direct comparisons between social and nonsocial cues may enrich our knowledge on whether such a moderating role of social exclusion on social attention is specific for social cues or generalizable to symbolic cues. For instance, further research should compare in the same experiment both the role of eye gaze and arrow (i.e., either with a within-participants design whereby both arrow and eye gaze are presented in the same experiment or with a between-participants design), in order to replicate and extend these results.

Furthermore, the current study did not involve a control group that was neither included nor excluded. Thus, we are not fully able to disentangle whether the observed differences between excluded and included groups are due to exclusion, inclusion, or both. However, previous studies found that being included in Cyberball has similar psychological consequences to watching a mountain sketch on the screen (Riva et al., 2014b). Nevertheless, future studies should consider avoiding this problem, for instance, by employing a nonsocial control group (see for instance Syrjämäki et al., 2018).

To sum up, the current work illustrated that social exclusion modulates the basic cognitive process of social attention; following exclusion, individuals showed reduced gaze-cueing effect toward faces portraying averted gaze, which represents a further sign of rejection.

# ETHICS STATEMENT

The procedures were approved by the Ethical Committee of the University of Milano-Bicocca, and were in accordance with the ethical standards of the 1964 Declaration of Helsinki and with the ethical standards recommended by the Italian Association of Psychology (AIP). All subjects gave written informed consent.

# AUTHOR CONTRIBUTIONS

RC drafted the first version of the manuscript and ran the studies. RC and PRiv conducted data analyses. PRiv, PRic, and SS conceived the study idea and the experimental paradigm, and supervised and coordinated the research. PRiv, PRic, and SS reviewed the final version of the manuscript.

Baumeister, R. F., and Leary, M. R. (1995). The need to belong: desire for interpersonal attachments as a fundamental human motivation. *Psychol. Bull.* 117, 497–529. doi: 10.1037/0033-2909.117.3.497


detection of real and fake smiles. *Psychol. Sci.* 19, 981–983. doi: 10.1111/j. 1467-9280.2008.02187.x


underlying peer influences on decision-making. *NeuroImage* 82, 23–34. doi: 10.1016/j.neuroimage.2013.05.061


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Capellini, Riva, Ricciardelli and Sacchi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# The Role of Emotional Expression and Eccentricity on Gaze Perception

#### Deema Awad\*, Nathan J. Emery and Isabelle Mareschal

Department of Biological and Experimental Psychology, School of Biological and Chemical Sciences, Queen Mary University of London, London, United Kingdom

The perception of another's gaze direction and facial expression complements verbal communication and modulates how we interact with other people. However, our perception of these two cues is not always accurate, even when we are looking directly at the person. In addition, in many cases social communication occurs within groups of people where we can't always look directly at every person in the group. Here, we sought to examine how the presence of other people influences our perception of a target face. We asked participants to judge the direction of gaze of the target face as either looking to their left, to their right or directly at them, when the face was viewed on its own or viewed within a group of other identity faces. The target face either had an angry or a neutral expression and was viewed directly (foveal experiment), or within peripheral vision (peripheral experiment). When the target was viewed within a group, the flanking faces also had either neutral or angry expressions and their gaze was in one of five different directions (from averted leftwards to averted rightwards in steps of 10◦ ). When the target face was viewed foveally there was no effect of target emotion on participants' judgments of its gaze direction. There was also no effect of the presence of flankers (regardless of expression) on the perception of the target gaze. When the target face was viewed peripherally, participants judged its direction of gaze to be direct over a wider range of gaze deviations than when viewed foveally, and more so for angry faces than neutral faces. We also find that flankers (regardless of emotional expression) did not influence performance. This suggests that observers judge that angry faces were looking at them over a broad range of gaze deviations in the periphery only, possibly resulting from increased uncertainty about the stimulus.

#### Keywords: gaze, faces, emotions, peripheral vision, contextual effects

# INTRODUCTION

We are sensitive to a variety of different facial cues during successful communication (Simpson and Crandall, 1972; Ekman, 1993; Engell and Haxby, 2007). Of the non-verbal cues that people exchange and decipher during these interactions, both eye gaze and facial expressions rapidly capture our attention, providing us with information about another's emotional state, focus of attention, intentions, and future behavior (Vuilleumier, 2005; Palermo and Rhodes, 2007).

From an early age (see McClure, 2000 for meta-analysis on emotional recognition from infancy to adolescence), and regardless of gender (Thayer and Johnsen, 2000), or cultural background (Ekman and Wallace, 1971), humans are experts in analyzing others' facial emotional expressions,

#### Edited by:

Paola Ricciardelli, University of Milano-Bicocca, Italy

#### Reviewed by:

John A. Greenwood, University College London, United Kingdom Giulia Prete, Università degli Studi G. d'Annunzio Chieti e Pescara, Italy

> \*Correspondence: Deema Awad d.awad@qmul.ac.uk

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 04 November 2018 Accepted: 29 April 2019 Published: 21 May 2019

#### Citation:

Awad D, Emery NJ and Mareschal I (2019) The Role of Emotional Expression and Eccentricity on Gaze Perception. Front. Psychol. 10:1129. doi: 10.3389/fpsyg.2019.01129

which serve as a reliable indicator of their internal emotional and mental states (Adolphs, 2003). From an evolutionary perspective, the rapid and accurate detection of facial expressions is seen as critical for survival (Darwin, 1872). This is supported by EEG studies demonstrating faster cortical activity during the perception of emotional faces compared to neutral faces, highlighting the role of the right dorsolateral prefrontal cortex in emotional processing (Prete et al., 2015; Prete et al., 2018). Support for the rapid detection and accurate perception of facial expression also comes from behavioral studies showing that negative facial emotions, (notably those eliciting feelings of threat and danger) draw attention quickly and involuntarily (Öhman et al., 2001; Armony and Dolan, 2002; Holmes et al., 2003), and that angry faces "pop out" in a crowd (Hansen and Hansen, 1988; Feldmann-Wüstefeld et al., 2011), even when the emotional content is not relevant to the task (Frühholz et al., 2011), or is viewed under conditions of restricted awareness (Mogg and Bradley, 1999). Rigoulot et al. (2012) proposed that the influence of facial expressions on behavior can result from two opposing sources. First, emotional expressions could improve performance by automatically capturing attention and directing it to the relevant part of the task, leading to improved performance, such as has been demonstrated in dot-probe paradigms (Armony and Dolan, 2002; Brosch et al., 2007). Second, emotional expressions could disrupt the processing of task-relevant information if participants struggle to disengage from the emotional stimuli, leading to impaired performance as occurs for example, in emotional Stroop tasks (Williams et al., 1996; Fox et al., 2001; Frühholz et al., 2011; Rigoulot et al., 2012).

The accurate perception of another's gaze direction is also essential to social interactions, providing important information about their focus of attention, future intentions, as well as their emotional states (Emery, 2000; Baron-Cohen et al., 2001; Calder et al., 2008; Mareschal et al., 2013a,b, 2014). As with the detection of facial emotions, the relevance of gaze has been demonstrated from a very early age. For example, infants and monkeys as young as 3 or 4 months old follow the eye gaze direction of others and regulate their own attention appropriately (Moore and Corkum, 1998; Ferrari et al., 2000). Although the direction of the eyes is important in judging gaze, converging evidence reveals that people use a combination of cues to judge where another person is looking, using information from the pupil and sclera (Kobayashi and Kohshima, 2001), head orientation (Langton, 2000; Otsuka et al., 2014), torso direction (Seyama and Nagayama, 2005), and their prior expectations (Mareschal et al., 2013a). In order to quantify how sensitive we are to another person's gaze, Gamer and Hecht (2007) measured the range of gaze deviations that an individual perceives as being directed toward them [often referred to as the cone of direct gaze (CoDG)]. This CoDG measure is approximately 8–10◦ in adults (e.g., Gamer and Hecht, 2007; Mareschal et al., 2013b; Florey et al., 2015), and is wider in children (Vida and Maurer, 2012; Mareschal et al., 2016). Recently Mareschal et al. (2013b) developed a CoDG model that quantifies the changes in gaze perception performance. The model accounts for performance with three parameters: (1) the peak of the CoDG or the bias of perceived direct gaze (the gaze deviation the participants judge most as being direct; in case of no bias, this value would be 0). (2) the width of the cone of direct gaze which is calculated as the distance between the category boundaries for direct and averted gaze deviations; which is the range of gaze deviations participants judged as being direct (3) the estimate of the internal noise (or uncertainty) associated with the perception of gaze. This parameter reflects the amount of uncertainty associated with the observers' internal representation of the gaze direction (see section Materials and Methods for more details).

In addition to cues from the eyes and head, our perception of where another person is looking can be influenced by secondary cues, such as facial expression (Mathews et al., 2003; Putman et al., 2006; Lobmaier et al., 2008; Ewbank et al., 2009). For example, Ewbank et al. (2009) asked their participants to judge the direction of gaze in angry, fearful and neutral faces. They found that the CoDG was wider when they used angry faces compared to fearful and neutral faces. Putman et al. (2006) investigated whether reflexive cueing of attention that occurs after perception of a gaze cue is greater for fearful than for happy faces in normal and anxious participants. They used a dynamic stimulus presentation displaying faces that simultaneously morphed from a neutral into a happy or fearful expression and whose eye gaze shifted from direct to averted while participants performed a cueing task. They found that fearful faces induced stronger gaze cueing than happy faces, and that the strength of this cueing effect was correlated with participants' anxiety levels. Additionally, Mathews et al. (2003) investigated whether a fearful expression enhances the effect of another's gaze in directing the attention of the observer. Anxious and non-anxious participants viewed faces with either direct or averted gaze, and the participants' task was to locate target letters in the display. They found a difference between the two groups. Notably, attention was guided by the direction of gaze in fearful faces more so than in neutral faces, but only in the anxietyprone individuals. Interestingly, these interactive effects of gaze and expression also seem present from an early age. Striano et al. (2006) measured larger amplitude event-related potentials from the scalps of 4 months old infants when they were presented with angry faces whose gaze was direct than when presented with angry faces whose gaze was averted.

Since both gaze and facial expressions can elicit rapid and automatic spatial orienting, this has led many to examine whether there are joint brain regions facilitating the perception of both types of facial cues. In a fMRI study, Engell and Haxby (2007) presented participants with neutral faces with either direct-gaze or averted-gaze, or emotionally expressive faces with direct-gaze. The authors found that the inferior occipital gyrus, fusiform gyrus, superior temporal sulcus (STS) and inferior frontal gyrus were strongly activated in the emotionally expressive faces conditions, and the right STS was more strongly activated in the averted-gaze than in the direct-gaze conditions. Further comparisons of the data in the right STS demonstrated that emotional expression and averted gaze activated distinct, though overlapping cortical regions in the STS. Thus, the authors argued that gaze-direction and expression are associated with dissociable but overlapping neural systems, and that the overlapping regions might be

responsible for the integration of emotional expression and gazedirection information.

Behavioral studies have also demonstrated the influence that eye-gaze has on the recognition of facial emotional expression (Adams and Kleck, 2003, 2005). For example, Adams and Kleck (2003) found that participants recognized angry and happy faces more quickly when the face had a forward facing (direct) gaze, whereas fearful and sad faces were categorized more quickly when their gaze was averted. In another study, Adams and Kleck (2005) asked participants to classify the emotion (anger, joy, fear, sadness) of faces with different eye gaze (averted or direct), and found that direct gaze enhanced the perception of approach-oriented emotions (anger and joy) and averted gaze enhanced the perception of avoidance-oriented emotions (fear and sadness). They argue that combining information about gaze and expression is critical for survival since this provides us with information that someone intends to approach (or avoid) us so that we react appropriately to them.

Most of the above studies examined the perception of faces (their direction of gaze or facial emotion) when they were viewed on their own. However, it is well established that a stimulus (for example, a face) can appear differently when surrounded by other objects than when viewed on its own (Palmer, 1975; Levitt and Lund, 1997; Todorovic, 2010 ´ ). For example, Walker and Vul (2014) showed participants photographs of women viewed either in a group or in isolation and found that participants judged the (same) woman to be significantly more attractive when she was part of a group than when viewed alone. They concluded that this was the result of the visual system averaging visual information (about the faces), resulting in a prototypical (more attractive) face. A different study by Haberman and Whitney (2009) investigated contextual effects on face perception. They presented participants with sets of faces varying in emotionality (e.g., happy to sad) followed by a test face. The participants' task was to either indicate whether the test face had been a member of the previously viewed set of faces, or whether the test face looked happier or sadder than the average of the set of faces. They found that participants were unable to determine whether the test face had been in the original set of faces, although they were able to judge whether it looked happier or sadder than the average of the set. Thus, the authors argued that although participants retained little information about the individual members of the set, they had a remarkable representation of the mean emotion of the set of faces, due to "ensemble averaging." For stimuli viewed in the periphery, averaging can be compulsory due to crowding whereby individual objects that are identifiable on their own become difficult to discriminate when presented with other objects. The strength of crowding depends on the similarity of the objects. For example, Parkes et al. (2001) presented participants with an oriented Gabor patch on its own or surrounded by another 8 Gabor patches of different orientations, and found that participants were unable to report the orientation of the central Gabor, although they were able to report the average orientation of the group. Thus, the authors argued that there is compulsory averaging of visual information in the periphery, whereby groups of (spatially proximal) objects are processed as an average rather than individually.

A question that arises from this is how peripheral vision affects our ability to process other types of information about faces? Bayle et al. (2009) investigated the processing of facial expressions for stimuli presented in the parafovea using magnetoencephalography. They recorded brain activity in response to centrally and parafoveally presented fearful faces and found that when the face was in the periphery, there was increased neural activation in the amygdala and fusiform gyrus when the face was fearful compared to when it was neutral, and that these faces were processed faster. Another study by Rigoulot et al. (2011) also investigated the impact of fearful faces when presented in the near and far periphery. Reaction times and event related potentials (ERPs) were recorded while participants were asked to categorize fearful and neutral faces presented at 15◦ and 30◦ to the left or right of fixation. Their findings showed a decrease in behavioral performance with eccentricity, and more importantly that fearful faces induced shorter reaction times than neutral faces. In a subsequent study, Rigoulot et al. (2012) investigated whether the implicit processing of faces in the far periphery could be modulated by their emotional expression. They presented happy, fearful and neutral faces also in the periphery (15◦ and 30◦ ). Participants had to categorize the gender of the faces (female/male) and both accuracy and reaction times were recorded. They found decreased accuracy and longer reaction times for emotional faces compared to neutral faces, thus they argue that their findings demonstrate that emotional facial expressions are automatically processed even in impoverished conditions of vision. However, it remains unknown whether facial expressions can influence our perception of other facial cues (such as eye gaze) in the periphery. A recent study examining peripheral processing of gaze found that the CoDG increased for peripherally presented stimuli compared to foveally viewed ones (Palanica and Itier, 2011; Florey et al., 2015), using neutral faces only.

The aim of this experiment is to examine how the emotional expression of a face, viewed foveally or in the periphery, influences how we process information about its gaze direction. We focus here on the expression of anger since it has been shown to consistently modulate performance across a number of different tasks. We will use the CoDG to measure gaze perception in the fovea and periphery for neutral and angry faces that are viewed either on their own or within a group of other faces. We expect that angry expressions will influence performance more in the periphery than in the fovea and that as a result, the CoDG will be larger for angry faces in the periphery than in the fovea. We also expect that flanker faces will lead to contextual effects and crowding, with effects more pronounced in the periphery than in the fovea for congruent facial emotions (between target and flankers) than incongruent facial emotions.

# MATERIALS AND METHODS

#### Participants

Fourteen participants were included in this study (2 males), ranging from age 21 to 48 years (M = 28; SD = 7.2). All participants had normal or corrected to normal vision. The experiment was approved by the ethics board of Queen Mary University of London, and participants gave written informed consent to take part in the study. Participants received a monetary compensation. Three participants were excluded because they failed to perform above chance in the peripheral presentation experiment.

#### Stimuli and Apparatus

fpsyg-10-01129 May 17, 2019 Time: 16:30 # 4

Faces: The stimuli consisted of four synthetic grayscale faces that were generated using Daz software<sup>1</sup> . Two male and two female faces were used throughout the experiment, all the faces were forward facing, and either had a neutral or angry expression. In order to vary eye gaze, the original eyes of the faces were removed and replaced with grayscale eye stimuli created using Matlab that allowed us to control the horizontal and vertical gaze deviations down to the nearest pixel. For each face, the following sequence was carried out: Each forward facing stimulus uploaded into Facegen was saved as an image file and opened in Photoshop. We took note of the position of the iris and pupils and then cut out the eyes in Photoshop. We then created iris and pupil stimuli in Matlab and positioned them at the location of the original eyes in the Facegen stimulus. The inter-ocular distance was kept the same as the original faces. We wrote Matlab functions that allowed us to move the eyes following a rotational trajectory on a 3d round surface.

Configuration: In the foveal experiment (eccentricity = 0◦ ), face stimuli were all the same size and subtended approximately 5.5◦ × 8 ◦ . In the peripheral experiment, The face stimuli in the peripheral experiment were M-scaled (**Figure 1**) using the formula from Duncan and Boynton (2003): 1/M = 0.065E + 0.054, where M is the scaling factor and E is eccentricity and subtended approximately 3.25◦ × 2.5◦ for the nearest flanker, 8 ◦ × 5.5◦ for the target and 17◦ × 11◦ for the further flanker. The fixation point was not presented centrally but rather 8◦ from the left edge of the monitor with the participant seated in front of the fixation point. We have previously found no difference between stimuli presented to the left or to the right of fixation (Florey et al., 2015), therefore we moved the fixation point to the left side of the monitor to ensure that the M-scaled stimuli could be properly displayed in the peripheral experiment. Participants were always seated in front of the fixation point and stimuli were presented in the right visual field (to the right of the fixation point). Note that in the peripheral experiment, part of the inner and outer flanker faces fell outside the traditional crowding zone defined by Bouma's law (Pelli and Tillman, 2008). In an initial pilot, we positioned the flankers closer to the target (eccentricities of 4.2◦ and 14.5◦ , resulting in a center to center spacing with the target of 4.1◦ and 6.75◦ respectively) and in all observers (6/6), their abilty to discriminate the most extreme gaze deviations in the target face fell to chance (see Pilot section in Results).

Stimulus presentation and response collection was controlled by a DELL PC running Matlab software (MathWords Ltd.) with Psychtoolbox (Brainard, 1997). The stimuli were presented on Iiyama vision master PRO 520 monitor (1600 × 1200 pixels, 60 Hz refresh rate). At the viewing distance of 57 cm, 1 pixel subtended 1.5 arcmin. Participants were in a dimly lit room and used a chinrest during the experiment.

## Procedure

Participants took part in a gaze categorization task, they were required to judge the direction of gaze of a target face, indicating whether the eye gaze direction was averted to the left, direct (forward facing), or averted to the right using the keyboard presses "j," "k," "l," respectively. The target face's gaze deviation was randomly selected from 9 possible deviations spanning from 20◦ to the left to 20◦ to the right, in steps of 5◦ (−20◦ , −15◦ , −10◦ , −5 ◦ , 0◦ , +5 ◦ , +10◦ , +15◦ , +20◦ ). Target faces were either neutral or angry and each target gaze deviation was repeated 10 times for the neutral target and angry target conditions, resulting in 180 trials.

In the flanked condition, the target face was surrounded by two flanker faces that were arranged horizontally on either side of it. Both flanker faces had the same gaze deviation that was randomly selected from 5 possible deviations (-20◦ , −10◦ , 0◦ , +10◦ , +20◦ ) and all flanker gaze deviations were presented an equal number of times. In addition, in the congruent conditions, the target and flanker faces could be either all neutral or all angry. In the incongruent conditions, the target face could be neutral and the flankers angry or the target could be angry and the flankers neutral. Each combination of target face gaze deviation, flankers' gaze deviation, target-face emotion and flankers' emotion was repeated 10 times using a method of constant stimuli resulting in 1800 trials. In the non-flanked conditions, each target gaze deviation was repeated 10 times for the neutral target and angry target conditions, resulting in an additional 180 trials. The flanked and non-flanked conditions were randomly interleaved resulting in a total of 1980 trials.

In order to investigate the effect of stimulus eccentricity on gaze perception, we ran the above procedure for the two eccentricity experiments separately.


<sup>1</sup>www.daz3d.com/

**Figure 2** illustrates the timeline of a trial in the foveal experiment (a similar timeline occurred in the peripheral experiment). Each trial began with a gray screen with a central fixation point for 1000 ms, followed immediately by the stimulus for 300 ms. After the extinction of the stimulus, a gray screen was presented again for a 200 ms wait period during which time no response was recorded. The next trial was initiated only after a response was recorded following the wait period. The fixation dot (10 pixels diameter) was present throughout in the peripheral condition, apart from the response collection screen. Reaction times were not measured. In the foveal condition, the fixation dot disappeared only when the stimulus appeared in the center of the screen to avoid overlap.

# Data Analysis

In order to quantify participants' gaze perception across the different conditions in the foveal and peripheral experiments, the data from separate conditions were compiled into the proportion of "left," "direct," and "right" responses and we fitted our psychophysical model to each participant's data (Mareschal et al., 2013b). As mentioned earlier, the model has three parameters (see **Figure 3**) to account for an observer's performance: (1) the peak of the CoDG or the bias of perceived direct gaze. This is the gaze deviation that participants judge most as being direct and is 0 if there is no bias, and significantly different from zero if there is a bias. For example, in the case of a rightwards bias, participants perceive a physically direct gaze as being rightwards and therefore a leftwards gaze as being direct.

(B) Sample biased data. The bias (peak) is roughly 2◦ to the right of 0◦ . (C) Standard deviation. This represents the noise of the sensory representation associated with a gaze deviation (here shown for a gaze deviation of 0◦ ). The dashed curve illustrates a high SD and the solid curve a low SD, and the vertical dashed line represents the left category boundary. In the case of the high SD, a stimulus at 0◦ will most likely elicit a "direct" response (area shaded in gray) whereas in the case of the low SD, there is a greater chance that it may elicit a "left" response. (D) Example of a low SD, a -15◦ stimulus has a narrow sensory representation and the probability of a direct response is low (area in gray).

Hence the sign of the peak parameter reflects the direction of the bias; (2) the boundary width (akin to the CoDG). Gaze judgments have been shown to arise from a three-channel process (a channel processing leftwards gaze deviations, one processing direct gaze deviations and one processing rightwards gaze deviations- e.g., Calder et al., 2007, 2008). The boundary width therefore represents the width between the categorical boundaries between the averted gaze deviations and direct gaze and reflects the range of gaze deviations that participants judge as being direct (e.g., **Figures 5**, **6**); (3) the standard deviation of participants' sensory representation of gaze. This represents the noise of the sensory representation associated with the gaze deviation. If the SD is high, this means that the same gaze deviation may elicit different responses from the observer (e.g., sometimes "left"; sometimes "direct"), whereas if the SD is low, the gaze deviation will elicit the same response from the observer. This estimate therefore reflects the (sensory) noise associated with the gaze perception process.

In order to examine the effects of eccentricity, flankers' eye gaze, and emotional content of both the target face and the flankers on the perception of the eye gaze of the target face, we performed a 5 (surround gaze deviation) × 2 (target emotional expression) × 2 (flankers' emotional expression) × 2 (eccentricity) repeated measures, four-way analysis of variance (ANOVA) on each of the three CoDG model parameters. We report results as significant at α = 0.05.

# RESULTS

## Pilot

Six participants performed the task for the periphery pilot. The CoDG model was fit to the averaged data across the six participants because it failed to fit the individual data in 4/6 participants (e.g., logistic functions failed to fit the data because responses across the different deviations were around 50%, even for the extreme deviations where the responses should have been close to 100 or 0%). Therefore, we calculated the proportion of correct responses at the extreme deviations in the averaged fits (e.g., L20 stimulus and "leftwards" response or R20 stimulus and "rightwards" response) and summed these across all flanker conditions. The extreme gaze deviations are the least ambiguous amongst the different gaze deviations (i.e., L20 is more clearly leftward gaze than L5), therefore, it is expected that gaze judgments should have the highest correct responses in those trials. However, data in **Figure 4** show that observers performed at chance for all flanker conditions (we include for comparison, percent correct performance on the main task from the next section).

# Experiment

Gaze categorization results are plotted in **Figure 5** for the foveal experiment and **Figure 6** for the peripheral experiment. Estimates of peak (bias), boundary width and standard deviation from the CoDG model are plotted in **Figure 7** for the foveal experiment and **Figure 8** for the peripheral experiment. Below we present the results for the ANOVA on each of the CoDG model parameters separately.

#### Bias (Peak)

We sought to determine the peak gaze deviation participants judged most as being direct (their bias), and whether this value significantly differed from 0 as a function of the

experimental conditions. For example, if the flankers' eye gaze direction influences perception of the target's gaze, this would be reflected by systematic shifts in the bias or peak. Specifically, if information about the target and flankers is being averaged, we would expect that the peak gaze deviation that participants' judge as being direct would be shifted away from the flankers gaze deviation (e.g., if the flankers are leftwards, L20) and there is averaging with the target, a physically direct target gaze would be judged as leftwards, and therefore the peak of direct responses would occur for a rightwards deviated target).

We found a significant main effect of eccentricity on the bias parameter F(1, 10) = 5.99, p = 0.034, η 2 <sup>p</sup> = 3.41, revealing a significant difference between the biases in the foveal condition (M = -0.58, SD = 0.21) and the peripheral condition (M = 0.66, SD = 0.47). Mauchly's test indicated that the assumption of sphericity had been violated for flankers' gaze deviation conditions, X<sup>2</sup> (9) = 19.66, p = 0.027, therefore Greenhouse-Geisser corrections are reported. We found a significant main effect of flankers' gaze deviation on the peak of the CoDG, F(2.4, 24.02) = 7.60, p = 0.002. There was no significant effect of the target emotional expression F(1, 10) = 0.37, p = 0.559, η 2 <sup>p</sup> = 0.96 or the flankers' emotional expression F(1, 10) = 0.162, p = 0.696, η 2 <sup>p</sup> = 0.43 on the peak of the CoDG.

There was a significant interaction between eccentricity and flankers' eye gaze deviations F(2.5, 10) = 6.34, p = 0.004. To explore this interaction, we conducted a series of pairedsample t-tests between each flanker's eye gaze deviation in the foveal experiment with the corresponding gaze deviation in the

values = leftward). Error bars represent 1 ± SEM.

peripheral experiment. After Bonferroni correction, we found a significant difference in the peak t(10) = 4.508, p = 0.005 between the flanker's eye gaze deviation L20 in the periphery (M = 2.71, SD = 2.51) and the fovea (M = -0.47, SD = 0.74). We also found a significant difference in the peak t(10) = 3.363, p = 0.036 between the flanker's eye gaze deviation L10 between the peripheral (M = 0.94, SD = 0.90) and foveal condition (M = -0.48, SD = 0.73). No other comparisons were significant (p > 0.05). As expected with averaging, the peak was shifted away from the flankers' gaze deviation although this effect was only significant for L20 and L10 and showed a trend in the predicted direction for R10 (M = -0.13, SD = 1.93) and R20 (M = -0.65, SD = 1.91).

In order to investigate whether there was a significant effect of the presence of flankers on the peak of the CoDG between the unflanked condition and the flanked conditions with the five different flankers gaze deviations, we ran a separate one-way ANOVA with six levels (5 flanker gaze deviations and the no surround condition). We found a significant difference between the six conditions F(2.63, 26.27) = 5.599, p = 0.006, however, following Bonferroni correction, none of the paired t-tests were significant (p > 0.05).

#### Boundary Width

We sought to determine whether the range of gaze deviations that participants judged as direct differed across the conditions. We found a significant main effect of eccentricity on the boundary width F(1, 10) = 38.307, p = 0.000, η 2 <sup>p</sup> =6.22, revealing a significantly narrower boundary width in the foveal experiment

non-flanked condition (No). Averaged data are in black.

(M = 9.012, SD = 1.321) than in the peripheral experiment (M = 21.515, SD = 2.520). There was also a significant main effect of target emotion on the boundary width, F(1, 10) = 9.185, p = 0.013, η 2 <sup>p</sup> =2.39, revealing a significantly wider boundary width for an angry target (M = 17.528, SD = 2.388) than for a neutral target (M = 13.001, SD = 1.213). There was also a significant main effect of flankers' gaze deviation on the boundary width, F(4, 40) = 3.371, p = 0.018, η 2 <sup>p</sup> =0.252. However, further t-tests revealed that the only significant comparisons were in the peripheral experiment between flankers' gaze deviation L20 (M = 19.52, SD = 7.17) and flankers' direct gaze deviation (0) (M = 22.63, SD = 9.61) t(10) = -2.26, p = 0.047, η 2 <sup>p</sup> = 0.37 and between L20 (M = 19.5, SD = 7.17) and R10 (M = 22.21, SD = 9.67) t(10) = -2.66, p = 0.024, η 2 <sup>p</sup> = 0.31. Lastly there was no significant effect of the flankers' emotion on the boundary width F(1, 10) = 0.161, p = 0.696, η 2 <sup>p</sup> = 0.12.

There was a significant interaction between eccentricity and target emotion on the boundary width F(1, 10) = 11.461, p = 0.007. To explore this interaction further, we ran paired t-tests between the different target emotion conditions in the fovea and the periphery. After Bonferroni correction, we found that the boundary width was significantly wider t(10) = 5.17, p = 0.00, η 2 <sup>p</sup> = 1.81 for an angry target presented in periphery (M = 26.09, SD = 12.63) than a neutral target presented in fovea (M = 9.06, SD = 4.21), and significantly widert(10) = 5.98, p = 0.00, η 2 <sup>p</sup> = 1.66 for a neutral face presented in periphery (M = 16.94, SD = 4.86) than an angry face presented in fovea (M = 8.97, SD = 4.72). Furthermore, we also found that in the periphery, the width of the CoDG was significantly wider t(10) = 3.26, p = 0.01, η 2 <sup>p</sup> = 0.96 for angry faces (M = 26.09, SD = 12.63) than neutral faces (M = 16.94, SD = 4.86). There was no significant difference between angry and neutral faces in fovea t(10) = -01.17, p = 0.87.

There was a significant interaction between eccentricity and flankers' eye gaze on the width of the CoDG F(4, 40) = 3.252, p = 0.021. Paired t-tests with Bonferroni corrections revealed that there was a significant difference in the CoDG width between the flankers' eye gaze deviation L20 t(10) = 5.90, p = 0.00, η 2 <sup>p</sup> = 1.71, in the periphery (M = 19.52, SD = 7.17) and the fovea (M = 9.21, SD = 4.60). We also found a significant difference for flankers' eye gaze deviation L10 t(10) = 6.49, p = 0.00, η 2 <sup>p</sup> = 1.93, in

the periphery (M = 21.44, SD = 7.82) compared to the fovea (M = 8.94, SD = 4.73), a significant difference between the flankers' direct eye gaze t(10) = 5.51, p = 0.00, η 2 <sup>p</sup> = 1.84, in the periphery (M = 22.63, SD = 9.60) and the fovea (M = 9.10, SD = 4.06), a significant difference between the flankers' eye gaze R10 t(10) = 6.01, p = 0.00, η 2 <sup>p</sup> = 1.85 in the periphery (M = 22.91, SD = 9.67) and the fovea (M = 8.90, SD = 4.58). Finally, a significant difference between the flankers' eye gaze R20 t(10) = 5.68, p = 0.00, η 2 <sup>p</sup> = 1.78, in the periphery (M = 21.10, SD = 8.56) and the fovea (M = 8.84, SD = 4.58).

In order to investigate whether there was a significant effect of the presence of flankers on the width of the CoDG between the unflanked condition and the flanked conditions with the five different flankers gaze deviations, we ran a separate one-way ANOVA with six levels (5 flaker gaze deviations and the no surround condition). However, there was no significant difference between the different conditions F(5, 50) = 1.78, p = 0.134.

#### Participants' Standard Deviation (SD)

We found a significant main effect of eccentricity on the SD F(1, 10) = 31.516, p = 0.00, η 2 <sup>p</sup> = 7.96, indicating that the SD was lower in the foveal experiment (M = 2.542, SD = 0.141) than in the peripheral experiment (M = 13.714, SD = 1.979). We also found a significant main effect of target emotion on the SD F(1, 10) = 31.346, p = 0.000, η 2 <sup>p</sup> =2.29, revealing that the SD was significantly greater in the angry target condition (M = 9.286, SD = 1.119), than the neutral target condition (M = 6.969, SD = 0.888). There was no significant effect of the flankers' emotion F(1, 10) = 1.169, p = 0.305 or flankers' gaze deviation F(4, 40) = 1.030, p = 0.404 on the SD.

There was a significant interaction between eccentricity and target emotion on the SD F(1, 10) = 14.761, p = 0.003. To further explore this interaction, we ran paired t-tests between the different target faces' emotions and the different eccentricities. After Bonferroni correction, we found that the SD was different in the periphery according to the emotional content of the target face t(10) = 5.003, p = 0.03, η 2 <sup>p</sup> = 0.60, revealing that the SD was greater for angry targets in periphery (M = 15.69, SD = 7.42) than neutral targets in the periphery (M = 11.74, SD = 5.88). The SD was also significantly higher t(10) = 5.67, p = 0.00, η 2 <sup>p</sup> = 2.43, for angry targets in periphery (M = 15.69, SD = 7.42) compared to angry targets in the fovea (M = 2.89, SD = 0.69). The SD also significantly increased t(10) = 5.96, p = 0.00, η 2 <sup>p</sup> = 2.56, for angry

targets in periphery (M = 15.69, SD = 7.42) compared to neutral targets in fovea (M = 2.20, SD = 0.63). The SD also significantly increased t(10) = 5.00, p = 0.00, η 2 <sup>p</sup> = 2.11 for neutral targets in periphery (M = 11.74, SD = 5.88) compared to angry targets in fovea (M = 2.89, SD = 0.69). And lastly, the SD significantly increased t(10) = 5.34, p = 0.00, η 2 <sup>p</sup> = 2.28 for neutral targets in the periphery (M = 11.74, SD = 5.88) compared to neutral targets in fovea (M = 2.20, SD = 0.63). We found no significant difference in SD between angry and neutral faces in fovea t(10) = 2.43, p = 0.21.

To investigate whether there was a difference on the SD between the unflanked condition and the flanked conditions with the five different flankers gaze deviations, we ran a separate one-way ANOVA with six levels (5 flanker gaze deviations and the no surround condition). However, there was no significant difference between the different conditions F(5, 50) = 1.208, p = 0.319.

To summarize our main results (see **Table 1** for summary of significant main results), we found an effect of eccentricity on perception of gaze, across all three model parameters. In the peripheral experiment, participants categorized gaze as direct over a wider range of gaze deviations than in the foveal experiment, across all conditions. When we examined for target emotion, we found that angry faces led to a significant increase in the boundary width and SD for angry targets in the periphery only. There was no effect of the flanker's emotional expression on target gaze categorization. We report weak evidence for the flankers' gaze deviation affecting perception of the target's gaze for leftwards gazing flankers only (peak shifts with L20 and L10, with a trend in the expected direction for R10 and R20), although there was no significant difference between the flanked and unflanked conditions.

# DISCUSSION

We examined whether the emotional expression of a face influences how we process information about its direction of gaze when viewed directly (foveal experiment) or not (peripheral experiment). We found that gaze perception depends on both the eccentricity and the emotional expression of the face being evaluated. Specifically, we found that observers judge a target face's gaze as being direct over a broader range of gaze deviations when the faces are presented in the periphery than in the fovea, consistent with previous findings (Florey et al., 2015). We report that angry faces are judged as looking direct over a wider range of gaze deviations than neutral faces when they are in the periphery only. Although we found some evidence that the presence of flankers affects performance by shifting the peak of the CoDG, there was no effect of flankers' emotion on the task.

#### Facial Emotional Effect

We found that angry target faces lead to a wider CoDG than neutral target faces, for faces presented in the periphery only. Although a wider CoDG with angry faces in the periphery is consistent with previous suggestions that threat stimuli elicit attention, our foveal result is at odds with earlier reports that faces displaying angry expressions are more likely to be perceived as looking direct when viewed centrally (Lobmaier et al., 2008; Ewbank et al., 2009). However, more recent results using highly anxious populations have found an increased tendency to orient a person's gaze direction for faces showing fearful or angry expression for anxious participants compared to non-anxious participants (Mathews et al., 2003; Holmes et al., 2006). The role of participant anxiety has been more recently linked to the CoDG where Jun et al. (2013) found that the width of the CoDG increased for anxious males only. Therefore, our results are in line with Jun et al. (2013), since the majority of our participants were female. However, given that we do find an effect of target emotion for stimuli in the periphery, this would suggest that the processing of emotional faces not viewed directly does not depend on gender or (potential) participant anxiety, perhaps reflecting a generic mechanism to respond to threat.

Why should the CoDG increase with facial emotion in the periphery? Eye gaze generally indicates that we are the object of another person's attention (Cary, 1978). If that person is displaying an angry expression this would be indicative of a threatening situation, so it might be beneficial to assume they are looking at us. Therefore, when the gaze is close to direct (for example 5◦ averted to the left or to the right), falsely perceiving that an angry face is looking at oneself is less costly and less dangerous than falsely missing it and ignoring an alarming threat. This is consistent with previous findings that angry emotional expressions and direct gaze are enhanced by approach-avoid self-preservation motivations (Adams and Kleck, 2005; Rigoulot et al., 2012). This is also in line with previous studies demonstrating that the perceived intensity of an angry face is increased when it displays direct gaze (Adams and Kleck, 2005). Taken together, this suggests that participants' sense that gaze is directed toward them with angry faces should increase because of the potential threat this stimulus represents. Emotional scenes (Calvo and Lang, 2005) and emotional faces (Rigoulot et al., 2012) are processed more quickly in peripheral vision suggesting an attention-grabbing mechanism, which is consistent with studies showing that it takes longer to disengage spatial visual attention from threatening stimuli compared to neutral stimuli (Stein et al., 2009). Rigoulot et al. (2012) propose that this attentional capture is due to the necessity to react to relevant stimulations (both negative and positive) in peripheral vision, even if attention is not consciously directed toward them. For example, Calvo and Lang (2005) investigated whether emotional visual scenes are more likely to attract a person's eye movements than neutral scenes. Pairs of emotional (either pleasant scenes such as people enjoying themselves or nonpleasant scenes such as people suffering harm) and neutral scenes (people performing a variety of daily activities) were presented parafoveally (2.1◦ or 2.5◦ from a fixation point) for 150–3000 ms, followed by an immediate recognition test (500 ms delay). They found that when the emotional and neutral scenes were presented simultaneously in parafoveal vision, the eyes moved to and fixated the emotional scene rather than the neutral scene, revealing initial orienting toward the emotional stimuli. Furthermore, the authors suggest that the meaning or content of the emotional scenes drew overt attention that is responsible for this early orienting effect.


TABLE 1 | Summary of significant main effects and interactions between the different experimental conditions and the model parameters.

Please refer to the results discussion for further details on t-tests when applicable.

However, it is also worth noting that angry faces are physically different from neutral faces that may increase their ambiguity. For example, the opening of the eyes in angry faces is smaller as a result of the furrowing of the brows. It is possible that those physical facial differences may have increased the ambiguity of the angry faces, leading to an increase in direct gaze judgments. Previous research has shown that we tend to judge gaze as being directed at us when it is ambiguous (Mareschal et al., 2013a, 2014), We find here that the SD is higher for angry faces in the periphery, consistent with the idea that participants were more uncertain in this condition, it is possible that the target's gaze was more ambiguous in this condition and that this leads to an increase in direct responses.

### Eccentricity Effect

We found that faces presented in the periphery led to an increased number of direct responses over a wider range of eye gaze deviations (a wider CoDG), consistent with previous results (Florey et al., 2015), and demonstrating that participants held fixation. This suggests that people assume they are being looked at when they are not directly viewing the faces, consistent with our findings here of increased uncertainty when making gaze judgments about faces in the periphery. One explanation for this increase in direct responses for faces in the periphery is that it is beneficial to assume that we are being looked at, notably in the case of threatening stimuli (e.g., Mareschal et al., 2013a, 2014). This decrease in accuracy of behavioral judgments (wider CoDG) with eccentricity is consistent with earlier results (Thorpe et al., 2001; Rigoulot et al., 2011, 2012). For example, Rigoulot et al. (2012) investigated implicit emotional processing in peripheral vision, and found a gradual decrease of behavioral performance with eccentricity, revealed by the lower rate of correct categorization and longer reactions times. However, it is important to note that their results also showed that emotional facial expressions could be automatically detected in peripheral vision (which is consistent with our target face emotional effect results discussed above). Similarly, Thorpe et al. (2001) investigated performance in peripheral vision by requiring participants to categorize pictures presented at different eccentricities. In their study, the authors found a linear decrease of performance with increasing eccentricity, supporting the suggestion that we make less accurate judgments about objects or faces when they are presented in the periphery.

#### Averaging

We expected to find significant differences in participants' bias across the different surround gaze conditions for stimuli presented in the periphery. We found that the peak of the direct responses for the target gaze categorization (their bias) shifted toward the gaze deviation of the surrounding flankers, reflecting averaging between the target gaze deviation and the flanker's gaze deviation, suggesting that participants were not ignoring the flankers. However, we also surprisingly found no difference in performance between the flanked and unflanked conditions. The lack of a difference between the unflanked and the flanked conditions suggests no crowding for gaze judgments. However, it is important to note that we had to move the inner face flanker further toward fixation, so that it was only partially within the crowding zone to increase performance from chance. Therefore gaze direction cannot be accurately reported when flanking faces are too close to a target face, presumably due to crowding effects. This is consistent with a recent study by Kalpadakis-Smith et al. (2018) who report crowding for judgments about the spacing of the two eyes within the face. Although participants were unable to perform our gaze task when the flankers were partially within the crowding zones, this may simply reflect the fact that gaze direction judgments require finer resolution than those made by the participants in Kalpadakis-Smith et al. (2018).

Despite no difference between the flanked and unflanked conditions on gaze judgments we find significant differences in the peaks with the different flankers' gaze deviations conditions. This could arise for a couple of reasons. First, this might reflect occasional flanker substitution (e.g., where participants erroneously report the flanker gaze deviation rather than the target's (Estes et al., 1976; Chastain, 1982; Põder and Wagemans, 2007; Freeman et al., 2012). Although this may have occurred some of the time, we can rule out that participants consistently reported the flanker gaze deviation, since those target gaze deviations that were in opposite direction to the flankers' deviations were rarely misclassified to the flankers' gaze (e.g., a target face with a leftwards gaze of 20◦ surrounded by rightwards flankers was misclassified as rightwards 10% of the time). Alternatively, since the fixation point was presented on the

left side of the monitor, and the faces were always presented to the right of the fixation point, when the flankers' gaze was leftwards (L20 and L10), this could have been perceived by the participants as being directed toward them, leading to different CoDG peaks. This is in line with our findings that participants judged the target face as looking more direct only for flankers' gaze deviation L20 and L10. This could also be influenced by the head orientation of the stimuli. In this experiment, all the faces were forward-facing. It has been suggested that observers first perform a symmetry judgment on faces in the periphery (forward-facing faces are symmetrical while turned faces are non-symmetrical), and that forward-facing faces are categorized as "direct" over a wider range of eye gaze deviations in the periphery in forward facing heads (Palanica and Itier, 2011; Florey et al., 2015). Second, since the participants were looking at a fixation point, and the leftward looking flankers could also be perceived as looking at the fixation point, this might resemble some form of joint attention leading to a greater number of direct responses. For example, Edwards et al. (2015) suggest that people rapidly orient their attention toward an individual with whom they have established joint attention, although this only really occurs for joint attention of a real object (not fixation points).

Finally, we found that the emotional expression of the target face significantly influenced eye gaze perception, but found no effect of flankers' emotional expression on the categorization of the target gaze deviation. We had expected that the flankers' emotional content could influence performance on the judgment of target gaze in two different ways. On the one hand, when the flankers and target face had the same emotional expression, we might have expected more of an influence of the flankers gaze on the perception of target gaze since crowding

#### REFERENCES


is most pronounced when the flankers resemble the targets (Kooi et al., 1994). On the other hand, we might expect less influence of the flankers gaze on the target's gaze since the targets might have popped out (e.g., "gaze in the crowd effect"). We failed to find any difference, suggesting that the flankers did not pop out (this aligns with informal reports from our observers that they were unaware of the flankers' emotions), possibly because of the attentional demands of the task at hand.

#### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of Queen Mary University of London (QMUL) Ethics committee. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by Queen Mary University of London (QMUL) Ethics committee.

### AUTHOR CONTRIBUTIONS

DA and IM: experiment design, stimuli preparation, programming of experiment, running experiment, data analysis, and writing the manuscript. NE: experiment design, and manuscript editing.

# FUNDING

Deema Awad was supported by an ESRC Ph.D. studentship.

sulcus and inferior parietal lobule. Curr. Biol. 17, 20–25. doi: 10.1016/j.cub.2006. 10.052


Ekman, P. (1993). Facial expression and emotion. Am. Psychol. 48, 384–392.




**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Awad, Emery and Mareschal. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Socially Communicative Eye Contact and Gender Affect Memory

*Sophie N. Lanthier1 \*, Michelle Jarick2 , Mona J. H. Zhu3 , Crystal S. J. Byun1 and Alan Kingstone1*

*1 Brain, Attention, and Reality Laboratory, Department of Psychology, University of British Columbia, Vancouver, BC, Canada, 2Atypical Perception Laboratory, Department of Psychology, MacEwan University, Edmonton, AB, Canada, 3Cognition and Natural Behaviour Laboratory, Department of Psychology, University of Waterloo, Waterloo, ON, Canada*

#### *Edited by:*

*Paola Ricciardelli, University of Milano-Bicocca, Italy*

#### *Reviewed by:*

*Agnieszka Wykowska, Istituto Italiano di Tecnologia, Italy Juan Lupiáñez, University of Granada, Spain*

> *\*Correspondence: Sophie N. Lanthier snlanthier@gmail.com*

#### *Specialty section:*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology*

*Received: 06 November 2018 Accepted: 29 April 2019 Published: 22 May 2019*

#### *Citation:*

*Lanthier SN, Jarick M, Zhu MJH, Byun CSJ and Kingstone A (2019) Socially Communicative Eye Contact and Gender Affect Memory. Front. Psychol. 10:1128. doi: 10.3389/fpsyg.2019.01128*

Because of their value as a socially communicative cue, researchers have strived to understand how the gaze of other people influences a variety of cognitive processes. Recent work in social attention suggests that the use of images of people in laboratory studies, as a substitute for real people, may not effectively test socially communicative aspects of eye gaze. As attention affects many other cognitive processes, it is likely that social attention between real individuals could also affect other cognitive processes, such as memory. However, from previous work alone, it is unclear whether, and if so how, socially communicative eye gaze affects memory. The present studies test the assumption that socially communicative aspects of eye gaze may impact memory by manipulating the eye gaze of a live speaker in the context of a traditional recognition paradigm used frequently in the laboratory. A female (Experiment 1) or male (Experiment 2) investigator read words aloud and varied whether eye contact was, or was not, made with a participant. With both female and male investigators, eye contact improved word recognition only for female participants and hindered word recognition in male participants. When a female investigator prolonged their eye contact (Experiment 3) to provide a longer opportunity to both observe and process the investigator's eye gaze, the results replicated the findings from Experiments 1 and 2. The findings from Experiments 1–3 suggest that females interpret and use the investigator's eye gaze differently than males. When key aspects from the previous experiments were replicated in a noncommunicative situation (i.e., when a video of a speaker is used instead of a live speaker; Experiment 4), the memory effects observed previously in response to eye gaze were eliminated. Together, these studies suggest that it is the socially communicative aspects of eye gaze from a real person that influence memory. The findings reveal the importance of using social cues that are communicative in nature (e.g., real people) when studying the relationship between social attention and memory.

Keywords: gaze, eye contact, attention, memory, gender

# INTRODUCTION

From the moment a child is born, she/he begins to engage with her/his mother to communicate her/his needs and to have those needs met. This is the earliest example of the importance of social interactions, and intuitively, the importance of interpreting social cues from others remains essential throughout one's life. Our eyes are central to social interaction, as they convey a wealth of information about our emotional and mental states which people use to decode our behaviors and intentions (Emery, 2000). During a social interaction, people tend to look at other peoples' eyes to gauge whether they are interested (Argyle et al., 1974; Ellsworth and Ross, 1975), paying attention (Kleinke et al., 1975), and what their intentions may be (Kleinke, 1986; Baron-Cohen, 1995; Emery, 2000; Shimojo et al., 2003; Ristic et al., 2005; Frischen and Tipper, 2006). Accordingly, it has been argued that one's ability to attend to the eyes of others plays a critical role in understanding and facilitating social interaction (Kleinke et al., 1975; Cary, 1978; Campbell et al., 1990; Perrett and Emery, 1994; Emery, 2000; Vertegaal et al., 2001; Tomasello et al., 2005). On the other hand, failing to properly attend to the eyes of others has been linked to deficits in social functioning in autism spectrum disorder (see Senju and Johnson, 2009a for a review) as well as social anxiety disorder (Wieser et al., 2009; Schneier et al., 2011). Indeed, researchers have theorized that eye gaze represents a special social attentional cue (Baron-Cohen, 1995) that may be processed by dedicated neural mechanisms (such as that revealed by activity in the superior temporal sulcus, Campbell et al., 1990; Itier and Batty, 2009).

Researchers have attempted to study the eyes' importance as a social attentional cue by using variants of classic visual attention paradigms in conjunction with socially relevant stimuli (e.g., an image of a face looking at you). In these laboratorybased tasks, such a stimulus is presented on a computer screen and a person's eye movements – and other attentional behaviors in response to the stimulus – are recorded. Using different tasks (e.g., *free viewing*: Yarbus, 1967; Walker-Smith et al., 1977; Mojzisch et al., 2006; Birmingham et al., 2008, 2009; Kuhn et al., 2009; Schrammel et al., 2009; Foulsham et al., 2010; Foulsham and Sanderson, 2013; *attentional cueing*: Friesen and Kingstone, 1998, 2003; Driver et al., 1999; Langton and Bruce, 1999; Ristic et al., 2002; Vuilleumier, 2002; Senju and Hasegawa, 2005; Zwickel and Võ, 2010; Palanica and Itier, 2011; Wiese et al., 2012; Kuhn et al., 2014; Wykowska et al., 2014; Rensink and Kuhn, 2015; *visual search*: von Grunau and Anston, 1995; Senju et al., 2005; Doi and Ueda, 2007; Senju and Csibra, 2008; Doi et al., 2009; Palanica and Itier, 2011; and *face detection*: Macrae et al., 2002; Pageler et al., 2003; Vuilleumier et al., 2005; Conty et al., 2006, 2007; Itier et al., 2007, 2011) and a variety of stimuli (e.g., *images of faces*: Laidlaw et al., 2012; *complex scenes*: Vuilleumier et al., 2005; Birmingham et al., 2008, 2009; and *dynamic videos*: Foulsham et al., 2010; Foulsham and Sanderson, 2013; Kuhn et al., 2014; Wykowska et al., 2014), these studies show that people prefer to look at the eyes over any other feature on the face (Birmingham et al., 2008, 2009; Laidlaw et al., 2012; Levy et al., 2012) and that individuals are extremely sensitive to the signals they convey (e.g., people attend to where other people look, especially when they look at them, von Grunau and Anston, 1995; Senju et al., 2005; Vuilleumier et al., 2005; Conty et al., 2007; Doi and Ueda, 2007; Senju and Johnson, 2009b; Freeth et al., 2013).

While these laboratory tasks have made use of a variety of different social stimuli that vary in complexity and approximation to real-life social interactions, the stimuli in these tasks are seldom real people. Recently, researchers have asked similar questions about how we attend to the eyes in more natural settings, where the stimuli are live people instead of static images (Zuckerman et al., 1983; Patterson et al., 2002, 2007; Kuhn and Tatler, 2005; Tatler and Kuhn, 2007; Kuhn et al., 2008, 2016; Foulsham et al., 2011; Laidlaw et al., 2011, 2016; Risko et al., 2012; Wesselmann et al., 2012; Gallup et al., 2012a,b, 2014; Freeth et al., 2013; Wu et al., 2013, 2014; Gobel et al., 2015; Kompatsiari et al., 2018). These studies reveal that the way people respond, both behaviorally and neurologically, to a real person that they can interact with (or a robot that makes head and eye movements that resemble those of a human, Kompatsiari et al., 2018) is often different than the way they attend to an image of a person. For instance, in socially communicative settings where interactions between live people can occur (i.e., people involved in the interaction are aware that they can both send signals to and receive signals from each other; Risko et al., 2012; Gobel et al., 2015; Jarick and Kingstone, 2015; Myllyneva and Hietanen, 2015, 2016; Nasiopoulos et al., 2015; Risko and Kingstone, 2015; Conty et al., 2016; Risko et al., 2016), people will only look at one another if it is socially acceptable to do so.

While these same concerns regarding the ecological validity of attention to social stimuli should also apply to memory for social stimuli, images of people are used as stimuli in most of the work investigating how eye gaze affects memory. Some of these studies report that memory for an image of face (Hood et al., 2003; Mason et al., 2004; Smith et al., 2006) and for words (Fry and Smith, 1975; Kelley and Gorham, 1988; Macrae et al., 2002; Falck-Ytter et al., 2014) is improved when these stimuli are associated with direct gaze (though see Beattie, 1981; Conty et al., 2010; Nemeth et al., 2013). In the investigations that have used live people as stimuli, a speaker's eye contact has been associated with improved memory for what the speaker has said (Otteson and Otteson, 1980; Sherwood, 1987; Fullwood and Doherty-Sneddon, 2006; Helminen et al., 2016). Some of the earliest studies examined how children's academic performance could be affected by a teacher's eye contact (Otteson and Otteson, 1980; Sherwood, 1987). For instance, Sherwood (1987) suggested that learning could be enhanced in a classroom when instructors made eye contact with members in the audience. In a more recent study, male participants remembered more details from a story told by a male storyteller who looked at them relative to a storyteller who looked away, but female participants did not (Helminen et al., 2016). The research using images and live people as stimuli seems to indicate that eye contact enhances the processing and retention of information. However, other laboratory research suggests eye contact may actually hinder performance (Beattie, 1981; Conty et al., 2010;

Nemeth et al., 2013) consistent with the notion that eye contact draws attention and other cognitive resources away from the task at hand (Nemeth et al., 2013).

An important limitation of all past studies using live speakers is that researchers have generally not measured and/or systematically manipulated when a given participant actually experiences eye contact with the investigator. For example, in past studies, listeners were normally exposed to either a speaker who never made eye contact with the listener(s) in an audience or a speaker who periodically made eye contact with some undefined subset of listeners (Otteson and Otteson, 1980; Sherwood, 1987; Fullwood and Doherty-Sneddon, 2006; Helminen et al., 2016). As such, it is unclear how much eye contact a listener actually made with the speaker (if they experienced any eye contact at all). More importantly, it is also unclear whether the specific information that a listener recalled was actually the information that was presented when the speaker made eye contact, as the temporal synchrony between the speaker's eye contact and the spoken information was not controlled (Otteson and Otteson, 1980; Sherwood, 1987; Fullwood and Doherty-Sneddon, 2006; Helminen et al., 2016). As a result, these studies cannot determine whether memory effects related to gaze reflect, for example, an enhancement or a decline from direct gaze. At best, the mixed results from the research that has used images of faces suggest that both factors may be in play. As such, it remains unclear whether mutual eye contact actually enhances or hinders memory for verbal information. What is needed is a paradigm that is controlled enough to study the effect of eye gaze, without compromising the signal that eye contact provides in a natural setting (Jarick and Kingstone, 2015; Myllyneva and Hietanen, 2015; Nasiopoulos et al., 2015; Risko and Kingstone, 2015; Conty et al., 2016; Helminen et al., 2016; Risko et al., 2016). The goal of the present work is to develop a rigorous paradigm that would avoid this limitation and enable us to investigate whether eye contact enhances or hinders memory for spoken information.

As previous laboratory work has successfully measured other gaze-related memory effects using recognition tests (e.g., gaze cuing to visual word stimuli presented on a computer screen, Fry and Smith, 1975; Kelley and Gorham, 1988; Macrae et al., 2002; Hood et al., 2003; Mason et al., 2004; Smith et al., 2006; Dodd et al., 2012; Falck-Ytter et al., 2014), the studies presented in this paper will use a variant of these classic recognition tasks. The basic methodology is as follows. In an initial study phase, a participant will be seated across from an investigator (or a video of an investigator) who reads words out loud. Critically, before the investigator reads each word s/he will either look up to make eye contact with the participant or keep gaze down at the computer screen to avoid eye contact. Afterward, the participant will perform a recognition test containing the words studied with eye contact, the words studied without eye contact, and new words. The key dependent measure will be recognition accuracy. During the study, and in order to systematically control what information is presented with eye contact, a laptop computer screen, that is only visible to the investigator, will indicate the word to be read aloud and instructions on whether or not to make eye contact with the participant on a given trial. Participants will also be instructed to make eye contact with the investigator during the experiment and to look at the investigator's eyes if making eye contact is not possible (i.e., the investigator was looking down at the screen rather than at the participant). The investigator will monitor whether the participant makes eye contact, and participants who fail to make eye contact throughout the experiment will be excluded. In previous work, direct eye gaze has enhanced and hindered memory performance, so the effect that gaze could have in the present studies was very much an open question.

Gender has been suggested as a modulating factor in the effect of eye gaze (e.g., Helminen et al., 2016). Often eye contact helps all participants recognize a face, regardless of the their gender (Macrae et al., 2002; Hood et al., 2003; Mason et al., 2004; Smith et al., 2006). However, in some contexts, one gender will benefit from eye contact, but the other will not (e.g., Otteson and Otteson, 1980; Goodman et al., 2012; Helminen et al., 2016). Finally, researchers have observed gender differences in how attentive participants are to the eyes (Connellan et al., 2000; Lutchmaya et al., 2002) and the nonverbal signals of others (Hall, 1978; Rosenthal et al., 1979; McClure, 2000), as well as how responsive participants are to these signals (e.g., females maintain more distance between themselves and a virtual agent that makes eye contact than males; Bailenson et al., 2001; Bayliss et al., 2005). Because of this, gender should be systematically controlled since it is a factor that could influence how eye gaze affects performance. The experiments reported here use either a female (Experiments 1, 3, and 4) or a male (Experiments 2 and 4) investigator who looks at male and female participants. This experimental setup has the added benefit of permitting an examination of whether gender will influence any observed eye gaze-induced memory effects.

# EXPERIMENT 1

It is currently unclear whether socially communicative eye contact helps or hinders memory. To determine this, we manipulated whether an investigator reading words aloud made eye contact with a participant or not and determined how this manipulation affected participants' word recognition. If the investigator's eye contact is helpful when a participant encodes information, then recognition performance would be best for words spoken while the investigator made eye contact. Alternatively, if the investigator's eye contact interferes with encoding, recognition performance would be worse for words spoken while the investigator made eye contact with the participant. Since the gender of both participants' and the gaze cue (e.g., investigator) have been reported to modulate the effect of gaze on memory, this factor was systematically manipulated across the studies reported in this paper. In Experiment 1, the investigator was female.

# Method

#### Participants

Eighty-four undergraduate students from the University of British Columbia (42 males, 42 females) received course credit for participating. All reported speaking English as their first language. All had normal or corrected to normal vision and were naive about the purpose of the experiment. All participants gave informed consent before participating and all associated methods were approved by the University of British Columbia's Research Ethics Board [Towards a More Natural Approach to Attention Research 1-200, certificate #H10-00527, & Research in Cognitive Ethology, #H04-80767].

#### Design

A 2 (Investigator gaze: eye contact and no eye contact) by 2 (Participant gender: male and female) mixed design was used, where investigator gaze was manipulated within participant and participant gender was a between-participant variable.

#### Apparatus

E-Prime 2.01 controlled the timing and presentation of stimuli read aloud by the investigator to the participant and logged response accuracy and response times (RTs) in the recognition test. The stimuli were presented on a 17-in. monitor with a 1920 × 1,080 pixel resolution.

#### Stimuli

The stimulus pool consisted of the 120 words from Macdonald and Macleod (1998). The words were nouns 5–10 letters long, with frequencies greater than 30 per million (Thorndike and Lorge, 1944). From the 120 words, 3 lists containing 40 words each were randomly generated. For a given participant, two lists were selected for study; one list was presented with eye contact and the other list without. The third list was reserved for a recognition test. List selection was counterbalanced across participants such that each word was presented in each of the different conditions (i.e., with eye contact, without eye contact, new words for recognition) an even number of times across participants.

#### Procedure

Participants heard words for a later memory test. Participants were not informed that they would complete a memory test after hearing the words. During the initial encoding phase, participants were seated ~40 in. across from a female investigator who read aloud words individually. Critically, while the investigator read the words, she either looked up to make eye contact briefly (less than a second) with the participant or kept gaze down at the computer screen to avoid eye contact. Eighty words in total were read aloud in random order to the participants, half of which were presented with eye contact and the other half without.

A laptop screen (only visible to the investigator) indicated when a word was to be read aloud and provided instructions on whether or not to make eye contact with the participant on a given trial. To begin each trial, a blank screen appeared for 1,500 ms. Next, the instruction to look up at the participant or look down at the laptop was presented to the investigator. After 1,000 ms, a word also appeared and remained on screen for 3,000 ms. As soon as the word appeared on screen, the investigator would then look as instructed either toward the participant or down at the computer screen while she read the word aloud. On trials where the investigator made eye contact, the investigator would look back down at the computer screen as she finished saying the word (~1 s). Next, a blank white screen would appear for 500 ms to alert the investigator of the end of the trial. The words and eye contact instructions were randomly intermixed. Since neither the investigator nor participant had knowledge of the trial sequence, this would prevent any systematic change in a participant's eye gaze throughout the experiment. The investigators (who were authors MZ and CB) were trained to maintain a neutral facial expression and a consistent tone of voice irrespective of the gaze condition, though some natural non-systematic variation within and across participants was expected. The investigators memorized the instruction script for each phase, followed the instruction prompt on the computer screen (i.e., "look at participant or look at screen"), and subsequently read the words to the participant. Additionally, investigators rehearsed before testing to help maintain consistency.

To further ensure that eye contact between the investigator and participant was controlled, participants were instructed to make eye contact with the investigator during the experiment, and if making eye contact was not possible (i.e., the investigator was looking down at the screen) to look at the investigator's face. This way, eye contact could be made easily when the investigator looked up at the participant. Thus, eye contact was monitored by the investigator on a trial-by-trial basis. In instances when a participant failed to make eye contact with the investigator consistently (e.g., on one or more trials), the participant was excluded. Any failures to make eye contact were extremely rare, and of the 2 (out of 84) participants who were excluded, there was no ambiguity, i.e., they consistently failed to make eye contact. It is important to note that that people are very sensitive at judging where people are looking and when eye contact is being made, even when judging that behavior *via* a static image or video (e.g., Anderson et al., 2011). The instructional sequence visible to the investigator during the encoding phase is presented in **Figure 1A**.

Once the encoding phase was complete, the investigator would open a recognition test on the laptop and turn the laptop to face the participant. After reading the recognition test instructions to the participant, the investigator would monitor the participant's performance as they completed four practice trials (which were excluded from the analysis). Next, the investigator left the participant alone in the room to complete the recognition test. The recognition test contained the words studied with eye contact, the words studied without eye contact, and 40 new words. The test words appeared on

<sup>1</sup> www.pstnet.com

new words have been plotted in this figure as a reference point but were not included in the analysis. Error bars represent the 95% confidence interval as defined by Masson and Loftus (2003).

a computer screen in white font against a black background and were presented in random order. A fixation cross was presented for 500 ms before each word. When a word appeared, the subjects were instructed to make a "new" or "old" response for each test word by pressing buttons labeled "New" and "Old" on the keyboard. There was a 500-ms blank interval before each word appeared on screen, and the word offset with the subject's key response. The response accuracy and response times were recorded. The trial sequence used during the recognition phase trial is presented in **Figure 1B**. Once the recognition task was complete, the participant remained seated until the investigator came back to the room.

#### Results

A two-way mixed ANOVA was conducted on response time (RT) and response accuracy (percentage correct), with investigator gaze (two levels: with eye contact and without eye contact) as the within-participant factor and participant gender (two levels: male and female) as the between-participant factor.

#### Response Time

Mean RTs are presented in **Figure 2**. There were no main effects of investigator gaze (*F*(1,82) = 0.04, MSE = 20,386.56, *p* = 0.84) or participant gender (*F*(1,82) = 1.08, MSE = 229,174.88, *p* = 0.30). Nor was there an interaction between investigator gaze and participant gender (*F*(1,82) = 0.36, MSE = 20,386.56, *p* = 0.55).

#### Percentage Correct

Analysis of the accuracy data (see **Figure 3**) revealed no main effect of investigator gaze (*F*(1,82) = 0.37, MSE = 36.69, *p* = 0.55) or participant gender (*F*(1,82) = 0.88, MSE = 321.28, *p* = 0.35). Critically, there was an interaction between investigator gaze and participant gender (*F*(1,82) = 15.84, MSE = 36.69, *p* < 0.001), such that female participants recognized *more* words that were spoken while the investigator made eye contact (79%) than

when they did not (75%; *t*(41) = 3.27, SEM = 1.31, *p* < 0.005). However, male participants recognized *fewer* words read while the investigator made eye contact (73%) than when they did not (76%; *t*(41) = 2.37, SEM = 1.33, *p* < 0.05).

confidence interval as defined by Masson and Loftus (2003).

### Discussion

The results from this initial experiment demonstrate that memory is improved for words that were encoded with eye contact, but this effect was only observed in female participants. These findings suggest that eye contact has differential effects on memory for verbal information in males and females. Given that females are more attentive and responsive to nonverbal behavior than males (Hall, 1978; Connellan et al., 2000; Bailenson et al., 2001; Lutchmaya et al., 2002; Bayliss et al., 2005; Yee et al., 2007; Marschner et al., 2015), it is possible that, in the context of the present study, females dedicated more attention to information delivered during eye contact than males, which may have resulted in deeper processing and better retention of words presented with than without eye contact (Craik and Tulving, 1975).

Alternatively, it could be the case that making eye contact with the opposite sex produces higher levels of arousal compared to making eye contact with the same sex (Argyle and Dean, 1965; Donovan and Leavitt, 1980). Research has demonstrated that eye contact elevates physiological arousal (Kleinke and Pohlen, 1971; Nichols and Champness, 1971; Gale et al., 1978; Wieser et al., 2009; Helminen et al., 2011, 2016), and that high levels of arousal can interfere with performance on similar tasks (Jelicic et al., 2004; Smeets et al., 2007). While it is possible that eye contact holds one's attention by increasing arousal (i.e., affective arousal theory; Kelley and Gorham, 1988; Senju and Johnson, 2009b;

Mather and Sutherland, 2011), eye contact between genders could produce excess arousal and anxiety, and thus, interfere with memory. Accordingly, male participants could have experienced more arousal than the female participants while making eye contact with the female investigator and were more distracted while words spoken with eye contact as a result (Nemeth et al., 2013). Since both the task and processing the eye contact are competing for cognitive resources, performance on the cognitive task suffers. We examined these possibilities in Experiment 2.

#### EXPERIMENT 2

In the previous experiment, only female participants benefited from the investigator's gaze. This finding could be attributed to the female investigator's eye contact distracting male participants from processing what is said (Nemeth et al., 2013). It could be the case that female participants were simply more attentive to the female investigator's eye contact than males irrespective of the investigator's gender (Hall, 1978; Connellan et al., 2000; Bailenson et al., 2001; Lutchmaya et al., 2002; Bayliss et al., 2005; Yee et al., 2007; Marschner et al., 2015). We seek to distinguish between these possibilities in Experiment 2 by using a male investigator. If males now benefit from eye contact (and females are possibly hindered by eye contact), this would support the idea that the investigator's gender contributes to the memory effect *vis-à-vis* its relation to the participant. However, if the results replicate Experiment 1, then a participant's gender is a contributing factor to how eye gaze influences memory, a finding that would also be consistent with the notion that females interpret nonverbal social cues differently than males.

#### Method

#### Participants

Eighty-four undergraduate students from the University of British Columbia (42 males, 42 females) who had not previously participated in Experiment 1 received course credit for participating. All had normal or corrected to normal vision and were naive about the purpose of the experiment.

#### Design, Apparatus, Stimuli, and Procedure

The design, apparatus, stimuli, and procedure were identical to those used in the previous study, with the exception that now a male investigator read the words aloud to the participants instead of a female investigator.

#### Results

Data analysis followed the same procedure that was used in Experiment 1.

#### Response Time

Mean RTs are presented in **Figure 4**. There was a marginally significant main effect of participant gender (*F*(1,82) = 3.35, MSE = 140,584.08, *p* = 0.07), such that females (967 ms) were

faster to respond than males (1,055 ms). No other main effects or interaction were significant (all other *F*'s < 1).

#### Percentage Correct

Analysis of the accuracy data (see **Figure 5**) revealed no main effect of investigator gaze (*F*(1,82) = 0.15, MSE = 41.95, *p* = 0.70) or participant gender (*F*(1,82) = 0.03, MSE = 463.44, *p* = 0.87). Critically, there was an interaction between investigator gaze and participant gender (*F*(1,82) = 15.22, MSE = 41.95, *p* < 0.001), such that female participants recognized *more* words that were spoken while the investigator made eye contact (77%) than when they did not (72%; *t*(41) = 3.68, SEM = 1.17, *p* < 0.001). However, male participants recognized *fewer* words read while the investigator made eye contact (73%) than when they did not (77%; *t*(41) = 2.16, SEM = 1.62, *p* < 0.05).

#### Discussion

Results from Experiment 2 replicate the findings reported in Experiment 1, wherein females benefited from eye contact on recognition tests and males did not. Furthermore, the present experiment rules out a same-gender explanation of Experiment 1, where memory was improved only when making eye contact with a person of the same gender (i.e., female investigator and female participants). Here, female participants showed a memory benefit from eye contact with a male investigator as well. These results are consistent with the notion that females are generally more attentive to gaze cues than males (Hall, 1978; Connellan et al., 2000; Bailenson et al., 2001; Lutchmaya et al., 2002; Bayliss et al., 2005; Yee et al., 2007; Marschner et al., 2015). It is possible that female participants decode the various signals that could be embedded within the investigator's eye contact (e.g., a signal to pay attention, a signal that the

investigator is watching you, information about the investigator's mental state, etc.) more readily than males. Thus, females quickly interpret this eye contact as a signal to pay attention, and dedicate more cognitive resources to words presented while the investigator looks at them (e.g., Otteson and Otteson, 1980; Sherwood, 1987). On the other hand, males may find interpreting the investigator's eye contact distracting since it is uninformative to the task, or perhaps they require more cognitive resources than females to process the different social signals. As a result, their performance on the task at hand is impaired.

point, but were not included in the analysis. Error bars represent the 95%

confidence interval as defined by Masson and Loftus (2003).

### EXPERIMENT 3

In the two previous experiments, the eye contact that was initiated by the investigator was quite brief (i.e., a quick glance (less than 1 s) up at the participant as the investigator said the word aloud). Although this brief glance may have provided enough time for females to decode the eye contact, it is possible that male participants needed longer periods of eye contact in order to decode this social cue. Indeed, in more natural settings, people tend to engage in eye contact with others between 1.7 and 3.6 s (Argyle and Dean, 1965; Helminen et al., 2011). To determine whether more eye contact can help males decode social cues more effectively, in the present study, the investigator made eye contact for a longer period of time (approximately 3 s). In order to maximize any effect of prolonged eye contact on the male participant's arousal, whether it is beneficial arousal or distracting arousal, we used a female investigator. If prolonged durations enable men to use eye contact as a social cue to enhance verbal information processing, then the present study should reduce or eliminate the effects of participant gender on word recognition.

# Method

#### Participants

Eighty-four undergraduate students from the University of British Columbia (42 males, 42 females) who had not previously participated in Experiment 1 or 2 received course credit for participating. All had normal or corrected to normal vision and were naive about the purpose of the experiment.

#### Design, Apparatus, Stimuli, and Procedure

The design, apparatus, stimuli, and procedure were identical to those used in Experiments 1 and 2, with the exception that the female investigator made *prolonged eye contact* when instructed to look at the participant instead of brief eye contact to further accentuate any effect of eye gaze. As in the previous experiments, a laptop that only the investigator could see provided the same instructions on what word to say and whether to make eye contact or not as the word was spoken. On trials where the investigator made eye contact with the participant, the investigator lifted her eyes and read the word when the laptop displayed the word and made continuous eye contact with the participant until the screen flashed white after 3,000 ms, at which point the investigator would return her gaze down to the computer screen.

#### Results

Data analysis followed the same procedure that was used in Experiments 1 and 2.

#### Response Time

Mean RTs are presented in **Figure 6**. There was a main effect of participant gender (*F*(1,82) = 15.71, MSE = 200,664.95, *p* < 0.001), such that females (1,019 ms) were faster to respond than males (1,341 ms). No other main effects or interactions were significant (all other *F's* < 1).

#### Percentage Correct

Analysis of the accuracy data (see **Figure 7**) revealed a main effect of investigator gaze (*F*(1,82) = 5.03, MSE = 47.31, *p* < 0.05), such that words presented with investigator eye contact (75%) were accurately recognized more than those presented without (73%). There was no main effect of participant gender (*F*(1,82) = 1.77, MSE = 389.94, *p* = 0.19). Critically, there was an interaction between investigator gaze and participant gender (*F*(1,82) = 6.37, MSE = 47.31, *p* < 0.02), such that female participants recognized *more* words that were spoken while the investigator made eye contact (79%) than when they did not (74%; *t*(41) = 3.45, SEM = 1.47, *p* < 0.001). However, male participants were no more likely to recognize words read with (72%) or without investigator eye contact (72%; *t*(41) = 0.19, SEM = 1.53, *p* = 0.54).

#### Comparison Between Experiments 1, 2, and 3

An additional follow-up analysis comparing all three experiments was run to reveal any differences (or similarities) in the effects eye contact had on memory in each experiment. A three-way mixed ANOVA was conducted on mean response accuracy with investigator gaze (two levels: with eye contact and without eye contact) as the within-participant factor and experiment (three levels: female investigator with brief glance, male investigator with brief glance, and female investigator with prolonged gaze) and participant gender (two levels: male and female) as between-participant factors. The analysis of the response accuracy data revealed a marginal main effect of investigator gaze (*F*(1,246) = 3.67, MSE = 42.00, *p* = 0.06), such that participants recognized more words that the investigator

included in the analysis. Error bars represent the 95% confidence interval as defined by Masson and Loftus (2003).

said while making eye contact than when they did not. Critically, this was qualified by a reliable interaction between investigator gaze and participant gender (*F*(1,246) = 35.16, MSE = 42.00, *p* < 0.001), such that female participants recognized *more* words that were spoken while the investigator made eye contact (78%) than when they did not (74%; *t*(125) = 6.00, SEM = 0.76, *p* < 0.001). However, male participants recognized *fewer* words read while the investigator made eye contact (73%) than when they did not (75%; *t*(125) = 2.67, SEM = 0.87, *p* < 0.01). No other main effects or interactions were significant (all *F*'s < 1.27). Note that the lack of a three-way interaction (*p* = 0.65) is supported by a Bayes factor estimated using Monte Carlo sampling *via* the BayesFactor package in R (Morey and Rouder, 2018). Specifically, the Bayes factor was 0.125:1 when comparing a model with the three-way interaction to a model without (i.e., only two-way interactions and main effects). In other words, there is about eight times more evidence against including a three-way interaction in the model.

#### Discussion

The results of Experiment 3 replicated the finding in both Experiments 1 and 2 that females recognized more words and were more sensitive to words presented with eye contact than without eye contact. Unlike the previous studies, male participants' recognition performance was not significantly different in the eye contact and no eye contact conditions. Further, the follow-up analysis comparing all three experiments revealed that the length of eye contact did not modify the effect eye contact had on memory in female participants.

The failure to observe any interaction between Experiment and Investigator gaze or Experiment, Investigator gaze, and Participant gender suggests that exaggerating eye contact does not help the males encode the nonverbal eye contact cues, nor did it enhance the eye contact benefit observed in female participants (as compared to female participants in Experiments 1 and 2)2 . These results also suggest that the memory benefits that arise from eye contact in Experiments 1–3 are unlikely to be mediated by an arousal response, since prolonging the eye contact in the present experiment would have, if anything, increased arousal, which could modify how eye contact affects performance3 . Instead, we suggest that these results are more consistent with the idea that eye contact provides a signal to pay attention, and that interpreting this social signal is responsible for enhancing information processing in the previous studies. If anything, it seems that lengthening the investigator's eye contact reduced the interference eye contact caused the males in Experiments 1 and 2. By providing the signal longer, males may have had enough resources to both process the eye contact and perform the memory task without these two tasks competing for cognitive resources. Since they did not perform better when the investigator made eye contact than when they did not, it seems that this cue may not have had social relevance for the males.

The interpretations discussed here all assume that the investigator's eye gaze is being interpreted (at least by female participants) as a socially communicative cue. While the live interaction between the investigator and the participant ensures that social communication can occur, it is also possible that a nonsocial cue associated with the investigator's eye gaze could also be driving the reported effects. Before concluding that socially communicative aspects of gaze produce these effects, it is important to exclude a nonsocial interpretation that could possibly account for the facilitatory effect of gaze. This will be addressed in Experiment 4 where the social communicative aspects of eye gaze will be dissociated from the purely perceptual cues by using a video of the investigator instead of a live investigator.

# EXPERIMENT 4

The previous studies demonstrated that females benefited from an investigator's gaze on a subsequent memory test, whereas males did not. These gender-specific memory effects could be driven by a socially communicative cue that is embedded in the investigator's eye contact (i.e., when someone looks at you, it is a signal to pay attention). According to this idea, females might have been sensitive to the social cue embedded in the investigator's eye contact, and could have used it to facilitate their performance on the recognition test. However, male participants might have failed to interpret and apply the investigator's eye contact as a signal to pay attention, and as a result their performance at test could have been hindered by the investigator's eye gaze.

A different possibility altogether is that there was nothing socially communicative about the investigator's gaze that drove the memory effects observed in the previous studies. For example, these effects could have arisen by observing the investigators shifting their gaze up from the computer monitor. In the previous studies, the investigators either kept their eyes on the computer screen while they read a word, or they lifted them to make eye contact just before saying a word. Observing just the movement of the eyes up from the computer screen could be an indicator that a word is about to be spoken, much in the same way the onset of a flashing light at a crosswalk indicates that one should pay attention for pedestrians. There is nothing inherently "social" about either of these cues, but they both serve the purpose of a warning cue that informs a participant to increase attention to an upcoming stimulus (i.e., a word or a pedestrian in the latter case). In fact, a variety of perceptual cues (i.e., arrows, flashes in the periphery, etc.) are known to generate changes in attention (e.g., Posner, 1980; Friesen et al., 2004; Bayliss et al., 2005; Hietanen et al., 2006; Ristic et al., 2007; Mulckhuyse and Theeuwes, 2010; Shin et al., 2011; Hayward and Ristic, 2015).

<sup>2</sup> We thank an anonymous reviewer for raising the possibility that the same effect might not be obtained with a male investigator and prolonged gaze. Based on the current data indicating that there is no effect of investigator gender or gaze duration, we think a 3-way interaction is unlikely, but we acknowledge that it is possible.

<sup>3</sup> Note that nervous system arousal was not directly measured or analyzed in this study and therefore we cannot be certain that arousal was mediating memory performance. Future studies should incorporate measures of arousal using galvanic skin response and/or pupil dilation to investigate this further.

Given that both perceptual and socially communicative cues were embedded in the live investigator's eye contact in the previous experiments, it is unclear which cue was actually driving the memory effects observed in the previous experiments.

The aim of Experiment 4 is to clarify whether socially communicative cues are responsible for the eye gaze-related effects observed in the previous experiments. One way to isolate the social aspects of eye gaze from the perceptual ones is to have observers watch a video of the investigator instead of interacting with a live investigator. Numerous studies have demonstrated that people respond differently, both behaviorally and neurologically, when looking at the eye gaze of people presented in images versus actual, physically present people (Hietanen et al., 2008; Itier and Batty, 2009; Teufel et al., 2010; Laidlaw et al., 2011; Pönkänen et al., 2011a,b; Risko et al., 2012, 2016; Schilbach et al., 2013; Schilbach, 2015). Furthermore, the eye gaze and gestures of people depicted in images and videos have less influence on the communication (Heath and Luff, 1993; Gullberg and Holmqvist, 2006) and attention (Varao-Sousa and Kingstone, 2015; Wammes and Smilek, 2017) of an observer than they typically would during an encounter with a live person. Presumably, this is because the people depicted in the images and videos cannot see the observer and therefore their gaze behavior is not actively communicating with the observer, and *vice versa* (De Jaegher et al., 2010; Schilbach, 2010; Risko et al., 2012).

By using a video recording of the investigator in the present study, the socially interactive context that was produced by using a live investigator in Experiments 1, 2, and 3 is removed. If the previous findings are replicated, it would suggest that perceptual cues derived from the eye gaze of someone in a video are enough to generate the memory benefits and deficits associated with eye contact, and that a socially communicative context is not required to generate these memory effects. However, eliminating eye-gaze related memory effects would be evidence for the idea that perceptual cues are not driving these previously observed effects. Instead, it would suggest that the socially communicative eye gaze from an individual that an observer could potentially interact with is required to produce these memory effects.

### Method

#### Participants

To examine investigator gender as a factor in one experiment rather than in two separate experiments, as was the case in Experiments 1 and 2, the sample size was doubled to 168 undergraduate students from the University of British Columbia (84 males, 84 females) who had not participated in any of the previous experiments. All received course credit for participating and had normal or corrected to normal vision and were naive about the purpose of the experiment.

#### Design

A 2 (Investigator gaze: eye contact and no eye contact) by 2 (Participant gender: male and female) by 2 (Investigator gender: male and female) mixed design was used, where investigator gaze was manipulated within participant and participant gender and investigator gender were a between-participant variables.

#### Apparatus and Stimuli

The apparatus and stimuli were identical to those reported in the Experiments 1 and 2; however, the participants now watched a video of the investigator from either Experiment 1 (female investigator) or Experiment 2 (male investigator). The videos shown to each participant were recorded by a camera that was placed in front of the investigator, on a tripod that was adjusted so that the camera was positioned at the investigator's eye level. This position was chosen to simulate the distance, height, and eye level of a participant who would have sat across from the live investigator in the previous experiments. A confederate also sat directly behind the camera and looked at the investigator's eyes. The intention was to simulate the same live interaction to ensure that any variation in expressiveness due to a live context (Experiments 1–3) was also generated when the investigator was videotaped. During the recordings, the investigator read the words aloud as in the previous experiments, i.e., when prompted by the laptop, the investigator either looked toward the computer screen or, to simulate eye contact for the viewer, briefly toward the camera lens. A total of six different videos were made to ensure that across participants, each word would appear in each condition evenly.

Videos were presented full screen at the recorded resolution (1,920 × 1,080 pixels), on a 17-in. monitor. Participants were seated approximately 60 cm from the screen. Sound from the videos was also played through speakers built into the computer.

#### Procedure

The procedure was identical to those used in Experiments 1 and 2, with the exception that a participant was first assigned to watch a video of either a male or female investigator saying the words out loud. Participants were also instructed to look at the investigators' eyes throughout the experiment. Based on a wealth of past work indicating that there is a preferential bias to look at the eyes of people when they are shown in photos or in videos (Laidlaw et al., 2011; Risko et al., 2012, 2016), we expected participants to readily comply with the instructions. Their performance during practice and self-report after testing support this prediction.

#### Results

A three-way mixed ANOVA was conducted on response time (RT) and response accuracy, with investigator gaze (two levels: eye contact and no eye contact) as the within-participant factor and participant gender (two levels: male and female) and investigator gender (two levels: male and female) as the betweenparticipant factors.

#### Response Time

Mean RTs are presented in **Figure 8**. There was a main effect of investigator gender (*F*(1,166) = 13.47, MSE = 214,475.60, *p* < 0.001), such that participants were faster to respond with the male investigator (1,018 ms) than the female investigator (1,203 ms). No other main effects or interactions were significant (all other *F's* < 1).

#### Percentage Correct

Analysis of the accuracy data (**Figure 9**) revealed a main effect of participant gender (*F*(1,166) = 4.43, MSE = 542.79, *p* < 0.05), such that female participants were more accurate (72%) than male participants (66%). No other main effects or interactions were significant (all other *F's* < 1.4).

#### Comparison Between Live and Videotaped Investigators

To reveal any difference in the memory effects generated by the eye gaze of a live investigator (in Experiments 1 and 2) and a videotaped investigator, a four-way mixed ANOVA was conducted on response accuracy with Investigator gaze (two levels: with eye contact and without eye contact) as the withinparticipant factor and Investigator presence (two levels: live and videotaped), Investigator gender (two levels: male and female) and Participant gender (two levels: male and female) as between-participant factors.

The analysis of mean RTs revealed that there was a main effect of investigator presence (*F*(1,328) = 12.19, MSE = 199,677.54, *p* < 0.001), such that participants were faster to recognize words that were said by the in-person investigator (987 ms) than the videotaped investigator (1,111 ms). There was also a main effect of investigator gender (*F*(1,328) = 11.77, MSE = 199,677.54, *p* < 0.001), such that participants were faster to recognize words that were said by the male investigator (991 ms) than the female investigator (1,109 ms). There was a marginal interaction between investigator presence and investigator gender (*F*(1,328) = 3.80, MSE = 199,677.54, *p* = 0.05), such that when the investigator appeared over video, participants recognized words that were said by a male investigator (1,018 ms) faster than a female investigator (1,203 ms). However, when the investigator was in-person, participants recognized words that were said by a male investigator (965 ms) as fast as words said by a female investigator (965 ms). No other main effects or interactions were significant (all *F*'s < 2.6).

The analysis of the accuracy data revealed a main effect of investigator presence (*F*(1,328)=13.43, MSE = 467.57, *p* < 0.001), such that participants recognized more words said by an in-person investigator (75%) than a videotaped investigator (69%). Critically, there was a three-way interaction between investigator gaze, investigator presence, and participant gender (*F*(1,328) = 15.88, MSE = 44.75, *p* < 0.001). When the investigator was in-person, there was an interaction between investigator gaze and participant gender (*F*(1,166) = 31.38, MSE = 38.85, *p* < 0.001) such that female participants recognized more words that were spoken while the investigator made eye contact (78%) than when they did not (74%; *t*(83) = 4.91, SEM = 0.87, *p* < 0.001). However, male participants recognized fewer words read while the investigator made eye contact (73%) than when they did not (76%; *t*(83) = 3.19, SEM = 1.04, *p* < 0.005). In contrast, the same analysis for the videotaped investigator, presented in Section "Percentage correct" (Experiment 4), yielded no effect of investigator gaze, nor an interaction between investigator gaze and participant gender. These findings indicate that the failure to observe any memory effects in Experiment 4 is because eye gaze from a live investigator is fundamentally different from the eye gaze of a videotaped investigator.

#### Discussion

In Experiment 4, there was no evidence that eye gaze displayed over video influenced memory. This was true regardless of

the investigator or participant's gender. This stands in direct contrast with the previous experiments that demonstrated that eye contact from a live investigator improved memory in female participants, and reduced memory in male participants. Taken together, the results demonstrate that people attend to the eye gaze of those they interact with in real life differently than the eye gaze of people depicted in images.

The current data dovetail with a growing body of research suggesting that viewing a live person elicits different neurological (Hietanen et al., 2006; Pönkänen et al., 2011a,b) and behavioral responses (Heath and Luff, 1992) than viewing an image of a person. Indeed, the finding that participants who listened to a live investigator recognized more words, faster than those who listened to a video of the same words, supports this notion. By using images, the present study stripped away the social cues that would typically be present during a live encounter, leaving only the perceptual cues associated with eye gaze. Since the effect of eye gaze was eliminated when presented over video, the implication is that the effects observed previously in Experiments 1–3 were driven by communicative cues associated with eye contact instead of noncommunicative perceptual cues.

#### GENERAL DISCUSSION

In the present studies, female participants benefited from the investigator's gaze more than male participants on a subsequent memory test. Specifically, female participants recognized more words in a subsequent memory test when the words were previously associated with eye contact from an investigator than when they were not. This was true regardless of the investigator's gender (female in Experiments 1 and 3 and male in Experiment 2), and whether the investigator's gaze was a quick glance (as in Experiments 1 and 2) or a prolonged stare (in Experiment 3). In contrast, male participants showed no benefit of the investigator's gaze on subsequent memory tests when the investigator's gaze was held longer (Experiment 3), and actually recognized fewer words on a subsequent memory test when they were associated with brief eye contact from the investigator (Experiments 1 and 2) relative to when they were not. While these findings suggest that eye gaze could provide a useful social cue to pay attention for females, this is not the case for males. Importantly, the data suggest it was something socially communicative about eye gaze (rather than something nonsocially communicative), since eye gaze effects disappeared in the absence of a communicative setting when the investigator was presented over video (Experiment 4).

Our finding that females and not males showed a memory enhancement to gaze contradicts a recent study by Helminen et al. (2016) who showed the opposite. Our methods were very different, such that Helminen et al. read aloud narratives, manipulated eye gaze spontaneously, and tested memory recall. Because of this, it is unclear whether the specific information that listeners recalled in Helminen et al. was spoken while the speaker made eye contact or not because the temporal synchrony between the speaker's eye contact and spoken information was not controlled. As a result, the data from Helminen et al. (2016) are equivocal as to whether memory effects related to gaze reflect an enhancement from direct gaze or a decline resulting from gaze aversion. Our methods, however, can determine whether memory for words is enhanced or diminished by directly pairing each word with direct or averted gaze, and comparing memory in those conditions to a neutral (no gaze) condition. Our design allowed us to discover that females benefited from eye contact by remembering those words more compared to the control (no gaze) condition, while males showed a reduction in memory for words paired with eye contact compared to the control. Thus, we believe our methods can more accurately depict the how eye gaze modulates memory in males and females.

### Gender Differences in Eye Gaze-Related Memory Effects

These findings converge with a body of literature that suggests males and females attend to social information differently. In comparison to males, females dedicate more attention to social stimuli, such as faces and eyes (Connellan et al., 2000; Lutchmaya et al., 2002) and more easily decode the nonverbal signals exhibited by others (Hall, 1978; Rosenthal et al., 1979; McClure, 2000). More importantly, previous studies have demonstrated that females are more responsive to eye gaze (Bailenson et al., 2001; Bayliss et al., 2005). These data suggest that females are more sensitive to social signals in general and eye gaze in particular, and thus the investigator's eye contact engaged their attention in the first three experiments (despite being irrelevant to the task).

The general interpretation of the memory effect observed in females is that when eye contact accompanies information, it is interpreted as signaling the intent to communicate information that warrants attention (Duncan, 1972; Niederehe and Duncan, 1974; Kampe et al., 2003; Csibra and Gergely, 2009; Senju and Johnson, 2009a). This is particularly noteworthy since the investigator's eye contact is actually irrelevant to the task at hand. As such, when people communicate with each other and their message is preceded or accompanied with eye contact, their eye contact serves to highlight the most important parts of their message. It would follow that in the present studies, when the female participant was with the speaker, information spoken with eye contact would be attended more and recognized better than information presented without eye contact.

While female participants noticed, and decoded the social signals associated with the investigator's gaze with apparent ease, male participants did not. It is possible that male participants were simply insensitive to the investigator's eye contact and/ or did not think the eye contact was an important cue, and dedicated the same amount of attention to words regardless of whether the investigator looked at them or not. However, this explanation seems unlikely given that in male participants, the investigator's eye contact, if anything, had a negative effect on their performance in Experiments 1 and 2. Moreover, participants were instructed to look at the investigator's eyes throughout the study, so it seems inconceivable that eye contact with the investigator went unnoticed.

Another intriguing possibility is that the males were sensitive to the investigator's eye gaze, but were unable to decode which signal the investigator intended to convey since eye contact provides a variety of different social signals. While eye contact may signal to pay attention, as previously discussed, eye contact also signals that one is being monitored (Guerin, 1986; Risko and Kingstone, 2011; Pönkänen et al., 2011b; Freeth et al., 2013; Baltazar et al., 2014; Marschner et al., 2015; Nasiopoulos et al., 2015; Hazem et al., 2017), and facilitates decoding of the emotional and intentional messages of others. Discerning which message is most important and/or appropriate (and requires the most attention) in a given situation may be more challenging for males than females. In the present context, males may have struggled to dissociate which signal embedded in the investigator's eye contact was most relevant (or if they did make this distinction, they appear not to have acted on it). As such, processing or actively ignoring the investigator's eye contact may have interfered with the males' ability to pay attention to information that was being spoken. This idea converges with the finding in previous work that the presence of eyes interfered with performance on a Stroop task (Beattie, 1981; Conty et al., 2010; Nemeth et al., 2013), presumably because processing the eyes required the same cognitive resources (e.g., selective attention) used to perform the task. According to this idea, the brief eye contact provided in Experiments 1 and 2 was too difficult for the males to decode while simultaneously completing another task. As a result, they were unable to dedicate enough attention to the task of attending to what the investigator said when they were looked at, and their performance suffered. However, the less subtle signal provided in Experiment 3 did not alter the previous data pattern, undermining the interpretation that males just need a more salient gaze signal for it to yield a performance benefit.

Another similar, though slightly different, explanation of how a participant's gender modified whether eye contact had a positive or negative impact on memory comes from a proposal put forward by Conty et al. (2016). The authors proposed that direct gaze first captures one's attention and then triggers selfreferential processing, i.e., a heightened processing of contextual information in relation with the self (Northoff et al., 2006). According to this account, direct gaze can have both positive and negative effects on performance since the tendency to pay attention to the direct gaze of others either facilitates or interferes with performance on a task (e.g., direct gaze may facilitate processing a face, but hinder processing information that is not related to the face). However, once direct gaze has triggered self-referential processing, any information associated with it would be prioritized. Indeed, a large body of research suggests that memory is improved for information processed in relation to oneself (i.e., the self-referential memory effect; Macrae et al., 2002; Northoff et al., 2006; Kim, 2012).

In Experiments 1–3, it is possible that the extent to which the speaker's eye contact triggered self-referential processing differed between males and females. Females may have processed both the speaker's eye contact and the self-referential cue it provides simultaneously, or simply processed these two signals more efficiently and sequentially. Thus, any interference (if any was experienced) in hearing what the speaker said, caused by simply processing the speaker's eye contact, was overridden by the self-referential processing benefit triggered through the speaker's eye contact. Males on the other hand may notice and process the speaker's eye contact, but not the self-referential cue it provides. As a result, the speaker's eye contact only interferes with processing what the speaker says. This could be due to interference caused by processing any self-referential cue in the context of the task (i.e., any self-referential cue could be distracting since it is irrelevant to the task of listening to everything the speaker says), or only self-referential cues conveyed through eye contact. Moving forward, it is still unclear which social signals communicated by the live investigator are being interpreted by participants. The findings are consistent with the idea that eye contact provides a social signal to pay attention, which results in memory benefits for information communicated with eye contact. However, there are many nonverbal social cues conveyed during a live encounter that can influence the way an observer pays attention. In fact, some research might suggest that head movements, rather than eye movements, are more important in eliciting attentional shifts in more natural contexts (Tomasello et al., 1998, 2005; Emery, 2000). Even though we have attributed the effects in Experiments 1, 2, and 3 to a social signal conveyed by eye gaze, it is possible that a different social signal that is also associated with the investigators' gaze could be driving these effects. Future work could clarify whether it was the investigator's eye contact or a general social cue that was associated with the eye contact that generated the memory benefits observed in the female participants in the previous studies.

Furthermore, although we have considered the no eye contact condition as the control condition, strictly speaking, it is theoretically possible that our observed memory effects reflect a performance decline without eye contact rather than an enhancement with eye contact. Recall that in previous research, listeners watched a speaker who never made eye contact with any listener in an audience or one who periodically made eye contact with some undefined listeners (Otteson and Otteson, 1980; Sherwood, 1987; Fullwood and Doherty-Sneddon, 2006). Even in instances where a single listener is present (in Experiments 1–3 and in Helminen et al., 2016), it is possible that a listener's memory was improved for the information spoken while making eye contact with the speaker, and it is also possible that a listener had worse memory for information presented while the speaker avoided eye contact. This reduction in memory could be due to the observer feeling excluded by the speaker (a possibility considered but not addressed by Fullwood and Doherty-Sneddon, 2006) or because the speaker's gaze directs the observer's attention elsewhere. That said, the fact that participants in our study performed better in the presence of a real investigator than a videotaped investigator, even in the no eye contact condition, suggests that performance is being enhanced in the eye contact condition rather than diminished in the no contact condition. It remains for future studies, however, to confirm this interpretation by testing the adequacy of our baseline, for instance, comparing it to a situation where the speaker could make eye contact with someone other than the participant.

# Contributions of Live and Non-live Settings to Eye Gaze-Related Memory Effects

By presenting a video of an investigator (Experiment 4) instead of a live investigator (Experiments 1–3), the socially communicative function of eye gaze was removed. A live investigator generates an interactive context in which both the investigator and the participants can convey and observe signals with their eyes (Risko et al., 2012, 2016; Baltazar et al., 2014; Gobel et al., 2015; Jarick and Kingstone, 2015; Myllyneva and Hietanen, 2015; Nasiopoulos et al., 2015; Risko and Kingstone, 2015; Conty et al., 2016; Hietanen, 2016; Hazem et al., 2017). This is not the case when the investigator is presented over video, since the investigator cannot observe any signals that the participants convey through their eye gaze. This notion is also supported by research showing that social centers in the brain are more activated when observing live people than when viewing images of people (Hietanen et al., 2006; Schilbach et al., 2010; Pönkänen et al., 2011a,b; Schilbach, 2015). Viewing a pre-recorded investigator enabled a strong test of whether nonsocial signals embedded in eye gaze could generate memory effects that were previously observed in response to a live investigator's eye gaze in Experiments 1–3. The results were unequivocal. The memory effects previously observed in response to a live investigator's eye gaze disappeared. Without a socially communicative context, eye gaze had no effect on memory. This finding provides strong support for the idea that socially communicative signals conveyed through eye gaze influence memory. This also stands in contrast with a noncommunicative explanation of how eye contact could affect memory. The notion above presents a challenge to researchers who have generally assumed that using images enables them to study social aspects of eye gaze present in real life with real people. However, if this assumption is misplaced, and indeed more and more research is suggesting this assumption may be, then there could be broad-reaching implications as researchers have been using images to study the social effects of eye gaze for decades.

# Memory Mechanisms Affected by Eye Contact

While the present work has demonstrated that manipulating a speaker's eye contact during encoding/consolidation can influence recognition memory for semantic (word) information, it has not explored or manipulated memory for other types of materials (e.g., faces) or different memory processes (e.g., retrieval). An important question related to memory retrieval is whether a speaker's eye contact affects *recall* as well recognition. While studies from natural settings suggest that viewers generally recall more information when speakers periodically make eye contact than when they do not (Otteson and Otteson, 1980; Sherwood, 1987; Fullwood and Doherty-Sneddon, 2006), this notion remains to be tested in a rigorous paradigm that permits one to assess who is, and is not, receiving eye contact, and what information specifically is being delivered in those moments.

Another interesting question relevant to memory retrieval is whether making eye contact during retrieval will help or hinder this process. Some research suggests that direct gaze during the retrieval process can enhance memory for a face (Hood et al., 2003; Smith et al., 2006). However, this question has yet to be tested in a paradigm that systematically manipulates a speaker's eye gaze during the retrieval process.

In sum, there are a number of different aspects of memory that could be influenced by eye contact. Future studies could extend the present work by exploring the effects of eye contact on all of the different components of memory mentioned at the outset of this section. They could also, for example, examine whether the effects are eye contact specific, or general to other visual (e.g., pointing) or nonvisual (e.g., verbal) cues.

# CONCLUSION

The significance of the eyes in human communication has fascinated scientists for centuries. While the present findings only begin to scratch the surface of this broad area of investigation, this work does highlight the importance of systematically examining gender and conducting studies in contexts where eye contact can be communicative. Indeed, in the absence of a communicative context, eye gaze did not modulate recognition performance. This conclusion has tremendous implications for social theories of human communication, memory, and cognition more broadly, as images of the eyes have been used to manipulate and measure social behavior and social neural mechanisms of various cognitive processes. Using real people in future studies will enable the assessment of the social effects of eye gaze in particular, and social signals in general, thereby enhancing our understanding of the cognitive and neural bases of human communication and social interaction.

# ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the University of British Columbia's Research Ethics Board [Towards a More Natural Approach to Attention Research 1-200, certificate #H10-00527, & Research in Cognitive Ethology, #H04-80767], with written and oral informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol

# REFERENCES


was approved by the University of British Columbia's Research Ethics Board in accordance with the guiding ethical principles of the Tri-Council Policy Statement (TCPS2 2014), the International Conference on Harmonization Good Clinical Practice Guidelines (ICH-GCP) and the requirements of the US Department of Health and Human Services, as set out in the Federal Policy for the Protection of Human Subjects, 45CFR Part 46, sub-part A.

# AUTHOR CONTRIBUTIONS

SL was the lead investigator for all of the projects reported in this dissertation and was primarily responsible for design conception, data management and analysis, and report composition. AK and MJ acted in a supervisory capacity during project conception and report composition. MZ and CB were involved in data collection, management, and organization.

# FUNDING

This project was funded by grants to AK from the Natural Sciences and Engineering Research Council of Canada, and the Social Sciences and Humanities Research Council of Canada. Financial support was also received by postgraduate scholarship to SL from the Natural Sciences and Engineering Research Council of Canada, and a fellowship from the University of British Columbia.

with superior temporal sulcus ablation. *Neuropsychologia* 28, 1123–1142. doi: 10.1016/0028-3932(90)90050-X


Conty, L., George, N., and Hietanen, J. K. (2016). Watching eyes effects: when others meet the self. *Conscious. Cogn.* 45, 184–197. doi: 10.1016/j.concog.2016.08.016


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Lanthier, Jarick, Zhu, Byun and Kingstone. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Perceived Gaze Direction Differentially Affects Discrimination of Facial Emotion, Attention, and Gender – An ERP Study

Sarah D. McCrackin\* and Roxane J. Itier\*

Department of Psychology, University of Waterloo, Waterloo, ON, Canada

The perception of eye-gaze is thought to be a key component of our everyday social interactions. While the neural correlates of direct and averted gaze processing have been investigated, there is little consensus about how these gaze directions may be processed differently as a function of the task being performed. In a within-subject design, we examined how perception of direct and averted gaze affected performance on tasks requiring participants to use directly available facial cues to infer the individuals' emotional state (emotion discrimination), direction of attention (attention discrimination) and gender (gender discrimination). Neural activity was recorded throughout the three tasks using EEG, and ERPs time-locked to face onset were analyzed. Participants were most accurate at discriminating emotions with direct gaze faces, but most accurate at discriminating attention with averted gaze faces, while gender discrimination was not affected by gaze direction. At the neural level, direct and averted gaze elicited different patterns of activation depending on the task over frontal sites, from approximately 220–290 ms. More positive amplitudes were seen for direct than averted gaze in the emotion discrimination task. In contrast, more positive amplitudes were seen for averted gaze than for direct gaze in the gender discrimination task. These findings are among the first direct evidence that perceived gaze direction modulates neural activity differently depending on task demands, and that at the behavioral level, specific gaze directions functionally overlap with emotion and attention discrimination, precursors to more elaborated theory of mind processes.

Keywords: gaze direction, attention discrimination, emotion discrimination, gender discrimination, face perception, ERP

# INTRODUCTION

Eye-gaze has long been considered one of the most important cues during social interactions and seems central to social cognition (Kleinke, 1986; Emery, 2000; George and Conty, 2008; Itier and Batty, 2009 for reviews). Perceiving eye-gaze is thought to be a key component of theory of mind, our ability to infer what others are feeling and thinking (Baron-Cohen and Cross, 1992). This "language of the eyes" informs how we respond and interact with those around us (Baron-Cohen et al., 1997). The importance of eye-gaze is especially evident in populations who display behavioral avoidance of the eye region as well as social impairment, including Autism Spectrum Disorder

#### Edited by:

Chris Ashwin, University of Bath, United Kingdom

#### Reviewed by:

Francisco J. Parada, Universidad Diego Portales, Chile Shota Uono, Kyoto University, Japan Maria Pia Viggiano, University of Florence, Italy

\*Correspondence:

Sarah D. McCrackin sdmccrac@uwaterloo.ca Roxane J. Itier ritier@uwaterloo.ca

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Neuroscience

Received: 30 November 2018 Accepted: 06 May 2019 Published: 24 May 2019

#### Citation:

McCrackin SD and Itier RJ (2019) Perceived Gaze Direction Differentially Affects Discrimination of Facial Emotion, Attention, and Gender – An ERP Study. Front. Neurosci. 13:517. doi: 10.3389/fnins.2019.00517

**152**

(Pelphrey et al., 2002; Senju and Johnson, 2009; Madipakkam et al., 2017) and Social Anxiety Disorder (Schneier et al., 2011).

There is support for the idea that key differences exist between the processing of direct and averted gaze. Direct gaze has been heavily implicated in emotion processing (see Hamilton, 2016 for a review), as it is associated with increased ventral striatum activation (Kampe et al., 2001; Strick et al., 2008; see Cardinal et al., 2002, for a review of the ventral striatum's implication in emotion processing). It is behaviorally more arousing than averted gaze (Nichols and Champness, 1971; Conty et al., 2010; McCrackin and Itier, 2018c) and it has been shown that participants are better at reporting their own emotional state after seeing direct gaze faces than averted gaze faces (Baltazar et al., 2014). While both gaze directions inform an observer about the gazer's attentional state, seeing averted gaze informs an observer about the object or environment that the gazer is looking at (George and Conty, 2008; Itier and Batty, 2009). Perceived averted gaze also spontaneously orients the perceiver's attention toward the gazed-at location (Friesen and Kingstone, 1998; Driver et al., 1999; see Frischen et al., 2007 for a review) and this gaze cueing is even faster if the gazer is smiling or fearful, which likely helps the perceiver attend faster to environmental threats or rewards (e.g., McCrackin and Itier, 2018a,b). In contrast, direct gaze is self-referential, indicating that the observer is the focus of attention (George and Conty, 2008; Itier and Batty, 2009; Conty et al., 2016), and direct gaze has been shown to produce similar brain activation as hearing one's name being called (Kampe et al., 2003).

Accumulating evidence from neuroimaging studies suggests that eye-gaze is processed by a complex brain network whose nodes include the superior temporal sulcus, the amygdala, the medial prefrontal cortex, the orbitofrontal cortex, and parietal regions such as the intraparietal sulcus (for reviews, see Grosbras et al., 2005; George and Conty, 2008; Itier and Batty, 2009; Nummenmaa and Calder, 2009). However, inconsistencies in brain activity linked to the processing of direct and averted gaze have been noted. For instance, some have found increased superior temporal sulcus activation for direct gaze relative to averted gaze (Calder et al., 2002; Wicker et al., 2003; Pelphrey et al., 2004) while others have found the opposite (Hoffman and Haxby, 2000), or no difference in activation between the two gaze types (Wicker et al., 1998; Pageler et al., 2003). Similarly, the orbitofrontal cortex has been reported to show increased activation for direct than averted gaze (Wicker et al., 2003), or no gaze difference (Wicker et al., 1998), and the amygdala has been found to be more active for direct than averted gaze (Kawashima et al., 1999; George et al., 2001), more active for averted than direct gaze (Hooker et al., 2003; Wicker et al., 2003), or not active at all (Pageler et al., 2003).

Most importantly for the present study, similar inconsistencies have been reported in the Event Related Potential (ERP) literature, which attempts to track the time-course of gaze processing. A large proportion of studies have focused on the N170, a face-sensitive ERP component that occurs approximately 130–200 ms post face presentation over occipitotemporal sites, and is thought to reflect the structural encoding of the face (Bentin et al., 1996; George et al., 1996; Eimer, 2000). Some have found this component to be larger for averted gaze faces or averted gaze shifts (Puce et al., 2000; Watanabe et al., 2002; Itier et al., 2007; Latinus et al., 2015; Rossi et al., 2015), while others have found it to be larger for direct gaze static faces or direct gaze shifts (Watanabe et al., 2006; Conty et al., 2007; Pönkänen et al., 2010; Burra et al., 2017), yet others have found no N170 gaze effect at all (Taylor et al., 2001; Schweinberger et al., 2007; Brefczynski-Lewis et al., 2011). Gaze modulations have also been reported before the N170, around 100–140 ms with both greater amplitudes for direct than averted gaze (e.g., Burra et al., 2018) and greater amplitudes for averted gaze than direct gaze (Schmitz et al., 2012). Finally, gaze effects have been reported after the N170, around 250–350 ms (adaptation study looking at left/right gaze directions, Schweinberger et al., 2007) or even 300–600 ms with greater direct gaze than averted gaze amplitudes (Conty et al., 2007; Itier et al., 2007; Burra et al., 2018) or vice versa (Carrick et al., 2007).

One likely contributor to these inconsistencies is the type of experimental paradigm used. Common tasks given to participants while they are shown direct and averted gaze images include oddball tasks (i.e., responding to an infrequent stimulus presented among frequent other stimuli; e.g., Brefczynski-Lewis et al., 2011; Rossi et al., 2015; Burra et al., 2018) and passive viewing tasks (Puce et al., 2000; George et al., 2001; Taylor et al., 2001; Watanabe et al., 2002, 2006; Pönkänen et al., 2010), as well as tasks requiring the discrimination of gender (Burra et al., 2018), gaze direction (Hoffman and Haxby, 2000; Hooker et al., 2003; Conty et al., 2007; Itier et al., 2007; Schweinberger et al., 2007; Latinus et al., 2015), emotional expression (Akechi et al., 2010), identity (Hoffman and Haxby, 2000), or head orientation (Itier et al., 2007). These different task demands likely contribute to the reported inconsistencies regarding which brain areas are more involved for which gaze direction, and the time course of this gaze processing difference. While both the ERP and the neuroimaging literatures have begun to explore how eye-gaze processing differs based on what participants are asked to do (Hoffman and Haxby, 2000; Hooker et al., 2003; Carrick et al., 2007; Latinus et al., 2015; Burra et al., 2018), few studies have employed direct task comparisons within the same participants. Within-subject designs are, however, more powerful statistically than between-subject designs and are necessary to draw conclusions regarding possible task effects on the neural processing of direct versus averted gaze.

As far as we know, the limited number of within-subject ERP studies that have directly compared tasks, have focused on the processing of facial expressions of emotion, using Gender Discrimination (GD) and Emotion Discrimination (ED) judgments. The stimuli used were eye-region stimuli (Sabbagh et al., 2004) or faces (Wronka and Walentowska, 2011; Rellecke et al., 2012; Itier and Neath-Tavares, 2017), but always with direct gaze. One exception includes the comparison of an ED task to judgments of looking direction and of object choice based on averted gaze faces only (Cao et al., 2012). These studies suggest that ED and GD tasks differentiate mainly after the N170 component. While Rellecke et al. (2012) and Wronka and Walentowska (2011) found no ERP difference between the two tasks, Sabbagh et al. (2004) found that the ED task resulted

in more negative ERPs than the GD task over inferior frontal and anterior temporal sites from 270 to 400 ms, which source localization suggested was driven by orbitofrontal and medial temporal activation. The ED task also resulted in more positive ERPs than the GD task from 300 to 500 ms over posterior central and parietal sites (Sabbagh et al., 2004), a similar finding to Itier and Neath-Tavares's (2017) report of more positive ERPs elicited by the GD task than the ED task over posterior sites from 200 to 350 ms (the latest tested time-window).

To the best of our knowledge, no current ERP study has directly investigated task effects on the processing of direct versus averted gaze faces in a within-subject design. The present study begins to fill this gap by examining the time-course of direct and averted gaze perception within three different discrimination tasks that have been commonly used in the gaze processing literature. Using the exact same stimuli for each task, i.e., male and female faces expressing anger or joy and with direct or averted eye-gaze, participants indicated whether the face expressed anger or joy (ED task), whether the face was male or female (GD task) and whether the face was attending to them or away from them (Attention Discrimination – AD task). Importantly, explicit processing of gaze direction was required by the AD task while gaze was irrelevant to the GD and ED tasks. ERPs time-locked to the presentation of the face stimuli were used to track the time-course of when gaze and task processing were occurring and interacting. If direct and averted gaze differentially impacted these three cognitive processes, we expected to see dissociations at the neural level, in spatial location (different electrodes) and/or in the time course of the interaction, as well as at the behavioral level.

Given the mixed findings reported on the N170 component as reviewed earlier, we analyzed a cluster of occipitotemporal electrodes during the time window encompassing this component (130–220 ms). However, the findings from the gaze and ERP literature on different tasks suggested that we might pick up a gaze and task interaction over frontal sites between 200 and 400 ms post-stimulus, after both gaze (e.g., Puce et al., 2000; Watanabe et al., 2002; Itier et al., 2007; Latinus et al., 2015; Rossi et al., 2015) and ED and GD task differences (Sabbagh et al., 2004) are processed. As gaze effects are traditionally picked up over parieto-occipital sites (Itier and Batty, 2009), and posterior central and parietal sites have been shown to discriminate between ED and GD tasks from 200 to 500 ms (Sabbagh et al., 2004; Cao et al., 2012; Itier and Neath-Tavares, 2017), we also hypothesized that we may find an interaction between gaze and task over posterior sites from 200 to 500 ms.

It has to be highlighted that the ERP field is witnessing a transition toward more robust data analyses. As Luck and Gaspelin (2017) recently demonstrated, examining the ERP waveforms (typically the group grand-average) before deciding which electrodes and time-windows to analyze, can massively inflate type I errors and lead to reporting false effects. Similarly, although using a priori hypotheses to select electrodes and timewindows provides resistance to type I errors, this approach can prevent the discovery of real effects at untested time-points. As most of the ERP literature on gaze processing employed both of these classic approaches, it is possible that a lot of the inconsistencies reported in the time course of the effect were also due, in addition to the various task demands, to the way the analyses were performed. While there is no perfect solution, the mass univariate approach shows promise in its capacity to reduce both types of error (Groppe et al., 2011; Pernet et al., 2011, 2015; Luck and Gaspelin, 2017; Fields and Kuperberg, 2018). With this approach, hypothesis testing can first be performed on a subset of a priori electrodes and time-points with a multiple comparison correction applied to control for type I errors (Groppe et al., 2011). Then, an exploratory analysis can be performed by testing each electrode at every time-point to enable the discovery of unpredicted effects, with the caveat that this type of analysis can have weak power because of the number of comparisons corrected for. Accordingly, we used the freely available Factorial Mass Univariate Toolbox (FMUT) extension (Fields, 2017) for the Mass Univariate Toolbox (MUT; Groppe et al., 2011) to perform a mass-univariate analysis in the present study. We first performed our hypothesis testing by running a mass univariate analysis on occipitotemporal sites from 130 to 200 ms to capture the N170, at frontal sites from 200 to 400 ms, and on parietooccipital sites from 300 to 500 ms. Then we performed an exploratory analysis over all electrodes and time-points.

# MATERIALS AND METHODS

# Participants

Thirty-six undergraduate students from the University of Waterloo (UW) participated in the study and received course credit upon completion. All were 18–29 years old, had normal or corrected-to-normal vision and had lived in Canada or the United States for the past 5 years or more. They reported no history of neurological or psychiatric illness and no drug use (psychiatric or otherwise). All participants rated themselves at least a 7 out of 10 on Likert-type scales when describing their ability to recognize people and emotional expressions (from 0-extremely poor to 10-extremely good). In total, ten participants were excluded before analysis due to technical issues during recording (N = 2), problems with eye-tracking calibration (N = 2), poor response accuracy (i.e., less than 80%; N = 2), or EEG data that had less than 50 trials per condition after cleaning (N = 4). This left a final sample of 26 participants (17 females, 9 males; mean age = 19.67, SD = 1.69) for analysis. The study received ethics clearance from the UW Research Ethics Board and all participants gave written informed consent in accordance with the Declaration of Helsinki.

# Face Stimuli

Five male and five female Caucasian identities were selected from the Radboud database (Langner et al., 2010).<sup>1</sup> Each individual displayed an angry expression and a happy expression with direct gaze, averted left gaze and averted right gaze (**Figure 1**). All gaze deviations were of equal magnitude. The images were

<sup>1</sup> Identities 10, 15, 19, 30, 31, 32, 33, 37 were used in the study blocks, while identities 7 and 14 were used in the practice block.

expressions displaying direct, averted left and averted right gaze (human image obtained from "Radboud Face dataset," used with permission – http://www.socsci.ru.nl:8180/RaFD2/RaFD?p=faq).

cropped with the GNU Image Manipulation Program (GIMP 2.8) so that only the individual's upper shoulders, head and neck were visible. All images were then mirrored to control for any asymmetry between the left and right image halves by creating a second set of images (e.g., an angry averted right image mirrored became a new angry averted left image). Images were equated on mean pixel intensity (M = 0.56, SD = 0.0003) and root mean square (RMS) contrast (M = 0.48, SD = 0.0002) with the SHINE package (Willenbockel et al., 2010). Custom MATLAB scripts were then used to add the color information back into each image for added realism.

# Experimental Design

Participants first provided informed consent, and then filled out a demographic questionnaire. They were fitted with an EEG cap and led to a sound-attenuated faraday cage with dim lighting for the experiment, which was presented on a CRT monitor with a refresh rate of 85 Hz and a resolution of 1280 × 960. A chinrest helped participants keep their heads still at a distance of 65 cm away from the monitor. Participants' dominant eyes were determined using the Miles test (Miles, 1930) and then tracked at a 1000 Hz sampling rate with an Eyelink 1000 eyetracker, which was recalibrated whenever necessary.

Participants were told that they would see pictures of individuals and complete three tasks, and that a prompt at the beginning of each trial would let them know which task to perform for that trial. The first task required indicating what emotional state the person was in (Emotion Discrimination Task, hereafter ED task; prompted by the words "Happy/Angry"). The second task required indicating whether the person was directing their attention at them (the participant) or away from them (Attention Discrimination task, hereafter AD task; prompted

by "At Me/Away" words). The third task required indicating whether the person was a male or female (Gender Discrimination task, hereafter GD task; prompted by "Male/Female" words). Participants were asked to indicate their answer when prompted using the left and right arrow keys.

**Figure 2** depicts a typical trial progression. At the trial start, the task prompt appeared, notifying the participant of the task and visually reminding them (with arrows) which answers corresponded to the left and right arrow keys. Task type was randomized and there were an equal number of trials for each task presented in each block. The response mapping for the arrow keys was counterbalanced between participants (i.e., half pressed the right arrow key for "angry," and half pressed the left arrow key; half pressed the right arrow key for direct gaze and half pressed the left arrow key; half pressed the right arrow key for male and half pressed the left arrow key). Participants were instructed to press the space bar when they had read the prompt, and this key press triggered the appearance of a white screen with a fixation cross (18.43◦ down on the horizontal midline). Participants were asked to fixate the cross for a minimum of 300 ms within a 1.92◦ × 1.92◦ margin to advance the trial to the face screen. This ensured that participants were fixated between the nasion and the nose when the face appeared. If ten seconds elapsed without this requirement being met, a drift correction occurred, canceling the trial. If the requirement was met, the trial advanced by presenting the face image (subtending 10.64◦ horizontally and 15.08◦ vertically) on a white background for 500 ms. There were an equal number of direct and averted gaze faces, with half of the averted gaze trials consisting of faces looking to the left and half to the right (all averted gaze trials were grouped together for analysis). Face identity was randomized, and each was presented an equal number of times within each block and within each condition. The face was followed by a 300 ms blank screen after which participants were prompted to indicate their answer by pressing the left or right arrow key. This procedure

ensured that the neural activity until 800 ms post face onset would not be contaminated by motor preparation and motor artifacts. However, in doing so, the response times collected were not clearly interpretable and are not further discussed.

SR Research's Experiment Builder 1.10.1385 was used to program and run the experiment. Participants completed a minimum of four practice trials to ensure they were comfortable with the tasks before starting the study blocks. In total, there were 8 blocks of 96 trials each. There were six within-subject conditions, corresponding to the face's gaze direction (direct or averted) in each of the three tasks performed (ED, AD, and GD), with facial expression trials collapsed. Thus, across the experiment, there were a total of 128 trials per each of the 6 conditions.

### Electroencephalography Recording

EEG data were recorded with the Active-two Biosemi EEG system at a sampling rate of 512 Hz, time-locked to the presentation of the face stimulus. Electrode offset was kept within a ±20 mV range. There were 66 electrodes on the custom-made caps under the 10/20 system, the 64 classic locations plus PO9 and PO10 electrodes added for increased posterior coverage. In addition, one electrode was placed over each mastoid, infra-orbital ridge, and the outer canthus of each eye, for a total of 72 recording electrodes. A Common Mode Sense (CMS) active-electrode and a Driven Right Leg (DRL) passive-electrode were used as the ground.<sup>2</sup>

#### Data Preprocessing and Cleaning

To ensure that participants read the task prompt on each trial, we used the eye-tracking data to exclude trials where participants did not fixate at least twice on the prompt screen within a rectangular region of interest (ROI) spanning the text (subtending 32.71◦ horizontally and 3.72◦ vertically, positioned 17.43◦ down and centered horizontally). This resulted in excluding an average of only 0.81 trials per participant (SD = 1.04). We also excluded trials in which participants did not fixate the spot encompassing the eyes, and nasion (a circular 5.50◦ ROI) that was cued by the fixation cross for at least the first 250 ms of face presentation. As the N170, the earliest face sensitive ERP component, can be modulated by what part of the face is fixated (de Lissa et al., 2014; Nemrodov et al., 2014; Neath and Itier, 2015; Neath-Tavares and Itier, 2016; Itier and Preston, 2018; Parkington and Itier, 2018), this step ensured that fixation location would not play a role in any N170 modulation and that participants were encoding the gaze direction for each face. This resulted in excluding an average of 3.23 trials per participant (SD = 4.98). Next, trials with incorrect responses were removed (an average of 4.72 trials/participant, SD = 2.09).

EEG data were processed using the EEGLab (version 13.6.5b; Delorme and Makeig, 2004) and ERPLab (version 5.1.1.0<sup>3</sup> ) toolboxes in MATLAB 2014b. An average reference was computed offline and data were band-pass filtered (0.01–30 Hz) and then cleaned. Trials were epoched from a −100 ms baseline (before the face) to 800 ms post-face. First, trials were removed if they exceed ±70 µV on any non-frontal and non-ocular channels (i.e., excluding: Fp1, Fpz, Fp2, AF3, AFz, AF4, AF8, AF7, IO1, IO2, LO1, and LO2). Any of these channels that were consistently noisy were removed for later interpolation. Then, data were visually inspected for eye-blinks and saccades. For cases where there were few eye artifacts, the data were manually cleaned, and any removed electrodes were added back in and interpolated with EEGlab's spherical splines tool. For cases where there were many eye-artifacts, Independent Component Analysis (ICA; using the EEGLab "runica" function) was used to remove saccades and eye-blinks before adding back and interpolating electrodes. Remaining noisy trials were then manually removed when necessary. An average of 97.29 trials/condition (SD = 22.34) were included in the final ERP waveforms.<sup>4</sup>

# Data Analysis

#### Behavioral Data Analysis

Correct answers for each condition were those in which the participant pressed the arrow key corresponding to the correct gender (GD task), emotional expression (ED task) or gaze direction (AD task). An ANOVA with the within-subjects factors of gaze direction (2; direct gaze, averted gaze) and task (3; GD, ED, AD) was run on participants' average accuracy using SPSS 25. Greenhouse-Geisser corrected degrees of freedom were reported when Mauchly's Test of sphericity was significant. The follow up t-tests for the gaze and task interactions were planned based on the theoretical motivation behind this paper. However, for transparency, the raw p-values for all follow-up paired t-tests are reported, such that those with p < 0.05 would be considered significant with Fischer's LSD test, and those with p < 0.016 would be considered significant after Bonferroni-correction (0.05/3 comparisons).

#### EEG Data Analysis

EEG data were analyzed using the Factorial Mass Univariate Toolbox (FMUT) extension (Fields, 2017) for the Mass Univariate Toolbox (MUT; Groppe et al., 2011). FMUT uses robust statistics to test each time-point included in the timewindow of interest for the selected electrodes, and then control for the familywise error rate. One ANOVA with the withinsubjects factors of gaze direction (2; direct gaze, averted gaze) and task (3; GD, ED and AD) was run over (i) a posterior cluster (P9, P10, PO9. PO10, P7, P8) between 130 and 200 ms encompassing the N170 component, (ii) a frontal electrode cluster (Fp1, Fp2, Fpz, AF3, AF4, AFz, F4, F3, F1, F2, Fz) from 200 to 400 ms, and (iii) parieto-occipital electrodes (Pz, POz, PO4, PO3, P1, P2, Oz, O1, O2) from 200 to 500 ms. The ANOVAs were corrected for multiple comparisons with the Permutation Based Cluster Mass technique (Maris and Oostenveld, 2007; Groppe et al., 2011). With this technique, data points that are spatially and temporally adjacent and that exceed the threshold for inclusion are considered a cluster. All F-values in the cluster

<sup>2</sup>The Biosemi Active-Two system does not use an actual recording reference site. <sup>3</sup>http://erpinfo.org/erplab

<sup>4</sup>Trials per condition: Direct GD = 99.46 (SD = 22.49), Averted GD = 98.50 (SD = 23.91), Direct ED = 98.07, SD = 24.00, Averted ED = 96.69 (SD = 28.25), Direct AD = 94.27 (SD = 21.03), Averted AD = 96.69 (SD = 28.25).

are then summed, and compared to a null distribution for cluster mass significance estimated with permutations. We used the recommended number of 100,000 permutations and alpha of 0.05, such that clusters exceeding the 1 – α percentile of the resulting distribution were considered significant. As discussed by Groppe et al. (2011) and Maris and Oostenveld (2007), true ERP effects are more likely than noise to occur across multiple adjacent electrodes and time-points, and thus ERP effects will typically stand out more clearly from noise using cluster-based statistics.

Based on the gaze direction by task interaction that we observed in the omnibus ANOVA at frontal sites during 200–400 ms, three follow-up ANOVAs were performed with FMUT to compare the activations associated with direct and averted gaze in each of the three tasks (the use of ANOVAs instead of t-tests as follow-up tests is recommended for the Permutation Based Cluster Mass technique; Fields, 2019). We performed these follow up ANOVAs over the frontal sites and time-points (220–290 ms) that were significant in the omnibus ANOVA with an alpha level set to 0.016 to correct for the three comparisons. As in the original ANOVA, 100,000 permutations were calculated.

Finally, we performed an exploratory analysis on all electrodes and relevant time-points (50–800 ms) post-face to allow for the discovery of unpredicted effects, again with 100,000 permutations and an alpha of 0.05. Based on the main effect of task that we observed in this analysis, we performed three follow-up task comparisons over the significant time-points (400–800 ms) and electrodes in the omnibus ANOVA with a corrected alpha level of 0.016.

# RESULTS

The datasets analyzed in the present study are available in the Open Science Framework Repository<sup>5</sup> .

#### Participant Accuracy

There was a main effect of task on response accuracy,<sup>6</sup> F(2,50) = 31.98, MSE = 30.16, p < 0.001, η 2 <sup>p</sup> = 0.56 (**Figure 3**), driven by greater accuracy in the GD than both the ED task [t(25) = 3.71, SE = 0.83, p = 0.001] and the AD task [t(25) = 7.61, SE = 1.12, p < 0.001], and by greater accuracy in the ED task than in the AD task [t(25) = 4.37, SE = 1.24, p < 0.001].

represent the accuracy for individual participants. Boxes encompass data points between the 25th and 75th percentiles, and within each box the mean (dotted horizontal line) and median (solid horizontal line) are indicated. ∗∗Indicates p < 0.016, which meets the threshold for significance with Bonferroni correction.

Although there was no main effect of gaze, F(1,25) = 2.82, MSE = 12.78, p = 0.11, η 2 <sup>p</sup> = 0.11, there was a strong interaction between gaze direction and task, F(1.37, 34.16) = 12.10, MSE = 18.70, p < 0.001, η 2 <sup>p</sup> = 0.33 (**Figure 3**). Planned paired comparisons comparing gaze conditions for each task revealed that participants were more accurate during the AD task in the averted gaze condition than in the direct gaze condition [t(25) = 3.18, SE = 1.77, p = 0.004]. In contrast, during the ED task, participants were more accurate in the direct gaze condition than in the averted gaze condition [t(25) = −3.51, SE = 0.67, p = 0.002]. Finally, there was no accuracy difference between the two gaze conditions for the GD task [t(25) = −0.81, SE = 0.52, p = 0.42]. The accuracy graph was created with BioVinci version 1.1.15 developed by BioTuring Inc.

# EEG Results N170 Analyses

The N170 ANOVA over posterior sites (P9, P10, PO9. PO10, P7, P8) from 130 to 200 ms did not reveal any significant effects of gaze direction, task, nor an interaction between the two.

#### Frontal and Parieto-Occipital Analyses

The omnibus ANOVA over frontal sites from 200 to 400 ms revealed an interaction between gaze direction and task on ERP amplitudes (**Figure 4**), but no main effect of gaze or task. While caution must be taken when making inferences about effect latency or location with cluster-based permutation tests (Sassenhagen and Draschkow, 2019), in this latency range the interaction was most pronounced from approximately 220–290 ms over electrodes F3, F1, AFz, and FPz. Our follow-up comparisons during that time window (with p < 0.016) of how direct and averted gaze are processed in each task revealed that in

<sup>5</sup>https://osf.io/am4zv/?view\_only=eac91ae8a07e44f7ab5aca550fc19da2

<sup>6</sup>For the interested reader, the RT time-locked to the onset of the answer prompt displayed a similar pattern as the accuracy data: the main effect of task [F(1.40,34.96) = 13.14, MSE = 13908.60, p < 0.001, η 2 <sup>p</sup> = 0.34] was driven by faster responses during the GD than both the ED [t(25) = −4.24, SE = 11.52, p < 0.001] and AD [t(25) = −4.29, SE = 23.09, p < 0.001] tasks, as well as faster responses during the ED than the AD task [t(25) = −2.35, SE = 21.36, p = 0.027]. There was no main effect of gaze (F = 1.02, p = 0.32), though there was a significant interaction between task and gaze [F(2,50) = 6.17, MSE = 3568.78, p = 0.004, η 2 <sup>p</sup> = 0.20]. RTs were faster for the averted gaze than the direct gaze condition during the AD task [t(25) = −2.72, SE = 17.96, p = 0.012]. The opposite pattern, thought not significant, was observed in the ED task, with faster RTs during the direct gaze than the averted gaze condition [t(25) = 1.80, SE = 14.57, p = 0.084]. There was no RT difference between gaze conditions for the GD task [t(25) = 0.024, SE = 9.94, p = 0.98].

the GD task, there were more positive ERP amplitudes for averted gaze than direct gaze (**Figure 5A**, left). This was most pronounced over F1 and AFz (**Figure 5A**, middle and right). In contrast, the opposite pattern was seen in the ED task (**Figure 5B**, left) with direct gaze producing more positive ERP amplitudes than averted gaze (**Figure 5B**, middle and right). Finally, there was no detectable effect of gaze direction in the AD task (**Figure 5C**, left, middle and right).

the F-value plotted according to the right-hand color bar.

There were no significant effects following the ANOVA over parieto-occipital sites (Pz, POz, PO4, PO3, P1, P2, Oz, O1, O2) from 200 to 500 ms.

#### Exploratory Analysis

The exploratory analysis over all electrodes and time-points (excluding the first 50 ms post-face, so between 50 and 800 ms) revealed a widespread main effect of task (**Figure 6**). It was most pronounced from 400 to 800 ms over posterior and fronto-central sites. Follow up comparisons indicated that this effect was driven by differences between the GD and ED tasks (**Figure 7A**), the GD and AD tasks (**Figure 7B**), and the ED and AD tasks (**Figure 7C**). Over posterior sites, ERP amplitudes were most negative in the AD task, intermediate in the ED task, and most positive in the GD task (**Figure 7D**, top). The opposite pattern was found over fronto-central sites (**Figure 7D**, bottom).

### DISCUSSION

The importance of eye-gaze processing during social interactions is undisputed (Kleinke, 1986; Emery, 2000; George and Conty, 2008; Itier and Batty, 2009 for reviews) and is particularly evident in disorders which feature both eye-gaze avoidance and social impairment, including Autism Spectrum Disorder (Pelphrey et al., 2002; Senju and Johnson, 2009; Madipakkam et al., 2017) and Social Anxiety Disorder (Schneier et al., 2011). The clinical significance of altered eye-gaze processing has led to a field of research devoted to understanding how direct and averted gaze are processed in the brain, and how we use them as cues to inform our social interactions.

While there has been much interest in examining the neural correlates of eye-gaze processing, there does not seem to be a consensus about where and when direct and averted gaze are differentiated in the brain. One of the likely reasons for this lack of consensus is that the experimental tasks in studies of gaze processing vary quite substantially (Hoffman and Haxby, 2000; Hooker et al., 2003; Carrick et al., 2007; Latinus et al., 2015; Burra et al., 2018). Given that direct and averted gaze can be interpreted differently in different social circumstances (Hamilton, 2016), it is likely that these gaze cues are processed differently depending on the type of task participants are asked to complete. To this end, we examined how viewing individuals with direct and averted gaze would affect performance during three different tasks commonly used in the field, in a within-subjects design. Those tasks have been previously used to study gaze processing in separate samples (one task at a time) and included an Emotion Discrimination (ED) task, where participants discriminated between two facial expressions, an Attention Discrimination (AD) task that required participants to infer the direction of the individual's attention based on gaze cues and a Gender Discrimination (GD) task. We found that direct and averted gaze elicited different behavioral effects depending on the task that participants were performing (**Figure 3**). Direct gaze was associated with better accuracy than averted gaze during the ED task, while averted gaze was associated with better accuracy in the AD task. However, there was no significant effect of gaze direction on performance in the GD task.

Although we believe our behavioral interactions between gaze and task reflect interactions between gaze processing and AD and ED task demands, other potential explanations should be noted. First, previous literature has reported that direct gaze has a facilitatory effect on a myriad of tasks including capturing attention (Yokoyama et al., 2014), facilitating recognition memory (Vuilleumier et al., 2005), and gender discrimination (Macrae et al., 2002; Burra et al., 2018; but see Vuilleumier et al., 2005). While it is possible that a general facilitatory effect of direct gaze may explain our behavioral findings in the ED task, we do not believe this is the case because no significant effect of gaze direction was found in the GD task. This would suggest that the facilitatory effect of direct gaze during the ED task was above any standard facilitation effect. Furthermore, the AD task was associated with worse performance for direct gaze, which goes against this explanation. It is important to highlight that all previous studies reporting facilitated effects for direct gaze studied only one task at a time, in contrast to the present within-subject design which directly compared three tasks in the same individuals.

A similar argument could be made regarding the possibility of gaze cuing effects influencing the results. Given that the gaze cuing literature suggests that spontaneous attention shifts

FIGURE 5 | A comparison of direct and averted gaze over frontal sites in the (A) gender discrimination (B) emotion discrimination, and (C) attention discrimination tasks. These post hoc analyses were run on the 220–290 ms time widow during which the main omnibus ANOVA yielded a significant interaction (see Figure 4). Left panels depict significant F-values corrected with a Permutation Based Cluster Mass technique at p < 0.016 (to account for the fact that three follow-up tests were run). Each electrode is plotted on the y-axis and each time point (post-face onset) is plotted along the x-axis. The color of the "blocks" in these left panels corresponds to the magnitude and direction of significance as indicated by the right-hand color bar. Middle panels depict mean ERP amplitudes and 95% confidence intervals for direct and averted gaze on electrodes F1 and AFz over which the interactions were maximum. Right panels depict the difference between the two gaze conditions (direct gaze amplitude – averted gaze amplitude) on F1 and AFz, with 95% confidence intervals.

with left panels depicting significant F-values corrected with a Permutation Based Cluster Mass technique at p < 0.05. Electrodes are plotted on the y-axis and time points following face presentation are plotted along the x-axis. Colored "blocks" represent significant F-values, with the magnitude of the F-value plotted according to the right-hand color bar.

occur toward gazed-at locations even when gaze direction is task irrelevant (Friesen and Kingstone, 1998; Driver et al., 1999), one could argue that averted gaze may have oriented participants' attention away from the stimuli during the tasks. However, there is no reason why this potential attention shift should have affected tasks differently, and because there was no effect of gaze direction on accuracy in the GD task, and opposite effects of gaze direction in the ED and AD tasks, it is unlikely that covert attention shifts in the direction of averted gaze could explain the pattern of results.

It must be noted that others have reported that direct gaze is associated with improved ED. Adams and Kleck (2003, 2005) and Sander et al. (2007) also found that angry and happy facial expressions (as used in the present study) were perceived more easily when paired with direct gaze than with averted gaze. However, they also found that fear and sadness were perceived more easily when paired with averted gaze than with direct gaze. Adams and Kleck (2003) proposed that direct gaze enhances the perception of facial expressions signaling behavioral approach from the gazer (e.g., angry and happy expressions), while averted gaze enhances the perception of facial expressions signaling behavioral avoidance (e.g., sadness and fear) due to a "shared signal" between gaze and emotion expression decoding. Although the support for the shared signal hypothesis was largely found to be tied to the specific stimuli used (Graham and LaBar, 2007; Bindemann et al., 2008), it is still possible that gaze direction may facilitate or impair ED differently depending on the emotional expression on the face. Replication of the present findings and extension to more facial expressions is needed to examine this possibility further.

There is also another potential explanation for our behavioral gaze effects, which concerns the inherently self-referential nature of direct gaze (Conty et al., 2016). Direct gaze signals to us that we are the direction of someone's attention (George and Conty, 2008; Itier and Batty, 2009; Conty et al., 2016), and has been shown to produce similar fMRI brain activation as hearing one's name being called (Kampe et al., 2003). Gaze processing has also been shown to interact with the self-relevance of contextual sentences at the ERP level (McCrackin and Itier, 2018c). In the attention discrimination task, participants indicated whether the individuals were directing their attention at them or away from them. This may have primed self-referential processing, which could have impacted how direct gaze was processed. However, if this was the case, one would expect participants to be more accurate at responding to direct gaze faces in the AD task, while the opposite was observed. In fact, if anything, the pattern of results (**Figure 3**) suggests that direct gaze hindered performance in the AD task (as opposed to a true accuracy benefit for the averted gaze condition).

We also found that gaze processing interacted with task at the ERP level, although the pattern of results did not map directly onto the pattern of behavioral results. Gaze processing differed between the three tasks from 200 to 400 ms over frontal sites. While there was no gaze difference in ERP amplitudes in the AD task over these sites, direct gaze elicited more positive amplitudes than averted gaze in the ED task, but less positive amplitudes than averted gaze in the GD task. The interaction between gaze direction and task indicated that these two effects overlapped in time, although the ED gaze effect appeared earlier (around 220 ms) than the GD gaze effect (around 255 ms). Interestingly, the ED gaze activity occurs in a time-window during which decoding of emotions typically occurs. The Early Posterior Negativity EPN – that typically differentiates between different facial expressions, in particular fearful and angry compared to happy facial expressions (e.g., Sato et al., 2001; Schupp et al., 2006; Herbert et al., 2008; Kissler et al., 2009; Wronka and Walentowska, 2011; Rellecke et al., 2012; Neath and Itier, 2015; Neath-Tavares and Itier, 2016), is often reported between 150 and 250 ms and up to 350 ms at posterior sites. Given that direct gaze has been implicated in emotion processing (Kampe et al., 2001; Strick et al., 2008; Hamilton, 2016) and affects participants' arousal (Nichols and Champness, 1971; Conty et al., 2010; McCrackin and Itier, 2018c) and introspective reporting of emotional state (Baltazar et al., 2014), the present frontal activation in the ED task may be indicative of overlap between the neural correlates associated with emotion processing and gaze processing.

Despite its excellent temporal resolution, EEG has poor spatial resolution, so caution must be taken when making inferences about possible neural generators. Nevertheless, we speculate

FIGURE 7 | Comparisons of the (A) gender and emotion discrimination tasks (B) gender and attention discrimination tasks, and (C) emotion and attention discrimination tasks. These post hoc analyses were run on the 400–800 ms time window during which there was a significant task effect in the omnibus ANOVA (see Figure 5). Left panels depict significant F-values corrected with a Permutation Based Cluster Mass technique at p < 0.016 (to account for the three follow-up tests). Electrodes are plotted on the y-axis and time after face onset is plotted on the x-axis. Colored "blocks" represent significant F-values, with the magnitude of the F-value plotted according to the right-hand color bar. (D) Mean ERP amplitudes from representative posterior (P10) and central (CP1) electrodes with 95% confidence intervals.

that the frontal activity recorded is linked to orbitofrontal (OFC) activity, given the involvement of the OFC in emotion processing, gaze processing and higher order theory of mind tasks (Calder et al., 2002; Amodio and Frith, 2006; Conty et al., 2007; Dixon et al., 2017). The 220–290 ms during which the task by gaze interaction was found significant at this frontal cluster falls in between timings reported by two independent studies to be sensitive to gaze (Conty et al., 2007) and task

(Sabbagh et al., 2004), respectively. Conty et al. (2007) reported OFC activation to differentiate between direct and averted gaze from 190 to 220 ms (picked up first over frontocentral and centroparietal sites, e.g., Fz, Cz, then later over occipital-temporal sites, e.g., P9, P10). In another study, source localization pointed to the OFC as the source of ERP amplitude differences found between 270 and 400 ms and differentiating between a GD task and an ED task close to our own (over frontal sites including FP2 and F4, as well as parieto-occipital sites), which asked participants to decode emotional state from eye-regions with direct gaze (Sabbagh et al., 2004). We thus find it plausible that the OFC would be involved in the gaze by task interaction picked up at frontal sites during similar timing.

One of the limitations of this study concerns the differences between the demands associated with each task, and it is unclear what differences between tasks are responsible for the differences in how gaze was processed during each. For example, while we assume that the key factor differentiating the ED from the GD and AD tasks is the recruitment of frontocentral emotion processing centers in the ED task, in particular the orbitofrontal cortex, the tasks also differ in terms of featural versus holistic processing. Indeed, the AD task may have required featural processing of the eyes, while both ED and GD judgments are generally considered to require holistic face processing (e.g., McKelvie, 1995; Prkachin, 2003; Calder and Jansen, 2005; Zhao and Hayward, 2010). However, as opposite gaze effects were seen between the GD and ED tasks at the neural level, this featural versus holistic processing difference cannot easily explain our neural interaction.

In contrast, a featural/holistic difference in processing may account for overall task differences found from 400 and 800 ms post-stimulus that may be related to task difficulty. Over occipitotemporal sites, the most positive ERP amplitudes were elicited by the GD task, intermediate amplitudes by the ED task, and the most negative amplitudes were elicited by the AD task. The reverse pattern was seen over centro-parietal sites, likely reflecting the opposite end of the same dipole. Similar task effects have been reported in studies in which participants used eyeregions (Sabbagh et al., 2004) or faces (Itier and Neath-Tavares, 2017 but see Rellecke et al., 2012 for null results) to complete similar ED and GD tasks. Itier and Neath-Tavares (2017)reported more positive ERPs in the GD than the ED task over posterior sites but at much earlier timings (from 200 to 350 ms, the latest measured time window due to much shorter response times). Sabbagh et al. (2004) reported more positive ERPs for the ED task than the GD task over posterior, central and parietal sites at a timing closer to our own timing (300–500 ms, where as our task effect began at 400 ms). These timing differences may be related to the fact that in the present study and the Sabbagh (2004) study, participants were asked to wait until the response prompt to press the keys while in the Itier and Neath-Tavares (2017) study, responses occurred as soon as possible after the presentation of the stimulus. Similar task effects have also been found when participants were asked to perform visual discrimination tasks with differing levels of complexity (Senkowski and Herrmann, 2002). Our behavioral data support the idea that task complexity might be responsible for these general effects of tasks, given the accuracy gradient followed the same pattern as the ERP amplitude gradient. Accuracy was indeed highest in the GD task, intermediate in the ED task, and worst in the AD task. Similar response time (Wronka and Walentowska, 2011; Rellecke et al., 2012) and accuracy (Wronka and Walentowska, 2011) gradients were previously reported by groups using similar GD and ED tasks. Overall, the general task effects seen at the ERP level seem related to task difficulty and future studies could investigate whether this difficulty is related to featural/holistic processing differences or to other task-specific factors.

We should also note that it was surprising to find neither a main effect of gaze direction, nor an interaction between gaze and task, over posterior sites during the 130–200 ms window encompassing the N1710, given past reports of gaze effects on this ERP component (Puce et al., 2000; Watanabe et al., 2002, 2006; Conty et al., 2007; Itier et al., 2007; George and Conty, 2008; Itier and Batty, 2009; Pönkänen et al., 2010; Latinus et al., 2015; Rossi et al., 2015; Burra et al., 2017). These previous reports have been quite mixed, with some finding enhanced N170 amplitudes in response to averted gaze (Puce et al., 2000; Watanabe et al., 2002; Itier et al., 2007; Latinus et al., 2015; Rossi et al., 2015), some to direct gaze (Watanabe et al., 2006; Conty et al., 2007; Pönkänen et al., 2010; Burra et al., 2017), and others, like the present study, finding no gaze effect at all (Taylor et al., 2001; Schweinberger et al., 2007; Brefczynski-Lewis et al., 2011). One possibility is that there is a lot of variation in how gaze is processed at the individual level over these sites (the N170 itself can range in latency from 130 to 200 ms between individuals). While there may be some similarities in timing and location, significant individual differences could have impacted our ability to detect gaze effects at the group level using a mass-univariate approach. Moreover, this literature on gaze effect almost always used neutral faces, while the present study used emotional expressions, which may have impacted the early processing of gaze. The other alternative is that previously reported findings regarding N170 modulations by gaze were type I errors that may be related to the lack of control of gaze position. Indeed, as far as we know, the present study is the first ERP study on gaze perception to have controlled for gaze position using a gazecontingent approach, a particularly important aspect given the growing literature showing modulations of the N170 amplitude with gaze fixation location, in particular to the eyes (de Lissa et al., 2014; Nemrodov et al., 2014; Neath and Itier, 2015; Neath-Tavares and Itier, 2016; Itier and Preston, 2018; Parkington and Itier, 2018). Those possible caveats represent an important topic for further research to address. In any case, from the present (and unique) within-subject design, there is no evidence of early gaze effects during the time window encompassing the N170 component, as least when using facial expressions of emotion.

In summary, the present study is one of the first ERP investigations demonstrating that direct and averted gaze are processed differently during emotion, attention and gender discrimination judgments performed by the same participants. Gaze direction did not affect GD task performance, while processing direct gaze facilitated emotion discrimination relative to averted gaze, and processing averted gaze facilitated the attention direction judgment relative to direct gaze. These results

provide support for the idea that gaze perception impacts attention and emotion discrimination judgments, which are likely key initial steps in our everyday theory of mind. If perceiving direct gaze facilitates ED and perceiving averted gaze facilitates AD, avoiding the eye-region will prevent this facilitation from occurring. Accordingly, our findings are in line with the assumption that the eye-gaze avoidance characteristic of autism spectrum disorder (e.g., Pelphrey et al., 2002; Senju and Johnson, 2009) may be contributing to impairments in emotion discrimination (Humphreys et al., 2007; Clark et al., 2008) and joint attention (Bruinsma et al., 2004), and perhaps even to the theory of mind impairments found in this condition (Baron-Cohen, 1995; Senju et al., 2009). Furthermore, our ERP findings provide a potential mechanism to explain how this may occur in ED: avoiding the eyes may result in less recruitment of frontal areas that process both gaze and emotion. If so, behavioral therapies encouraging exploration of the eye-region may have the added benefit of improving emotion discrimination and potentially theory of mind.

# ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the UW Research Ethics Board, with written informed consent from all subjects. All participants gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the UW Research Ethics Board.

# REFERENCES


# AUTHOR CONTRIBUTIONS

SM and RI involved in the early conceptualization and experimental design for this project and revised the draft numerous times together. SM programmed the experiments, created the stimuli, ran the participants, processed the data, analyzed the data with advice from RI, and wrote the initial draft of the manuscript.

# FUNDING

This work was supported by a Queen Elizabeth II Graduate Scholarship for Science and Technology (QEII-GSST) awarded to SM and by grants from the Natural Sciences and Engineering Research Council of Canada (NSERC Discovery Grant #418431), the Canada Foundation for Innovation (CFI, #213322), and the Canada Research Chair Program (CRC, #213322 and #230407), awarded to RI.

# ACKNOWLEDGMENTS

We would like to extend our thanks to everyone who helped with EEG capping, including Tracy Duncan, Sarika Soomal, Mariem Hamdy, and Peter Black. We warmly thank Cyril Pernet for his helpful correspondence about the LIMO package used to graph our ERPs, and Amie Durston for her help with the analyses.




Puce, A., Smith, A., and Allison, T. (2000). ERPs evoked by viewing facial movements. Cogn. Neuropsychol. 17, 221–239. doi: 10.1080/026432900380580


in social anxiety. Biol. Psychol. 91, 263–269. doi: 10.1016/j.biopsycho.2012. 07.004


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 McCrackin and Itier. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Investigating Eye Contact Effect on People's Name Retrieval in Normal Aging and in Alzheimer's Disease

#### Desirée Lopis\* and Laurence Conty

Laboratory of Human and Artificial Cognition (CHArt EA4004), Paris Nanterre University, Nanterre, France

#### Edited by:

Chris Ashwin, University of Bath, United Kingdom

#### Reviewed by:

Gillian Slessor, University of Aberdeen, United Kingdom George Stothart, University of Bath, United Kingdom

> \*Correspondence: Desirée Lopis desiree.lopis@gmail.com

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 22 November 2018 Accepted: 08 May 2019 Published: 29 May 2019

#### Citation:

Lopis D and Conty L (2019) Investigating Eye Contact Effect on People's Name Retrieval in Normal Aging and in Alzheimer's Disease. Front. Psychol. 10:1218. doi: 10.3389/fpsyg.2019.01218 Difficulty in recalling people's name is one of the most universally experienced changes in old age and would also constitute one of the earliest symptom of Alzheimer's disease (AD). Direct gaze, i.e., another individual's gaze directed to the observer that leads to eye contact, has been shown to improve memory for faces and concomitant verbal information. Here, we investigated whether this effect extends to memory for Face-Name association and can thus enhance names' retrieval in normal aging and in AD, at the early stage of the disease. Twenty AD patients, 20 older adults and 25 young adults participated in our study. Subjects were presented with faces displaying either direct or averted gaze in association with a name presented orally. They were then asked to perform a surprise recognition test for each pair of stimuli, in a sequential fashion (i.e., first categorizing a face as old or new and then associating a name using a forcedchoice procedure). Results showed that direct gaze does not improve memory for Face-Name association. Yet, we observed an overall direct gaze memory effect over faces and names independently, across our populations, showing that eye contact enhances the encoding of concomitantly presented stimuli. Our results are the first empirical evidence that eye contact benefits memory throughout the course of aging and lead to better delimit the actual power of eye contact on memory.

Keywords: face-name memory, associative memory, eye contact effects, aging, Alzheimer's disease

# INTRODUCTION

A global decline in memory abilities is commonly observed in aging (e.g., Craik and Rose, 2012; Nyberg et al., 2012), with episodic memory being particularly affected (e.g., Cansino, 2009; Tromp et al., 2015). Difficulty in recalling people's name is one of the most universally experienced changes in old age and a common everyday cognitive complaint of elderly individuals (e.g., Reese et al., 1999; Cargin et al., 2008). In some questionnaire studies on everyday memory ability, this difficulty to recall names is "singled out by many of the elderly respondents as the most noticeable and most frustrating change in cognitive ability" (Cohen and Faulkner, 1984, p. 50). Previous studies suggest that older adults show difficulties in naming pictured celebrities more often than young adult

(e.g., Maylor, 1990; Cross and Burke, 2004), and that agerelated decline is larger for producing proper names than common nouns (Evrard, 2002; Rendell et al., 2005). Because names must be both remembered and mentally associated to the corresponding faces, this difficulty would mostly rely on deficits in associative memory, i.e., the memory for associations between concomitantly presented information (Naveh-Benjamin et al., 2004; Old and Naveh-Benjamin, 2008). The associative deficit hypothesis (ADH) thus "attributes age-related declines in associative memory to older adults' inability to encode and retrieve the relationships between single units of information" (Naveh-Benjamin et al., 2009, p. 221).

Forgetting names is also the most common early symptom of Alzheimer's disease (AD) (Irish et al., 2011; Tak and Hong, 2014). This is not surprising since the underlying neuroanatomical structures supporting the associative memory is the hippocampus which is known to be affected early in the process of AD (Erk et al., 2011). Also, in healthy individuals, face-name associative memory performance was found to be inversely correlated with amyloid burden (i.e., one of the central neuropathological features of AD) in brain region associated with memory systems (Rentz et al., 2011), leading the authors to suggest that this type of memory might be a sensitive marker to detect preclinical stages of AD. Moreover, the ability to remember names and faces decreases concomitantly with the progression of the disease (Tak and Hong, 2014), eventually leading a person with AD to not only forget peoples' names but also to ignore the identity of the person she or he is talking to. This results in experiencing shame and loss of self-confidence, eventually promoting social withdrawal among persons with AD.

Human faces convey critical socio-emotional signals in everyday life. Gaze, in particular, plays a critical role in the regulation of inter-individual exchanges (Kleinke, 1986). Among all gaze directions, it has been shown that direct gaze, i.e., another individual's gaze directed to the observer that leads to eye contact, implicitly influences a wide range of cognitive processes and behaviors (Senju and Johnson, 2009). Among all the socalled "Watching eyes effects" (W.E. effects, Conty et al., 2016), direct gaze has been robustly shown to increase memory for face identity in young adults (Hood et al., 2003; Vuilleumier et al., 2005; Conty and Grèzes, 2011). However, the use of eye contact during interaction not only improves memory for faces but would also enhance memory for concomitant verbal information. For example, results from two different studies conducted in a school context converge in showing that students remember more instructions or details from a story when the teacher gaze at them more frequently (Fry and Smith, 1975; Ottenson and Ottenson, 1979). Fullwood and Doherty-Sneddon (2006) also found positive effects of mutual gaze upon verbal information recall: establishing eye contact with the listener during verbal presentations improves memory for verbal content, compared to presentations delivered with no mutual gaze.

Altogether, these findings point toward the possibility that direct gaze facilitates the face-name association retrieval. If true, eye contact could be used during interactions to compensate the difficulty in recalling names emerging in normal aging and in AD (Baltazar, 2015; Conty et al., 2016). To the best of our knowledge, this hypothesis has never been tested. The purpose of the present study was to investigate the existence of the W.E. effects on memory for face-name association retrieval in healthy young adults, older adults and in AD patients at the early to mild stage of the disease.

It has been argued that the W.E. effects rely on a unique self-referential mechanism, suggesting that they should evolve together during lifespan (see Conty et al., 2016 for more details). Yet, there is a lack of data about the persistence of these effects in normal and pathological aging. We recently started to investigate this issue by focusing on the impact of direct gaze on the modulation of others' appraisal (Kleinke, 1986; Kuzmanovic et al., 2009) and on memory for faces in healthy young adults, healthy old adults and in old adults with AD (Lopis et al., 2017). We exposed participants to faces with different eye directions (direct vs. averted) and asked them to rate each face's degree of likeability. Participants were then asked to identify the previously seen faces during a surprise recognition test. The results revealed for the first time that the effect of direct gaze on other's appraisal (i.e., rating faces with direct gaze as more likeable than faces with averted gaze) is preserved in normal aging as well as until the mild stage of AD. However, the effect on memory for faces emerged exclusively in young participants. On one hand, these findings are encouraging as they showed that some of the W.E. effects still emerge in normal and pathological aging. On the other hand, this invalidated the self-referential model of the W.E. effects, thus calling for further investigations. In our previous study, one explanation for the absence of memory effect of eye contact in older adults – with or without AD – can rely on the lack of salience of the face stimuli.

In our previous study, we used standardized digital grayscale portraits as facial stimuli. Yet, increasing environmental support can improve memory (Craik et al., 1986). More specifically, Bender et al. (2017) recently showed that, regardless of the observer's age, highly distinctive, colored face stimuli and accompanied by multiple non-facial details, as well as salient characteristics such as eye color and head rotation, are easier to recognize and require shorter processing time than less distinctive, grayscale face stimuli. Thus, enriching stimuli by adding external contextual cues – namely, keeping the colored version of the portraits and attributing names to the individuals – may be a plausible way to enhance their ecological validity and therefore increase their salience for older adults.

We thus exposed participants to colored pictures of unfamiliar faces with different eye directions (direct versus averted). Concomitantly to each face presentation, participants heard the name of the individual portrayed in the picture. We asked them to make a decision about whether the name "fitted" or "fitted very well" the face it was associated to Naveh-Benjamin et al. (2009). After an interfering task, participants were submitted to a two-stage surprise recognition test; they were first asked to recognize the previously seen faces, then their names. We predicted that the face-name associations assigned to individuals initially displayed with a direct gaze would be better recognized compared to the associations initially assigned to individuals gazing away from the participant. We recruited AD patients, matched older participants without cognitive impairment and

healthy young subjects, in order to distinguish effects pertaining to normal or pathological aging.

# MATERIALS AND METHODS

# Participants

A total of 65, right-handed, native French-speaking participants were included in the study: 20 patients with a diagnosed AD (14 women; mean age ± standard deviation = 81.8 ± 5.8 years), 20 community-dwelling healthy older adults (OA, 14 women; mean age = 79.9 ± 4.8 years) and 25 healthy young adults (YA, 12 women; mean age = 22.8 ± 3.4 years). We based our sample size calculation on the previous results obtained in our groups regarding the W.E. effects (Lopis et al., 2017). The minimum effect size f in these studies was 0.62 (i.e., η 2 p = 0.28). Based on this value, we computed a total sample size of 45 for a power of 0.9, at alpha 0.05 using the software G∗Power 3 (Faul et al., 2007). We then recruited 25 participants for each group with the aim of accounting for potential exclusion of participants due to technical errors or meeting of exclusion criteria (i.e., neuropsychological tests' or depression scores, see details below). One AD patient was excluded because he did not meet the GDS maximum criterion score of 6. Two older adults were also excluded: one who did not meet the MMSE minimum criterion score of 26 and another for technical issues in collecting the data. Lastly, four supplementary AD patients and three supplementary OA had to be excluded in order to meet the matching criteria for age, gender distribution, years of education and level of depression between the AD patients and OA groups. None of the young participants was excluded.

Young adults were recruited by advertisements spread on a French internet database of volunteers willing to participate in psychology or neuroscience research. OA were community dwelling and were recruited by advertisements and notices distributed through senior citizen organizations in the Paris areas. The patients with AD were recruited from a local memory center and were at the early to mild stage of the disease (MMSE between 19 and 24; Feldman and Woodward, 2005). All participants had normal or corrected-to-normal vision and were naive to the aim of the experiment. They provided written informed consent according to institutional guidelines of the local research ethics committee (who stated on the compliance with the Declaration of Helsinki). The whole procedure was approved by the local ethics committee (Comité de Protection des Personnes Ile-de-France-X, protocole 02(2) 2015 – ALCOM No. 2014-A01141-46).

All participants underwent structured interviews and neuropsychological testing to assess cognitive functioning. A full description of the groups of participants is presented in **Table 1**. The diagnosis of probable or possible AD was assigned to patients by a neurologist according to the criteria of the National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer's disease and Related Disorders Associations (NINCDS/ADRDA; McKhann et al., 2011). AD patients were excluded if they were judged to be unable to understand task instructions. None of the AD patients was reported to have prosopagnosia. For controls, the following exclusion criteria were applied: history of neurological disorders, traumatic brain injury with loss of consciousness and significant history of psychological or psychiatric disorders.

The neuropsychological evaluation consisted in exploring global cognition with the Mini Mental State Examination (MMSE; Folstein et al., 1975), frontal lobe and executive functions with the Frontal Assessment Battery (Dubois et al., 2000), episodic memory with the 5-words test (Dubois et al., 2002), attention and working memory with the forward and backward digit spans (Wechsler, 1997). As for psychiatric evaluation, older participants (OA and AD patients) fulfilled the 15-items Geriatric Depression Scale (Yesavage et al., 1983) and those who scored 7 or more on this scale were excluded from the study. The Mini International Psychiatric Interview 5.0.0 (French version, Lecrubier et al., 1998) was administered to YA to screen for present major depression.

All healthy participants had performances within the normal range in all neuropsychological screening tests (i.e., having a score no more different than 1.65 SD compared to the mean of their group of reference, as provided in the norms of each test). All healthy OA had an MMS score superior or equal to 26. None of them expressed any complaints about their memory. All were paid for their participation.

Older adult and AD groups were matched for age, gender distribution, years of education, and level of depression (see details in **Table 1**).

# Stimuli

#### Facial Stimuli

Seventy static colored photographs of 30 individuals (15 men/15 women) were selected from a database of digitized portraits of adult faces (see Vuilleumier et al., 2005; Conty et al., 2007). All faces had neutral expression and involved individuals unknown to our participants. The age of each individual ranged from 20 to 60 years and our stimuli selection included approximately 1/3 of young-looking faces, 1/3 of middle-agedlooking faces, and 1/3 of old-looking faces. Head direction was always oriented straight toward the observer. All the 30 individuals were photographed with closed eyes. Twenty individuals out of 30 (10 men/10 women) were randomly selected and considered as Target-Faces. They were available in two supplementary views: one with the eyes directed straight toward the observer (Direct Gaze condition), one with the eyes averted by 30◦ toward the right side from the observer position (Averted Gaze condition). Face stimuli with averted gaze were mirrored to obtain both left-averted and rightaverted gaze pictures. Two sets of these 20 Target-Faces were created, F1 and F2. On each set, half of the individuals were shown with direct gaze and the other half with averted gaze (right-averted for half of the participants and left-averted for the other half). The association between Target-Faces and gaze direction compiled for the set F1 was reversed for


TABLE 1 | Means and SDs of demographics, general neuropsychological efficiency and depression scores.

Two-tailed t-tests for independent samples were used. SD, standard deviation; n.s., not significant; AD, Alzheimer's disease; MMSE, Mini Mental State Examination; FAB, Frontal Assessment Battery; GDS, Geriatric Depression Scale.

<sup>a</sup>Gender distribution across groups was tested by using a two-sided Fisher exact test for Count Data.

<sup>b</sup>Maximum possible score.

<sup>c</sup>Not applicable. Young adults were screened for present major depression by using the Mini International Psychiatric Interview 5.0.0 (French version, Lecrubier et al., 1998).

the set F2. The 10 remaining individuals constituted a set of "New" Faces.

#### Name Stimuli

Forty name (20 males/20 females) were extracted from the French National Institute of Statistics and Economic Studies website<sup>1</sup> , with a specific tool allowing to select the most frequently attributed names to children born in the birth cohorts from 1960 to 2000, ranked on popularity. Each selected name was uttered by a female voice and recorded by a professional Digital Voice recorder. The volume was normalized with the AUDACITY 2.0.6 freeware. Two sets of 20 names were created (10 males/10 females): a set of Target Names and a set of New Names. The two sets matched in terms of duration (ranging from 600 to 900 ms), frequency (medium frequency of occurrence of 10,000 to 30,000 times in France between 1960 and 2000) and phonetic properties (same number of syllables, with no more than three syllables), overall and across gender.

## Procedure

Participants sat approximately at 70 cm in front of a Dell computer with a 15.6 inches screen (with a resolution of 1366 × 768 pixels) on which face stimuli were shown on a black background. The name stimuli were presented through speakers plugged to the computer. E-Prime <sup>R</sup> 2.0 software was used to control stimulus presentation, response recording, and latency (Schneider et al., 2002).

The experiment was divided into three parts: an initial encoding task (Study phase), a 5-min interfering phase and a surprise recognition task (Test phase) (**Figure 1**). Four versions of the experiment, corresponding to different 20 Target Face-Target Name pairs were created. Two of them with the set of 20 Target-Faces F1, the other 2 with the set of 20 Target-Faces F2. The 4 versions were created using the set of 20 Target Names. In each version, each Target Name was randomly associated with a Target Face, with the following constraints: (a) face-name gender matching, (b) across the versions, each Target Name was associated with a different Target Face, 2 of them presented with direct gaze and the other 2 with averted gaze. Each participant processed only one version of the experiment, and the four versions were equally processed across participants.

#### Study Phase

During the initial Study phase, participants were presented with 20 different Target Name-Target face pairs (i.e., 20 trials), one by one, in a randomized order. For each face, participants were asked to perform a forced-choice decision about whether the name "fitted" or "fitted very well" the face it was associated to (Naveh-Benjamin et al., 2009; Carr et al., 2017). We choose this incident task to favor the association between the two components, i.e., the face and the name. Participants were told that there was no correct or wrong answer for this task and that the aim was to examine subjective perception of how names and faces match together. Choices were made via a key press system (see details below).

Each trial started with a 1000 ms presentation of a fixation cross (visual angle of 2◦ × 2 ◦ ) which was located at the level of the to-be-presented face's eyes. Then the Target Face appeared on the screen, covering a visual angle of approximately 13◦ horizontally and 16◦ vertically. After 500 ms, the face remained displayed on the screen while the Target Name was presented orally. 1500 ms after the start of name presentation, the question "How do you

<sup>1</sup>https://www.insee.fr/fr/statistiques/3532172

think this name fits him/her?" along with the option choices' tags "Well" and "Very well" appeared on the screen, under the face. The response cannot be provided before the appearance of the question, so that each individual's face was seen during 2000 ms by each participant before she provided her response. The face remained on the screen until a response was given via a key press system (see details below). Immediately after the participant's response, a black screen was displayed during 1000 ms, preceding the fixation cross of the next trial.

#### Interfering Task

After the Study phase, an approximately 5-min Interfering phase followed. Participants were submitted to a 3 min counting backward task, a commonly used interfering task during episodic memory evaluation (Wechsler, 1945; Folstein et al., 1975; Grober et al., 1987). They were asked to begin with 150 and count backward by 3 until 0. This interfering phase was directly followed by a surprise recognition test for each face-name pair (Test phase).

#### Test Phase

During the Test phase, the recognition of the face and of the associated name were processed in a sequential fashion. Participants were first asked to perform an Old-New recognition task on faces, i.e., to say, for each presented face, if they thought they have seen it before or not.

Participants saw a total of 30 faces with closed eyes (i.e., 30 trials), among which 20 "OLD" faces, i.e., the Target Faces previously seen by the participant during the Study phase, and 10 "NEW" faces, i.e., distractor-faces unknown to the participant (see Smith et al., 2006 for similar procedure). Closed eyes were used in order to specifically test the recognition of the identity of the face and to prevent participants from doing a superficial picture-matching task (Bruce, 1982). The order of face presentation was random. Each trial started with the presentation of a fixation cross during 500 ms at the center of the screen. Then, an individual's face with closed eyes appeared on the screen, covering a visual angle of approximately 13◦ horizontally and 16◦ vertically. Such stimulus could be either an "OLD" face or a "NEW" face. A dialog box was concomitantly displayed, reporting the question "Have you already seen this person?" along with the response boxes "Yes" and "No." Once the response was entered (see below for details), participants were then asked to perform a recognition task over names. For this second task, we had the choice to employ either a free recall paradigm (see, for example, Amariglio et al., 2012) or a forced-choice decision

task in which the participant has to choose among two presented names, the one belonging to the face shown (see Polcher et al., 2017 for a similar procedure). We choose the latter to avoid floor effects in participants with cognitive deficits. In addition, this response format may also reduce potential frustration in older adults. Lastly, we did not employ the commonly used paradigm to investigate the ADH in elderly (i.e., submitting the participants to three different memory tests, two for the components and one for their associations, sequentially (Naveh-Benjamin et al., 2004, 2009) because of its duration and redundancy, which may have caused early fatigability in AD participants.

Concretely, as the face was still on the screen, another dialog box appeared, displaying two possible questions, depending on participants' previous answer (whether it was correct or not). If they indicated that they actually recognized the face shown on the screen, they were asked "What was his/her name?" If they reported to never have seen the face before, the displayed question was "What might be his/her name?" Response boxes always displayed two possible names. If the face was an "OLD" one (whatever the response given by the participants), participants were given the choice between the Correct Name (i.e., the name with which the face was actually associated during the Study phase) and a Distractor Name which was always an "OLD' name as well, but previously associated with direct gaze (50% of the trials) or with averted gaze (50% of the trials) and that was of the same gender as the target face. Thus, in half of the trials the Correct Name and the Distractor Name were associated with the same gaze direction during the Study Phase (Distractor Name – Same) while in the other half they were associated with different gaze direction (Distractor Name – Opposite). Distractor Name – Same were pre-selected, but changed for each of the four versions of the experiment. If the face was a "NEW" one, a choice was given to participants between two "NEW" names (i.e., stimuli that participants never heard of). These new names were randomly selected from the set of 20 New Names (see section "Stimuli"), with the following constraints: face/name gender matching; each name only seen once during the Testing phase. Once the response was entered (see below for details), a black screen appeared during 1000 ms, then the next trial begun.

#### Participants' Response

In each experimental phase, dialog boxes were always displayed on the screen, concomitantly and under the face. Each item of the pair of response boxes ("Well/Very well," "Yes/No," "Correct Name/Distractor Name") was always located respectively on the left and on the right side of the screen. The place of each item was fixed except for the Correct Name, which was randomly located, with the constraint to appear half of the cases on the left side of the screen for every participant. Participants were asked to answer all the questions by using a two-choice button press. A cover placed on the computer keyboard allowed the participants to only use two keys to enter their response: one located on the left side keyboard and the other one located on the right side.

#### Debriefing Interview

At the end of the experiment, participants were asked whether any of the faces used in the experiment was previously known to them and if they anticipated the incoming surprise recognition task during the Study phase. They were also asked to report any feeling of inconsistency regarding the face-name associations. All participants confirmed that none of the faces was familiar prior to testing. None of them pointed out the presence of inconsistent or unusual face-name pairs, nor did they anticipate the subsequent recognition task.

# Statistical Analysis

#### Demographic and Neuropsychological Data

Twenty AD patients, 20 healthy OA, and 25 YA were included in the analyses. In order to examine group differences in the total sample (N = 65), we applied a two-sided Fisher exact test for Count Data for categorical and analyses of variance (ANOVAs) for continuous variables, respectively. Following the ANOVA, we performed planned comparisons by using bilateral Student's t-test when main effects or interactions were observed (significance level < 0.05).

#### Variables of Interest

We conducted repeated measures ANOVAs with Gaze Direction (direct/averted) as within-subjects factor and Group (AD/OA/YA) as between-subjects factor on the following variables of interest: mean time of exposure to the faces during the study phase (TEx, i.e., response time to the task + 2000 ms corresponding to the minimal exposure time), percentage of correct face recognition (Hits for faces) and associated RTs (RTs of hits for faces) and percentage of correct name recognition following OLD face correct recognition (Hits for complete face – name recognition, which was considered as the overt evidence of a successful Face-Name association). We also computed the percentage of correct name recognition following OLD face presentation, independently from its correct recognition (Hits for Name recognition) and associated RTs (RTs of Hits for Name recognition). This third score included all the trials where participants either recognized the OLD face or not, but still picked the correct associated name. We therefore assumed that it may still reveal that the association between the face and the name is successful, even if the face has not been systematically, explicitly recognized. This last variable was submitted to an ANOVA with Gaze Direction (direct/averted) and type of Distractor Name (Same or Opposite, see section "Materials and Methods") as within-subjects factor and Group (AD/OA/YA) as between-subjects factor.

Partial Eta-squared (η 2 p ) are reported as effect size indexes. As suggested by Cohen (1988), we considered effect sizes as being small for η 2 <sup>p</sup> < 0.06, medium for 0.06 ≤ η 2 <sup>p</sup> < 0.14, and marked for η 2 <sup>p</sup> ≥ 0.14. For significant comparisons, Cohen's d was used to determine effect size with d < 0.3 corresponding to a small effect, 0.3 < d < 0.8 to a medium effect and d > 0.8 to a large effect (Cohen, 1988).

Lastly, we computed the mean d' parameter and the decision criterion C for each group of participants in order to assess groups' discrimination performances and control for biased response criteria respectively (Green and Swets, 1966). This was done exclusively for the Old-New recognition task for faces since

the forced choice decision paradigm we employed to test name recognition does not allow response bias computation.

In AD group, two participants didn't recognize any "OLD" face during the Test phase. The computation of the Hits for complete face-name recognition and for Name Recognition were therefore impossible for them, so they were excluded from this analysis.

## RESULTS

Descriptive statistics of the three groups are listed in **Table 1**.

# Study Phase

#### Time of Exposure to the Face (TEx)

The ANOVA with Gaze Direction as within-subjects factor and Group as between-subjects factor performed on the TEx (i.e., 2000 ms + response time, see section "Materials and Methods") revealed a main effect of Group (F(2,62) = 28.52; p < 0.0001; η 2 <sup>p</sup> = 0.47). Planned comparisons showed that AD patients were slower in performing the task than healthy OA (respectively: mean = 5331 ± 1388 vs. 3839 ± 798 ms, t(38) = 4.16; p < 0.001, d = 1.3), who were slower than YA (mean = 3213 ± 538, t(43) = 3.13; p = 0.003, d = 0.9). Crucially, no effect of Gaze Direction, or interaction between Gaze Direction and Group, was found on this variable (all p<sup>s</sup> > 0.1), indicating that all participants were exposed during the same amount of time to faces with direct and averted gaze.

# Test Phase

#### Hits for Faces

As expected, AD patients showed poorer discrimination performances (d <sup>0</sup> = 0.33) than OA (0.57) who, in turn, showed poorer discrimination performances than YA (d <sup>0</sup> = 1.60). Moreover, the decision criterion C was centrally placed for each group of participants (ranging from 0.00 to 0.09), showing that groups' response criteria were not biased. These results supports the relevance of analyzing Hit rates as a sensitivity index (Azzopardi and Cowey, 1998).

The ANOVA with Gaze Direction as within-subjects factor and Group as between-subjects factor performed on the Hits for faces revealed a main effect of Group on this variable (F(2,62) = 3.14, p = 0.05; η 2 <sup>p</sup> = 0.09). Planned comparisons revealed that YA (mean = 76 ± 9%) recognized significantly more faces than OA (mean = 63 ± 9%, t(43) = 4.46; p < 0.0001, d = 1.3) and AD (mean = 56 ± 14%, t(43) = 5.51; p < 0.0001, d = 1.7). However, OA only showed a tendency to perform better than AD patients (t(38) = 1.86; p = 0.06, d = 0.6). We also observed a main effect of Gaze direction (F(1,62) = 4.89, p = 0.03; η 2 <sup>p</sup> = 0.07), showing that, overall, participants recognized significantly more faces initially displayed with direct (mean = 67 ± 25%) than averted gaze (mean = 62 ± 26%) (**Figure 2A**). There was no interaction between Gaze Direction and Group (F(2,62) = 0.34; p > 0.1).

#### Complete Face – Name Recognition

The ANOVA with Gaze Direction as within-subjects factor and Group as between-subjects factor performed on the percentage of complete face-name recognition (i.e., hits for OLD faces and associated names) revealed a main effect of group (F(2,59) = 4.72, p = 0.01; η 2 <sup>p</sup> = 0.13). Planned comparisons showed that YA (mean = 71 ± 15%) reached greater complete face/name recognition than healthy OA (55 ± 20%, t(43) = 2.95; p = 0.005, d > 0.9) and AD patients (mean = 56 ± 23%, t(41) = 2.55; p = 0.01, d > 0.7). There were no differences between healthy OA and AD patients (t(36) < 1 – **Figure 2B**). There was no overall effect of gaze direction, or interaction between Gaze Direction and Group (all t < 1; all p > 0.1).

#### Hits for Name Recognition

The ANOVA with Gaze Direction and Type of Distractor Name as within-subjects factors and Group as between-subjects factor performed on the percentage of correct Name Recognition (i.e., names correct recognitions percentage after each OLD face presentation, independently from its correct recognition ) revealed a main effect of Group (F(2,62) = 5.28, p = 0.007; η 2 <sup>p</sup> = 0.14). Planned comparisons revealed that, overall, YA (mean = 69 ± 15%) better associated names to faces than healthy OA (56 ± 16%, t(43) = 2.68; p = 0.01, d > 0.8) and AD patients (mean = 58 ± 11%, t(43) = 2.76; p = 0.008, d > 0.8). No differences were observed between healthy OA and AD patients (t(38) = −0.27; p > 0.1). The ANOVA also revealed a trend for a gaze direction effect on this variable (F(1,62) = 3.45 p = 0.06, η 2 <sup>p</sup> = 0.05) showing that participants tended to better associate names to faces previously seen with direct (mean = 64 ± 18%) compared to averted gaze (59 ± 19%) (**Figure 2C**). This effect did not depend on Group (F < 1; p > 1) but on the Type of Distractor Name (F(1,62) = 4.06, p = 0.04; η 2 <sup>p</sup> = 0.06). Planned comparisons revealed that participants better associated names to faces seen with direct gaze (mean = 68 ± 23%) when confronted to a distractor name that was associated to the opposite (averted) gaze direction (mean = 57 ± 25%, t(64) = 2.82; p = 0.006, d > 0.4), but not when confronted to a distractor name associated to the same (direct) gaze condition (mean = 61 ± 23% vs. mean = 61 ± 27%, t(64) = −0.0; p = 1) (**Figure 3**). No triple interaction between Gaze Direction, type of Distractor Name and Group was found (all F < 1; all p > 0.1). No other main effect or interaction were observed.

#### RTs of Hits for Faces

The ANOVA with Gaze Direction as within-subjects factor and Group as between-subjects factor performed on the RT for hits for faces revealed a main effect of Group on this variable (F(2,58) = 31.66; p < 0.0001; η 2 <sup>p</sup> = 0.52). Planned comparisons showed that, overall, AD participants were slower than healthy OA (respectively, mean = 4744 ± 1325 vs. 3712 ± 1082 ms; t(38) = 2.69; p = 0.01, d = 0.8) who were slower than YA (mean = 2094 ± 499 ms; t(43) = 6.65; p < 0.0001, d = 2.0). No effects of Gaze Direction, or interaction between Gaze Direction and Group, were found on RT's for Hits for Faces (all p > 0.1) (**Table 2**).

#### RTs of Hits for Name Recognition

The ANOVA with Gaze Direction as within-subjects factor and Group as between-subjects factor performed on this variable

FIGURE 2 | Young adults (YAs), healthy older adults (OAs) and AD patients' behavioral results. Mean, standard error to the mean, raw data and the density curve showing the full data distribution are reported. (A) Percentage of recognition for faces presented with direct and averted gaze. A main effect of Group (p = 0.05) and a main effect of Gaze Direction (p = 0.03) are observed. Significant differences detected by planned comparisons: YA vs. OA and AD, trend for OA vs. AD patients. (B) Percentage for Complete Face-Name Recognition. A main effect of Group (p = 0.01) is observed. Significant differences detected by planned comparisons: YA vs. OA and AD. (C) Percentage of recognition for names. A main effect of Group (p = 0.007) and a trend for a main effect of Gaze Direction (p = 0.06) are observed. Significant differences detected by planned comparisons: YA vs. OA and AD.

between Gaze Direction and Type of Distractor Name was observed (p = 0.006). Significant planned comparison is indicated with ∗∗p < 0.01. (B) Direct gaze effect on names' recognition percentage for the condition Distractor Name – Opposite. The effect indicated in (A) with ∗∗ is illustrated here in each group of participants (YA, OA, AD).

revealed a main effect of Group (F(2,58) = 19.2; p < 0.0001; η 2 <sup>p</sup> = 0.39). Planned comparisons showed that, overall, AD participants were slower than healthy OA (respectively, mean = 5161 ± 1935 vs. 4025 ± 1097 ms; t(36) = 2.25; p = 0.03, d = 0.7) who were slower than YA (mean = 2702 ± 882 ms; t(43) = 4.48; p < 0.0001, d = 1.9). No effects of Gaze Direction, or interaction between Gaze Direction and Group, were found (all p > 0.1) (**Table 2**).

# DISCUSSION

Difficulty in recalling people's names is a major concern for healthy older adults and people with AD. Previous works have shown that the use of eye contact during social interaction enhances memory for faces (Mason et al., 2004; Vuilleumier et al., 2005; Conty and Grèzes, 2011) and concomitant verbal information (Fry and Smith, 1975;


 patients. independent

<0.05; ∗∗∗p <0.001.

> ∗p

Sneddon, 2006). We aimed to extend our knowledge about W.E. memory effects and test whether eye contact can improve memory for Face-Name association in healthy YA, OA, and AD patients at the early to mild stage of the disease. Our results do not support this hypothesis, as direct gaze failed to increase memory for a complete face-name recognition in the three populations. Interestingly however, our data revealed a better encoding of stimuli (here faces and names) concomitantly presented with direct, as compared to averted gaze, overall on the three populations. This supports the view that eye contact context benefits memory throughout the course of aging. Contrary to our hypothesis, the direct gaze condition did

Ottenson and Ottenson, 1979; Fullwood and Doherty-

not enhance "Complete Face-Name recognition" scores in any of our groups, suggesting that eye contact does not enhance memory for Face-Name association. One critical reason could be that names do not require semantic treatment. The mere action to form face-name associations is known to be particularly difficult, owing to the inherent lack of relation between a face with a name (Werheid and Clare, 2007). By contrast, forming an association between a face and another biographical information (i.e., professions, hobbies) is easier (McWeeny et al., 1987; Cohen, 1990). McWeeny called this phenomenon the "Baker-Baker paradox": "[...] names remained much harder to recall than occupations. This was true even for ambiguous labels that could be used as names or as occupations. It is much harder to recall that a person's surname is Baker than to recall that a person is a baker" (McWeeny et al., 1987, p. 143).

The absence of any contextual properties requires indeed much more efforts and a higher level of cognitive demand to formulate an associative link, i.e., to bind a proper name to a unique face. It is thus possible that a minimal amount of semantic processing is necessary for direct gaze to benefit associative memory.

However, when focusing on the Name recognition score, our data showed that, overall, participants tended to better recognize names that have previously been presented in direct, as compared to averted gaze condition. At first view, this may have reflected a sort of covert benefit of direct gaze on face-name association. Yet, this trend was actually driven by the context of the recognition task. Participants recognized more names encoded in the direct gaze condition, only when the name was confronted to another name encoded in the averted gaze (opposite) condition. No effect was observed when the participant had to choose between two names previously encoded in the direct gaze condition. Thus, this effect actually revealed an overall benefit in encoding stimuli concomitantly presented with direct gaze (as compared to averted), instead of a benefit on associative memory. Such pattern was observed across all our groups and converges with the few existing studies that have investigated eye contact effects on memory for faces and concomitant verbal information using either teacher's instructions (Fry and Smith, 1975), children's tales (Ottenson and Ottenson, 1979) or sales information about fictitious product (Fullwood and Doherty-Sneddon, 2006) as

TABLE 2


Means and SDs of response times (RTs) for correct answers in the surprise recognition

 test for faces (RT for Hits for faces) and for names (RT for Hits for names) of AD patients, Older adults and Young

fpsyg-10-01218 May 27, 2019 Time: 14:39 # 9

to-be-remembered stimuli. In sum, if the present results do not validate the hypothesis that eye contact can enhance facename association retrieval, they still support the view that eye contact benefits encoding capacities for concomitantly conveyed information. Yet, from a theoretical perspective, it would be interesting to further the investigation of W.E. effects on Face-Word association by varying the semantic value of the word (by using for example names vs. word related to hobbies vs. word unrelated to individuals).

Importantly also, the present data revealed a persistence of the W.E. effects on memory in normal and pathological aging. In a previous study, we suggested that the memory for faces' effect is disrupted under the effect of aging since we only observed its emergence in YA, but not in OA and in AD patients (Lopis et al., 2017). In the present study, when improving the salience of the facial stimuli by adding contextual details (i.e., keeping the colored version of the portraits and attributing names to the individuals), we observe an overall increase of recognition performances for faces, on one hand, and names, on the other hand, previously presented with direct, as compared to averted gaze. These effects were observed across groups, suggesting that the mechanisms underlying the W.E. effects on memory may actually still be functional in healthy OA and in AD patients.

However, it is noteworthy that the overall face recognition percentage for YA, i.e., in young normal cognition, was unusually low as compared to data reported by other works (i.e., 76% in the present data vs. over 80 and 89% respectively in Hood et al., 2003 and Lopis et al., 2017 with similar sample size). So far, studies that have investigated the W.E. effects on memory for faces have asked participants: either to perform a gender identification or an age-classification task (Mason et al., 2004; Vuilleumier et al., 2005; Okruszek et al., 2017), to indicate whether the actor was physically addressing them or not (Conty and Grèzes, 2011), to simply look at the faces (Hood et al., 2003; Smith et al., 2006) or to express likability judgment over them (Lopis et al., 2017). To the best of our knowledge, the present study is the first one to propose a task requiring participants to split their attentional resources between two different kinds of stimuli (i.e., face and name). The attentional resources allocated to process face's morphology were likely diminished. It is therefore possible that the modulation of the attentional load at the encoding stage also plays a key-role in the emergence of the W.E. memory effect in older populations. Further investigations are needed in order to clarify this question.

As expected, our results showed that, overall, AD patients were significantly slower than healthy OA who, in turn, were slower than YA in performing the two recognition tasks. These effects can be related to the general age-associated weakening of executive functions and information processing speed (for a review, see Harada et al., 2013). However, when focusing on percentage of correct response, OA performed only marginally better than AD patients on face recognition and, most importantly, did not differ from the AD patients neither on Complete Face-Name Recognition nor on Name Recognition. At a first view, this was unexpected, especially when considering that several Face-Name association memory tasks (methodologically similar to ours) have been proposed as promising tools for the early detection of cognitive deficits that may constitute early stages of AD (Rentz et al., 2011; Amariglio et al., 2012; Polcher et al., 2017). However, contradictory results have also been reported and future research should endeavor to address this issue (see Rubiño and Andrés, 2018 for a review). For example, advanced age and/or low education have also been associated with a decline of performance on these kinds of tasks (Amariglio et al., 2012; Papp et al., 2014; Sanabria et al., 2018). Plus, a revised version of the Face-Name Association Memory Exam (FNAME) – the most tested tool – has also been shown to poorly discriminate between cognitively healthy older adults and people presenting a mild cognitive impairment (MCI) (Alegret et al., 2015). In the light of these data, the lack of difference between our groups of OA and AD patients is less surprising and confirms the struggle elderly people experience in retrieving names.

# CONCLUSION

By investigating whether eye contact can improve memory for Face-Name association in normal aging and in AD, our study allowed to delimit the actual power of W.E. effect on memory. Our data do not support the view that W.E. can improve other's name retrieval. However, we showed that an eye contact context can still enhance the encoding of concomitantly presented stimuli (here face and name independently). Further, our results are the first empirical evidence that the W.E. memory effect are preserved in normal aging and in AD. However, further investigations are needed to elucidate the conditions that favor the emergence of the W.E. effect on memory in older people.

# ETHICS STATEMENT

All participants provided written informed consent according to institutional guidelines of the local research ethics committee (who stated on the compliance with the Declaration of Helsinki). The whole procedure was approved by the local ethics committee (CPP IdF-X, protocole 02(2) 2015 – ALCOM No. 2014-A01141-46).

# AUTHOR CONTRIBUTIONS

LC and DL conceived and designed the experiments and wrote the manuscript. DL collected the data and performed the analysis.

# FUNDING

This study was funded by the French "Agence Nationale de la Recherche" (ANR) ("AlCom" project ANR-13-JSH2-0001-01).

#### ACKNOWLEDGMENTS

fpsyg-10-01218 May 27, 2019 Time: 14:39 # 11

The authors thank Jérôme Blin, MD, Claire Gautherot and Oriane Breysse, neuropsychologists, and all the staff of the

### REFERENCES


private geriatric hospital "Les Magnolias," as well as Severine Rose, psychologist, and all the staff of the day care center for people living with dementia "Mémoire Plus" for their kind and helpful assistance.



memory exam (S-FNAME) performance is related to amyloid burden in subjective cognitive decline. Sci. Rep. 8, 1–9. doi: 10.1038/s41598-018-21644-y


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Lopis and Conty. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Eye Contact Is a Two-Way Street: Arousal Is Elicited by the Sending and Receiving of Eye Gaze Information

#### *Michelle Jarick\* and Renee Bencic*

*Atypical Perception Laboratory, Department of Psychology, MacEwan University, Edmonton, AB, Canada*

Research shows that arousal is significantly enhanced while participants make eye contact with a live person compared to viewing a picture of direct or averted gaze. Recent research has pointed toward the potential for social interaction as a possible driving force behind the arousal enhancement. That is, eye gaze is not only a signal perceived but also a signal sent out in order to communicate with others. This study aimed to test this by having dyads engage in eye contact and averted gaze naturally, while wearing sunglasses, and while blindfolded; such that the gaze signals were clear, degraded, and blocked, respectively. Autonomic nervous system arousal was measured *via* skin conductance response and level. The results showed that dyads exhibited the highest degree of arousal (increased skin conductance) while making eye contact (send/receive) compared to sendonly or receive-only gaze trials; however, this was only the case if eye contact was clear. Once gaze information became degraded (by sunglasses or blindfold), arousal significantly decreased and was no longer modulated by the sending and receiving of gaze. Therefore, the arousal enhancement observed during eye contact is not only caused by receiving gaze signals (the focus of previous research) and should be more accurately attributed to the subtle interplay between sending and receiving gaze signals.

Keywords: gaze perception, eye contact, skin conductance, social interaction, eye contact effect

# INTRODUCTION

Eye gaze is a rich source of social information. Much research has shown that gaze direction is particularly useful to understand where someone's attention is focused (e.g., Friesen and Kingstone, 1998; Hietanen, 1999; Frischen et al., 2007 for a review), to predict someone's behavior and respond accordingly (e.g., avoiding someone walking toward you; Pelphrey et al., 2004), and most importantly, to know when the lines of communication are open or closed (e.g., Cary, 1978; Ho et al., 2015, for a review Risko et al., 2012; Canigueral and Hamilton, 2019). Indeed, humans have evolved to have eyes that are easily tracked, where our dark pupil is centered on a white sclera (Kobayashi and Kohshima, 1997). This high contrast between the pupil and sclera makes it easy for others to decipher in which direction the eyes are moving. In fact, research has shown that humans are as accurate as 1° of visual angle in determining others' eye movements (Anderson et al., 2011). The high contrast of the eyes

#### *Edited by:*

*Chris Ashwin, University of Bath, United Kingdom*

#### *Reviewed by:*

*Zoë Charlotte Leonora Freeman, University of Bath, United Kingdom Elisa De Stefani, University of Parma, Italy Anneli Kylliainen, University of Tampere, Finland*

*\*Correspondence:* 

*Michelle Jarick jarickm@macewan.ca; michellejarick@gmail.com*

#### *Specialty section:*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology*

*Received: 05 November 2018 Accepted: 13 May 2019 Published: 04 June 2019*

#### *Citation:*

*Jarick M and Bencic R (2019) Eye Contact Is a Two-Way Street: Arousal Is Elicited by the Sending and Receiving of Eye Gaze Information. Front. Psychol. 10:1262. doi: 10.3389/fpsyg.2019.01262*

**178**

also attracts attention due to the complexity of the information the eyes portray. This attention-capturing effect of gaze has been observed from birth, with neonates preferring direct gaze over averted gaze of their mothers (Farroni et al., 2002; Senju and Csibra, 2008). In adults, looking at a photo of a person with direct gaze results in quicker processing of the face, as demonstrated by faster face detection (Senju et al., 2005) and facial gender discrimination accuracy (Macrae et al., 2002), compared to photos of averted gaze.

Recently, the literature on eye gaze has been more focused on face-to-face eye gaze, rather than traditional photos/videos presented in the laboratory, which have revealed some important differences (for reviews, see Risko et al., 2012; Canigueral and Hamilton, 2019). For instance, previous evidence suggests that when instructed to look at an image of a human face, participants visually attend more to the face (Castelhano et al., 2007), and particularly to the eyes of the social stimulus (Pelphrey et al., 2002; Birmingham et al., 2008, 2009) than to other features in the image. However, Laidlaw et al. (2011) tracked participants' eyes while they were seated in a waiting room with a live confederate or a video of the confederate and found that participants fixated on the live confederate fewer times and for shorter durations relative to the video of the confederate. Thus, when the eyes were able to look back, participants' gaze behavior changed. This difference in looking behavior between images and real-world contexts has also been supported in studies that measured autonomic nervous system arousal. For instance, research measuring skin conductance response (SCR) while participants viewed static images of faces with direct and averted gaze has shown only slight changes in SCR between gaze directions (Kampe et al., 2003; Kylliäinen and Hietanen, 2006; Joseph et al., 2008). Yet, Hietanen and colleagues have found a significant enhancement in SCR magnitude, when participants are exposed to direct over averted gaze with a live confederate. For example, Hietanen et al. (2008) found an increase in SCR as well as subjective reports of heightened arousal in participants exposed to direct eye gaze compared to averted gaze. Importantly, Hietanen et al. (2008) compared this SCR effect between live and static stimuli and found that the increase in SCR was only observed during direct gaze with the live confederate and not when the confederate was presented as a static image (Hietanen et al., 2008).

Furthermore, engagement with a real person has been shown to elicit different brain responses compared to an image/video. In multiple studies, Hietanen and colleagues used electroencephalography (EEG) to measure brain activity during direct versus averted gaze with a live person or a photo of a person. In two studies, the researchers took measures of hemispheric asymmetry in the frontal brain regions and found direct gaze with a real person elicited more EEG activation in the left hemisphere indicative of approach motivation compared to averted gaze that elicited more rightward hemispheric activation indicative of avoidance motivation (Hietanen et al., 2008; Pönkänen et al., 2011). Notably, these patterns of hemispheric activation were not observed for eye gaze with a person in a photo. In another study, Pönkänen et al. (2011) found enhanced face/eye-selective event-related brain wave (N170) to be significantly enhanced to direct gaze compared to averted gaze or closed eyes, but only when viewed from a live person. Similar differences between real people and images have been shown using functional magnetic resonance imaging (fMRI). Cavallo et al. (2015) had participants lie down in an MRI scanner while making direct or averted gaze with a photo of a person, a real person in the room (through a mirror), or with themselves in a mirror. They found that faceto-face gaze (direct and averted) elicited significant activation in brain areas involved in language comprehension and production (inferior frontal gyrus (IFG), premotor cortex, and supplementary motor area). Interestingly, the brain areas involved in inferring mental states during social interactions (anterior rostral medial prefrontal cortex or arMPFC) were only active when participants made direct gaze (eye contact) with a real person. Further analysis showed that there was an increase in connectivity between the IFG and arMPFC during live eye contact, suggesting that live gaze triggers a network of brain regions involved in the detection of communicative intentions and language.

Similar interpretations of real face-to-face interactions have been proposed. For example, Laidlaw et al. (2011) suggested that eye gaze with a live person opens up the possibility for interaction than when viewing someone in a video where interaction is not possible. In other words, a live person can look back at you and communicate social information that video stimuli are devoid of. Hietanen and colleagues have interpreted the attentional, physiological, and neurological differences between in direct gaze with a real person as reflecting the increase in self-awareness caused by being the focus of someone else's gaze. This increase in self-awareness is proposed to encompass affective states, perception of another's attention, self-referential processing, and reciprocal attention/interaction mechanisms (for a review, see Hietanen, 2018). As Gallagher (2014) put it, eye gaze in a "live encounter" is more than just a visual representation and encompasses the impact on the observers' own system for action, which presents a "unique type of interaction."

The notion of eye contact eliciting self-awareness was recently examined by Myllyneva and Hietanen (2015, 2016). The researchers measured changes in physiology [skin conductance response (SCR) and heart rate], brain waves (ERP; frontal P3 waveforms) as well as self-report measures of self-awareness while participants viewed another live person (model) behind a voltage-sensitive LC shutter. The visibility of the model was manipulated such that participants could: (1) clearly see the model and the model could see them, (2) believed the model could see them but they could not see the model, and (3) could not see each other. The key condition being the "belief " that someone could see them. The findings from Myllyneva and Hietanen (2015) showed a significant increase in SCR and P3 amplitude, as well as heart rate deceleration when participants "believed" the model could see them, but they could not see the model and self-awareness ratings were higher as well. These findings were replicated when the model wore sunglasses that either degraded eye gaze or blocked it completely (Experiment 2), where SCR increased when the eyes were visible and degraded but not when they were blocked. In Myllyneva and Hietanen (2016), the findings were contradictory. Those results revealed that self-awareness ratings were higher when participants could see the model or believed the model could see them, but physiological responses (skin conductance increase and heart rate deceleration) only differed when the participants could see the model. Thus, the "belief " that the model could see them was enough to increase the subjective experience of self-awareness, but not the objective physiological response associated with it. Myllyneva and Hietanen concluded that despite the experience of self-awareness, a social encounter must satisfy two conditions: (1) looking at another person and (2) being looked at by another person, to elicit physiological and neurological responses. However, the experience of self-awareness is a complex, high level, cognitive state, and likely encompasses many factors, which may have been represented differently between the two experiments conducted by Myllyneva and Hietanen.

We propose that a cleaner, low-level, perceptual explanation of gaze processing during live social interactions could be that people are simultaneously attending to two gaze signals: the gaze signal from others while at the same time monitoring their own gaze signals (i.e., being self-aware). While a video depicting social interaction only involves attending to the gaze signals coming from the person in the video. Furthermore, a live setting involves continuous, real-time monitoring, in which case the two sources of information could correlate or be independent depending on the context. Gobel et al. (2015) demonstrated the "sending" aspect of gaze behavior by having people filmed while watching videos of higher versus lower ranked individuals. The participants believed that their viewing behavior would later be watched by the individual in the video or that they would not be seen by anyone. When participants thought that their behavior was going to be later observed, they looked at the eyes less if the person was a higher ranked individual compared to lower ranked. Thus, those viewing higher ranked authoritative people were more controlled in their viewing behavior. In other words, participants were sensitive to the gaze signals they themselves were conveying when they believed someone might analyze them.

While mounting research has shown a definitive enhancement in physiological arousal when mutual eye gaze is made with a live person, it has yet to be shown whether arousal is being elicited by the eye gaze from others or the self-monitoring of our own gaze, or both. Here, we aimed to systematically examine the relationship between gaze signals and physiological arousal by manipulating the degree to which gaze signals are sent and received during a live social interaction. We measured the level of arousal of two strangers as they sat side-by-side on a couch and performed four different gaze "poses" (or trials) for 1 min each: (1) looked away from one another (*baseline/no-gaze trials*), (2) looked at their partner's profile (*sent-only trials*), (3) had their partner look at them (*received-only trials*), and (4) made eye contact (*sent/ received trials*). We then manipulated the clarity by which the gaze signals could be sent/received by either degrading them (one participant worse tinted sunglasses) or blocking them (one participant was blindfolded). We have three main hypotheses, one for each clarity condition. First, we believe that the significant enhancement of arousal observed in studies with live interactions is likely elicited by both *sending* gaze signals out (and the self-monitoring that goes along with that) as well as *receiving* gaze signals from others (and the interpretations that go along with that). Hence, our first prediction is that arousal will be enhanced the most when participants make eye contact because participants will be *sending* and *receiving* gaze signals, which has been already shown numerous times in previous research. We also predict that the sent-only and received-only trials will have a significant boost in arousal compared to the no-gaze trials, if arousal is associated with sending information. Note that there is already evidence that arousal is associated with receiving gaze from photos, but currently no research regarding sending gaze only. We also wanted to test whether the gaze signals need to be clear in order for those signals to be interpreted. For instance, situations do arise where the gaze signals are hard to receive even though eye contact is being made (e.g., imagine making eye contact with someone wearing sunglasses). Thus, our second prediction is that arousal during eye contact trials when gaze is degraded (with sunglasses) will be similar to the sent-only and received-only trials, where only one signal is influencing arousal. In the blindfolded condition, there are no signals sent or received and therefore, our third prediction is that the arousal during eye contact will not differ from the no-gaze trials. However, participants who are blindfolded are still aware that someone is looking at them and they are the focus of someone's attention. Thus, if arousal is associated with the mental attribution of self-awareness (suggested by Myllyneva and Hietanen, 2015), then we might see a boost in arousal comparative to the sent-only trials.

There are few unique aspects of our design that warrant justification. To boost ecological validity, we had two participants perform the eye gaze trials together where previous studies typically used a participant and confederate. We chose not to use a confederate because we have observed in our previous (unpublished) studies that eye contact is something that people can quickly and easily habituate to. If confederates habituate to the gaze trials after one or two participants, then the eye gaze experience could be diminished for future participants, thereby giving them a different eye gaze experience. This habituation effect is also why we chose to only present each gaze trial (*away, sent-only, received-only, eye contact*) once per condition. In our most recent (unpublished) research, we have observed that participants arousal for eye contact becomes less and less the more times that they do it, and after three repetitions, arousal is no longer elicited to the same degree as it was in the first trial. Thus, we wanted to limit our trial number to three repetitions of each gaze trial. Lastly, we have an unusually long duration of eye contact (1 min) that is not typical of everyday eye contact that lasts only 3–5 s (Helminen et al., 2001). However, our question was not in relation to "making eye contact" *per se*, but rather associated with the *signals* sent and received, which would usually happen over multiple eye contact experiences during a conversation. Rather than having participants make eye contact naturally and unpredictably for a longer duration, we decided that we could encourage faster signal transfer if participants held eye contact for more than 5 s. While 1 min seems like a long duration, participants seem to be able to do it well and it allows us to get an idea of how arousal changes over time by evaluating skin conductance level, rather than just skin conductance responses. By taking advantage of this, we believe this to be the first study to show arousal as a function of time during social interactions.

#### EXPERIMENT 1

# Method

#### Participants

Sixty-four MacEwan University undergraduate students (13 males, 47 females, mean age of 19 years old) were recruited in pairs (dyads) to participate in the study. There were 17 same-sex female dyads and 13 male-female opposite-sex dyads, all right-handed, and had normal or corrected-to-normal vision and hearing. Participants were compensated with 2% course credit toward their psychology course. All participants reported not knowing their partner, except for two same-sex dyads (*n* = 4) who indicated that they were friends and their data was excluded from the analysis. Experimental procedures were approved by the MacEwan Research Ethics Board. All participants gave informed, written consent prior to participation.

#### Materials and Procedure

Upon arrival to the laboratory, participant dyads were greeted by a female investigator and asked to first sit next to one another on the same couch. Tape was used to indicate the desired physical proximity between participants on the couch (this distance was approximately 30 cm apart). The investigator sat in front of both participants at a distance of 115 cm behind a table with a laptop computer. Participants were then fitted with physiological monitoring equipment (*Thought Technology, Inc.*), whereby two Ag/AgCl electrodes were attached to the palmar surface and of the distal phalanxes of their ring and index finger of their left hand. Their skin conductance level (SCL) was collected at a sampling rate of 256 samples per second. Participants were informed that their nervous system arousal would be monitored during different gaze trials and to try and remain as still as possible to prevent movement artifacts. Participants were also to try and stay as neutral as possible by keeping a neutral facial expression and withholding laughter or talking. All participants were able to remain fairly neutral with ease. Designated rest periods were inserted as 1–2 min breaks between each of the gaze trials, where participants could move, talk, or laugh during those breaks if need be. However, the investigator noted that participants did not talk much with each other during these breaks, but they would occasionally smile or laugh at the investigator.

The experiment was conducted in three blocks, one for each condition: clear (gaze was clearly observed), degraded (gaze was degraded by sunglasses), and blocked (gaze was blocked by a blindfold). The order of the blocks was counterbalanced in an ABC, BCA, CAB for each dyad. See **Figure 1** for a schematic representation of the three conditions. For the *clear* condition, participants performed the gaze trials normally without any obstruction of gaze information. Both participants A (on the left) and B (on the right) could send and receive gaze information clearly. For the *degraded* condition, participant B was asked to wear sunglasses while performing the gaze trials. Note that in this condition, participant A would not be able to send or receive gaze information to participant B very well, since participant B's gaze would be degraded by the tint of the sunglasses. Participant B, however, could send and receive gaze information from participant A just fine. For the *blocked* condition, participant B wore a blindfold while performing the gaze trials. In this condition, participants A and B could no longer send or receive gaze information with each other. However, it is important to note that the participants were still instructed on which gaze trial they were to complete, so when asked to make eye contact, for example, they would still turn their heads toward each other as if to make eye contact. Thus, the participant with the blindfold knew that there was someone looking at them during those trials.

Each block consisted of four gaze trials (*no-gaze, send-only, receive-only*, and *send/receive or eye contact*), where participants performed four "poses" for 1 min each: (1) *no-gaze* - participants looked away from each other by turning their head in the opposite direction from their partner, (2) *send-only trials* participant looked at their partner's profile by turning their head toward their partner, (3) *receive-only trials*—participant was looked at by their partner while looking straight ahead and, (4) *eye contact trials*—both participants engaged in eye contact by only turning their head toward one another. Participants began and ended each trial on a verbal command from the investigator (i.e., "ready, set, go" and "stop"), who kept track of the 1-min interval and event-marked the SCL recording to coincide with the start and stop of each gaze trial. Participants were not aware of the trial order or duration of gaze. This was due to previous studies in our lab showing anticipatory arousal (i.e., increase in SCL) for the eye contact trials before the trial started, which we wanted to mitigate as much as possible. Thus, participants were given the instruction to perform each "pose" before each gaze trial. The order of the gaze trials remained the same for each block but was counterbalanced across blocks. For example, the clear condition had the trial order of A (no-gaze), B (send), C (receive), and D (eye contact); the degraded condition had A (no-gaze), C (receive), D (eye contact), and B (send); and blocked condition had A (no-gaze), D (eye contact), C (receive), and B (send). The no-gaze trials were always first because they acted as a baseline measure of arousal for each condition, since electrodermal activity has been shown to steadily change (increase in some and decrease in others) over the course of the experimental session. In other words, we wanted to evaluate the base level of arousal for each participant at the beginning of each testing block, and then be able to compare that arousal level to the following key gaze trials within that block.

The experiment took approximately 45 min to complete. Upon completion, participants had the physiological equipment removed and were verbally debriefed.

#### Data Analysis

The skin conductance data was manually reviewed offline using the *Thought Technology* software *Physiology Suite* and any visible artefacts were removed (less than 1% of each participants data was removed). No high-pass or low-pass filters were needed. The data was then exported to Excel and imported into a custom Matlab program where it was epoched by participant, condition, and gaze trial. This program also baseline-corrected the data to 8-s before the start of each trial. We chose a longer baseline to include the electrodermal change from the start of the trial, which included the investigator giving the gaze instructions. Data were included even if participants demonstrated little change in skin conductance, but would have been removed if participants demonstrated a change too soon (0.1 μS within the first second) after the trial started, since this response would not have been elicited by the stimulus (Dawson et al., 2000). However, this did not occur in any of the data collected and therefore none was removed.

The data was analyzed in terms of both skin conductance responses (SCRs) and skin conductance levels (SCLs) in SPSS. SCRs were defined as the mean amplitude across the first 10 s of each gaze trial, while the SCLs were calculated as the average of the entire 60-s epoch. Mean SCRs and mean SCLs were submitted to a 2 (*Participant*: A or B) × 3 *(Condition*: clear, degraded, blocked) × 4 (*Gaze trial*: no-gaze, send-only, receive-only, send/receive) mixed Analysis of Variance (ANOVA) with the between-subject factor *Participant* and within-subject factors *Condition* and *Gaze Trial*, where sphericity was violated, a Greenhouse-Geiser correction was used.

#### Results

Our hypothesis was that the arousal (SCR and SCL) in response to eye contact would be significantly higher in the natural condition (clear *sending* and *receiving* of gaze signals that significantly differs from other trials), lower for the sunglasses condition (degraded gaze signals, resembling more *send* or *receive* gaze trials), and lowest for the blindfolded condition (blocked gaze signals, resembling *no-gaze* trials).

#### Skin Conductance Response Analysis

**Figure 2** shows the mean skin conductance responses for each gaze trial in each condition. The ANOVA revealed significant main effects for Condition, *F*(2, 116) = 7.755, *p* = 0.001, *η*<sup>2</sup> = 0.12 and Gaze Trial, *F*(3, 174) = 3.42, *p* = 0.05, *η*<sup>2</sup> = 0.06. Most importantly, there was a significant interaction between Condition and Gaze Trial, *F*(6, 348) = 12.344, *p* < 0.001, *η*<sup>2</sup> = 0.18. Pairwise Bonferroni comparisons showed a significant difference between the natural and blindfold conditions (*p* < 0.001), such that the natural condition elicited the overall highest SCR (*M* = 1.15 μS, SE = 0.14), while the blindfolded condition showed the lowest overall SCR (*M* = 0.353 μS, SE *=* 0.11). The ANOVA also

revealed Pairwise Bonferroni comparisons showed a significant difference between eye contact and away trials, whereby the eye contact trial elicited the largest SCR (*M* = 1.12 μS, SE *=* 0.16) and the away trial showed the lowest SCR (*M* = 0.559 μS, SE *=* 0.15). There was no between-subjects difference in SCR between participants A and B.

#### Skin Conductance Level Analysis

**Figure 3** shows the mean skin conductance levels (average of the 60-s) for each gaze trial in each condition. The ANOVA revealed significant main effects of Condition, [*F*(2, 112) = 5.951, *p* = 0.01, *η*<sup>2</sup> = 0.10] a marginally significant main effect of Gaze Trial, [*F*(3, 168) = 2.35, *p* = 0.07, *η*<sup>2</sup> = 0.04]. Most importantly, there was a significant interaction between Condition and Gaze Trial, [*F*(6, 336) = 12.728, *p* < 0.001, *η*<sup>2</sup> = 0.19]. There was no between-subjects difference in SCR between participants A and B.

Since the hypotheses were separate by condition, we decided to investigate the significant interaction by conducting separate one-way ANOVAs for each condition separately, as well as a one-way ANOVA for the eye contact trials (*send/receive*) across the three conditions. A Bonferroni correction was used for multiple comparisons (*alpha of p <* 0.012).

#### *Clear Condition*

In this condition, we predicted that eye contact (*send/receive trials*) would elicit significantly higher arousal responses compared to all of the other gaze trials (*send-only, receive-only,* or *no-gaze*). The ANOVA revealed a significant SCR difference across Gaze Trials [*F*(3, 174) = 10.547, *p* < 0.001, *η*<sup>2</sup> = 0.15] and SCLs [*F*(3, 174) = 9.77, *p* < 0.001, *η*<sup>2</sup> = 0.14]. Pairwise Bonferroni comparisons showed that eye contact trials elicited higher SCRs compared to away trials (*p* < 0.001), send trials (*p* = 0.015), and receive trials (*p* = 0.004). The SCL analysis mirrored that, with eye contact being significantly higher compared to away trials (*p* < 0.001), send trials (*p* = 0.01), and receive trials (*p* = 0.006). There were no between-subjects differences between participants A and B.

#### *Degraded Condition*

In this condition, eye gaze was degraded by sunglasses, and thus it might be hard for participants to send or receive gaze signals at the same time. As such, we predicted that eye contact would elicit a similar arousal response as the *send-only* and *receive-only* trials. Consistent with our prediction, the ANOVA revealed only a marginal SRC effect for Gaze Trial [*F*(3, 174) = 2.88, *p* = 0.065] and no SCL effect [*F*(3, 174) = 1.267, *p* = 0.287, *η*<sup>2</sup> = 0.02]. Pairwise Bonferroni comparisons on the SCRs showed there was a significant difference only between the eye contract trials and away trials (*p* < 0.05). There were no between-subjects differences between participants A and B.

#### *Blocked Condition*

In this condition, eye gaze was blocked by a blindfold. Since eye contact cannot be made in this case, we predicted arousal to be the same across all Gaze Trials. As expected, the ANOVA did not reveal a significant SCR effect for Gaze Trial [*F*(3, 174) = 0.522, *p* = 0.668, *η*<sup>2</sup> = 0.009] nor for SCLs [*F*(3, 168) = 0.701, *p* = 0.553, *η*<sup>2</sup> = 0.01]. However, unlike the previous conditions, both SCRs [*F*(1, 58) = 4.838, *p* < 0.05] and SCLs [*F*(1, 56) = 3.98, *p* = 0.05] revealed a marginally significant difference between participant A and participant B, such that Participant B showed higher SCRs and SCLs (*MSCR* = 0.667 μS, *MSCL* = 0.297 μS) for all gaze trials compared to Participant A (*MSCR* = −0.105 μS, *MSCL* = −0.084 μS). This finding is likely due to Participant B being the one blindfolded and therefore subject of attention.

#### Eye Contact Across Conditions

**Figure 4** shows the mean skin conductance responses for eye contact trials in each condition. Here, we predicted that

*degraded, and blocked*). The error bars represent the 95% confidence intervals. \* represents *p* < 0.001.

arousal elicited by eye contact would be modulated by the clarity of the gaze signals. As such, the clear condition should elicit the highest arousal gaze signals can be both sent and received clearly. However, once the gaze signals are degraded, there will be less arousal because there might only be one gaze signal (send or received) that is processed and when signals are blocked completely with no gaze signals involved, arousal should be lowest. The ANOVA compared SCRs for eye contact trials across each condition (*clear, degraded, and blocked*). The results showed a significant main effect of condition, *F*(2, 116) = 25.83, *p* < 0.001, *η*<sup>2</sup> = 0.31. Pairwise Bonferroni comparisons showed significant differences between every condition, with the clear condition eliciting the highest arousal compared to degraded (*p* < 0.01) and blocked (*p* < 0.001). The degraded condition also showed significantly higher arousal than blocked (*p* < 0.01). There were no betweensubjects differences between participants A and B.

#### Arousal as a Function of Time

The mean skin conductance level for each participant was epoched into six time-windows of 10 s each. The data for participants A and B can be seen in **Figure 5**. The means for each epoch were submitted to a 2 (*Participant*: A or B) × 4 (*Gaze trial*: no-gaze, send-only, receive-only, send/receive) × 6 (*Time:* 1, 2, 3, 4, 5, 6) mixed Analysis of Variance (ANOVA) with the between-subject factor *Participant* and within-subject factors *Gaze Trial* and *Time*. A Bonferroni correction was used for multiple comparisons.

#### *Clear Condition*

The ANOVA revealed a significant main effect of Gaze Trial [*F*(3, 174) = 8.13, *p* < 0.001, *η*<sup>2</sup> = 0.12], a main effect of Time [*F*(5, 290) = 53.22, *p* < 0.001, *η*<sup>2</sup> = 0.48], and an interaction between Gaze Trial and Time [*F*(15, 870) = 3.19, *p* < 0.001, *η*<sup>2</sup> = 0.05]. Within-subject contrasts of Gaze Trial showed that eye contact trials were significantly higher in SCL compared to the other trials, *F*(1, 58) = 17.43, *p* < 0.001, *η*2 = 0.23. Within-subject contrasts of Time showed that every time-window was significantly lower than the one before it (all *p*'*s* < 0.001) suggesting that SCL continually declined over 1 min. Within-subject contrasts involving the interaction showed a significant higher SCL between the eye contact trials and the rest during time-windows 5 (*p* < 0.005) and 6 (*p* < 0.02), suggesting that the SCL during eye contact stayed high during 1 min. There was no difference between participants A and B.

#### *Degraded Condition*

The ANOVA revealed a significant main effect of Time [*F*(5, 290) = 56.37, *p* < 0.001, *η*<sup>2</sup> = 0.49], and an interaction between Gaze Trial and Time [*F*(15, 870) = 3.97, *p* < 0.001, *η*<sup>2</sup> = 0.06], but no main effect of Gaze Trial. Within-subject contrasts of Time showed that every time-window was significantly lower than the one before it (all *p's* < 0.001) suggesting that SCL continually declined over 1 min. Within-subject contrasts involving the interaction showed a significant SCL difference for eye contact trials from the rest of the gaze trials until time-window 4 (*p* < 0.005). This suggests that the SCL for eye contact declined to the level of the other gaze trials after time about 30 s. There was no difference between participants A and B.

#### *Blocked Condition*

The ANOVA revealed a significant main effect of Time [*F*(5, 290) = 60.75, *p* < 0.001, *η*<sup>2</sup> = 0.52], and an interaction between Gaze Trial and Time [*F*(15, 870) = 4.95, *p* < 0.001, *η*<sup>2</sup> = 0.08], but no main effect of Gaze Trial. Within-subject contrasts of Time showed that every time-window was significantly lower than the one before it (all *p* < 0.001) suggesting that SCL continually declined over 1 min. Within-subject contrasts involving the interaction showed a significant SCL difference from the rest of the gaze trials during time-windows 2, (*p* < 0.005), 3 (*p* < 0.001), 4 (*p* < 0.001), and 6 (*p* < 0.002). This interaction suggests that eye contact trials began a significant decline more than the other gaze trials. There was no difference between participants A and B.

#### EXPERIMENT 2

It could be argued that the heightened autonomic nervous system arousal observed when participants made eye contact was *not* due to the sending and receiving of gaze information (and an increased opportunity for social interaction) but instead a consequence of viewing another person's eyes. The eyes are salient attention-capturing stimuli and this alone may be responsible for the increase in SCRs. Moreover, one could argue that the blocked and degraded conditions elicited lower SCRs not because they hindered the participants' ability to successfully send and receive gaze information, but instead because they degraded the visibility (and thus the saliency) of the eyes. To eliminate the arousal associated with just seeing the eyes, we conducted a *post hoc* experiment where participants performed the gaze trials with themselves through a mirror. Since there would be no need to send or receive gaze signals with oneself, we assumed that any potential for social interaction would be eliminated. Yet in this scenario, participants are still making eye contact with a pair of real eyes and thus, the saliency of the eyes remains constant. If making eye contact with oneself in the mirror is less arousing than making eye contact with another person, then that would demonstrate there is something beyond the saliency of the eyes that is driving the enhanced arousal seen in Experiment 1. We hypothesized there to be no significant difference in physiological response between any of the gaze trials because there would be no need to send and/or receive signals when making eye contact with oneself. If our predictions are supported, the results would more strongly speak to the notion that the heightened arousal found in Experiment 1 is due to both individuals sending *and* receiving gaze information, which contributes to the potential for further social interaction.

### Method

#### Participants

Eleven MacEwan University undergraduates were recruited to participate (2 males, 8 females; average age = 21 years old). All were right-handed with normal or corrected-to-normal vision and hearing. Data from one participant was not included in the analysis due to instructions not being followed (they moved their whole body during the gaze trials instead of just their head as instructed).

#### Materials and Procedure

Participants were situated in front of a 27″ iMac computer where a picture was taken of their averted face (i.e., their profile) using Photo Booth. Participants were then fitted with physiological monitoring equipment (*Thought Technology, Inc.*) whereby two Ag/AgCl electrodes were attached to the palmar surface and of the distal phalanxes of their ring and index finger of their left hand. Their skin conductance level (SCL) was collected at a sampling rate of 256 samples per second. Participants engaged in three gaze trials that attempted to replicate those in Experiment 1: (1) looked at the wall (*nogaze trials*), (2) looked at their own averted face on the computer screen (*send-only trials*), and (3) made eye contact with themselves in a mirror (*send/receive or eye contact trials*). Each trial lasted for 1 min and was signaled by the experimenter to begin and end. Participants remained relatively still throughout the experiment to prevent movement artifacts, with 1–2 min breaks between trials to allow for movement if needed. Following the three gaze trials, participants were then detached from the physiological equipment and filled out a brief questionnaire that assessed the degree to which looking at themselves in the mirror provoked negative (e.g. disgust, awkward) or positive (e.g. attraction, content) emotions. The questionnaire was used to ascertain whether any arousal observed was associated with making eye contact with oneself in the mirror or whether it could have been attributed to the emotions elicited by looking at oneself in the mirror. The experiment took approximately 15 min to complete.

#### Results

Similar to Experiment 1, data was pre-processed and artifactchecked for each gaze trial with the removal of an 8-s anticipatory phase during the instructions. Mean skin conductance responses (SCRs) were calculated by averaging the amplitude across the first 10 s while the skin conductance levels (SCLs) were calculated as the average amplitude of the entire 1-min epoch. Mean SCRs and SCLs were baselinecorrected to 1 s before anticipatory began. Data was included even if participants demonstrated little change in skin conductance across the trials. The average SCRs across the gaze trials can be seen in **Figure 5**.

Mean SCRs and SCLs were submitted to a one-way Analysis of Variance (ANOVA) with the within-subjects factor Gaze Trial (*no-gaze, send-only, and send/receive*). As predicted, the ANOVAs revealed no significant main effect across Gaze Trials for SCRs [*F*(2, 18) = 2.117, *p* = *n.s.*] and SCLs [*F*(2, 18) = 1.656, *p = n.s.*], such that making eye contact with oneself was no more arousing than the send or away trials. It should be noted however that the sample size is small and observed power was low (0.3). The small sample also lends itself to greater variability which may impact the ANOVA results. However, the raw data for each participant can be seen in **Table 1**, showing that only two participants had even a hint of the trend toward greater SCR for eye contact than the other trials (represented by \*). All other participants showed either no change in SCR over each trial or in the opposite direction expected.

TABLE 1 | Individual mean skin conductance responses (SCRs) for the mirror condition (Experiment 2).


*The last column represents the skin conductance change between the no-gaze and eye contact trials. Only two participants showed even a hint of the eye contact trials eliciting higher SCRs, indicated by the \*.*

#### Discussion

The purpose of this study was two-fold: first to demonstrate that live eye contact between two strangers can elicit heightened autonomic nervous system arousal due to the sending *and* receiving of gaze signals; and second, to evaluate whether arousal can be modulated by the clarity of gaze information. For example, can the eyes elicit arousal even if the information received from them is unclear? To test this, we monitored participants' skin conductance level while they maintained gaze for 1 min in a clear condition, degraded condition (sunglasses), and blocked condition (blindfolded). Hence, our hypotheses centered around how eye contact (*send/receive trials*) would activate nervous system arousal in each other the three clarity conditions. Thus, our predictions were threefold: (1) we predicted that arousal will be enhanced the most when participants make eye contact over any other gaze trial (*send-only, receive-only, or no-gaze*) because participants will be *sending* and *receiving* gaze signals; (2) we predicted that arousal during degraded eye contact trials (with sunglasses) will be similar to the send-only and receive-only trials, where only one signal is being monitored/processed; and (3) lastly, we predicted that the arousal for the blindfolded eye contact trials will not differ from the no-gaze trials, since no gaze signals are being sent or received.

There are two unique characteristics of our design that warrant a second mention. One is that this might be the first study to observe and measure autonomic nervous system arousal of two live participants engaged in different gaze conditions—typically research has measured a participant response to a live confederate, or response of participant to a static image of direct gaze. As such, our design has strong ecological validity. Second, our design required participants to maintain the gaze (e.g., eye contact) for 60-s intervals—a time period that is much longer in duration than what is typically used in the eye gaze literature (~3–5 s; Helminen et al., 2001). This prolonged time period allows the flexibility to analyze the skin conductance response (SCR) when eye contact is made (i.e., initial gaze response), as well as the change in skin conductance level (SCL) over time due to continuous sending and receiving of gaze information (i.e., gaze communication). Thus, we afforded the opportunity to evaluate arousal to gaze as a function of time, giving more insight into a complex process.

Our main finding was that arousal was highest when participants made eye contact and when the gaze signals were clearly sent and received simultaneously. We believe that the arousal enhancement was due to the combined, simultaneous tasks of monitoring one's own gaze signals and interpreting the gaze signals of others. This finding is consistent with the recent eye tracking data from Hessels et al. (2019) who simultaneously recorded gaze from two interacting participants. They reported that gaze depends on the sub-task, such as speaking versus listening. The results showed that people will monitor gaze for cues about when speaking will commence. This gaze monitoring is analogous to what we mean by *receiving* gaze signals. For example, while one person is sending the gaze signal regarding speaking, the other is sending out signals regarding listening. Interestingly, eye contact trials in the clear condition were the only trials that significantly differed in skin conductance level (SCL) from the other gaze trials. Thus, not only did eye contact initially elevate arousal, but it stayed elevated for the 1-min epoch (see **Figure 5**). In terms of sending and receiving gaze signals, this finding suggests that we keep processing and interpreting gaze information continually while eye contact is made. In many animals, prolonged gaze is typically associated with aggression or intimidation (Emery, 2000; Skuse, 2003). In humans, sustained eye contact has been linked to expressing control/dominance or love/inclusion (Argyle et al., 1974; Kellerman et al., 1989; Hall et al., 2005; for a review, see Hietanen, 2018). Although there was no reason for participants in this study to convey emotional feelings, especially when they were clearly instructed to remain neutral, it is possible that some emotional processing was going on between the eyes. Measuring the affective relationship with arousal level and gaze duration in humans is an avenue for future research.

With regards to signal clarity, we found that as the gaze signals became degraded, arousal was diminished. That is, when participants made eye contact with someone wearing sunglasses (where the eyes can be seen but the signals cannot be received), arousal was significantly lower than in the clear condition. Indeed, the arousal associated with eye contact in the degraded condition was on the same level (not significantly different) from the arousal elicited by the send-only or receiveonly conditions. This finding suggests that when gaze signals are degraded, only one process (either sending *or* receiving occurs). For instance, if you are the person wearing the sunglasses, then you do not need to self-monitor your own gaze because those signals cannot be clearly received by an observer. Similarly, if you are the person looking at someone wearing sunglasses, you can send gaze signals and need to self-monitor, but you cannot receive the signals clearly and therefore do not need to process or interpret those signals. Thus, arousal in this case might only be associated with either self-monitoring one's own gaze signals, *or* interpreting others gaze signals, but not both. This finding is not consistent with Myllyneva and Hietanen (2015) who found an increase in arousal (skin conductance response) for eye contact compared to averted gaze when the gaze signals were clear and when they were degraded by sunglasses (but not when blocked). However, depending on the tint of the sunglasses and the distance of the model, gaze signals might have been clear enough to interpret. In our study here, sunglasses were tinted to the degree that they eyes were noticeable, but eye movements were not able to be tracked. Given the limited number of trials in our study, we might have found an effect similar to Myllyneva and Hietanen (2015) had we tested additional participants.

We did find a similar result as Myllyneva and Hietanen (2015) when the gaze signals were blocked in the blindfold condition, where no gaze signals could be sent or received. Myllyneva and Hietanen (2015) found that eye contact with a model wearing opaque glasses was the same as the model averted. We found the same result, such that the arousal in the eye contact trials was not significantly different from the *no-gaze* trials. However, planned comparisons did show an interesting effect of Participant A (not manipulated) compared to B (wore the blindfold). Participant B showed higher SCRs and SCLs for all gaze trials compared to Participant A. These results could have been due to participant B "believing" that they were the center of someone's attention, as described in Myllyneva and Hietanen (2015, 2016). Myllyneva and Hietanen suggested that even the thought that someone was looking at you could be enough to generate an arousal response associated with an increase in self-awareness. Perhaps that was the experience of Participant B, but more research is needed to directly test these speculations.

In Experiment 2, the heightened arousal found during eye contact in the clear condition in Experiment 1 was not observed when participants locked eyes with themselves in a mirror. Instead, as seen in **Figure 6**, participants SCRs when looking in the mirror looked very similar in magnitude to the SCRs observed in the sunglasses condition when the gaze signal was degraded. That said, there is one similarity between the mirror and sunglasses conditions–the degree to which the eyes could be perceived. In both cases, the eyes were visible (to a degree) and looking back. While in the sunglasses condition the potential to interact with another person was still available, in the mirror condition this potential was nonexistent. However, signals could either be sent or received in the sunglasses condition, whereas sending/receiving gaze signals with oneself is redundant. Thus, the arousal observed in the eye contact trials during the sunglasses condition could be simply due to the saliency of the eyes perceived through the sunglasses, or due to either sending gaze or receiving gaze. Given the small sample size (*n* = 10) and variability in the data, we are cautious to make any strong conclusions from this *post hoc* control experiment. However, future research could examine eye gaze with oneself in a mirror more thoroughly to see how gaze is interpreted and modulated. Perhaps gazing at one's own reflection would increase self-awareness and cause heightened arousal under certain conditions.

Altogether these findings support the notion that there is something special about making eye contact with a live person and the arousal observed is likely the result of both sending and receiving gaze information. Research has proposed that making eye contact with a live person opens the door for social interaction (Laidlaw et al., 2011), and is likely that initial message that is responsible for the immediate boost in skin conductance response. In other words, making eye contact with another person might at first be processed as an approach signal to socially interact. One theory put forth by Hietanen et al. (2008) is that direct gaze may signal to the receiver that there is an intent to approach, while averted gaze may signal the intent to avoid. Hietanen et al. (2008) found that eye contact with a confederate resulted in heightened activity in the left frontal cortex and enhanced arousal, both of which are responses associated with the motivational tendency to approach. The same study found that perception of an averted face elicited greater activity in the right frontal cortex, indicative of the motivational tendency to avoid (Hietanen et al., 2008). Congruent with these findings were participants' subjective reports of higher arousal in the eye contact condition and the increase in approach motivation compared to the averted condition (Hietanen et al., 2008). However, the differential gaze effect was only significant when participants made eye contact with a live confederate but not when they made eye contact with an image of the confederate (Hietanen et al., 2008). Since the effect was only observed with a live confederate, this provides evidence that we only use eye contact to signal approach if there is someone to approach and interact with.

This approach/avoidance theory is not new and was first observed by Cary (1978) who video-taped conditions where a naïve participant was seated in a waiting room while another person entered the room. Social interaction (brief or continuous) was more likely to be observed if mutual gaze (or eye contact) between dyads occurred once the participant entered the room. Continuous conversation was even more likely to occur if mutual gaze occurred a second time upon entry. Alternatively, Cary (1978) found that the absence of mutual gaze upon entry predicted little to no social interaction between dyads. While this was an observational study, it does support the notion that it is the mutual exchange (i.e., sending and receiving) of approach signals that prompts the potential for social interaction, and a mutual exchange of avoidance signals (if one or both individuals avert their gaze from each other) decreases the potential for social interaction.

The intention to approach someone and socialize would be suitable for activating the nervous system within the first few seconds (SCR), but we found here that eye contact maintains the heightened arousal response for the 60-s duration (SCL). What signals could be sent and received on a continuous basis as to sustain an elevated level of arousal? Some researchers have suggested the existence of "a social brain network" (Johnson et al., 2005; Adolphs, 2009) specialized in processing social information that is modulated by eye contact (Senju and Johnson, 2009). In line with this, Conty et al. (2016) put forth the Watching Eyes model, which suggests that gaze is processed in two stages: first, the eyes capture attention and we processes whether the eyes are looking at us or not, and then the second stage activates internal processing generated by eye contact that self-referential in nature and can be associated with pro-social behaviors and positive appraisals of others. This second stage is consistent with Hietanen and colleagues, who have shown that self-referential processing could occur when we just "believe" someone is looking at us, regardless of seeing the eyes (Myllyneva and Hietanen, 2015; although see Myllyneva and Hietanen, 2016). Also in line with this, Cavallo et al. (2015) found that live gaze (direct and averted) elicited significant activation in brain areas involved in inferring mental states during social interactions (anterior rostral medial prefrontal cortex or arMPFC) as well as language comprehension and production [inferior frontal gyrus (IFG), premotor cortex, and supplementary motor area]. Further analysis showed that there was an increase in connectivity between the IFG and arMPFC during live eye contact, suggesting that it triggers a network of brain regions involved in the detection of communicative intentions and language. Similarly, a desynchronization of alpha-band activity was observed when infants looked at an object together with an adult during a social interaction involving eye contact (Hoehl et al., 2014). No such effect was observed when infants and adults were not engaged in eye contact. Thus, it is likely that the *received* gaze signals that we refer to in our study are analogous to these mentalizing processes, such that we are continually interpreting others gaze to understand their intentions, desires, beliefs, and knowledge (Conty et al., 2016). It is also likely that the *sent* gaze signals are related more to an increase is self-awareness (Conty et al., 2016), and an overall heightened attention to monitor what gaze signals we want to be public. For example, if we are lying, we might conceal gaze signals to not show the truth. Or if we are angry or sad, we might avoid gaze with others as to hide our feelings that could be communicated unwillingly through eye contact.

Altogether, our findings contribute to the previous literature by showing that arousal is elicited most strongly in the first 10 s during eye gaze if it is clear that the eyes are looking back (consistent with Watching Eyes model stage 1), and then the individual maintains a high level of arousal for the duration of the eye gaze, likely in response to mentalizing processes (self-reference, self-monitoring, communication, etc.) that occur thereafter (consistent with Watching Eyes model stage 2). This arousal pattern elicited in the clear condition, but once the gaze signals could not be interpreted clearly, like when someone wears sunglasses in the degraded condition, arousal dropped back down to baseline levels after about 30 s. Thus, we believe that the arousal level sustained over the entire minute was not due to the observer just selfmanaging (Silver and Shaw, 2018), but also due to the online and consistent perceiving and interpreting other's gaze for information related to their mental state, emotion, intention, attention, etc. (Baron-Cohen, 1995).

# LIMITATIONS

Measuring the behavior of two live participants, while rich in data, is not without its limitations. For instance, participants who wore the sunglasses (participant B) verbally mentioned that they were uncertain of the extent to which the sunglasses disguised their eyes to the other participant. Based on informal conversations, the majority of participants who wore the sunglasses assumed their eyes were quite visible and thus, they would believe they could send gaze information to the other person. However, this should have been consistent with every participant either believing that their eyes were visible or not. Another potential limitation was the withinsubjects design, such that participants took part in all three conditions—clear, degraded, and blocked. While the order of conditions was counterbalanced, previous (unpublished) research in our lab has shown that participants habituate to eye contact over time and show less and less arousal with repeated exposure. Thus, our data may have been stronger if we had enough participants to analyze the data as a between-subjects design. Lastly, in the blindfold condition it was assumed that the blindfold would prevent all gaze signals from being sent and received between dyads because participant B's eyes were entirely concealed. Since both participants were expected to have no ability to send or receive gaze information, no differences in the SCR between partners were expected to emerge in the eye contact trials. As mentioned in the results, there was a difference between participants arousal levels within the blindfold condition, such that participant B (wearing the blindfold) showed significantly higher arousal across all gaze trials. One possibility is that simply being blindfolded increased arousal because of the knowledge of being the focus of someone's attention. Thus, being the object of someone's attention could have been driving the arousal response.

# CONCLUSION

The current study demonstrated that arousal from eye contact is associated with the sending *and* receiving of gaze signals, and as the ability to exchange gaze signals decreases (by degrading the visibility of the eyes with sunglasses or a blindfold), so does arousal and the possibility for social interaction. We also tried to rule out the argument that arousal from eye contact is due to the saliency of the eyes by demonstrating no arousal enhancement when participants made eye contact with themselves in a mirror.

These findings could have implications for individuals who wear sunglasses in our everyday life. From the sender's point of view, it might be helpful to know that while wearing sunglasses during a social interaction (e.g., interview, business deal, romantic date, etc.), gaze information might not be communicated clearly, if at all. This lack of gaze information could hinder the communicative process by decreasing arousal and in turn reduce attention to, interest in, and excitement for what is being said. From the receiver's point of view, it might be helpful to know that if someone is wearing sunglasses they might be doing so because they are not open to engaging in social interaction. That is, they might be trying to conceal their eyes in order to reduce the gaze signals of approach. On the other hand, if you wish to engage in riveting social interaction, then perhaps sunglasses should be avoided.

#### ETHICS STATEMENT

All procedures were approved by the MacEwan University Research Ethics Board and participants gave informed consent.

#### REFERENCES


#### AUTHOR CONTRIBUTIONS

MJ was the lead investigator for all of the projects reported and was primarily responsible for design conception, data management and analysis, and report composition. RB was involved in data collection, management, and organization.

# FUNDING

This project was funded by a dissemination grant to MJ from the Office of Research Services, MacEwan University.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Jarick and Bencic. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*