Event Abstract

Back to Event

Mutual Gaze: Implications for Human-Robot Interaction

Frank Broz1*, Hagen Lehmann1, Chrystopher L. Nehaniv1 and Kerstin Dautenhahn1
  • 1 University of Hertfordshire, Computer Science, United Kingdom

Abstract

In order for humanoid robots to interact more naturally with people, they need models that allow them to produce realistic social gaze behavior. Mutual gaze is an important aspect of face-to-face social interaction that arises from the interaction of the gaze of two individuals. The approach taken in this work is to collect data from human conversational pairs with the goal of gaining insight into this behavior and eventually learning a controller for robot gaze from human data. In this preliminary analysis, we identify factors that influence gaze between pairs of people, indicating that gaze behavior depends on the characteristics of the interaction rather than on either individual considered in isolation. We also investigate gaze directed at specific facial features during episodes of mutual gaze.

I. INTRODUCTION

Mutual gaze is an ongoing process between two interactors jointly regulating their eye contact, rather than the result of an individual’s action [1]. This behavior is of social importance from an early developmental stage; it seems to be the basis of and precursor to more complex task-oriented gaze behaviors such as visual joint attention [2]. Mutual gaze is also important for face-to-face communication. It is a component of turn-taking ”proto-conversations” between infants and caretakers that set the stage for language learning [3] and is known to play a role in regulating conversational turn-taking in adults [4].

There have recently been a number of studies on people’s responses to mutual gaze with robots in conversational interaction tasks with robot gaze policies that are not based on human gaze behavior [5], [6], [7]. Because mutual gaze is an interaction, unrealistic robot gaze can have an impact on the human’s behavior and the impressions they form about the robot. Building models using data collected from human-human pairs is likely to improve the quality of interaction with robots that gaze.

II. METHOD

For the experiment, 32 pairs of participants were recruited from the university campus. The two requirements for participation were that the members of each pair knew one another and that they were able to engage in a fifteen minute conversation in english. The first requirement was chosen to limit a possible source of variability in the data, since it has been shown that strangers exhibit less mutual gaze than people who are familiar with one another. The second requirement was chosen to enable future linguistic research on the dataset.

The pairs were seated approximately 1 meter apart with a desk between them. Before the experiment started they were informed that they would engage in an unconstrained conversation for fifteen minutes while gaze and video data was recorded. The participants were told they could choose their topic of conversation freely. They were given a list of possible topics as suggestions, including: hobbies, a recent vacation, restaurants, television shows, or movies.

At the beginning of the session the participants were ask to fill out a consent form and answer a questionnaire about their demographic information and level of familiarity with their partner. Next, each participant was guided through the calibration procedure for the gaze tracking system by the experimenter. At the beginning and end of the conversation, the experimenter clapped his hands over the table at a level visible to the video cameras for both gaze trackers. During data collection the experimenter stayed behind a divider out of sight of the participants in order to minimize possible bias created by his presence. At the end of the session, participants also completed a short personality inventory (TIPI) [8].

III. SYSTEM OVERVIEW

ASL MobileEye gaze tracking systems were used to collect the gaze direction data [9]. Video output of the scene camera of each system was input into face-tracking software based on the faceAPI library [10]. Gaze direction and the location of the partner’s facial features (in image pixel coordinates) were logged for each participant. The gaze and face data recorded were labeled with the corresponding frame number of their video input and manually aligned by locating the frames in which handclaps occurred at the start and end of video recording.

IV. RESULTS

A. Automated Classification of Gaze States

For each pair, 12 minutes of the interaction were selected for analysis to eliminate distracted periods at the beginning and end of sessions. The data was classified into high-level behavioral states depending on participants’ face-directed gaze (see Figure 1). In all pairs observed, one participant looked at their partner noticeably more than the other. The participant with the
high face-directed gaze level is referred to as the “High” gaze participant and the partner with the lower level of face-directed gaze is referred to as “Low”. The gaze states and their descriptions are given below.

• Mutual - both participants’ look at one another’s face
• At Low - the High partner looks at the face of the Low partner while they look away
• At High - the Low partner looks at the face of the High partner while they look away
• Away - both look away from their partner’s face
• Missing - state could not classified due to missing readings

Automated analysis was performed only on pairs with fewer than 30% of their timesteps containing missing readings. This resulted in the selection of 15 pairs.

B. Factors Influencing Gaze Behavior

Using Pearson’s correlation coefficient, a negative correlation was found be- tween the Mutual and At Low states (r=-0.516, p<0.05). This can be interpreted as that the more the “High” gaze participant gazes at the other, the more the mutual gaze between the two participants during the conversation is inhibited. This has strong implications for human robot interaction research. It seems to be very important to look at the dyadic gaze patterns and not only at gaze on the individual or population level. Robots staring at their human interaction partner are very likely to inhibit mutual gaze and therefore a natural and comfortable social interaction.
In order to analyze the relationship between gaze and personality, extroversion, agreeableness, openness, conscientiousness and neuroticism scores were calculated for each participant from the TIPI. Positive correlation was found between the difference in time spent gazing at the conversational partner and the difference in the neuroticism score of the two participants (r=0.473, p<0.1). In this experiment, it seems to be the case that the more unequal the conversational partners are in their emotional stability, the larger the difference in the time they spent gazing gazing at each other. This preliminary result linking interactions between personalities to gaze is interesting from a psychological perspective and warrants further investigation.

C. Mutual Gaze and Facial Features

For the Mutual state, data was further classified according to the facial features each participant gazed at using feature information from the face tracker (see Figure 2). The results are interesting in that there does not seem to be much direct eye contact during mutual gaze. Also, which features are focused on seems to relate to a participant’s overall gaze behavior during the interaction. The Low gaze participant looked at High’s mouth far more often than vice versa. These results indicate a possible relationship between gaze and speaker role that might be explained through further analysis of the dataset including speech information.

Figure 1
Image
Figure 2
Image

Acknowledgements

This research was conducted within the EU Integrated Project ITALK (Integration and Transfer of Action and Language in Robots) funded by the European Commission under contract number FP7-214668.

References

[1] M. Argyle, Bodily communication, 2nd ed. Routledge, 1988.
[2] T. Farroni, “Infants perceiving and acting on the eyes: Tests of an evolutionary hypothesis,” Journal of Experimental Child Psychology, vol. 85, no. 3,
pp. 199–212, July 2003.
[3] C. Trevarthen and K. J. Aitken, “Infant intersubjectivity: Research, theory, and clinical applications,” The Journal of Child Psychology and Psychiatry
and Allied Disciplines, vol. 42, no. 01, pp. 3–48, 2001.
[4] C. Kleinke, “Gaze and eye contact: A research review.” Psychological Bulletin, vol. 100, no. 1, pp. 78–100, 1986.
[5]Y.Yoshikawa,K.Shinozawa,H.Ishiguro,N.Hagita,andT.Miyamoto,“The effects of responsive eyemovement and blinking behavior in a communication
robot,” in IROS, 2006, pp. 4564–4569.
[6] C. Yu, M. Scheutz, and P. Schermerhorn, “Investigating multimodal real-time patterns of joint attention in an hri word learning task,” in HRI ’10: 5th
ACM/IEEE international conference on Human-robot interaction. New York, NY, USA: ACM, 2010, pp. 309–316.
[7] A.-L. Vollmer, K. S. Lohan, K. Fischer, Y. Nagai, K. Pitsch, J. Fritsch, K. J. Rohlfing, and B. Wredek, “People modify their tutoring behavior in robot- directed interaction for action learning,” in DEVLRN ’09: Proceedings of the 2009 IEEE 8th International Conference on Development and Learning.
Washington, DC, USA: IEEE Computer Society, 2009, pp. 1–6.
[8] S. D. Gosling, P. J. Rentfrow, and J. Swann, W. B., “A very brief measure of the big five personality domains,” Journal of Research in Personality,
vol. 37, pp. 504–528, 2003.
[9] Applied Science Laboratories, “Mobile eye gaze tracking system.” [Online]. Available: http://asleyetracking.com/
[10] Seeing Machines, Inc., “faceapi.” [Online]. Available: http://seeingmachines.com/

Keywords: gaze, human-robot interaction, mutual gaze, Psychology, social robotics

Conference: IEEE ICDL-EPIROB 2011, Frankfurt, Germany, 24 Aug - 27 Aug, 2011.

Presentation Type: Poster Presentation

Topic: Social development

Citation: Broz F, Lehmann H, Nehaniv CL and Dautenhahn K (2011). Mutual Gaze: Implications for Human-Robot Interaction. Front. Comput. Neurosci. Conference Abstract: IEEE ICDL-EPIROB 2011. doi: 10.3389/conf.fncom.2011.52.00024

Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.

The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.

Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.

For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.

Received: 11 Apr 2011; Published Online: 12 Jul 2011.

* Correspondence: Dr. Frank Broz, University of Hertfordshire, Computer Science, Hatfield, AL10 9AB, United Kingdom, f.broz@tudelft.nl